2019 on Ricky

How to Create Temporary Files in Bash: mktemp and trap

Mon, 30 Dec 2019 14:31:45 +0800

How to Create Temporary Files in Bash: mktemp and trap

`mktemp` command

The generated temp file name is random and readable/writable only by the user.
To ensure creation succeeds, append OR (||) after mktemp to exit on failure.
To delete temp files on script exit, use trap to register cleanup.

#!/bin/bash

trap 'rm -f "$TMPFILE"' EXIT

TMPFILE=$(mktemp) || exit 1
echo "Our temp file is $TMPFILE"

mktemp options

-d creates a temporary directory.
-p specifies the directory for temp files. Default is $TMPDIR. If not set, /tmp is used.
-t specifies a filename template. The template must end with at least three consecutive X characters for randomness; six X are recommended. The default template is tmp. followed by ten random characters.

`trap` usage

The trap command handles system signals in Bash scripts.

Getting Started with GitHub Actions

Mon, 23 Dec 2019 10:16:01 +0800

Getting Started with GitHub Actions

How Does Nginx Defend Against DDoS?

Fri, 20 Dec 2019 09:42:50 +0800

ngx_http_limit_req_module

limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;

Golang Service Exceeded File Handle Limit (too many open files)

Mon, 16 Dec 2019 10:14:33 +0800

Golang Service Exceeded File Handle Limit (too many open files)

Check system config: ulimit -a | grep open. The system config is normal.
Check the service file limit: cat /proc/40636/limits. The service did not inherit the system settings and is still limited to 1024.
Check open file count: lsof -p 40636 | wc -l. It exceeds the limit, so the error occurs.
Check which connections are open: lsof -p 40636 > openfiles.log. Many HTTP connections were left open to the alerting service. The root cause is that the app failed to parse config, sent errors to the alerting service, and did not close connections, leading to too many open files. Why it did not inherit system limits needed more investigation.

Trellis Ansible Bad Interpreter Error

Fri, 06 Dec 2019 22:34:31 +0800

Trellis Ansible Bad Interpreter Error

Bad Interpreter Error

Error we got using Ansible was a bad interpreter error. Python 2.7 is not to be found:

zsh: /usr/local/bin/ansible-vault: bad interpreter: /usr/local/opt/python@2/bin/python2.7: no such file or directory

And that was correct because when we checked /usr/local/opt we only had Python 3.

Installation Python 2

brew install python@2

Python Crashing Hard

Next, on Ansible version check I got another error

➜ trellis git:(master) ansible --version
[1] 19153 abort ansible --version

It was somehow crashing Python 2.7 though it should just work with it. I decided to upgrade Ansible as well

Load Balancing with iptables and ip rule

Wed, 04 Dec 2019 11:08:04 +0800

Load Balancing with iptables and ip rule

Steps

This example uses an Arch Linux device with two Internet uplinks: eth0 and eth1. The mapping is:

Mark 10 (0xa) - Routing table #110 - use eth0
Mark 11 (0xb) - Routing table #111 - use eth1

We decide which uplink to use based on the packet mark. First, use ip rule to map each mark to its routing table.

The default routing table priority is 32768. To ensure our tables are used, set a higher priority (for example 31000).

htop explained

Thu, 14 Nov 2019 10:05:35 +0800

htop explained

Some Jenkinsfile examples

Wed, 06 Nov 2019 17:29:46 +0800

Some Jenkinsfile examples

Parallel

#!/usr/bin/env groovy
pipeline {
agent any
stages {
stage("Build") {
steps {
sh 'mvn -v'
}
}
stage("Testing") {
parallel {
stage("Unit Tests") {
agent { docker 'openjdk:7-jdk-alpine' }
steps {
sh 'java -version'
}
}
stage("Functional Tests") {
agent { docker 'openjdk:8-jdk-alpine' }
steps {
sh 'java -version'
}
}
stage("Integration Tests") {
steps {
sh 'java -version'
}
}
}
}
stage("Deploy") {
steps {
echo "Deploy!"
}
}
}
}

When

#!/usr/bin/env groovy
pipeline {
agent any
environment {
VALUE_ONE = '1'
VALUE_TWO = '2'
VALUE_THREE = '3'
}
stages {
// skip a stage while creating the pipeline
stage("A stage to be skipped") {
when {
expression { false } //skip this stage
}
steps {
echo 'This step will never be run'
}
}
// Execute when branch = 'master'
stage("BASIC WHEN - Branch") {
when {
branch 'master'
}
steps {
echo 'BASIC WHEN - Master Branch!'
}
}
// Expression based when example with AND
stage('WHEN EXPRESSION with AND') {
when {
expression {
VALUE_ONE == '1' && VALUE_THREE == '3'
}
}
steps {
echo 'WHEN with AND expression works!'
}
}
// Expression based when example
stage('WHEN EXPRESSION with OR') {
when {
expression {
VALUE_ONE == '1' || VALUE_THREE == '2'
}
}
steps {
echo 'WHEN with OR expression works!'
}
}
// When - AllOf Example
stage("AllOf") {
when {
allOf {
environment name:'VALUE_ONE', value: '1'
environment name:'VALUE_TWO', value: '2'
}
}
steps {
echo "AllOf Works!!"
}
}
// When - Not AnyOf Example
stage("Not AnyOf") {
when {
not {
anyOf {
branch "development"
environment name:'VALUE_TWO', value: '4'
}
}
}
steps {
echo "Not AnyOf - Works!"
}
}
}
}

Jinja docx template, avoiding new line in nested for

Tue, 29 Oct 2019 10:17:35 +0800

Jinja docx template, avoiding new line in nested for

What the f*ck Python! 🐍

Fri, 11 Oct 2019 14:32:20 +0800

What the f*ck Python! 🐍

Fighting ISP Cache Hijacking Again with iptables

Mon, 07 Oct 2019 10:41:08 +0800

Fighting ISP Cache Hijacking Again with iptables

Cause

The fight against the carrier cache problem started two years ago. The carrier even cached cnpm data. Worse, their cache servers were not only slow like a turtle in a marathon, they also crashed frequently, so I just wanted to write code but had to face a wall of red errors.

Fix

iptables -I FORWARD -p tcp -m tcp -m ttl --ttl-gt 20 -m ttl --ttl-lt 30 -j DROP

Use Nginx and mod_pagespeed to Convert Images to WebP on the Fly

Mon, 07 Oct 2019 10:35:22 +0800

Use Nginx and mod_pagespeed to Convert Images to WebP on the Fly

Compile ngx_pagespeed

First make sure Nginx is built with --with-compat, so we do not need to rebuild Nginx from scratch.

incubator: https://github.com/apache/incubator-pagespeed-ngx.git

# Switch to the nginx source directory and configure the build
./configure --with-compat --add-dynamic-module=../incubator-pagespeed-ngx

# Build modules
make modules

# Copy the built module into the nginx modules directory
sudo cp objs/ngx_pagespeed.so /etc/nginx/modules/

# Create the cache directory for converted images
sudo mkdir -p /var/ngx_pagespeed_cache
sudo chown -R www-data:www-data /var/ngx_pagespeed_cache

load_module modules/ngx_pagespeed.so;


# enable pagespeed module on this server block
pagespeed on;

# Needs to exist and be writable by nginx. Use tmpfs for best performance.
pagespeed FileCachePath /var/ngx_pagespeed_cache;

# Ensure requests for pagespeed optimized resources go to the pagespeed handler
# and no extraneous headers get set.
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
 add_header "" "";
}
location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }

pagespeed RewriteLevel CoreFilters;

The last line (pagespeed RewriteLevel CoreFilters;) specifies the enabled optimizations. It includes basic filters such as:

Is there a regular expression to detect a valid regular expression?

Fri, 04 Oct 2019 14:44:38 +0800

Is there a regular expression to detect a valid regular expression?

This is a recursive regex, and is not supported by many regex engines. PCRE based ones should support it.

/
^ # start of string
( # first group start
(?:
(?:[^?+*{}()[\]\\|]+ # literals and ^, $
| \\. # escaped characters
| \[ (?: \^?\\. | \^[^\\] | [^\\^] ) # character classes
(?: [^\]\\]+ | \\. )* \]
| \( (?:\?[:=!]|\?<[=!]|\?>)? (?1)?? \) # parenthesis, with recursive content
| \(\? (?:R|[+-]?\d+) \) # recursive matching
)
(?: (?:[?+*]|\{\d+(?:,\d*)?\}) [?+]? )? # quantifiers
| \| # alternative
)* # repeat content
) # end first group
$ # end of string
/

Without whitespace and comments

Force file download with Nginx

Mon, 19 Aug 2019 12:12:32 +0800

Force file download with Nginx

add_header Content-Disposition 'attachment;';

server {
 listen 80;
 server_name my.domain.com;
 location ~ ^.*/(?P<request_basename>[^/]+\.(mp3))$ {
 root /path/to/mp3/
 add_header Content-Disposition 'attachment; filename="$request_basename"';
 }
}

{
 listen 80;
 server_name backup.baifu-tech.net;
 root /data/backup/rechargecent-mago;
 location / {
 auth_basic "baifu backup center";
 auth_basic_user_file /etc/nginx/ssl/htpasswd;
 autoindex on;
 autoindex_exact_size off;
 autoindex_localtime on;
 }
}

Shell Script Study Notes

Wed, 07 Aug 2019 15:14:08 +0800

Shell Script Study Notes

Arithmetic operations

val=`expr $a + $b`

Operators

Symbol	Description	Example
!	NOT	[ ! false ]
-o	OR	[ $a -lt 20 -o $b -gt 20 ]
-a	AND	[ $a -lt 20 -a $b -gt 20 ]
=	equality check	[ $a = $b ]
!=	inequality check	[ $a != $b ]
-z	string length is 0, returns true if 0	[ -z $a ]
-n	string length is not 0, returns true if not 0	[ -n $a ]
str	check whether string is empty, true if not	[ $a ]
-b	check whether file is a block device	[ -b $file ]
-c	check whether file is a character device	..
-d	check whether file is a directory	[ -d $file ]
-f	check whether file is a regular file	[ -f $file ]
-r	check whether file is readable	..
-w	check whether file is writable	..
-x	check whether file is executable	..
-s	check whether file is empty	..
-e	check whether file exists	..

Special variables

Variable	Meaning
$0	file name of the current script
$n	arguments passed to a script or function; n is the position
$#	number of arguments passed to a script or function
$*	all arguments as a single word, e.g., “1 2 3”
$@	all arguments as separate words, e.g., “1” “2” “3”
$?	exit status of the last command or return value of a function
$$	current shell process ID; for scripts, the PID of the script process

POSIX program exit statuses

Code	Meaning
0	command exited successfully
> 0	failure during redirection or word expansion (~, variables, commands, arithmetic, and word splitting)
1 - 125	command exited unsuccessfully; specific meanings are command-defined
126	command found but file is not executable
127	command not found
> 128	command died from a signal

Input/output redirection

Command	Description
command > file	redirect output to file
command > file	redirect output to file by appending
n > file	redirect file descriptor n to file
n » file	redirect file descriptor n to file by appending
n >& m	merge output file m and n
n <& m	merge input file m and n
« tag	use content between start tag and end tag as input

File include

Use . or source to include files.

Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

Thu, 25 Jul 2019 14:25:56 +0800

Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

mattrobenolt:

Was just going to propose this. :) Unfortunately, this is just how this feature works. If it’s not useful and is just wasting resources, it’s easy to disable globally by setting SENTRY_SCRAPE_JAVASCRIPT_CONTEXT = False in your sentry.conf.py or per project in the UI.

Other than that, there’s not much else we can do here since the feature is effectively working as intended.

Python Telegram Bot

Thu, 11 Jul 2019 14:03:25 +0800

Fixing Disk Space Not Freed on Linux

Wed, 10 Jul 2019 09:57:33 +0800

Fixing Disk Space Not Freed on Linux

Use `df -ah` and `du -h --max-depth=1`

The total from du is far smaller than the total reported by df.

When a process deletes files but keeps running, the files are not actually removed, so disk space is not freed, and those files are not counted.

lsof |grep delete

Sentry Source Code Development Notes

Wed, 03 Jul 2019 11:53:15 +0800

OpenResty + Redis: Block High-Frequency IPs

Thu, 27 Jun 2019 11:18:46 +0800

OpenResty + Redis: Block High-Frequency IPs

init_by_lua_block {
 redis = require "redis"
 client = redis.connect('127.0.0.1', 6379)
}
server {
 listen 8080;
 location / {
 access_by_lua_file /usr/local/nginx/conf/lua/block.lua;
 proxy_pass http://192.168.1.102:8000;
 }
}

-- Redis-based IP rate limiting / blocking for OpenResty (ngx_lua)

-- NOTE:
-- This script assumes a global `client` variable is used/stored.
-- Make sure `redis` module is available and `client` is initialized somewhere.

local function isConnected()
 return client:ping()
end

local function createRedisConnection()
 return redis.connect("127.0.0.1", 6379)
end

-- If the Redis connection fails, stop blocking (allow traffic)
if pcall(isConnected) then
 -- already connected (or ping succeeded)
else
 -- not connected; try reconnect
 if pcall(createRedisConnection) then
 -- Reconnect: this will reconnect Redis on every request
 -- For high traffic, consider disabling reconnect (skip pcall), and allow/terminate directly via ngx.exit
 client = createRedisConnection()
 else
 ngx.exit(ngx.OK)
 end
end

local ttl = 60 -- sampling window (seconds)
local bktimes = 30 -- requests within window to trigger block
local block_ttl = 600 -- block duration after trigger (seconds)

local ip = ngx.var.remote_addr
local ipvtimes = client:get(ip)

if ipvtimes then
 if ipvtimes == "-1" then
 -- blocked
 return ngx.exit(403)
 else
 local last_ttl = client:ttl(ip)
 -- ngx.say("key exist.ttl is ", last_ttl)

 if last_ttl == -1 then
 client:set(ip, 0)
 client:expire(ip, ttl)
 -- ngx.say("ttl & vtimes recount")
 return ngx.exit(ngx.OK)
 end

 local vtimes = tonumber(client:get(ip)) + 1

 if vtimes < bktimes then
 client:set(ip, vtimes)
 client:expire(ip, last_ttl)
 -- ngx.say(ip, " view ", vtimes, " times")
 return ngx.exit(ngx.OK)
 else
 -- ngx.say(ip, " will be block next time.")
 client:set(ip, -1)
 client:expire(ip, block_ttl)
 return ngx.exit(ngx.OK)
 end
 end
else
 -- key does not exist
 client:set(ip, 1)
 -- ngx.say(ip, " view 1 times")
 client:expire(ip, ttl)
 return ngx.exit(ngx.OK)
end

Terraform Getting Started Notes

Wed, 26 Jun 2019 17:22:52 +0800

Terraform Getting Started Notes

設定 Haproxy 以防止 DDOS 攻擊

Mon, 17 Jun 2019 11:07:13 +0800

設定 Haproxy 以防止 DDOS 攻擊

TCP syn flood attacks

vi /etc/sysctl.conf

# Protection from SYN flood
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_max_syn_backlog = 1024

Slowloris like attacks

defaults
option http-server-close
timeout http-request 5s
timeout connect 5s
timeout client 30s
timeout server 10s
timeout tunnel 1h

限制每一個 user 的連線數量

普通用戶瀏覽網站的網頁，或是從網站下載東西時，瀏覽器一般會建立 5-7 個 TCP 鏈接。當一個惡意 client 打開了大量 TCP 連線時，耗費大量資源，因此我們必須要限制同一個用戶的連線數量。

但如果有很多使用者，是從某一個私有網段，透過 NAT 的方式連線到 Server 時，且實際上我們也不知道，到底哪一個會是 NAT 的轉址後的 IP，不知道該將哪個 IP 設定為白名單，這樣的限制就會造成問題，因此我們認為實際的環境，這樣的設定應該要保留不處理。

以下是一個設定的範例，最重要的地方是在 frontend ft_web 區塊的設定。

global
stats socket ./haproxy.stats level admin
defaults
option http-server-close
mode http
timeout http-request 5s
timeout connect 5s
timeout server 10s
timeout client 30s
listen stats
bind 0.0.0.0:8880
stats enable
stats hide-version
stats uri /
stats realm HAProxy Statistics
stats auth admin:admin
frontend ft_web
bind 0.0.0.0:8080
# Table definition
stick-table type ip size 100k expire 30s store conn_cur
# Allow clean known IPs to bypass the filter
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src
# Split static and dynamic traffic since these requests have different impacts on the servers
use_backend bk_web_static if { path_end .jpg .png .gif .css .js }
default_backend bk_web
# Dynamic part of the application
backend bk_web
balance roundrobin
cookie MYSRV insert indirect nocache
server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
server srv2 192.168.1.3:80 check cookie srv2 maxconn 100
# Static objects
backend bk_web_static
balance roundrobin
server srv1 192.168.1.2:80 check maxconn 1000
server srv2 192.168.1.3:80 check maxconn 1000

限制每個 user 產生新連線的速率 Limiting the connection rate per user

惡意的使用者會在短時間內建立很多連線，但如果產生新連線的速度太高，就會消耗掉過多的資源服務一個使用者。

Kubernetes Runtime Explained in Plain Language

Thu, 13 Jun 2019 11:28:26 +0800

Kubernetes Runtime Explained in Plain Language

Do you understand the Nginx request processing flow?

Thu, 07 Mar 2019 14:05:55 +0800

Do you understand the Nginx request processing flow?

11 processing phases

NGX_HTTP_POST_READ_PHASE:

A phase after receiving the full HTTP headers. It is before URI rewrite. Very few modules register in this phase, and it is skipped by default.

NGX_HTTP_SERVER_REWRITE_PHASE:

The phase that modifies the URI before matching the location, used for redirects. This is where rewrite directives in the server block but outside location are executed. While reading request headers, nginx selects the virtual host by host and port.

2019 on Ricky

How to Create Temporary Files in Bash: mktemp and trap

mktemp command

mktemp options

trap usage

Getting Started with GitHub Actions

How Does Nginx Defend Against DDoS?

ngx_http_limit_req_module

Golang Service Exceeded File Handle Limit (too many open files)

Trellis Ansible Bad Interpreter Error

Bad Interpreter Error

Installation Python 2

Python Crashing Hard

Load Balancing with iptables and ip rule

Steps

htop explained

Some Jenkinsfile examples

Parallel

When

Jinja docx template, avoiding new line in nested for

What the f*ck Python! 🐍

Fighting ISP Cache Hijacking Again with iptables

Cause

Fix

Use Nginx and mod_pagespeed to Convert Images to WebP on the Fly

Compile ngx_pagespeed

Is there a regular expression to detect a valid regular expression?

Force file download with Nginx

Shell Script Study Notes

Arithmetic operations

Operators

Special variables

POSIX program exit statuses

Input/output redirection

File include

Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

Excessive Errors from sentry.lang.javascript.processor: SoftTimeLimitExceeded()

Python Telegram Bot

Fixing Disk Space Not Freed on Linux

Use df -ah and du -h --max-depth=1

Sentry Source Code Development Notes

OpenResty + Redis: Block High-Frequency IPs

Terraform Getting Started Notes

設定 Haproxy 以防止 DDOS 攻擊

TCP syn flood attacks

Slowloris like attacks

限制每一個 user 的連線數量

限制每個 user 產生新連線的速率 Limiting the connection rate per user

Kubernetes Runtime Explained in Plain Language

Do you understand the Nginx request processing flow?

11 processing phases

`mktemp` command

`trap` usage

Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

Use `df -ah` and `du -h --max-depth=1`