2021 on Ricky

How Nginx Hides Upstream Errors

Mon, 27 Dec 2021 15:47:12 +0800

How Nginx Hides Upstream Errors

Nginx allows enabling next upstream for the following seven retryable error codes

403 Forbidden
404 Not Found
429 Too Many Requests
500 Internal Server Error
502 Bad Gateway
503 Server Unavailable
504 Gateway Timeout

When upstream returns 404, return a 200 response with a not-found image

You can use proxy_intercept_errors to achieve this.\n> When proxy_intercept_errors is enabled, requests with upstream response codes >= 300 can be further handled via the error_page directive.

Argo CD ApplicationSet Controller: The World Turns for Me!

Mon, 27 Dec 2021 09:41:03 +0800

# Install kind; see the official docs for other platforms
# Run a lightweight k8s cluster locally
~$ brew install kind
# Install kubectx; see the official docs for other platforms
# Use it to switch between k8s contexts
~$ brew install kubectx
# Install helm; see the official docs for other platforms
# K8s package manager
~$ brew install helm
# Install kubectl; see the official docs for other platforms
# Communicate with the k8s cluster API server
~$ brew install kubectl
# Install argocd cli; see the official docs for other platforms
# Communicate with Argo CD
~$ brew install argocd

Go AES Encryption and Decryption in Three Modes (CBC/ECB/CFB)

Wed, 22 Dec 2021 17:40:52 +0800

Understanding Cilium Series (1): Introduction to Cilium

Tue, 21 Dec 2021 13:04:38 +0800

Understanding Cilium Series (1): Introduction to Cilium

Current status of k8s Service load balancing implementations

Before Cilium, Services were implemented by kube-proxy in three modes: userspace, iptables, and ipvs.

Userspace

In this mode, kube-proxy acts as a reverse proxy and listens on random ports. It redirects traffic to the proxy port via iptables rules, and kube-proxy forwards the traffic to backend pods. Service requests go from user space into kernel iptables and then back to user space, which is costly and has poor performance.

Kubernetes Without kube-proxy

Mon, 20 Dec 2021 17:57:13 +0800

Kubernetes Without kube-proxy

Quick-Start

kubeadm init --skip-phases=addon/kube-proxy

# Setup Helm repository
helm repo add cilium https://helm.cilium.io/


helm install cilium cilium/cilium --version 1.9.18 \
 --namespace kube-system \
 --set kubeProxyReplacement=strict \
 --set k8sServiceHost=REPLACE_WITH_API_SERVER_IP \
 --set k8sServicePort=REPLACE_WITH_API_SERVER_PORT

Zsh tab-completion not working

Fri, 10 Dec 2021 17:44:49 +0800

Zsh tab-completion not working

Question

Although I’ve used Oh-My-Zsh in the past, I decided this time around (i.e. setting up a new computer) I’d try to avoid installing it to keep things a bit leaner. Right now I’m trying to cherry-pick Oh-My-Zsh’s insensitive tab-completion feature. Digging around its source repo, I found the following line: zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z}' 'r:|=*' 'l:|=* r:|=*'

I’m fairly confident this is the line Oh-My-Zsh executes to do what I want, so I tried adding it to my .zshrc. Restarted my terminal. Typed cd desk, then hit [tab]. No dice – I didn’t get cd Desktop/.

Linkerd 2.8 - Build a Simple and Secure Multi-Cluster Kubernetes Architecture

Thu, 09 Dec 2021 09:25:44 +0800

Linkerd 2.8 - Build a Simple and Secure Multi-Cluster Kubernetes Architecture

Best Practices for Writing Bash Scripts

Mon, 06 Dec 2021 11:11:54 +0800

#!/usr/bin/env bash
# Bash3 Boilerplate. Copyright (c) 2014, kvz.io
set -o errexit
set -o pipefail
set -o nounset
# set -o xtrace

# Set magic variables for current file & dir
__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[0]}")"
__base="$(basename ${__file} .sh)"
__root="$(cd "$(dirname "${__dir}")" && pwd)" # <-- change this as it depends on your app

arg1="${1:-}"

Use shift to read function arguments

This makes it easier to reorder arguments, if you change your mind later.

Day 28 - Introduction to Useful Third-Party Kubernetes Tools

Thu, 02 Dec 2021 13:28:09 +0800

Day 28 - Introduction to Useful Third-Party Kubernetes Tools

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
ithome-6564f65698-947rv 1/1 Running 0 84s
ithome-6564f65698-fglr9 1/1 Running 0 84s
ithome-6564f65698-k5wtg 1/1 Running 0 84s
ithome-6564f65698-rrvk4 1/1 Running 0 84s
ithome-6564f65698-zhwlj 1/1 Running 0 84s

Stern/Kail

The names of created pods often contain some unreadable random strings.

If you use kubectl to observe logs for individual pods, you need to switch between different pods.

There are many tools for this, such as Stern, Kube-tail, and Kail.

5 DevSecOps open source projects to know

Tue, 30 Nov 2021 12:41:13 +0800

5 DevSecOps open source projects to know

Clair

Sigstore

KubeLinter

Open Policy Agent and Gatekeeper

Falco

[Kubernetes] Service Overview

Wed, 24 Nov 2021 14:00:45 +0800

[Kubernetes] Service Overview

Define Service

With selector

Since you need Pods before you define a Service, assume there are Pods in the cluster (exposing TCP port 9376) with the label app=MyApp. You can define a Service as an abstraction layer in front of those pods and provide the service through a domain name.

kind: Service
apiVersion: v1
metadata:
 name: my-service
spec:
 # type has four options (ClusterIP, NodePort, LoadBalancer, ExternalName)
 # default is ClusterIP
 type: ClusterIP
 # select pods with "app=MyApp"
 selector:
 app: MyApp
 # Service port configuration
 ports:
 - protocol: TCP
 port: 80
 # This is the port number exposed by the Pod
 targetPort: 9376

Pod <---> Endpoint(tcp:9376) <---> Service(tcp:80, with VIP)

How Go Reads a 16GB File in 25 Seconds

Mon, 22 Nov 2021 11:07:17 +0800

After opening the file, we have two options:

Read the file line by line, which reduces memory pressure but takes more time.

Read the entire file into memory and process it, which uses more memory but significantly reduces time.

Because the file is too large (16 GB), we cannot load the entire file into memory. But the first option is also not feasible for us because we want to process the file within seconds.

Bind JSON with jsoniter in Gin

Mon, 22 Nov 2021 10:27:39 +0800

Bind JSON with jsoniter in Gin

simple

go build -tags=jsoniter ./...

custom

implement `BindingBody` interface

// github.com/gin-gonic/gin@v1.6.3/binding/binding.go:36
// Binding describes the interface which needs to be implemented for binding the
// data present in the request such as JSON request body, query parameters or
// the form POST.
type Binding interface {
 Name() string
 Bind(*http.Request, interface{}) error
}
// BindingBody adds BindBody method to Binding. BindBody is similar with Bind,
// but it reads the body from supplied bytes instead of req.Body.
type BindingBody interface {
 Binding
 BindBody([]byte, interface{}) error
}

package custom
import (
 "bytes"
 "fmt"
 "io"
 "net/http"
 jsoniter "github.com/json-iterator/go"
 "github.com/gin-gonic/gin/binding"
)
// BindingJSON replaces Gin's default binding and supports richer JSON features
var BindingJSON = jsonBinding{}
// You can customize jsoniter config or add plugins
var json = jsoniter.ConfigCompatibleWithStandardLibrary
type jsonBinding struct{}
func (jsonBinding) Name() string {
 return "json"
}
func (jsonBinding) Bind(req *http.Request, obj interface{}) error {
 if req == nil || req.Body == nil {
 return fmt.Errorf("invalid request")
 }
 return decodeJSON(req.Body, obj)
}
func (jsonBinding) BindBody(body []byte, obj interface{}) error {
 return decodeJSON(bytes.NewReader(body), obj)
}
func decodeJSON(r io.Reader, obj interface{}) error {
 decoder := json.NewDecoder(r)
 if binding.EnableDecoderUseNumber {
 decoder.UseNumber()
 }
 if binding.EnableDecoderDisallowUnknownFields {
 decoder.DisallowUnknownFields()
 }
 if err := decoder.Decode(obj); err != nil {
 return err
 }
 return validate(obj)
}
func validate(obj interface{}) error {
 if binding.Validator == nil {
 return nil
 }
 return binding.Validator.ValidateStruct(obj)
}

// Replace binding.JSON with the custom one
ctx.ShouldBindWith(ms, binding.JSON)
ctx.ShouldBindBodyWith(ms, binding.JSON)

Nginx notes

Fri, 19 Nov 2021 14:35:58 +0800

Record Nginx configuration file and explanation.

files structure

.
├── geoip.conf
├── nginx.conf
├── sites-available
│ ├── default.conf
├── sites-enabled
│ ├── default.conf -> ../sites-available/default.conf
├── upstream.conf

geoip.conf

## module: ngx_http_geoip2_module
## https://github.com/leev/ngx_http_geoip2_module
## Read the GeoIP database and set variables
geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
 auto_reload 60m;
 $geoip2_metadata_country_build metadata build_epoch;
 ## Set $geoip2_data_country_code to the ISO 3116 country code for $remote_addr
 $geoip2_data_country_code source=$remote_addr country iso_code;
 ## Set $geoip2_data_country_name to the corresponding English city name
 $geoip2_data_country_name country names en;
}

upstream.conf

## module: ngx_http_upstream_module
## Define server groups
upstream to_nodejs1 {
 ## server address [parameters]; define a server
 ## parameters:
 ## weight=number defines the weight, default is 1
 ## max_fails=number sets max retries to the upstream server, default is 1
 ## fail_timeout=time sets the time to stop sending requests to this upstream server after reaching max_fails, default is 10 seconds
 ## backup marks this upstream server as a backup when others are unavailable
 ## down marks this upstream server as unavailable
 server 10.7.0.12:9000 max_fails=3 fail_timeout=5s;
 server 10.7.0.12:9001 max_fails=3 fail_timeout=5s backup;
}

upstream to_nodejs2 {
 server 10.7.0.12:9002 max_fails=3 fail_timeout=5s;
 server 10.7.0.12:9003 max_fails=3 fail_timeout=5s backup;
}

upstream to_nodejs9005 {
 server 10.7.0.12:9005 max_fails=3 fail_timeout=5s;
}

## module: ngx_http_map_module
## map string $variable { ... } creates a new variable
map $arg_agent $game_api {
 ## $arg_agent is the agent value in the request (https://abc.com/?agent=123)
 ## When agent=123, $game_api is to_nodejs95
 123 to_nodejs95;
 ## If agent ends with 1, 2, 3, or 4, $game_api is to_nodejs1
 ~*1$ to_nodejs1;
 ~*2$ to_nodejs1;
 ~*3$ to_nodejs1;
 ~*4$ to_nodejs1;
 ## If agent does not match the rules above, $game_api defaults to to_nodejs2
 default to_nodejs2;
}

default.conf

## module: ngx_http_limit_req_module
## Limit request handling
## limit_req_zone key zone=name:size rate=rate [sync]; defines request limiting rules
limit_req_zone $binary_remote_addr$server_name zone=websocket:10m rate=1r/m;
## limit_req_status code; sets HTTP status code for rejected connections, default is 503
limit_req_status 502;

## Configure virtual host
server {
 ## listen port [default_server] [ssl] [http2 | spdy] [proxy_protocol] [setfib=number] [fastopen=number] [backlog=number] [rcvbuf=size] [sndbuf=size] [accept_filter=filter] [deferred] [bind] [ipv6only=on|off] [reuseport] [so_keepalive=on|off|[keepidle]:[keepintvl]:[keepcnt]];
 ## Set the listen port, default is *:80
 ## Below listens on port 80 and is the default virtual host
 listen 80 default_server;
 ## server_name name ...; set virtual host name, regex allowed, default is ""
 server_name _;

 access_log logs/default/default.log json;
 error_log logs/default/default.error.log warn;

 ## module: ngx_http_access_module
 ## allow address | CIDR | unix: | all; allow IP access
 allow 1.1.1.1;
 ## deny address | CIDR | unix: | all; deny IP access
 deny 12.34.56.78;

 ## Set the root directory for requests
 root /usr/share/nginx/html;
 ## limit_req zone=name [burst=number] [nodelay | delay=number]; set request limiting zone
 limit_req zone=websocket nodelay;
 ## limit_req_log_level info | notice | warn | error; set log level for rejected requests, default is error
 limit_req_log_level warn;

 ## location [ = | ~ | ~* | ^~ ] uri { ... }
 ## location @name { ... } configure based on the request URI
 location / {
 default_type application/json;
 ## Return HTTP status code 200 with a string
 return 200 '{"Code": "$status", "IP": "$remote_addr"}';
 }
}


server {
 ## Below listens on port 443 and is the default virtual host; all connections use SSL
 listen 443 default_server ssl;
 server_name _;

 access_log logs/default/default.log json;
 error_log logs/default/default.error.log warn;
 ## module: ngx_http_ssl_module
 ## Set the PEM-format certificate
 ssl_certificate /etc/ssl/hddv1.com.crt;
 ## Set the PEM-format key
 ssl_certificate_key /etc/ssl/hddv1.com.key;
 ## Set SSL versions, default is TLSv1 TLSv1.1 TLSv1.2
 ssl_protocols TLSv1.2 TLSv1.3;
 ## Set enabled ciphers, default is HIGH:!aNULL:!MD5
 ssl_ciphers "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC2:!RC4:!aNULL:!eNULL:!LOW:!IDEA:!DES:!TDES:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!EXPORT:!ANON";
 ## Specify the DH parameter file for DHE ciphers
 ssl_dhparam /etc/ssl/dhparams.pem;
 ## Prefer server ciphers, default is off
 ssl_prefer_server_ciphers on;
 ## ssl_session_cache off | none | [builtin[:size]] [shared:name:size];
 ## Set cache and size, default is none
 ssl_session_cache shared:SSL:1m;
 ## Set session reuse time, default is 5 minutes
 ssl_session_timeout 5m;
 add_header X-Frame-Options "SAMEORIGIN";
 add_header X-XSS-Protection "1; mode=block";
 add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";

 root /usr/share/nginx/html;
 limit_req zone=websocket nodelay;
 limit_req_log_level warn;
 default_type application/json;

 location / {
 default_type application/json;
 return 200 '{"Code": "$status", "IP": "$remote_addr"}';
 }
}

nginx.conf

## module: ngx_core_module
## worker_processes number | auto; number of Nginx worker processes, auto equals CPU count
worker_processes auto;
## worker_rlimit_nofile number; max open files for workers, default is system RLIMIT_NOFILE
worker_rlimit_nofile 131072;
## worker_shutdown_timeout time; shutdown timeout for reloads and related commands
worker_shutdown_timeout 60;

## error_log file [level]; set error log path
## debug, info, notice, warn, error, crit, alert, emerg
error_log logs/error.log warn;

## pid file; master process ID file location
pid logs/nginx.pid;

## module: ngx_core_module
## Connection handling
events {
 ## worker_connections number; max concurrent connections per worker, default is 512, must be less than worker_rlimit_nofile
 ## max connections = worker_connections * worker_processes
 worker_connections 102400;

 ## accept_mutex on | off; default is off
 ## When on, only one worker accepts new connections while others remain idle
 ## When off, all workers wake up; one accepts, the rest go back to sleep
 ## With TCP long connections and high traffic, off performs better for throughput and QPS
 accept_mutex off;

 ## multi_accept on | off; accept all connections at once, default is off
 multi_accept on;
}

## module: ngx_http_core_module
## HTTP server settings
http {
 ## module: ngx_core_module
 ## include file | mask; include settings from file
 ## Below sets MIME types, defined in the mime.type file
 include mime.types;

 ## default_type mime-type; default MIME type, default is text/plain
 default_type application/octet-stream;
 ## server_names_hash_max_size size; max size of the server_name hash table, default is 512k
 server_names_hash_max_size 2048;
 ## Size of the server_name hash table for fast lookup, default depends on CPU L1 cache
 server_names_hash_bucket_size 256;
 ## server_tokens on | off | build | string; show Nginx version on error pages, default is on
 server_tokens off;
 ## Log 404 in the error log
 log_not_found off;
 ## Enable sendfile() for file transfer efficiency, default is off
 sendfile on;
 ## Use full packets for file sending, default is off
 tcp_nopush on;
 ## Send data as soon as possible, default is on
 tcp_nodelay on;
 ## Set keepalive timeout seconds; Nginx closes after timeout, default is 75
 keepalive_timeout 70;
 ## client_max_body_size size; max allowed request body size
 client_max_body_size 64M;

 ## module: ngx_http_gzip_module
 ## Enable gzip compression, default is off
 gzip on;
 ## Minimum Content-Length to gzip, default is 20
 gzip_min_length 1k;
 ## Gzip buffer size, default is one memory page
 ## gzip_buffers number size;
 gzip_buffers 4 32k;
 ## Compression level, range 1-9, default is 1
 gzip_comp_level 7;
 ## MIME types to compress, default is text/html
 gzip_types text/plain application/x-javascript text/css application/xml text/javascript application/x-httpd-php application/json;
 ## Add Vary: Accept-Encoding to HTTP response headers, default is off
 gzip_vary on;
 ## Disable compression for specific User-Agent
 ## Below disables IE 6
 gzip_disable "MSIE [1-6]\.";

 ## resolver address ... [valid=time] [ipv6=on|off] [status_zone=zone]; use the specified DNS servers for server_name and upstreams
 resolver 114.114.114.114 8.8.8.8 1.1.1.1;

 ## module: ngx_http_headers_module
 ## add_header name value [always]; add fields to HTTP response headers
 ## Below allows CORS
 add_header Access-Control-Allow-Origin *;
 add_header Access-Control-Allow-Headers DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type;
 add_header Access-Control-Allow-Methods GET,POST,OPTIONS;
 add_header Access-Control-Expose-Headers 'WWW-Authenticate,Server-Authorization,User-Identity-Token';

 ## module: ngx_http_realip_module
 ## set_real_ip_from address | CIDR | unix:; set trusted proxy IPs such as reverse proxies
 set_real_ip_from 10.0.0.0/8;
 set_real_ip_from 172.16.0.0/12;
 set_real_ip_from 192.168.0.0/16;
 ## real_ip_header field | X-Real-IP | X-Forwarded-For | proxy_protocol; define which header provides client IP, default is X-Real-IP
 real_ip_header X-Forwarded-For;
 ## Use the last non-trusted IP or last IP in real_ip_header as the real IP, default is off
 real_ip_recursive on;

 ## module: ngx_http_log_module
 ## log_format name [escape=default|json|none] string ...; set log format
 log_format json escape=json '{"@timestamp":"$time_iso8601",'
 '"@source":"$server_addr",'
 '"ip":"$http_x_forwarded_for",'
 '"client":"$remote_addr",'
 '"request_method":"$request_method",'
 '"scheme":"$scheme",'
 '"domain":"$server_name",'
 '"client_host":"$host",'
 '"referer":"$http_referer",'
 '"request":"$request_uri",'
 '"args":"$args",'
 '"sent_bytes":$body_bytes_sent,'
 '"status":$status,'
 '"responsetime":$request_time,'
 '"upstreamtime":"$upstream_response_time",'
 '"upstreamaddr":"$upstream_addr",'
 '"http_user_agent":"$http_user_agent",'
 '"Country":"$geoip2_data_country_name",'
 '"State":"$geoip2_data_state_name",'
 '"City":"$geoip2_data_city_name",'
 '"https":"$https"'
 '}';
 log_format main '$remote_addr - $remote_user [$time_local] "$request" '
 '$status $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';
 ## access_log path [format [buffer=size] [gzip[=level]] [flush=time] [if=condition]]; set log path and format name
 ## access_log off; disable logging
 access_log logs/access.log json;
}

Golang os/exec usage (notes)

Fri, 12 Nov 2021 11:34:30 +0800

Golang os/exec usage (notes)

Add extra environment variables before running a command

cmd := exec.Command("programToExecute")
additionalEnv := "FOO=bar"
newEnv := append(os.Environ(), additionalEnv)
cmd.Env = newEnv
out, err := cmd.CombinedOutput()
if err != nil {
 log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("%s", out)

Use os/exec with pipes or bash expressions on Linux

rcmd := `iw dev | awk '$1=="Interface"{print $2}'`
cmd := exec.Command("bash", "-c", rcmd)
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

Use os/exec with batch expressions on Windows

cmd := exec.Command("cmd", "/c", "ffmpeg -i myfile.mp4 myfile.mp3 && del myfile.mp4")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

Fix local language issues in Windows CMD with os/exec

cmd := exec.Command("cmd", "/c", "chcp 65001 && netsh WLAN show drivers")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

When using this method, remember to remove the first line of output (the chcp response) before further processing.

Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management

Wed, 10 Nov 2021 13:12:35 +0800

Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management

Planning considerations

During search, each shard uses one thread. Too many shards can slow things down.

A node has multiple thread pools by function. The search (count/search/suggest) pool size is int((# of allocated processors * 3) / 2) + 1 and the default queue_size is 1000. The write (index/delete/update/bulk) pool size is # of allocated processors and the default queue_size is also 1000. (Official docs - Thread pools)

Cloudflare Traffic sequence

Wed, 03 Nov 2021 13:11:29 +0800

Traffic sequence

Traffic to your application runs through the following sequence on Cloudflare’s edge:

DDoS
URL Rewrites
Page Rules
IP Access Rules
Bots
Firewall Rules
Rate Limiting
Managed Rules
Header Modification
Access
Workers

Regex for Markdown Syntax

Fri, 29 Oct 2021 10:19:41 +0800

# Headings
headerOne = `(#{1}\s)(.*)`
headeTwo = `(#{2}\s)(.*)`
headerThree = `(#{3}\s)(.*)`
headerFour = `(#{4}\s)(.*)`
headerFive = `(#{5}\s)(.*)`
headerSix = `(#{6}\s)(.*)`
# Bold and Italic Text
boldItalicText = `(\*|\_)+(\S+)(\*|\_)+`
# Links
linkText = `(\[.*\])(\((http)(?:s)?(\:\/\/).*\))`
# Images
imageFile = `(\!)(\[(?:.*)?\])(\(.*(\.(jpg|png|gif|tiff|bmp))(?:(\s\"|\')(\w|\W|\d)+(\"|\'))?\)`
# Unordered List
listText = `(^(\W{1})(\s)(.*)(?:$)?)+`
# Numbered Text
numberedListText = `(^(\d+\.)(\s)(.*)(?:$)?)+`
# Block Quotes
blockQuote = `((^(\>{1})(\s)(.*)(?:$)?)+`
# Inline Code
inlineCode = "(\\`{1})(.*)(\\`{1})"
# Code Block
codeBlock = "(\\`{3}\\n+)(.*)(\\n+\\`{3})"
# Horizontal Line
horizontalLine = `(\=|\-|\*){3}`
# Email Text
emailText = `(\<{1})(\S+@\S+)(\>{1})`
# TABLES
tableText = `(((\|)([a-zA-Z\d+\s#!@'"():;\\\/.\[\]\^<={$}>?(?!-))]+))+(\|))(?:\n)?((\|)(-+))+(\|)(\n)((\|)(\W+|\w+|\S+))+(\|$)`

Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection.

Wed, 20 Oct 2021 16:15:48 +0800

Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection.

# This will install GUI and make it as a default startup option and then restart the machine.
$ sudo yum install xrdp tigervnc-server

~# sudo su
~# passwd
~# systemctl enable --now xrdp

~# netstat -antup | grep xrdp
tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN 10202/xrdp
tcp 0 0 127.0.0.1:3350 0.0.0.0:* LISTEN 10201/xrdp-sesman

[Notes] Build an Ubuntu remote desktop environment on GCP

Wed, 20 Oct 2021 16:14:47 +0800

[Notes] Build an Ubuntu remote desktop environment on GCP

Install

# dependency
sudo apt-get install ubuntu-desktop gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal

# VNC Server
sudo apt-get install vnc4server

# After install, run vncserver and set the password when prompted
vncserver

Modify `~/.vnc/xstartup`

#!/bin/sh

# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
x-window-manager &

gnome-panel &
gnome-settings-daemon &
metacity &
nautilus &

Execute

# Kill the current vncserver and restart
vncserver -kill :1
# vncserver defaults to port 5900; :1 is 5901, and so on
vncserver :1

Start vncserver on reboot

@reboot /usr/bin/vncserver :1

Understand the crypto trio: Encode, Encrypt, and Hash

Mon, 18 Oct 2021 14:24:01 +0800

Understand the crypto trio: Encode, Encrypt, and Hash

Encoding

It does not change the data and has no encryption effect. It simply represents the data in another form. A classic example is Morse code.

JavaScript has two useful functions: encodeURI and decodeURI, which encode special characters in URLs (spaces, punctuation, etc.) into URL-safe formats.

Base64 encodes binary data into ASCII characters.

Huffman Coding is a lossless compression encoding algorithm. In short, it compresses by abbreviating frequently used symbols.

Encryption

Encryption and decryption require a key. In the simplest Caesar cipher, each letter is shifted by an offset, and that offset is the key for encryption/decryption.

Fix Nginx 500 errors (too many open files, connection)

Sat, 09 Oct 2021 11:51:06 +0800

Fix Nginx 500 errors (too many open files, connection)

Nginx 500 errors only show up in logs. Two common cases:

socket() failed (24: Too many open files) while connecting to upstream

$ sudo su - www-data
$ ulimit -n # check current limit (ulimit -a shows all params)
1024

# vim /etc/security/limits.conf # set nofile (max number of open files)
# add/modify the following two lines
* soft nofile 655360
* hard nofile 655360

ulimit -n # log out and log back in to see the new value
655360

# If ulimit -n is not 655360, run ulimit -n 655360 to force set it
# Then verify with ulimit -n or ulimit -Sn (soft) and ulimit -Hn (hard) (or ulimit -a).

# Calculate and set from system level
lsof | wc -l # count open files
sudo vim /etc/sysctl.conf
fs.file-max = 3268890
sudo sysctl -p

512 worker_connections are not enough while connecting to upstream

# /etc/nginx/nginx.conf
worker_connections 10240;

# Refer to Nginx CoreModule
# worker_processes 2;
# worker_rlimit_nofile 10240;
# events {
# # worker_connections 10240;
# }

# Increasing Nginx connections can slow down overall speed because php-cgi is not enough.
# Adjust as follows.

# php-cgi was started with phpfcgid_children="10" and phpfcgid_requests="500"
# ab was run on another server, connect via a switch using GBit ethernet
# http://till.klampaeckel.de/blog/archives/30-PHP-performance-III-Running-nginx.html

# vim /etc/nginx/nginx.conf
worker_connections 10240;
worker_rlimit_nofile

# vim /etc/init.d/php-fcgi
PHP_FCGI_CHILDREN=15
PHP_FCGI_MAX_REQUESTS=1000
change to
PHP_FCGI_CHILDREN=512 # or 150 and increase gradually, watch MySQL connections
PHP_FCGI_MAX_REQUESTS=10240

# The article's phpfcgid_stop() function is good and can be used if needed.
# phpfcgid_stop() {
# echo "Stopping $name."
# pids=`pgrep php-cgi`
# pkill php-cgi
# wait_for_pids $pids
# }

Super easy! Build a beautiful and handy zsh terminal

Thu, 07 Oct 2021 08:44:18 +0800

Sign git commits with GPG

Tue, 05 Oct 2021 18:28:06 +0800

Sign git commits with GPG

Gitlab-CI Introduction

Fri, 24 Sep 2021 11:11:03 +0800

Gitlab CI

Concept

Gitlab

DevOps
GitOps

Workflow

code push -> pipeline -> stage -> job

Design

plan -> code -> build -> test -> release -> deploy -> operate -> monitor -> plan

Runner Executors

Shell
VirtualBox
Docker
Docker Machine
Kubernetes
Else…

References

Runner

Register

gitlab-runner register

After register

concurrent = 1
check_interval = 0

[session_server]
 session_timeout = 1800

[[runners]]
 name = "public-shell"
 url = "https://gitlab.go2cloudten.com/"
 token = "-mdH9OAOzG5yPsf_AVnW"
 executor = "shell"

[[runners]]
 name = "public-docker"
 url = "https://gitlab.go2cloudten.com/"
 token = "AcEGPPKTS1uuQ_A_qpWy"
 executor = "docker"
 [runners.docker]
 dns = ["192.168.185.5", "192.168.185.6"]
 tls_verify = false
 image = "registry.go2cloudten.com/it/office_sop/node:12.13.0"
 privileged = true
 disable_entrypoint_overwrite = false
 oom_kill_disable = false
 disable_cache = false
 shm_size = 0
 pull_policy = "if-not-present"
 volumes = ["/cache"]

Repository

.gitlab-ci.yaml

stages:
 - domain

check-icp:
 stage: domain
 image: registry.go2cloudten.com/it/office_sop/icp
 tags:
 - docker
 script:
 - domains=$(awk -F '|' '{if($6 ~ "Y" && ($7 ~ "West" || $7 ~ "Yuqu")) print $3}' domains-info.md | sed 's/ //g' | sort | uniq)
 - if [[ "${domains}" == "" ]]; then telegram.sh 'There is no domain in list' ; else telegram.sh 'Start checking ICP.' ; fi
 - for i in ${domains}; do result=$(checkicp ${i}); if [[ "${result}" == "未备案" ]];then telegram.sh "${i} 未备案"; sleep 1 ;fi;done
 - telegram.sh 'ICP check completed.'
 only:
 - schedules

Common shell scripting tips

Wed, 22 Sep 2021 13:01:11 +0800

Docker Introduction

Fri, 17 Sep 2021 14:11:03 +0800

Docker

Concept

VM vs Container

VM - Base on OS
Container - Base on Application (Linux Kernel: Namespace and Cgroup)

Client to Server

Docker daemon - containerd, docker-containerd-shim, docker-runc
Docker client - cli command

docker cli -> docker daemon -> containerd -> runc -> namespace & cgroup

Image

Snapshots

Container

Read-Only processes on image

Hub / Registry

Store images

References

Docker commands

Dockerfile

ARG dist="/tmp/password"
ARG projectDir="/password"

FROM golang:1.16-alpine3.14 AS builder
RUN apk add build-base upx
ARG dist
ARG projectDir
WORKDIR ${projectDir}
COPY . .
RUN go build -trimpath -o main cmd/main.go
RUN upx -9 -o ${dist} main

FROM scratch
ARG dist
ENV TZ=Asia/Taipei
COPY --from=builder ${dist} /usr/local/bin/password

Dockerfile1

FROM alpine
CMD ["nc","-l","12345"]

Dockerfile2

FROM alpine
CMD ["echo","DOCKER"]

docker build command

docker build . -t program

docker build . -f Dockerfile -t test_mysql

docker build . -t hello:v1.1 --build-arg dist=/tmp/hello --build-arg projectDir=/hello

docker build

. docker/status
echo -e "${GREEN}Before build${RESET}"
docker image ls
docker build . -f docker/Dockerfile1 -t test1
docker build . -f docker/Dockerfile2 -t test2

docker image

. docker/status
echo -e "${GREEN}After build${RESET}"
docker image ls

docker run AND rm

. docker/status
echo -e "${GREEN}Run container1${RESET}"
docker run -d --name container1 test1
echo -e "${GREEN}Run container2${RESET}"
docker run -d --name container2 test2
echo -e "${GREEN}List alive containers${RESET}"
docker ps
echo -e "${GREEN}List all containers${RESET}"
docker ps -a
echo -e "${GREEN}Remove alive container${RESET}"
docker rm -f container1
echo -e "${GREEN}List all containers${RESET}"
docker ps -a
echo -e "${GREEN}Remove exit container${RESET}"
docker rm container2
echo -e "${GREEN}List all containers${RESET}"
docker ps -a

docker pull AND rmi

. docker/status
echo -e "${GREEN}List all image${RESET}"
docker image ls
echo -e "${GREEN}Pull alpine image${RESET}"
docker pull alpine
echo -e "${GREEN}List all image${RESET}"
docker image ls

docker rmi

. docker/status
echo -e "${GREEN}Remove alpine image${RESET}"
docker rmi alpine
echo -e "${GREEN}List all image${RESET}"
docker image ls

prune

docker system prune -f --volumes

docker history

. docker/status
echo -e "${GREEN}History of test1${RESET}"
docker history test1
echo -e "${GREEN}History of mysql:8${RESET}"
docker history mysql:8

Docker remote

Edit service file

# /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock -H tcp://0.0.0.0:2375

Restart service

systemctl daemon-reload
systemctl restart docker

Specify DOCKER_HOST

. docker/status
echo -e "${GREEN}List images on 192.168.185.9${RESET}"
DOCKER_HOST=192.168.185.9:2375 docker images

Docker-compose

version: "3"

services:
 svn:
 image: zeyanlin/svn
 environment:
 - LDAP_HOSTS=${LDAP_HOSTS}
 - LDAP_BASE_DN=${LDAP_BASE_DN}
 - LDAP_BIND_DN=${LDAP_BIND_DN}
 - LDAP_ADMIN_PASS=${LDAP_ADMIN_PASS}
 ports:
 - 8000:80
 - 3690:3690
 depends_on:
 - ldap
 ldap:
 image: zeyanlin/openldap
 environment:
 - LDAP_DOMAIN=${LDAP_DOMAIN}
 - LDAP_ADMIN_PASS=${LDAP_ADMIN_PASS}
 ports:
 - 389:389
 - 636:636
 php:
 image: zeyanlin/phpldapadmin
 environment:
 - LDAP_HOSTS=${LDAP_HOSTS}
 ports:
 - 80:80
 depends_on:
 - ldap

Env

LDAP_HOSTS=ldap
LDAP_DOMAIN="knowhow.fun"
LDAP_BASE_DN="dc=knowhow,dc=fun"
LDAP_BIND_DN="cn=admin"
LDAP_ADMIN_PASS="123qwe"

Golang benchmarks

Thu, 02 Sep 2021 12:50:33 +0800

Golang benchmarks

Basics

Benchmarks are used for performance testing. Functions should import the testing package and define functions that start with Benchmark. The parameter type is testing.B, and the target function is called repeatedly inside the benchmark loop.

➜ go test -bench=. -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 250 4682682 ns/op
PASS
ok pkg06 1.875s
➜ go test -bench=. -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 249 4686452 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 1.854s

How bench works

The benchmark function keeps running until b.N is no longer valid; it is the number of iterations.
b.N starts at 1. If the benchmark completes within 1 second (default), b.N increases and the benchmark runs again.
b.N increases in the sequence 1,2,5,10,20,50,... and the benchmark reruns.
The result above means it ran 250 times in 1 second, with each run taking 4682682 ns.
The -12 suffix relates to GOMAXPROCS. The default value is the number of CPUs visible to the Go process at startup. You can change it with -cpu, and pass multiple values to run multiple benchmarks.

Run with multiple CPU counts

➜ go test -bench=. -cpu=1,2,4 -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib 244 4694667 ns/op 0 B/op 0 allocs/op
BenchmarkFib-2 255 4721201 ns/op 0 B/op 0 allocs/op
BenchmarkFib-4 256 4756392 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 5.826s

Run benchmarks multiple times with count

Due to CPU throttling, memory locality, background work, GC activity, etc., a single run may be noisy. It is common to run benchmarks multiple times.

Back up databases with mydumper

Wed, 01 Sep 2021 17:58:48 +0800

Docker Security Best Practices: Cheat Sheet

Fri, 20 Aug 2021 23:14:30 +0800

Docker Security Best Practices: Cheat Sheet

Image Security

Use Trusted Images

Unprivileged User

FROM base_image
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

User ID Namespace

Segregate namespaces to prevent container privilege escalation from affecting the host.

To mitigate this risk, configure your host and the Docker daemon to use a separate namespace with the --userns-remap option.

Container Runtime Security

Forbid New Privileges

For enhanced security, it is recommended to explicitly forbid the addition of new privileges after container creation using this option: --security-opt=no-new-privileges.

[Translated] Go inline strategy and limitations

Sat, 14 Aug 2021 20:33:41 +0800

package main

import "fmt"

func main() {
 n := []float32{120.4, -46.7, 32.50, 34.65, -67.45}
 fmt.Printf("The total is %.02f\n", sum(n))
}

func sum(s []float32) float32 {
 var t float32
 for _, v := range s {
 if t < 0 {
 t = add(t, v)
 } else {
 t = sub(t, v)
 }
 }
 return t
}

func add(a, b float32) float32 { return a + b }
func sub(a, b float32) float32 { return a - b }

Run with -gflags="-m" to show which functions are inlined.

SQL Join types explained visually

Fri, 13 Aug 2021 16:16:54 +0800

SQL Join types explained visually

Little Git Tricks: Use .mailmap to Merge Different Authors

Thu, 05 Aug 2021 13:42:14 +0800

Little Git Tricks: Use .mailmap to Merge Different Authors

.mailmap file ==> Name you want to keep <email> Name you no longer want <email>

John Doe John Doe
John Doe John Doe
John Doe john doe
Max Example Max

before

$ git shortlog -se
 1 John Doe <John@Doe.org>
 1 John Doe <John@doe.org>
 2 John Doe <john@doe.org>
 1 Max <max@test.co.uk>
 3 Max Example <hi@test.com>
 1 john doe <john@doe.org>

after

$ git shortlog -se
 5 John Doe <john@doe.org>
 4 Max Example <hi@test.com>

Go Modules workflow for private Git repositories

Fri, 23 Jul 2021 09:40:39 +0800

Go Modules workflow for private Git repositories

go env -w GOPRIVATE=github.com/appleboy
1. go env -w GONOPROXY=github.com/appleboy
2. go env -w GONOSUMDB=github.com/appleboy
git config --global url."https://$USERNAME:$ACCESS_TOKEN@github.com".insteadOf "https://github.com"
1. git config --global url.ssh://git@your.private.git/.insteadOf https://your.private.git/

Gin documentation (Chinese)

Thu, 22 Jul 2021 15:55:48 +0800

Customize route log format

default

[GIN-debug] POST /foo --> main.main.func1 (3 handlers)
[GIN-debug] GET /bar --> main.main.func2 (3 handlers)
[GIN-debug] GET /status --> main.main.func3 (3 handlers)

import (
 "log"
 "net/http"

 "github.com/gin-gonic/gin"
)

func main() {
 r := gin.Default()
 gin.DebugPrintRouteFunc = func(httpMethod, absolutePath, handlerName string, nuHandlers int) {
 log.Printf("endpoint %v %v %v %v\n", httpMethod, absolutePath, handlerName, nuHandlers)
 }

 r.POST("/foo", func(c *gin.Context) {
 c.JSON(http.StatusOK, "foo")
 })

 r.GET("/bar", func(c *gin.Context) {
 c.JSON(http.StatusOK, "bar")
 })

 r.GET("/status", func(c *gin.Context) {
 c.JSON(http.StatusOK, "ok")
 })

 // Listen and Server in http://0.0.0.0:8080
 r.Run()
}

Run multiple services with Gin

package main

import (
 "log"
 "net/http"
 "time"

 "github.com/gin-gonic/gin"
 "golang.org/x/sync/errgroup"
)

var (
 g errgroup.Group
)

func router01() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 01",
 },
 )
 })

 return e
}

func router02() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 02",
 },
 )
 })

 return e
}

func main() {
 server01 := &http.Server{
 Addr: ":8080",
 Handler: router01(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 server02 := &http.Server{
 Addr: ":8081",
 Handler: router02(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 g.Go(func() error {
 return server01.ListenAndServe()
 })

 g.Go(func() error {
 return server02.ListenAndServe()
 })

 if err := g.Wait(); err != nil {
 log.Fatal(err)
 }
}

XML, JSON, YAML, and ProtoBuf rendering (output formats)

SecureJSON

SecureJSON prevents JSON hijacking. If the response is an array, it will prefix the output with “while(1)” by default.

docker-compose yaml problem

Mon, 19 Jul 2021 15:40:36 +0800

issue

# ./docker-compose up -d
Creating network "gogs_default" with the default driver
Creating gogs_mysql_1 ... done
Creating gogs_gogs_1 ... error

ERROR: for gogs_gogs_1 Cannot create container for service gogs: invalid port specification: "133342"

ERROR: for gogs Cannot create container for service gogs: invalid port specification: "133342"
ERROR: Encountered errors while bringing up the project.

services:
 gogs:
 ports:
 - 2222:22

YAML supports so-called “base-60 floating-point numbers,” which is useful for time calculations.

Therefore, 2222:22 is interpreted as 2222 * 60 + 22, which is 133342.

What happens after you type a URL in the browser and press Enter?

Sun, 18 Jul 2021 23:45:45 +0800

What happens after you type a URL in the browser and press Enter?

I can feel the speed — optimize zsh and oh my zsh cold start

Thu, 15 Jul 2021 18:19:06 +0800

I can feel the speed — optimize zsh and oh my zsh cold start

Profiling

# .zshrc
zmodload zsh/zprof

$ /bin/zsh
$ zprof

num calls time self name
-----------------------------------------------------------------------------------
 1) 1 395.66 395.66 33.10% 395.59 395.59 33.09% _zsh_nvm_auto_use
 2) 1 216.22 216.22 18.09% 216.13 216.13 18.08% nvm_die_on_prefix
 3) 1 648.00 648.00 54.20% 168.85 168.85 14.12% nvm_auto
 4) 2 479.15 239.57 40.08% 160.50 80.25 13.43% nvm
 5) 1 102.30 102.30 8.56% 84.99 84.99 7.11% nvm_ensure_version_installed
 6) 2 51.21 25.60 4.28% 29.55 14.78 2.47% compinit
 7) 1 680.18 680.18 56.89% 22.17 22.17 1.85% _zsh_nvm_load
 8) 2 21.66 10.83 1.81% 21.66 10.83 1.81% compaudit
 9) 1 17.31 17.31 1.45% 17.31 17.31 1.45% nvm_is_version_installed
10) 193 17.43 0.09 1.46% 14.50 0.08 1.21% _zsh_autosuggest_bind_widget
[Redacted]

The zprof module only reports the time for each zsh function, so it is useful for finding oh-my-zsh plugins that slow down cold start. To profile the entire .zshrc, use xtrace. Add the following at the top of .zshrc:

[Community] Cisco UCS CIMC series

Thu, 15 Jul 2021 17:16:52 +0800

SQL queries don't start with SELECT

Wed, 14 Jul 2021 18:11:46 +0800

SQL queries don’t start with SELECT

SQL queries happen in this order

FROM/JOIN and all the ON conditions
WHERE
GROUP BY
HAVING
SELECT (including window functions)
ORDER BY
LIMIT

6 YAML Features most programmers don't know

Wed, 07 Jul 2021 15:08:31 +0800

6 YAML Features most programmers don’t know

There are more examples that are similarly dangerous as Tom Ritchford pointed out

013 is mapped to 11 as the leading zero triggers the octal notation
4:30 is mapped to 270. Max Werner Kaul-Gothe and Niklas Baumstark informed me that this is automatically converted to minutes (or seconds?) as it is interpreted as a duration: 4*60 + 30 = 270 . Interestingly, this pattern still “works” with 1:1:1:1:1:1:1:1:4:30 .

Multi-Line String

mail_signature: |
 Martin Thoma
 Tel. +49 123 4567

{ "mail_signature": "Martin Thoma\nTel. +49 123 4567" }

Anchor

The & defined a variable emailAddress with the value "info@example.de". The * then indicated that the name of a variable follows.

Draw Diagrams With Markdown

Tue, 29 Jun 2021 15:55:35 +0800

Draw Diagrams With Markdown

ansible.builtin.slurp - read file content

Mon, 31 May 2021 17:15:04 +0800

ansible.builtin.slurp - read file content

---
- name: Use HTTP POST to upload file
 hosts: all

 tasks:
 - name: Read binary file content
 slurp:
 path: "/bin/ls"
 register: bin_file

 - name: Send HTTP POST Request
 uri:
 url: "https://your_server/upload.php"
 headers:
 Accept: "application/json"
 Content-Type: "application/octet-stream"
 method: POST
 validate_certs: false
 body: "{{ bin_file.content }}"
 status_code:
 - 200
 - 201
 register: upload_result

 - name: Display upload_result
 debug:
 var: upload_result

Using slurp avoids the limitations of lookup(‘file’).

You can read files on managed hosts, or use delegate_to: localhost to read files on the controller.
You can read binary files for further processing, such as base64 encoding.

Nginx request_time and upstream_response_time explained

Fri, 14 May 2021 16:04:04 +0800

Nginx request_time and upstream_response_time explained

Time definitions

request_time

Time from the first byte of the client request to the completion of response data being sent. $request_time includes time to receive the request, time for the upstream to respond, and time to send the response (excluding log write time).

upstream_response_time

Time from Nginx establishing a connection to the upstream until all data is received and the connection is closed.

upstream_connect_time

Time to connect to the upstream server. If using an encrypted protocol, this includes handshake time.

How to use a here documents to write data to a file in bash script

Fri, 07 May 2021 11:37:35 +0800

How to use a here documents to write data to a file in bash script

allow here-documents within shell scripts to be indented in a natural fashion using `EOF<<-`

command <<-EOF
 msg1
 msg2
 $var on line
EOF

Disabling pathname/parameter/variable expansion, command substitution, arithmetic expansion with `'EOF'`

#!/bin/bash
OUT=/tmp/output.txt

echo "Starting my script..."
echo "Doing something..."
# No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word.
# If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document
# are not expanded. So EOF is quoted as follows
cat <<'EOF' >$OUT
 Status of backup as on $(date)
 Backing up files $HOME and /etc/
EOF

echo "Starting backup using rsync..."

Upgrade database failed after ugrade to Zabbix 5...

Mon, 03 May 2021 16:33:23 +0800

High Performance Go Workshop

Wed, 28 Apr 2021 16:37:05 +0800

High Performance Go Workshop

Bypass X-Frame-Options with Nginx

Mon, 26 Apr 2021 17:39:33 +0800

Bypass X-Frame-Options with Nginx

The X-Frame-Options HTTP response header tells the browser whether a page can be displayed inside <frame>, <iframe>, <embed>, or <object>. Sites can prevent clickjacking by ensuring their pages are not embedded elsewhere. By using Nginx as a forward proxy, we can bypass X-Frame-Options and embed a third-party page in our own page.

X-Frame-Options has three possible values:

deny: the page cannot be displayed in a frame, even on the same origin.
sameorigin: the page can be displayed in a frame on the same origin.
allow-from uri: the page can be displayed in a frame only from the specified origin.

When Chrome tries to load frame content and X-Frame-Options denies it, the console shows an error like: Refuse to display 'http://192.168.20.101:8080' in a frame because it set 'X-Frame-Options' to 'deny'.

Setting up JWT Authentication

Fri, 23 Apr 2021 11:13:34 +0800

SSH failing with Error : fatal: daemon() failed: No such device

Thu, 04 Mar 2021 18:48:39 +0800

SSH failing with Error : fatal: daemon() failed: No such device

/var/log/secure

Oct 10 10:58:05 vps sshd[23799]: fatal: daemon() failed: No such device

# rm -vf /dev/null
removed `/dev/null`
-bash-3.2# mknod /dev/null c 1 3
Started SSH and the SSH started responding:

# service sshd restart
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
-bash-3.2# service sshd status
openssh-daemon (pid 30608) is running…

Enable GitLab project integration: Mattermost slash commands

Thu, 28 Jan 2021 11:50:41 +0800

Enable GitLab project integration: Mattermost slash commands

Nginx SSL/TLS configuration with TLSv1.2 and TLSv1.3 - ECDHE and strong ciphers suite (Openssl 1.1.1)

Fri, 22 Jan 2021 13:49:17 +0800

Nginx SSL/TLS configuration with TLSv1.2 and TLSv1.3 - ECDHE and strong ciphers suite (Openssl 1.1.1)

ssl.conf

##
# SSL Settings (TLSv1.2 and TLSv1.3)
##
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'TLS13+AESGCM+AES128:EECDH+AES128';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_ecdh_curve X25519:sect571r1:secp521r1:secp384r1

universal-ssl.conf

##
# SSL Settings (TLSv1.0 + TLSv1.1 + TLSv1.2 + TLSv1.3)
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers 'TLS13+AESG+AES128:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES25GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-R-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-S:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_ecdh_curve X25519:sect571r1:secp521r1:secp384r1;

Let Zsh read macOS system proxy settings and set env vars

Mon, 11 Jan 2021 14:50:11 +0800

Let Zsh read macOS system proxy settings and set env vars

$ system_profiler SPNetworkDataType # get full network configuration

$ networksetup -listallnetworkservices # list all network services
$ networksetup -getwebproxy Wi-Fi # get system proxy settings for a specific service

$ scutil --proxy # get enabled proxy settings (wrapper for system_profiler)

2021 on Ricky

How Nginx Hides Upstream Errors

Nginx allows enabling next upstream for the following seven retryable error codes

When upstream returns 404, return a 200 response with a not-found image

Argo CD ApplicationSet Controller: The World Turns for Me!

Go AES Encryption and Decryption in Three Modes (CBC/ECB/CFB)

Understanding Cilium Series (1): Introduction to Cilium

Current status of k8s Service load balancing implementations

Userspace

Kubernetes Without kube-proxy

Quick-Start

Zsh tab-completion not working

Question

Linkerd 2.8 - Build a Simple and Secure Multi-Cluster Kubernetes Architecture

Best Practices for Writing Bash Scripts

Use shift to read function arguments

Day 28 - Introduction to Useful Third-Party Kubernetes Tools

Stern/Kail

5 DevSecOps open source projects to know

Clair

Sigstore

KubeLinter

Open Policy Agent and Gatekeeper

Falco

[Kubernetes] Service Overview

Define Service

With selector

How Go Reads a 16GB File in 25 Seconds

Bind JSON with jsoniter in Gin

simple

custom

implement BindingBody interface

Nginx notes

Record Nginx configuration file and explanation.

files structure

geoip.conf

upstream.conf

default.conf

nginx.conf

Golang os/exec usage (notes)

Add extra environment variables before running a command

Use os/exec with pipes or bash expressions on Linux

Use os/exec with batch expressions on Windows

Fix local language issues in Windows CMD with os/exec

Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management

Planning considerations

During search, each shard uses one thread. Too many shards can slow things down.

Cloudflare Traffic sequence

Traffic sequence

Regex for Markdown Syntax

Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection.

[Notes] Build an Ubuntu remote desktop environment on GCP

Install

Modify ~/.vnc/xstartup

Execute

Start vncserver on reboot

Understand the crypto trio: Encode, Encrypt, and Hash

Encoding

Encryption

Fix Nginx 500 errors (too many open files, connection)

socket() failed (24: Too many open files) while connecting to upstream

512 worker_connections are not enough while connecting to upstream

Super easy! Build a beautiful and handy zsh terminal

Sign git commits with GPG

Gitlab-CI Introduction

Gitlab CI

Concept

Gitlab

Workflow

Design

Runner Executors

References

Runner

Register

After register

Repository

.gitlab-ci.yaml

Common shell scripting tips

Docker Introduction

Docker

implement `BindingBody` interface

Modify `~/.vnc/xstartup`

allow here-documents within shell scripts to be indented in a natural fashion using `EOF<<-`

Disabling pathname/parameter/variable expansion, command substitution, arithmetic expansion with `'EOF'`