[{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/tags/commandline/","section":"Tags","summary":"","title":"Commandline","type":"tags"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/","section":"Jin Li","summary":"","title":"Jin Li","type":"page"},{"content":" Why migrate? # My previous shell setup used Oh My Zsh with Powerlevel10k. It was polished and fast, but it tied the prompt to Zsh. I wanted the same visual language on macOS, Linux, HPC clusters, and Windows PowerShell, so I moved the prompt to Oh My Posh.\nOh My Posh separates the prompt theme from the shell. One JSON configuration can therefore be initialized by both Zsh and PowerShell. The result is my Koi-fish (锦鲤) theme, jinli.omp.json, with Catppuccin-inspired colors.\nWhat the Jinli theme shows # The theme is designed to keep useful context visible without turning the prompt into a wall of information:\nThe path segment uses context-aware icons for home, locked directories, GitHub, Git, npm, Downloads, Pictures, and ordinary folders. Git information shows the repository, branch, worktree state, and change counts. Python and Node versions appear on the right when they are relevant. SSH sessions receive a remote icon, making it obvious when a shell is not local. Battery, time, command duration, and exit status adapt to the machine and remain aligned on the right. The same theme works in Zsh and PowerShell, while the surrounding setup also supports Linux, macOS, Windows, and NixOS. The configuration model # The important change is separating stable shared configuration from local configuration that tools are allowed to modify:\nThe repository’s .zshrc.common contains the shared Zsh setup. ~/.zshrc is a normal, user-managed file created by the installer. It adds the Oh My Posh executable directory before sourcing .zshrc.common. Tool installers such as Conda and OpenClaw can append to ~/.zshrc without changing the Git-managed shared file. An existing ~/.zshrc.local is still sourced afterward for backward compatibility with older installations. The theme is linked to ~/.config/oh-my-posh/jinli.omp.json on macOS/Linux. For example, machine-specific settings belong in ~/.zshrc:\n1 2 export PATH=\u0026#34;$HOME/.local/bin:$PATH\u0026#34; alias connect-hpc=\u0026#39;ssh user@example.org\u0026#39; This arrangement keeps private variables, cluster modules, workstation-only aliases, and local tool paths out of the shared repository configuration.\nOh My Zsh remains optional # Oh My Posh is the prompt engine, not a replacement for every Zsh plugin. The shared configuration enables zsh-autosuggestions and fast-syntax-highlighting.\nIf Oh My Zsh is already installed, the installer places and loads these plugins through its custom plugin directory. Otherwise, it installs them under ~/.local/share/zsh/plugins and loads them directly. Oh My Zsh itself is not installed just to enable these plugins.\nThe shared Zsh setup also provides case-insensitive, substring-aware completion. For example, cd dev\u0026lt;Tab\u0026gt; can complete a directory such as itrip-dev-doc. It enables a directory stack through AUTO_PUSHD and PUSHD_IGNORE_DUPS, so cd -\u0026lt;number\u0026gt; and dirs -v can be useful when moving between projects.\nInstallation # macOS and Linux # Run the installer as a normal user. On macOS, install Homebrew first:\n1 2 3 curl -fsSLO https://raw.githubusercontent.com/jin-li/ShellConfig/main/install-oh-my-posh.sh chmod +x install-oh-my-posh.sh ./install-oh-my-posh.sh The installer asks where to clone the repository. Press Enter to use ~/Documents/GitHub/ShellConfig, or enter another directory. You can set SHELL_CONFIG_DIR beforehand to use it as the suggested destination.\nThe script installs Oh My Posh and its Zsh dependencies, installs the plugins, creates the theme link, creates the local ~/.zshrc, and offers to install Meslo Nerd Font. Restart the terminal afterward or run exec zsh.\nTo update an existing installation, run the updater from the repository itself:\n1 2 cd /path/to/ShellConfig ./update.sh The updater uses its own location to find the repository, so it continues to work even when the repository was installed somewhere other than the default directory. It updates the checkout and theme link while preserving the local ~/.zshrc.\nWindows PowerShell # Run the installer as your normal user:\n1 2 3 Set-ExecutionPolicy -Scope Process Bypass Invoke-WebRequest https://raw.githubusercontent.com/jin-li/ShellConfig/main/install-oh-my-posh.ps1 -OutFile install-oh-my-posh.ps1 .\\install-oh-my-posh.ps1 The PowerShell script uses WinGet when Oh My Posh or Git is missing, asks where to clone the repository, and updates only the ShellConfig-managed block in $PROFILE. Existing user and tool configuration in the profile is preserved. The theme is loaded directly from the selected repository directory; Windows does not use the Unix ~/.config theme link.\nNixOS # The repository also exposes a NixOS module through its flake:\n1 2 3 4 5 inputs.shell-config.url = \u0026#34;github:jin-li/ShellConfig\u0026#34;; modules = [ inputs.shell-config.nixosModules.default ]; programs.shellConfig.enable = true; users.users.\u0026lt;username\u0026gt;.shell = pkgs.zsh; Then rebuild the host and start a new session:\n1 2 sudo nixos-rebuild switch --flake .#\u0026lt;host\u0026gt; exec zsh Fonts and terminal setup # The theme uses Nerd Font glyphs. Install and select MesloLGM Nerd Font in the terminal profile; otherwise icons may appear as boxes. For WSL, run the Linux installer inside WSL but install and configure the font on Windows.\nHPC and machines without sudo # The macOS/Linux installer asks about sudo before attempting package installation. It checks for curl, git, unzip, and zsh. If sudo is unavailable and only Zsh is missing, it can build ncurses and Zsh under ~/.local by default, or under $SHELL_CONFIG_PREFIX when that variable is set. A compiler and make must already be available, usually through cluster modules.\nOther missing dependencies must be provided by the cluster or installed in the user’s own environment. The local ~/.zshrc is the right place to load modules and add user-local executable directories to PATH.\nKeeping Neovim separate # The LazyVim installer is deliberately separate from the prompt installer. Oh My Posh configures the shell prompt, while LazyVim manages the Neovim configuration and editor plugins.\n1 2 3 curl -fsSLO https://raw.githubusercontent.com/jin-li/ShellConfig/main/install_LazyVim.sh chmod +x install_LazyVim.sh ./install_LazyVim.sh The installer checks for Neovim 0.11.2 or newer, backs up existing Neovim configuration and data, and clones the LazyVim starter into ~/.config/nvim. It no longer manages a legacy ~/.vimrc. Start nvim afterward to install plugins, then run :checkhealth or :LazyHealth.\nResult # The prompt now has one theme across Zsh and PowerShell, the shared pieces live in Git, and machine-specific tools can safely edit the local ~/.zshrc. That structure makes the configuration easier to carry between a laptop, Linux workstation, HPC cluster, and Windows without turning every machine-specific change into a shared-config conflict.\n","date":"5 August 2026","externalUrl":null,"permalink":"/en/p/migrating-from-powerlevel10k-to-oh-my-posh/","section":"Posts","summary":" Why migrate? # My previous shell setup used Oh My Zsh with Powerlevel10k. It was polished and fast, but it tied the prompt to Zsh. I wanted the same visual language on macOS, Linux, HPC clusters, and Windows PowerShell, so I moved the prompt to Oh My Posh.\nOh My Posh separates the prompt theme from the shell. One JSON configuration can therefore be initialized by both Zsh and PowerShell. The result is my Koi-fish (锦鲤) theme, jinli.omp.json, with Catppuccin-inspired colors.\n","title":"Migrating from Powerlevel10k to Oh My Posh","type":"post"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/tags/oh-my-posh/","section":"Tags","summary":"","title":"Oh-My-Posh","type":"tags"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/post/","section":"Posts","summary":"","title":"Posts","type":"post"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/tags/powershell/","section":"Tags","summary":"","title":"PowerShell","type":"tags"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/categories/software/","section":"Categories","summary":"","title":"Software","type":"categories"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"5 August 2026","externalUrl":null,"permalink":"/en/tags/zsh/","section":"Tags","summary":"","title":"Zsh","type":"tags"},{"content":"","date":"2026年8月5日","externalUrl":null,"permalink":"/tags/%E5%91%BD%E4%BB%A4%E8%A1%8C/","section":"Tags","summary":"","title":"命令行","type":"tags"},{"content":"","date":"2026年8月5日","externalUrl":null,"permalink":"/categories/%E8%BD%AF%E4%BB%B6/","section":"Categories","summary":"","title":"软件","type":"categories"},{"content":" Background # I was previously using Cloudflare Tunnel (free plan) to reverse proxy my Nextcloud container. While convenient, the free tier has a 100MB upload limitation that became problematic for my usage.\nThis article documents my migration from Cloudflare Tunnel to direct Traefik reverse proxy with Let\u0026rsquo;s Encrypt TLS certificates, including the complete setup, issues encountered, and troubleshooting steps.\nWhy Migrate? # Cloudflare Tunnel (Free Plan) Pros # ✅ Automatic SSL/TLS termination ✅ No port forwarding needed ✅ Works with dynamic IPs ✅ Built-in DDoS protection Cloudflare Tunnel (Free Plan) Cons # ❌ 100MB upload limit (major issue) ❌ Bandwidth limits ❌ Tunnel overhead adds latency ❌ Dependency on Cloudflare infrastructure Why Traefik + Let\u0026rsquo;s Encrypt # ✅ No upload limits (configurable) ✅ Direct connection (lower latency) ✅ Automatic certificate management ✅ Full control over routing ✅ Can handle multiple services Current Architecture # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ┌────────────────────────────────────────────────────────────┐ │ Internet │ └─────────────────────────────┬──────────────────────────────┘ │ ┌───────────▼───────────┐ │ Router │ │ Port Forwarding │ │ 80, 443 → Server │ └────────────┬────────────┘ │ ┌────────────▼───────────┐ │ Traefik │ │ (TLS Termination) │ └────────────┬────────────┘ │ ┌────────────▼───────────┐ │ Nextcloud │ │ (Docker) │ └─────────────────────────┘ Before: Internet → Cloudflare Tunnel → Traefik → Nextcloud\nAfter: Internet → Traefik → Nextcloud\nPrerequisites # Docker and docker-compose installed Traefik container running Domain name (e.g., nextcloud2.example.com) Cloudflare account with domain DNS managed Server with public IP (or dynamic DNS) Step 1: Remove Cloudflare Tunnel # Stop the cloudflared container # 1 2 cd /path/to/cloudflared docker compose down Update cloudflared config (optional) # If you want to keep cloudflared for other services, remove the Nextcloud entry from config.yaml:\n1 2 3 4 5 6 7 8 9 10 11 12 ingress: # Remove these lines: # - hostname: \u0026#34;cloud.example.com\u0026#34; # service: https://traefik:443 # originRequest: # noTLSVerify: true # - hostname: \u0026#34;nextcloud2.example.com\u0026#34; # service: https://traefik:443 # originRequest: # noTLSVerify: true # Keep other entries... - service: http_status:404 Step 2: Update DNS in Cloudflare # For each domain you want to use with Traefik:\nGo to Cloudflare Dashboard → DNS Find the DNS record (e.g., nextcloud2.example.com) Change from Orange Cloud (Proxied) to Gray Cloud (DNS Only) Point to your server\u0026rsquo;s public IP address Important: For Let\u0026rsquo;s Encrypt DNS challenge to work, traffic must go directly to your server, not through Cloudflare\u0026rsquo;s proxy.\nStep 3: Configure Traefik\u0026rsquo;s Let\u0026rsquo;s Encrypt DNS Challenge # Update traefik.yml # Ensure you have the ACME configuration with Cloudflare DNS challenge:\n1 2 3 4 5 6 7 8 9 10 certificatesResolvers: le: acme: email: your-email@example.com storage: /letsencrypt/acme.json dnsChallenge: provider: cloudflare resolvers: - \u0026#34;1.1.1.1:53\u0026#34; - \u0026#34;1.0.0.1:53\u0026#34; Set Cloudflare API Token # Create or update .env file in the traefik directory:\n1 2 3 CF_API_KEY=your_cloudflare_api_token CF_API_EMAIL=your-email@example.com TZ=UTC Note: For DNS challenge, you need a Cloudflare API Token (not Global API Key) with the following permissions:\nZone DNS: Edit zone DNS Update docker-compose.yml # Ensure environment variables are passed to Traefik:\n1 2 3 4 environment: - CF_API_KEY=${CF_API_KEY} - CF_API_EMAIL=${CF_API_EMAIL} - TZ=${TZ:-UTC} Step 4: Configure Dynamic Router with TLS # Create a file in traefik/dynamic/ directory (e.g., nextcloud2.yml):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 http: routers: nextcloud2-router-http: rule: \u0026#34;Host(`nextcloud2.example.com`)\u0026#34; entryPoints: - web middlewares: - redirect-https service: nextcloud2-service nextcloud2-router-https: rule: \u0026#34;Host(`nextcloud2.example.com`)\u0026#34; entryPoints: - websecure tls: certResolver: le # Important: specifies Let\u0026#39;s Encrypt resolver service: nextcloud2-service services: nextcloud2-service: loadBalancer: servers: - url: \u0026#34;http://nextcloud:80\u0026#34; # Nextcloud container name passHostHeader: true middlewares: redirect-https: redirectScheme: scheme: https permanent: true Key Configuration Points # Two routers needed:\nHTTP router for redirect (port 80) HTTPS router for actual traffic (port 443) certResolver: le is crucial - without this, Traefik uses its internal self-signed certificate\npassHostHeader: true - preserves the original Host header\nStep 5: Restart Traefik # 1 2 cd /path/to/traefik docker compose up -d --force-recreate traefik Traefik will:\nWatch for new config files Request Let\u0026rsquo;s Encrypt certificate via DNS challenge Create _acme-challenge.nextcloud2.example.com TXT record Wait for DNS propagation Verify ownership Download certificate Step 6: Verify Setup # Check if certificate was issued # 1 2 3 4 5 # View all certificates cat /path/to/traefik/acme.json | python3 -m json.tool # Check specific domain cat /path/to/traefik/acme.json | grep -A 5 \u0026#39;nextcloud2\u0026#39; Test HTTPS connection # 1 2 3 4 5 6 7 8 # Verify certificate echo | openssl s_client -connect nextcloud2.example.com:443 -servername nextcloud2.example.com 2\u0026gt;/dev/null | openssl x509 -noout -issuer -dates # Verify domain matches echo | openssl s_client -connect nextcloud2.example.com:443 -servername nextcloud2.example.com 2\u0026gt;/dev/null | openssl x509 -noout -subject -text | grep -E \u0026#34;(Subject:|DNS:)\u0026#34; # Test connection curl -I https://nextcloud2.example.com Add More Domains # To add additional domains, simply create more config files:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 # Create config for another domain cat \u0026gt; /path/to/traefik/dynamic/blog.yml \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; http: routers: blog-router-http: rule: \u0026#34;Host(`blog.example.com`)\u0026#34; entryPoints: - web middlewares: - redirect-https service: blog-service blog-router-https: rule: \u0026#34;Host(`blog.example.com`)\u0026#34; entryPoints: - websecure tls: certResolver: le service: blog-service services: blog-service: loadBalancer: servers: - url: \u0026#34;http://blog:80\u0026#34; passHostHeader: true middlewares: redirect-https: redirectScheme: scheme: https permanent: true EOF Traefik auto-reloads every 10 seconds, so no restart needed.\nCommon Issues \u0026amp; Solutions # 1. Certificate Shows as Self-Signed # Problem: Browser shows \u0026ldquo;Not Secure\u0026rdquo; or self-signed certificate warning\nSolution: Add certResolver: le to your router config. Without this, Traefik uses its internal certificate.\n2. DNS Challenge Failed # Problem: Certificate not being issued, errors in logs\nSolution:\nVerify Cloudflare API token has correct permissions Check Traefik has access to API token via environment variables Ensure DNS is not proxied through Cloudflare (gray cloud) Check logs: docker logs traefik 3. Certificate Exists but Not Valid for Domain # Problem: Let\u0026rsquo;s Encrypt cert exists, but not for your domain\nSolution: Traefik creates certificates on-demand. Simply access the domain and Traefik will automatically request the certificate.\n4. HTTP to HTTPS Redirect Not Working # Problem: HTTP requests don\u0026rsquo;t redirect to HTTPS\nSolution: Ensure you have both HTTP and HTTPS routers configured with the redirect middleware.\n5. Nextcloud Shows \u0026ldquo;Trusted Domain Error\u0026rdquo; # Problem: Nextcloud doesn\u0026rsquo;t trust the new domain\nSolution: Update Nextcloud\u0026rsquo;s trusted domains:\nIn nextcloud/docker-compose.yml, update:\n1 2 environment: - NEXTCLOUD_TRUSTED_DOMAINS=nextcloud2.example.com,cloud.example.com,previous-domain.com Or in Nextcloud config (config/config.php):\n1 2 3 4 5 \u0026#39;trusted_domains\u0026#39; =\u0026gt; [ \u0026#39;nextcloud2.example.com\u0026#39;, \u0026#39;cloud.example.com\u0026#39;, \u0026#39;previous-domain.com\u0026#39;, ], Advanced Configuration # Enable Debug Logging # 1 2 3 # traefik.yml log: level: DEBUG Configure Certificate TTL # 1 2 3 4 5 6 7 8 9 certificatesResolvers: le: acme: email: your-email@example.com storage: /letsencrypt/acme.json dnsChallenge: provider: cloudflare caServer: \u0026#34;https://acme-v02.api.letsencrypt.org/directory\u0026#34; # Production # caServer: \u0026#34;https://acme-staging-v02.api.letsencrypt.org/directory\u0026#34; # Staging Multiple Domains for Single Service # 1 2 3 4 5 6 7 8 9 http: routers: multi-domain-router-https: rule: \u0026#34;Host(`domain1.com`) || Host(`domain2.com`) || Host(`domain3.com`)\u0026#34; entryPoints: - websecure tls: certResolver: le service: my-service Monitoring and Maintenance # Check Certificate Expiry # 1 2 3 4 5 6 7 8 # Script to check all certificates cat /path/to/traefik/acme.json | python3 \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; import json, sys, datetime data = json.load(sys.stdin) for domain in data[\u0026#39;le\u0026#39;][\u0026#39;Certificates\u0026#39;]: cert = domain[\u0026#39;certificate\u0026#39;] print(f\u0026#34;Domain: {domain[\u0026#39;domain\u0026#39;][\u0026#39;main\u0026#39;]}\u0026#34;) EOF Manual Certificate Renewal # Let\u0026rsquo;s Encrypt certs auto-renew 30 days before expiry. To force renewal:\n1 docker compose exec traefik traefik renew-certs Backup acme.json # 1 cp /path/to/traefik/acme.json /path/to/traefik/acme.json.backup-$(date +%Y%m%d) Performance Comparison # Metric Cloudflare Tunnel Traefik + Let\u0026rsquo;s Encrypt Upload Limit 100MB ❌ Unlimited ✅ Latency Higher (tunnel overhead) Lower (direct) ✅ SSL Certs Automatic ✅ Automatic ✅ Cost Free (limited) Free (unlimited) ✅ Configuration Simple Moderate DDoS Protection Built-in ✅ Must add separately Conclusion # Migrating from Cloudflare Tunnel to Traefik with Let\u0026rsquo;s Encrypt was straightforward and provides:\nNo upload limitations Better performance Full control Automatic certificate management The key takeaways:\nRemove Cloudflare Tunnel for the domain Update DNS to DNS only (not proxied) Configure Traefik with ACME + Cloudflare DNS challenge Set certResolver: le in router config Restart Traefik to request certificates This setup is ideal for self-hosted services and scales well as you add more domains.\nReferences # Traefik ACME Documentation Let\u0026rsquo;s Encrypt DNS Challenge Cloudflare DNS Challenge Provider Changelog # 2026-07-05 - Initial publication, documented the migration process ","date":"5 July 2026","externalUrl":null,"permalink":"/en/p/automatic-tls-certificates-with-traefik-and-lets-encrypt/","section":"Posts","summary":" Background # I was previously using Cloudflare Tunnel (free plan) to reverse proxy my Nextcloud container. While convenient, the free tier has a 100MB upload limitation that became problematic for my usage.\nThis article documents my migration from Cloudflare Tunnel to direct Traefik reverse proxy with Let’s Encrypt TLS certificates, including the complete setup, issues encountered, and troubleshooting steps.\n","title":"Automatic TLS Certificates with Traefik and Let's Encrypt","type":"post"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/tags/cloudflare-dns-challenge/","section":"Tags","summary":"","title":"Cloudflare DNS Challenge","type":"tags"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/tags/lets-encrypt/","section":"Tags","summary":"","title":"Let's Encrypt","type":"tags"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/categories/network/","section":"Categories","summary":"","title":"Network","type":"categories"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/series/reverse-proxy-series/","section":"Series","summary":"","title":"Reverse Proxy Series","type":"series"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/tags/ssl-certificate/","section":"Tags","summary":"","title":"SSL Certificate","type":"tags"},{"content":"","date":"2026年7月5日","externalUrl":null,"permalink":"/tags/ssl%E8%AF%81%E4%B9%A6/","section":"Tags","summary":"","title":"SSL证书","type":"tags"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/tags/tls/","section":"Tags","summary":"","title":"TLS","type":"tags"},{"content":"","date":"5 July 2026","externalUrl":null,"permalink":"/en/tags/traefik/","section":"Tags","summary":"","title":"Traefik","type":"tags"},{"content":"","date":"2026年7月5日","externalUrl":null,"permalink":"/series/%E5%8F%8D%E5%90%91%E4%BB%A3%E7%90%86%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"反向代理系列","type":"series"},{"content":"","date":"2026年7月5日","externalUrl":null,"permalink":"/categories/%E7%BD%91%E7%BB%9C/","section":"Categories","summary":"","title":"网络","type":"categories"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/categories/ai/","section":"Categories","summary":"","title":"AI","type":"categories"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/tags/deepseek/","section":"Tags","summary":"","title":"DeepSeek","type":"tags"},{"content":" Introduction # Running large language models (LLMs) locally has become increasingly practical as quantization techniques and hardware capabilities have advanced. For agent frameworks like OpenClaw and Hermes, having a local LLM with a large context window (100k+ tokens) is essential for maintaining conversation history and executing complex multi-step tasks.\nThis guide covers the best models available in June 2026 for machines with 128GB VRAM (such as Strix Halo setups), how to deploy them using Ollama in Docker, and how to leverage speculative decoding with Multi-Token Prediction (MTP) for faster inference.\nWhy Run LLMs Locally? # Before diving into specific models, here are the key reasons to run LLMs locally for agent use:\nPrivacy: Your data never leaves your machine Latency: No network round-trips means faster response times Cost: No API fees after the initial hardware investment Customization: Fine-tune and modify models without restrictions Offline capability: Agents work without internet connectivity Best Models for 128GB VRAM (June 2026) # Here are the top models that can run on a 128GB VRAM machine with context windows of 100k+ tokens:\nCohere Command A+ (218B total, 25B active) # Context: 128k input / 64k output VRAM: ~110GB at 4-bit quantization Strengths: Specifically trained for conversational tool use, making it ideal for agent frameworks that need to call external APIs and tools DeepSeek Coder V2 (236B total, 20.9B active) # Context: 128k VRAM: ~118GB at 4-bit quantization Strengths: Specialized for code generation with 338-language support, perfect for code-centric agents DeepSeek V4 Flash (284B total, 13B active) # Context: 128k native (up to 1M with speculative decoding) VRAM: ~80GB at 4-bit quantization Strengths: Flagship MoE model with excellent reasoning capabilities Gemma 4 26B/31B # Context: 128k (E4B) to 256k (26B/31B) VRAM: ~33GB at 4-bit quantization Strengths: Google\u0026rsquo;s latest open model with excellent long-context support Qwen3-8B-128K # Context: 128k VRAM: ~4GB at 4-bit quantization Strengths: Ultra-lightweight, perfect for running multiple agent instances Deploying with Ollama in Docker # The easiest way to run these models locally is using Ollama in a Docker container. Here\u0026rsquo;s how to get started:\nStep 1: Pull the GGUF Model # Most models are available as GGUF files on Hugging Face. For example, to get the DeepSeek V4 Flash Q2 model:\n1 2 3 git clone https://github.com/antirez/ds4 cd ds4 ./download_model.sh q2 # Downloads ~80GB Q2 quantized model Step 2: Set Up Ollama in Docker # Create a directory for your models and start the Ollama container:\n1 2 3 4 5 6 7 8 9 10 11 # Create models directory mkdir -p ~/models cp ds4/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf ~/models/ # Start Ollama container with model mount docker run -d \\ --name ollama \\ -p 11434:11434 \\ -v ~/models:/models \\ -e OLLAMA_HOST=0.0.0.0 \\ ollama/ollama Step 3: Import the Model into Ollama # 1 2 3 4 5 6 # Import the GGUF file docker exec -it ollama bash -c \\ \u0026#34;ollama import deepseek-v4-flash-q2 /models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf\u0026#34; # Run the model docker exec -it ollama bash -c \u0026#34;ollama run deepseek-v4-flash-q2\u0026#34; Understanding Quantization: Q2 vs Q4 # When deploying large models, quantization is essential for fitting them into VRAM. Here\u0026rsquo;s what the quantization levels mean:\nQuantization File Size VRAM Usage Quality FP16 ~570GB ~600GB Baseline Q4 ~150GB ~160GB Minimal loss Q2 ~80GB ~90GB Noticeable degradation For a 128GB VRAM machine:\nQ4 quantization is recommended for models up to ~236B parameters Q2 quantization allows you to run even larger models (284B+) with some quality trade-off Imatrix vs Standard Quantization # Some models offer \u0026ldquo;imatrix\u0026rdquo; variants that use importance matrices during quantization:\nStandard Q2: Uses a synthetic fallback heuristic based on weight energy Q2-imatrix: Uses real activation statistics from a calibration dataset The imatrix version provides better quality at the same file size because it allocates bits more intelligently to the most important parameters.\nSpeculative Decoding with MTP # One of the most exciting advances in local LLM inference is Multi-Token Prediction (MTP) combined with speculative decoding. Here\u0026rsquo;s how it works:\nThe Problem with Standard Inference # Traditional autoregressive generation produces one token at a time:\n1 2 3 4 Step 1: Predict token 1 (full forward pass) Step 2: Predict token 2 (full forward pass) Step 3: Predict token 3 (full forward pass) ... This is sequential and doesn\u0026rsquo;t utilize GPU parallelism effectively.\nHow MTP + Speculative Decoding Works # Draft Phase: MTP heads predict multiple candidate tokens in parallel using a single forward pass Verify Phase: The base model verifies all candidates in one additional forward pass Accept/Reject: Tokens that match are kept; mismatches trigger regeneration from that point The Speed-up # With an 85-90% acceptance rate (as reported by DeepSeek), you can achieve:\n1.8x throughput improvement (tokens per second) Reduced latency for multi-token generation Using MTP with Ollama # To enable MTP in Ollama, create a Modelfile that references both the base model and the MTP helper:\n1 2 FROM /models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf MTP /models/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf Then create and run the model:\n1 2 ollama create deepseek-v4-flash-q2-mtp -f Modelfile ollama run deepseek-v4-flash-q2-mtp Choosing the Right Model for Your Use Case # Here\u0026rsquo;s a quick reference guide:\nUse Case Recommended Model Why Agent orchestration with tool calling Cohere Command A+ Trained specifically for tool use Code generation and debugging DeepSeek Coder V2 Specialized for code, 128k context Fast, low-latency responses Qwen3-8B-128K Only 4GB VRAM, can run multiple instances Balanced reasoning + long context Gemma 4 26B/31B Up to 256k context, moderate VRAM Maximum performance DeepSeek V4 Flash 13B active, supports MTP for speed Performance Tips # Quantize to 4-bit for the best balance of quality and VRAM usage Use GGUF format with llama.cpp-based runners for automatic CPU offloading Enable MTP for DeepSeek V4 Flash to get 1.8x speed improvement Run multiple smaller models instead of one large model if you need concurrent agents Monitor VRAM usage with nvidia-smi to avoid out-of-memory errors Conclusion # Running local LLMs for agent use has never been more accessible. With 128GB VRAM, you have the flexibility to choose from several excellent models with 128k+ context windows. Whether you prioritize tool-use capabilities (Command A+), code generation (DeepSeek Coder V2), or raw performance (DeepSeek V4 Flash), there\u0026rsquo;s a model that fits your needs.\nThe combination of Ollama for easy deployment and speculative decoding for faster inference makes local agent deployment both practical and efficient. As quantization techniques continue to improve, we can expect even larger models to become accessible on consumer hardware.\nReferences # Cohere Command A+ Model Card DeepSeek V4 Flash GGUF Gemma 4 Model Card Ollama Documentation Speculative Decoding Research ","date":"2 June 2026","externalUrl":null,"permalink":"/en/p/deploying-local-llms-for-agent-use-a-comprehensive-guide/","section":"Posts","summary":" Introduction # Running large language models (LLMs) locally has become increasingly practical as quantization techniques and hardware capabilities have advanced. For agent frameworks like OpenClaw and Hermes, having a local LLM with a large context window (100k+ tokens) is essential for maintaining conversation history and executing complex multi-step tasks.\nThis guide covers the best models available in June 2026 for machines with 128GB VRAM (such as Strix Halo setups), how to deploy them using Ollama in Docker, and how to leverage speculative decoding with Multi-Token Prediction (MTP) for faster inference.\n","title":"Deploying Local LLMs for Agent Use: A Comprehensive Guide","type":"post"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/tags/gemma/","section":"Tags","summary":"","title":"Gemma","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/tags/llm/","section":"Tags","summary":"","title":"LLM","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/tags/local-ai/","section":"Tags","summary":"","title":"Local AI","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/tags/ollama/","section":"Tags","summary":"","title":"Ollama","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/tags/openclaw/","section":"Tags","summary":"","title":"OpenClaw","type":"tags"},{"content":"","date":"2 June 2026","externalUrl":null,"permalink":"/en/tags/speculative-decoding/","section":"Tags","summary":"","title":"Speculative Decoding","type":"tags"},{"content":"","date":"2026年6月2日","externalUrl":null,"permalink":"/tags/%E6%8E%A8%E6%B5%8B%E8%A7%A3%E7%A0%81/","section":"Tags","summary":"","title":"推测解码","type":"tags"},{"content":"","date":"2026年6月2日","externalUrl":null,"permalink":"/tags/%E6%9C%AC%E5%9C%B0ai/","section":"Tags","summary":"","title":"本地AI","type":"tags"},{"content":"","date":"12 May 2026","externalUrl":null,"permalink":"/en/tags/ai/","section":"Tags","summary":"","title":"AI","type":"tags"},{"content":"","date":"12 May 2026","externalUrl":null,"permalink":"/en/series/ai-assistant-series/","section":"Series","summary":"","title":"AI Assistant Series","type":"series"},{"content":"","date":"2026年5月12日","externalUrl":null,"permalink":"/series/ai%E5%8A%A9%E6%89%8B%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"AI助手系列","type":"series"},{"content":"","date":"12 May 2026","externalUrl":null,"permalink":"/en/categories/artificial-intelligence/","section":"Categories","summary":"","title":"Artificial Intelligence","type":"categories"},{"content":" Background # After setting up my AI assistant OpenClaw, I\u0026rsquo;m documenting the entire setup process including configuring online models, local models, testing model performance, and adding skills.\nPrerequisites # A server or local computer Docker and docker-compose installed Network configured (Tailscale VPN) Ollama installed (for running local models) OpenClaw Overview # OpenClaw is an AI assistant framework supporting multiple model providers, including online and local models. Key features:\nMultiple online model providers (OpenRouter, DeepSeek, etc.) Local models via Ollama Skills system Multi-node deployment Configuring Online Models # OpenRouter # Configure OpenRouter as online model provider for access to various open-source and commercial models:\nopenrouter/qwen3.6-plus:free - Free Qwen model openrouter/free - Free model pool deepseek/deepseek-chat - DeepSeek model Model Configuration # In ~/.openclaw/openclaw.json, configure model providers and model lists:\n1 2 3 4 5 6 7 8 9 10 11 12 { \u0026#34;models\u0026#34;: { \u0026#34;providers\u0026#34;: { \u0026#34;openrouter\u0026#34;: { \u0026#34;models\u0026#34;: [\u0026#34;qwen3.6-plus:free\u0026#34;, \u0026#34;free\u0026#34;] }, \u0026#34;deepseek\u0026#34;: { \u0026#34;models\u0026#34;: [\u0026#34;deepseek-chat\u0026#34;] } } } } Model Fallback Chain # Configure model fallback chain for automatic switching when primary model is unavailable:\nopenrouter/qwen3.6-plus:free ollama/qwen3.5-35b-iq2 (local model) ollama/gemma-4-e4b-q5 (local model) openrouter/free deepseek/deepseek-chat Configuring Local Models # Ollama Configuration # Ollama runs in Docker container at 100.64.0.3:11434:\n1 2 3 4 5 6 7 8 9 services: ollama: container_name: ollama image: ollama/ollama ports: - \u0026#34;11434:11434\u0026#34; volumes: - ollama:/root/.ollama restart: always Local Model List # qwen3.5-35b-iq2 # Architecture: MoE (Mixture of Experts), 35B total params, ~3B active Quantization: UD-IQ2_XXS, 10.7GB Advantage: Lightweight compute, fits 16GB VRAM Tool calling: Requires RENDERER qwen3.5 + PARSER qwen3.5 in Modelfile gemma-4-e4b-q5 # Architecture: Dense E4B, 7.5B total, 4.5B effective Quantization: Unsloth Q5_K_M, 6.3GB Advantage: 128K context, native tool calling Note: Requires Ollama v0.20+ Modelfile Configuration # Custom GGUF models need RENDERER and PARSER:\n1 2 3 FROM ./qwen3.5-35b-iq2.gguf RENDERER qwen3.5 PARSER qwen3.5 Testing Model Performance # Long Context Test # Tested gemma-4-e4b-q5 with 128K context:\n5/5 needles at 96K tokens Test duration: 51.8s Reasoning Test # Qwen MoE performs better on Einstein riddle, Gemma 4 excels in long context scenarios.\nTool Calling Test # Qwen 3.5-35b-IQ2: Better structured output and tool use Gemma 4: Good tool calling with long context Adding Skills # deep-read Skill # Location: skills/deep-read/\nStructured deep reading workflow for books, papers, articles → atomic notes + methods + connections.\ncf-fetcher Skill # Location: skills/cf-fetcher/\nReduces HTML→Markdown token consumption by 65-80%, best for news sites, blogs, forums.\nBrowser Automation Skill # Location: skills/browser-automation/\nFor controlling web pages with multi-step flows.\nSkill Configuration # In ~/.openclaw/openclaw.json, register skills:\n1 2 3 4 5 6 7 8 9 10 { \u0026#34;skills\u0026#34;: { \u0026#34;deep-read\u0026#34;: { \u0026#34;location\u0026#34;: \u0026#34;skills/deep-read/\u0026#34; }, \u0026#34;cf-fetcher\u0026#34;: { \u0026#34;location\u0026#34;: \u0026#34;skills/cf-fetcher/\u0026#34; } } } Xiaomi MiMo Models # Subscribed to Xiaomi token plan ($6/mo, 60M credits):\nmimo-v2-pro - 1M context, reasoning, credit = 2× token (ctx\u0026lt;256K) mimo-v2-omni - 1M context, multimodal (text+image) mimo-v2-tts - 32K context, TTS, max 1024 tokens Key Lessons Learned # Config source of truth: ~/.openclaw/openclaw.json, NOT ~/.openclaw/agents/main/agent/models.json MoE models: Key is fitting fully in VRAM, IQ2_XXS acceptable as only ~3B params active Gemma 4 GGUF: Requires Ollama v0.20+, old versions can\u0026rsquo;t load Model fallback: Only triggers on auth failures, rate limits, timeouts, billing errors Node config: ws:// connections need OPENCLAW_ALLOW_INSECURE_PRIVATE_WS=1 Ollama Modelfile: Custom GGUF must include RENDERER and PARSER Related Links # OpenClaw Docs OpenClaw GitHub Ollama OpenRouter ","date":"12 May 2026","externalUrl":null,"permalink":"/en/p/openclaw-ai-assistant-setup-summary/","section":"Posts","summary":" Background # After setting up my AI assistant OpenClaw, I’m documenting the entire setup process including configuring online models, local models, testing model performance, and adding skills.\nPrerequisites # A server or local computer Docker and docker-compose installed Network configured (Tailscale VPN) Ollama installed (for running local models) OpenClaw Overview # OpenClaw is an AI assistant framework supporting multiple model providers, including online and local models. Key features:\n","title":"OpenClaw AI Assistant Setup Summary","type":"post"},{"content":"","date":"12 May 2026","externalUrl":null,"permalink":"/en/tags/openrouter/","section":"Tags","summary":"","title":"OpenRouter","type":"tags"},{"content":"","date":"12 May 2026","externalUrl":null,"permalink":"/en/tags/skills/","section":"Tags","summary":"","title":"Skills","type":"tags"},{"content":"","date":"2026年5月12日","externalUrl":null,"permalink":"/categories/%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD/","section":"Categories","summary":"","title":"人工智能","type":"categories"},{"content":"","date":"2026年5月12日","externalUrl":null,"permalink":"/tags/%E5%A4%A7%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B/","section":"Tags","summary":"","title":"大语言模型","type":"tags"},{"content":"","date":"2026年5月12日","externalUrl":null,"permalink":"/tags/%E6%8A%80%E8%83%BD/","section":"Tags","summary":"","title":"技能","type":"tags"},{"content":"","date":"2026年3月17日","externalUrl":null,"permalink":"/tags/portainer/","section":"Tags","summary":"","title":"Portainer","type":"tags"},{"content":" 缘起 # 我们之前介绍了使用Portainer来监控和管理电脑上运行的容器，Portainer提供了可视化的界面，可以方便地查看容器日志、重启容器、管理镜像文件等。在配置了反向代理后，我们可以方便地使用域名来远程访问，例如https://portainer.example.com。\n但是之前的文章只介绍了用Portainer监控本地电脑上的容器。后来我又在我的其他电脑上安装运行了Docker容器，如果也想用Portainer来监控，一个自然的想法是，在每台运行Docker容器的电脑上都安装Portainer，用不同的域名来反向代理，例如portainer1.example.com、portainer2.example.com、portainer3.example.com等等。\n这种方法当然是行得通的，就是比较碎片化。Portainer是支持远程监控其他电脑上的Docker容器的，这就是Portainer里的Agent。\n简介 # Docker提供了一个位于/var/run/docker.sock的套接口（socket）,用于Docker守护进程与其他工具之间的通信。其他工具可以通过这个套接口获取Docker容器的运行状态，也可以向这个套接口发送命令来管理Docker容器。\n工作原理 # Portainer可以通过访问远程电脑上的Docker套接口来管理远程电脑上的Docker容器。具体过程如下：\n首先我们确定一个主Portainer，例如本地电脑A上运行的Portainer。其他远程电脑运行Agent。 Agent1向主Portainer报告自己这台电脑上的Docker容器的状态，主Portainer将其显示在Portainer的UI界面上。用户可以在这个界面上看到其他电脑上运行的容器状态。 若用户想要管理其他电脑上的容器，则直接在Portainer的UI界面上操作，Portainer会将用户的操作发送给Agent。Agent完成相应的操作后将新的状态再次报告给主Portainer。 实现方式 # 上述工作原理的实现方式很多，常见的有两种：Portainer Agent和Portainer Edge Agent。\nPortainer Agent # 在远程电脑B上运行一个Portainer Agent容器，这个Agent读取电脑B上的/var/run/docker.sock。 本地电脑A上运行Portainer Server (即普通的Portainer)，添加一个远程环境指向电脑B，向电脑B上的Agent容器索要电脑B上的容器状态，向电脑B上的Agent发送操作指令。电脑B上的Agent执行电脑A通过Portainer发送来的指令。 这种实现方式的关键是本地电脑A的Portainer要能主动访问到远程电脑B的Agent。如果A和B在同一个局域网里，那只需要通过局域网里的IP地址和端口直接访问就行了；如果A和B不在同一个局域网里，我们就需要将电脑B上的Agent容器暴露到公网，可以通过域名反向代理来暴露。\nPortainer Edge Agent # 上述Agent方式将远程电脑的Agent都暴露在公网的方法显然不太安全，尽管我们可以设置口令进行加密，但依然可能被攻击。\n于是有了一种更安全的方法，远程电脑B上的Agent不需要暴露到公网，本地电脑A也不会主动去找B。反之，远程电脑B上的Agent每隔一段时间（默认是5秒）向电脑A上的Portainer汇报一次，并查看是否有什么任务交给自己。\n在本地电脑A看来，它不知道远程电脑B在哪，只知道自己会定时获取到电脑B交给自己的状态，自己给B列的任务也会被B拿走执行。\n这样，我们就只暴露了本地电脑A的Portainer，这个Portainer只负责收取其他远程Agent的状态更新和向其他Agent发布任务。无论是本地电脑A还是其他电脑，他们都不接受别的电脑的指令，这样自然更安全。这种Agent称为Edge Agent。\n对比 # 普通的Agent是本地电脑主动找远程电脑，按需指挥它们；Edge Agent是远程电脑定时向本地电脑汇报，并领取任务。\n因此，普通Agent模式更高效，没有任务时不会去找远程电脑。Edge Agent则是不管远程电脑有没有任务，都定时去找本地电脑。不过，这种通信并不会太消耗电脑性能资源，如果我们设置的汇报时间间隔大一点儿（例如30秒），那么Edge Agent模式的性能开销就微乎其微了。\n综上，这里我们选择使用Edge Agent模式。\nPortainer Edge Agent部署 # 这里我们选用的工具链如下：\nPortainer: 免费的社区版portainer-ce 反向代理：Traefik和免费版的Cloudflare Tunnel Portainer Server配置 # Portainer Server实际上就是我们在之前的文章中讲的Portainer的部署，只不过为了让Edge Agent能连上，我们需要添加一些反向代理的设置。\nPortainer容器的docker-compose.yml文件\n设置Traefik路由规则portainer.yml\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 http: routers: portainer: rule: \u0026#34;Host(`550w.jinli.io`)\u0026#34; entryPoints: - websecure service: portainer-api tls: {} portainer-edge: rule: \u0026#34;Host(`edge.jinli.io`)\u0026#34; entryPoints: - websecure tls: {} service: portainer-edge services: portainer-api: loadBalancer: servers: - url: \u0026#34;http://portainer:9000\u0026#34; portainer-edge: loadBalancer: servers: - url: \u0026#34;http://portainer:8000\u0026#34; 之前的路由规则只暴露了9000端口，用于Portainer的网页界面。 现在我们暴露8000端口，用于Edge Agent的连接。\n在Portainer中创建Edge Agent环境时，Portainer默认使用Portainer的域名加8000端口（如portainer.example.com:8000）进行连接。但由于我们用Cloudflare Tunnel来做反向代理，Cloudflare Tunnel不会监听8000端口，因此，我们需要再给portainer:8000分配一个域名（如edge.example.com，并且让Edge Agent连接新的域名，而非portainer.example.com:8000。\nCloudflare Tunnel\n先在Cloudflare网页客户端中为edge.example.com添加DNS记录（如果之前没添加portainer.example.com的话，也创建一个），记录是指向Cloudflare Tunnel的CNAME。\nCloudflare Tunnel里我们需要同时代理域名portainer.example.com和edge.example.com，将下面的代码加入到原来的config.yml中：\n1 2 3 4 5 6 7 8 - hostname: edge.jinli.io service: https://traefik:443 originRequest: noTLSVerify: true - hostname: edge.jinli.io service: https://traefik:443 originRequest: noTLSVerify: true 这些全部设置好之后，启动这些容器服务，就可以通过https://portainer.example.com在浏览器中访问Portainer界面了。\nPortainer网页界面添加Edge Agent环境 # 在Portainer的网页界面左侧，管理员（Administration）区域有个Environment-related选项，打开这个选项就可以看到如下界面。\n点击“添加环境”（Add Environment），选择Docker Standalone，点击Start Wizard，选择Edge Agent，根据提示创建一个Edge Agent。\n创建完成后，会显示一个Docker命令告诉你怎样在远程电脑上创建Edge Agent容器。由于我们使用docker-compose.yml来管理容器，所以我们不用这个命令。但我们需要记下来这个命令里出现的EDGE_ID和EDGE_KEY，将这两个值保存下来。\n创建Edge Agent # 重新编码EDGE_KEY\nEDGE_KEY保存了Portainer Server的信息，Edge Server就靠这里面的信息来寻找Portainer Server。\n在命令行运行如下命令对EDGE_KEY进行解码（把命令中的your_edge_key换成你在上一步获取的EDGE_KEY的值）：\n1 echo your_edge_key | base64 -d 你会得到格式如下的一串字符：\n1 https://portainer.example.com|portainer.example.com:8000|xxxxxxxxfinger_print|3 其中第一段https://portainer.example.com就是告诉Edge Server去https://portainer.example.com找Portainer Server。\n第二段的portainer.example.com:8000表示在Edge Agent找到Portainer Server后，向portainer.example.com:8000发送请求建立连接。\n我们刚才说了，我们的portainer.example.com是用Cloudflare Tunnel代理的，Cloudflare Tunnel不会监听8000端口。所以我们创建了个新域名edge.example.com来代理portainer:8000。于是我们用新域名替换掉第二段里的portainer.example.com:8000，替换后的字符串长这样：\n1 https://portainer.example.com|edge.example.com|xxxxxxxxfinger_print|3 最后，我们把替换后的字符串重新编码：\n1 echo -n https://portainer.example.com|edge.example.com|xxxxxxxxfinger_print|3 | base64 -w0 将重新编码的字符串保存下来，这就是新的EDGE_KEY。\n注意，重新编码的字符串末尾如果有=，Edge Agent容器很可能不认，可以删掉。\nEdge Agent容器部署\n接下来我们在远程电脑上部署Edge Agent。\n先创建一个.env文件，写入EDGE_ID和魔改后的EDGE_KEY，也可以把要多长时间汇报一次状态也定义到环境变量里（这里是每30秒）：\n1 2 3 EDGE_ID=xxxxxxxxxx EDGE_KEY=xxxxxxxxxxxxxxxxxxxxxxx EDGE_POLL_INTERVAL=30 在创建一个docker-compose.yml文件:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 version: \u0026#34;3.8\u0026#34; services: portainer-edge-agent: image: portainer/agent:2.39.0 container_name: portainer-edge-agent restart: unless-stopped environment: EDGE: \u0026#34;1\u0026#34; EDGE_ID: ${EDGE_ID} EDGE_KEY: ${EDGE_KEY} EDGE_INSECURE_POLL: \u0026#34;1\u0026#34; EDGE_POLL_INTERVAL: ${EDGE_POLL_INTERVAL} volumes: - /var/run/docker.sock:/var/run/docker.sock - /var/lib/docker/volumes:/var/lib/docker/volumes 直接启动这个容器就行了，不需要设置代理，因为Portainer Server不需要去找它，它会根据EDGE_KEY里的信息去寻找Portainer Server。\n部署完毕 # 完成上述步骤后，就可以在本地电脑的Portainer界面里看到远程电脑的容器了:\n点击进入，看到的界面和我们在本地电脑上的Portainer容器管理界面是一样的。\n添加更多Edge Agent # 如果你想把更多电脑添加到本地的Portainer界面，只需要重复上面的步骤就行了。\n注意，添加每台电脑时都需要重新创建一个新的环境，并获取新的EDGE_ID和EDGE_KEY，环境不能共用。\n常见问题 # 如果在部署Portainer Edge Agent容器后容器内报类似下面所示的错误：\n1 2026/03/19 08:30:37.506PM FTL github.com/portainer/agent/cmd/agent/main.go:262 \u0026gt; unable to associate Edge key | error=\u0026#34;invalid key format\u0026#34; 这说明在对EDGE_KEY重新编码时使用的base64设置不对。我们需要将EDGE_KEY编码为Base64URL格式，需要使用base64url命令，但Linux默认只有base64命令，为此我们加了-w0参数，但这个参数只是为了禁用换行功能。\n如果出现了上述错误，可以使用某些可以进行Base64URL编码的网站来编码，例如：https://www.base64encode.org/，记得勾选Perform URL safe encoding选项。\n","date":"2026年3月17日","externalUrl":null,"permalink":"/p/%E5%AE%B9%E5%99%A88%E4%BD%BF%E7%94%A8portainer-agent%E7%9B%91%E6%8E%A7%E8%BF%9C%E7%A8%8B%E7%94%B5%E8%84%91%E4%B8%8A%E7%9A%84docker%E5%AE%B9%E5%99%A8/","section":"Posts","summary":" 缘起 # 我们之前介绍了使用Portainer来监控和管理电脑上运行的容器，Portainer提供了可视化的界面，可以方便地查看容器日志、重启容器、管理镜像文件等。在配置了反向代理后，我们可以方便地使用域名来远程访问，例如https://portainer.example.com。\n","title":"容器（8）：使用Portainer Agent监控远程电脑上的Docker容器","type":"post"},{"content":"","date":"2026年3月17日","externalUrl":null,"permalink":"/series/%E5%AE%B9%E5%99%A8%E6%8A%80%E6%9C%AF%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"容器技术系列","type":"series"},{"content":"","date":"2026年3月17日","externalUrl":null,"permalink":"/categories/%E5%BC%80%E5%8F%91/","section":"Categories","summary":"","title":"开发","type":"categories"},{"content":"","date":"13 March 2026","externalUrl":null,"permalink":"/en/series/large-language-model-series/","section":"Series","summary":"","title":"Large Language Model Series","type":"series"},{"content":" Introduction # Previously, we discussed building private large language model services, where we used Ollama as the backend service and LobeChat as the frontend, communicating through API calls to the backend Ollama to implement conversations.\nRecently, I have been trying to use OpenClaw, but its token consumption rate is too fast, so I thought about running a local large language model to provide tokens. After talking with ChatGPT, it told me that in this case, it\u0026rsquo;s better to use vLLM rather than Ollama, because vLLM provides API compatibility with OpenAI, supports Copilot, and has better performance. So I decided to switch to vLLM.\nIntroduction # vLLM # vLLM and Ollama are both tools for running large language models (LLM) on your own computer or on a server, but their use cases differ. Simply put, vLLM is focused on efficiently running large language models, while Ollama focuses on making it easy to run large language models.\nAdditionally, vLLM provides API compatibility with OpenAI. If the services we want to run depend on commercial model APIs like OpenAI\u0026rsquo;s, and support custom model providers, we can connect to our own deployed vLLM.\nOllama # For an introduction and deployment instructions for Ollama, please refer to the first article in this series.\nSince Ollama uses some custom REST APIs that are incompatible with OpenAI\u0026rsquo;s API, it cannot directly connect to many services that use large language models. However, developers have created a tool LiteLLM that can call Ollama and is compatible with OpenAI\u0026rsquo;s API.\nWe only need to deploy this tool along with the Ollama container and make some settings, and it will be able to provide API compatibility with OpenAI\u0026rsquo;s API.\nComparison and Considerations for vLLM and Ollama # GPU Memory Usage\nWhen running, the biggest difference we can perceive between vLLM and Ollama is that deploying vLLM requires loading the entire model into the GPU\u0026rsquo;s memory, even when the model is idle (not being called), vLLM will still occupy this portion of memory.\nIn other words, vLLM can hardly run alongside other programs that heavily use the graphics card. For example, after running vLLM, if you want to play games, you need to take vLLM offline first.\nOllama, on the other hand, does not have this problem. Ollama only loads the model into memory when it is called. When you want to play games, you don\u0026rsquo;t need to shut down Ollama, as long as you don\u0026rsquo;t call Ollama during your gaming session, it can coexist peacefully with your game.\nNumber of Callable Models\nAfter deploying Ollama, we can download multiple models, and users can decide which one to call.\nHowever, after deploying vLLM, the model loaded into GPU memory is fixed — one vLLM container can only load one model. To choose between models, you would need to deploy multiple vLLM containers simultaneously. This is clearly impractical for personal users, as our GPUs generally cannot handle that. If you want to load multiple models, you can only use very small ones, which defeats the purpose.\nDeploying the Large Language Models and Frontend # Overview # I have two computers each with a GPU with 16GB of VRAM. I chose to deploy vLLM on Computer A and deploy Ollama with a vLLM-compatible interface on Computer B (let\u0026rsquo;s call it vOllama for now). The interfaces of vLLM and vOllama are exposed to the public internet via domain names.\nThe Open WebUI frontend is deployed on any one of the computers (either A or B, or another Computer C), which makes it easy to connect to our self-deployed large language models.\nDeploying vLLM # docker-compose # The docker-compose.yml file is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 services: vllm: image: vllm/vllm-openai:v0.17.0 container_name: vllm restart: unless-stopped runtime: nvidia deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] environment: - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN} - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True - HF_HUB_DISABLE_XET=1 - HF_HUB_ENABLE_HF_TRANSFER=0 - VLLM_API_KEY=${VLLM_API_KEY:-dummy} - TZ=${TZ:-UTC} volumes: - ~/.cache/huggingface:/root/.cache/huggingface ipc: host command: \u0026gt; ${MODEL} ${MODEL_PARAMETERS} networks: traefik-net: aliases: - vllm networks: traefik-net: external: true The MODEL, MODEL_PARAMETERS, and other variables are defined in the environment variable file .env:\n1 2 3 4 5 HF_TOKEN=hf_your_token MODEL=Qwen/Qwen2.5-Coder-14B-Instruct-AWQ MODEL_PARAMETERS=\u0026#34;--host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.81 --dtype auto --kv-cache-dtype fp8 --enforce-eager\u0026#34; VLLM_API_KEY=your_api_key TZ=Europe/Paris Here, HF_TOKEN is the Hugging Face token, which we need to download large language models from Hugging Face.\nvLLM can only load one model at a time. We chose Qwen/Qwen2.5-Coder-14B-Instruct-AWQ, which requires approximately 14GB of VRAM to load.\nTesting # Enter a command like the following in the terminal to test whether vLLM has been deployed successfully:\n1 2 3 4 5 6 7 curl https://vllm.example/v1/chat/completions \\ -H \u0026#34;Authorization: Bearer your_vllm_api_key\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{ \u0026#34;model\u0026#34;: \u0026#34;Qwen/Qwen2.5-Coder-14B-Instruct-AWQ\u0026#34;, \u0026#34;messages\u0026#34;: [{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Hello\u0026#34;}] }\u0026#39; If you receive a response similar to the following, vLLM has been successfully deployed and is accessible via the domain name:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 { \u0026#34;id\u0026#34;: \u0026#34;chatcmpl-82d9387336a446bd\u0026#34;, \u0026#34;object\u0026#34;: \u0026#34;chat.completion\u0026#34;, \u0026#34;created\u0026#34;: 1774276907, \u0026#34;model\u0026#34;: \u0026#34;Qwen/Qwen2.5-Coder-14B-Instruct-AWQ\u0026#34;, \u0026#34;choices\u0026#34;: [ { \u0026#34;index\u0026#34;: 0, \u0026#34;message\u0026#34;: { \u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Hello! How can I assist you today?\u0026#34;, \u0026#34;refusal\u0026#34;: null, \u0026#34;annotations\u0026#34;: null, \u0026#34;audio\u0026#34;: null, \u0026#34;function_call\u0026#34;: null, \u0026#34;tool_calls\u0026#34;: [], \u0026#34;reasoning\u0026#34;: null }, \u0026#34;logprobs\u0026#34;: null, \u0026#34;finish_reason\u0026#34;: \u0026#34;stop\u0026#34;, \u0026#34;stop_reason\u0026#34;: null, \u0026#34;token_ids\u0026#34;: null } ], \u0026#34;service_tier\u0026#34;: null, \u0026#34;system_fingerprint\u0026#34;: null, \u0026#34;usage\u0026#34;: { \u0026#34;prompt_tokens\u0026#34;: 30, \u0026#34;total_tokens\u0026#34;: 40, \u0026#34;completion_tokens\u0026#34;: 10, \u0026#34;prompt_tokens_details\u0026#34;: null }, \u0026#34;prompt_logprobs\u0026#34;: null, \u0026#34;prompt_token_ids\u0026#34;: null, \u0026#34;kv_transfer_params\u0026#34;: null } In the terminal, this JSON will be displayed compressed on a single line.\nDeploying vOllama # Configuration Files # The docker-compose.yml file is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 services: ollama: container_name: ollama image: docker.1ms.run/ollama/ollama:0.17.7 environment: - OLLAMA_ORIGINS=* - OLLAMA_HOST=0.0.0.0 - TZ=America/New_York deploy: resources: reservations: devices: - driver: nvidia capabilities: [\u0026#34;gpu\u0026#34;] count: all volumes: - ./ollama-data:/root/.ollama restart: always networks: - traefik-net litellm: image: ghcr.io/berriai/litellm:main-latest container_name: litellm restart: unless-stopped command: [\u0026#34;--config\u0026#34;, \u0026#34;/app/config.yaml\u0026#34;] volumes: - ./litellm-config.yaml:/app/config.yaml networks: - traefik-net networks: traefik-net: external: true Here we deploy two containers, ollama and litellm, both using the traefik-net network we created earlier. We still use Cloudflare Tunnel to reverse proxy the vOllama service.\nlitellm requires a configuration file litellm-config.yml, as shown below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 model_list: - model_name: qwen3.5:0.8b litellm_params: model: ollama/qwen3.5:0.8b api_base: http://ollama:11434 - model_name: mistral:7b litellm_params: model: ollama/mistral:7b api_base: http://ollama:11434 general_settings: master_key: your_ollama_api_key The configuration file serves two main purposes:\nmodel_list tells litellm which large language models we have running in ollama. When we download and install additional models in Ollama, we need to update this file to include the new models. Since we are exposing the models on the public internet, we need to set an API key (i.e., master_key) to control access. Anyone who does not provide the API key will not be able to call our models. Installing Large Language Models # Use docker compose up -d to deploy the Ollama container. After a successful deployment, Ollama is empty with no models downloaded. Use docker exec -it ollama ollama run qwen3.5:0.8b to download and run the qwen3.5:0.8b model. After installation, you can continue downloading other models. Use docker exec -it ollama ollama list to list the installed models. Testing # Use the following command to test whether the deployment was successful:\n1 2 3 4 5 6 7 curl https://ollama.example/v1/chat/completions \\ -H \u0026#34;Authorization: Bearer your_ollama_api_key\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{ \u0026#34;model\u0026#34;: \u0026#34;qwen3.5:0.8b\u0026#34;, \u0026#34;messages\u0026#34;: [{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Hello\u0026#34;}] }\u0026#39; If you receive a response similar to the following, vOllama has been successfully deployed and is accessible via the domain name:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 { \u0026#34;id\u0026#34;: \u0026#34;chatcmpl-5844b507-f967-4412-b8fa-815c4e3235b1\u0026#34;, \u0026#34;created\u0026#34;: 1774276549, \u0026#34;model\u0026#34;: \u0026#34;qwen3.5:0.8b\u0026#34;, \u0026#34;object\u0026#34;: \u0026#34;chat.completion\u0026#34;, \u0026#34;choices\u0026#34;: [ { \u0026#34;finish_reason\u0026#34;: \u0026#34;stop\u0026#34;, \u0026#34;index\u0026#34;: 0, \u0026#34;message\u0026#34;: { \u0026#34;content\u0026#34;: \u0026#34;Greetings! How\u0026#39;s going? 😊 Feel free to ask me anything—whether you need help, just wanted to say hello, or maybe something else. I\u0026#39;m here for fun and helpful!\u0026#34;, \u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34; } } ], \u0026#34;usage\u0026#34;: { \u0026#34;completion_tokens\u0026#34;: 111, \u0026#34;prompt_tokens\u0026#34;: 15, \u0026#34;total_tokens\u0026#34;: 126 } } In the terminal, this JSON will be displayed compressed on a single line.\nDeploying Open WebUI # Using only curl or APIs to call the deployed models is clearly inconvenient — we need to deploy a web-based client. Here we choose the open-source client Open WebUI.\nIntroduction to Open WebUI # Open WebUI is an open-source web client for large language models, with a style similar to OpenAI\u0026rsquo;s ChatGPT interface. It allows us to connect to our self-deployed vLLM or Ollama models, and also supports connecting to various commercial models (API key required).\nDeploying Open WebUI # The docker-compose.yml file is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 version: \u0026#34;3.9\u0026#34; services: open-webui: image: ghcr.io/open-webui/open-webui:0.8.10 container_name: open-webui volumes: - ${DATA_DIR}:/app/backend/data environment: - OPENAI_API_BASE_URL=http://vllm:8000/v1 - OPENAI_API_KEY=${VLLM_API_KEY:-dummy} - TZ=${TZ:-UTC} restart: unless-stopped networks: traefik-net: aliases: - open-webui networks: traefik-net: external: true The OPENAI_API_BASE_URL and OPENAI_API_KEY here point to our locally deployed vLLM. These can be left blank and configured later in the graphical interface.\nUse the following command to start the open-webui container:\n1 docker compose up -d Open WebUI Configuration # After setting up the reverse proxy, we can access Open WebUI via the domain name. On first access, Open WebUI will ask us to set up an administrator account and password.\nAfter logging in with the administrator account, click the avatar in the upper right corner and you will see the \u0026ldquo;Admin Panel\u0026rdquo; option. Click to enter and configure the backend model services that Open WebUI needs to connect to.\nIn the \u0026ldquo;Settings\u0026rdquo; page of the \u0026ldquo;Admin Panel\u0026rdquo;, there is a \u0026ldquo;Connections\u0026rdquo; option where you can add the Ollama service, vLLM service, or vOllama service we deployed earlier, as well as commercial large language model services (such as OpenAI, Azure, Anthropic, etc.).\nAfter adding them, click the \u0026ldquo;Models\u0026rdquo; option in the \u0026ldquo;Settings\u0026rdquo; page, and you can see the available models in the Ollama or vLLM service. These models are set to \u0026ldquo;private\u0026rdquo; by default. You can click the edit button on the right side of a model and click the \u0026ldquo;Access\u0026rdquo; option in the model settings to configure which users can access that model.\nFinally, after configuring the models, you can use Open WebUI just like ChatGPT. After entering a question, Open WebUI will call the vLLM or Ollama service we deployed earlier to get a response:\nTroubleshooting # Installing Qwen3.6 with Ollama # When using Ollama to install the uncensored Qwen3.6 model Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive, you may encounter installation failures. The error message is as follows:\n1 llama_model_load: error loading model: error loading model architecture: unknown model architecture: \u0026#39;qwen35moe\u0026#39; This issue has been discussed on GitHub:\nhttps://github.com/ollama/ollama/issues/14503\nThe reason is that after downloading the Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive model, the Modelfile generated by Ollama is incorrect. It contains two From /root/.ollama/xxx lines, indicating that Ollama is trying to load two model files when in fact only one is needed. We need to delete the second From /root/.ollama/xxx line or comment it out with #, then save and re-run the model.\nHowever, the Modelfile does not seem to be explicitly created. We need to enter the container first:\n1 docker exec -it ollama bash Then execute the following command inside the container to generate the file:\n1 ollama show --modelfile hf.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:IQ2_M | cat \u0026gt; /root/Modelfile After commenting out or deleting the second From /root/.ollama/xxx line, save and exit, then recreate the model:\n1 ollama create Qwen3.6-35B-A3B-Uncensored-IQ2M -f /root/Modelfile ","date":"13 March 2026","externalUrl":null,"permalink":"/en/p/local-large-language-model-2-comparison-of-vllm-and-ollama-and-deploy-open-webui-frontend/","section":"Posts","summary":" Introduction # Previously, we discussed building private large language model services, where we used Ollama as the backend service and LobeChat as the frontend, communicating through API calls to the backend Ollama to implement conversations.\nRecently, I have been trying to use OpenClaw, but its token consumption rate is too fast, so I thought about running a local large language model to provide tokens. After talking with ChatGPT, it told me that in this case, it’s better to use vLLM rather than Ollama, because vLLM provides API compatibility with OpenAI, supports Copilot, and has better performance. So I decided to switch to vLLM.\n","title":"Local Large Language Model (2): Comparison of vLLM and Ollama and Deploy Open WebUI Frontend","type":"post"},{"content":"","date":"13 March 2026","externalUrl":null,"permalink":"/en/tags/open-webui/","section":"Tags","summary":"","title":"Open WebUI","type":"tags"},{"content":"","date":"13 March 2026","externalUrl":null,"permalink":"/en/tags/vllm/","section":"Tags","summary":"","title":"VLLM","type":"tags"},{"content":"","date":"2026年3月13日","externalUrl":null,"permalink":"/series/%E5%A4%A7%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"大语言模型系列","type":"series"},{"content":"This is Jin Li\u0026rsquo;s personal blog.\n","date":"4 March 2026","externalUrl":null,"permalink":"/en/about/","section":"Pages","summary":"This is Jin Li’s personal blog.\n","title":"About","type":"page"},{"content":"","date":"4 March 2026","externalUrl":null,"permalink":"/en/page/","section":"Pages","summary":"","title":"Pages","type":"page"},{"content":"","date":"11 February 2026","externalUrl":null,"permalink":"/en/tags/container/","section":"Tags","summary":"","title":"Container","type":"tags"},{"content":" Motivation # Due to the network environment issues in China, we cannot directly pull container images from Docker Hub. To facilitate the deployment of some container services on computers within China, I decided to build my own container image registry (i.e., Registry) on a server located overseas.\nSolution # Docker official provides the image of container image registry Registry, we can use this image to deploy our own container image registry.\nWe also need to configure a graphical interface for this registry to facilitate viewing. There are quite a few third-party graphical interfaces available online, and here I chose Joxit/docker-registry-ui.\nDeployment, Configuration and Usage # Deployment of Image Registry Service # Deploy Container\nUse the following docker-compose.yml file to deploy the image registry service:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 services: registry-ui: image: joxit/docker-registry-ui:main restart: always ports: - 8002:80 environment: - SINGLE_REGISTRY=true - REGISTRY_TITLE=Docker Registry UI - DELETE_IMAGES=true - SHOW_CONTENT_DIGEST=true - NGINX_PROXY_PASS_URL=http://registry:5000 - SHOW_CATALOG_NB_TAGS=true - CATALOG_MIN_BRANCHES=1 - CATALOG_MAX_BRANCHES=1 - TAGLIST_PAGE_SIZE=100 - REGISTRY_SECURED=false - CATALOG_ELEMENTS_LIMIT=1000 container_name: registry-ui networks: traefik-net: aliases: - registry-ui registry: image: registry:3 container_name: registry restart: always environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry REGISTRY_HTTP_HEADERS_Access-Control-Allow-Origin: \u0026#39;[http://registry-ui.jinli.io]\u0026#39; REGISTRY_HTTP_HEADERS_Access-Control-Allow-Methods: \u0026#39;[HEAD,GET,OPTIONS,DELETE]\u0026#39; REGISTRY_HTTP_HEADERS_Access-Control-Allow-Credentials: \u0026#39;[true]\u0026#39; REGISTRY_HTTP_HEADERS_Access-Control-Allow-Headers: \u0026#39;[Authorization,Accept,Cache-Control]\u0026#39; REGISTRY_HTTP_HEADERS_Access-Control-Expose-Headers: \u0026#39;[Docker-Content-Digest]\u0026#39; REGISTRY_STORAGE_DELETE_ENABLED: \u0026#39;true\u0026#39; volumes: - ${DATA_DIR}:/var/lib/registry networks: traefik-net: aliases: - registry networks: traefik-net: external: true Here we use two images:\nregistry: Docker official provides the image of container image registry docker-registry-ui: Third-party graphical interface for container image registry developed by Joxit Reverse Proxy\nWe still use the previous Traefik plus Cloudflare Tunnel solution, and we need to configure Traefik for both the registry service and the graphical interface service.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 http: routers: registry-router: rule: \u0026#34;Host(`registry.example.com`)\u0026#34; entryPoints: - web service: registry-service #tls: # certResolver: le services: registry-service: loadBalancer: servers: - url: \u0026#34;http://registry:5000\u0026#34; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 http: routers: registry-ui-router: rule: \u0026#34;Host(`registry-ui.example.com`)\u0026#34; entryPoints: - web service: registry-ui-service #tls: # certResolver: le services: registry-ui-service: loadBalancer: servers: - url: \u0026#34;http://registry-ui:80\u0026#34; Testing # After deployment, we can test whether we can access the graphical interface. For example, the above settings will publish the graphical interface to https://registry-ui.example.com. Open the browser and enter the above URL. If all previous settings are correct, you will see the following page:\nWe haven\u0026rsquo;t pushed any images to the registry yet, so the registry appears empty.\nUsage # Assuming the container image registry service is deployed on computer A, and we want to pull and use a certain image on computer B. The steps are as follows:\nComputer A: Pushing Images to the Registry # On computer A, you need to perform the following operations:\nBuild the image locally, or pull the image from platforms like Docker Hub or GitHub. For example, pull the registry image from Docker Hub: 1 docker pull registry:3 Tag the image with a new label: 1 docker tag registry:3 registry.example.com/registry:3 Push the image to the self-hosted registry: 1 docker push registry.example.com/registry:3 After completing the above operations, you can see the registry image has been successfully uploaded on https://registry-ui.example.com:\nComputer B: Pulling Images from Our Registry # When using Docker on computer B, you only need to add our registry name registry.example.com/ before the image name you want to use, for example:\n1 docker pull registry.example.com/registry:3 If using docker compose, it\u0026rsquo;s the same, just replace the image names in docker-compose.yml accordingly.\n","date":"11 February 2026","externalUrl":null,"permalink":"/en/p/container-7-building-your-own-container-image-registry/","section":"Posts","summary":" Motivation # Due to the network environment issues in China, we cannot directly pull container images from Docker Hub. To facilitate the deployment of some container services on computers within China, I decided to build my own container image registry (i.e., Registry) on a server located overseas.\nSolution # Docker official provides the image of container image registry Registry, we can use this image to deploy our own container image registry.\n","title":"Container (7): Building Your Own Container Image Registry","type":"post"},{"content":"","date":"11 February 2026","externalUrl":null,"permalink":"/en/series/container-technology-series/","section":"Series","summary":"","title":"Container Technology Series","type":"series"},{"content":"","date":"11 February 2026","externalUrl":null,"permalink":"/en/categories/development/","section":"Categories","summary":"","title":"Development","type":"categories"},{"content":"","date":"11 February 2026","externalUrl":null,"permalink":"/en/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":" Motivation # This website has been online for five years, consistently using the Hugo framework. Hugo is an excellent framework, but after years of use, I\u0026rsquo;ve encountered some inconveniences.\nSpecifically, since my website is deployed using Docker containers and the blog content is managed through a GitHub repository, every time I need to update the blog content, I have to modify markdown files on my local computer, commit to the GitHub repository, and then pull the updates on the server, rebuild the container, and restart it.\nWhile this workflow isn\u0026rsquo;t terribly complicated, the main drawback is that I must have the blog\u0026rsquo;s Git repository on the computer I\u0026rsquo;m using to write. Sometimes I get inspiration and want to write something, but if the blog\u0026rsquo;s Git repository isn\u0026rsquo;t available on my phone or current computer, I have to write it elsewhere and copy it into the blog\u0026rsquo;s markdown files later. This is somewhat inconvenient, but what\u0026rsquo;s more frustrating is when I want to continue writing a draft blog post—if I don\u0026rsquo;t have the Git repository available, I can\u0026rsquo;t see the previous content and thus can\u0026rsquo;t continue writing.\nSo I started thinking: is there a way to add an editing system to the blog that allows me to save drafts on the server? This way, from any computer, or even from my phone, I could open the draft and continue writing, and then publish it to the blog when finished.\nFollowing ChatGPT\u0026rsquo;s suggestions, such a system is called a Content Management System (CMS), and it recommended using DeCap CMS, an open-source CMS system to build the blog\u0026rsquo;s editing system.\nIntroduction to DeCap CMS # DeCap CMS is a Git-based content management system that supports multiple static website generators (such as Hugo, Jekyll, Gatsby, etc.). It provides a user-friendly interface that allows users to edit and manage blog content through a browser and commit changes directly to a Git repository.\nInstalling DeCap CMS is quite simple—you just need to place an admin folder in the root directory of the static website with an HTML file and a configuration file inside. The more complex part is configuring a backend service that communicates between DeCap CMS and the GitHub repository.\nHow DeCap CMS Works # DeCap CMS Workflow # The DeCap CMS workflow can be divided into roughly 4 steps:\nAdmin Login Interface: By placing an admin folder in the root directory of the static website, you can access the DeCap CMS login interface by visiting https://blog.example.com/admin/. User Authentication: DeCap CMS supports multiple authentication methods, including GitHub OAuth, GitLab OAuth, Bitbucket OAuth, and others. Users can choose the authentication method that suits them to log in. Content Editing: After successful login, users can edit blog content through the DeCap CMS interface. DeCap CMS provides a WYSIWYG (What You See Is What You Get) editor, allowing users to edit blog content directly in the browser and preview the results. Content Publishing: When users finish editing, they can click the publish button. Using the access token we provide to DeCap CMS to access the Git repository, DeCap CMS will commit the changes to the Git repository, trigger the static website generator\u0026rsquo;s (like Hugo) build process, and finally publish the updated blog content to the website. Components of DeCap CMS # From this, we can see that we need to prepare 4 things:\nDeCap CMS Frontend Files: You need to create an admin folder in the blog\u0026rsquo;s root directory and place DeCap CMS frontend files—an HTML file and a configuration file—inside. Login Authentication: You need to choose an authentication method and configure the corresponding authentication information. Git Repository Access Token: You need to generate a token that can access the blog\u0026rsquo;s Git repository and configure it in DeCap CMS. Proxy Service to Forward DeCap CMS Requests to the Backend: You need to configure a backend service to handle DeCap CMS requests and forward them to the Git repository. In my case, I use Hugo as the blog framework and the blog content is stored in a GitHub repository, so I need to configure DeCap CMS to support Hugo. Since DeCap CMS supports GitHub OAuth authentication, I can choose to use GitHub OAuth for user authentication. Finally, I need to configure a backend service to handle DeCap CMS requests and forward them to the GitHub repository.\nInstalling and Configuring DeCap CMS # Installing DeCap CMS Frontend Files # For the Hugo framework, DeCap CMS requires frontend files to be placed in the static folder of the Hugo project\u0026rsquo;s root directory (for other frameworks, they might need to be placed in public, src, site, etc., depending on DeCap CMS\u0026rsquo;s documentation). Therefore, we create a static/admin folder in the blog\u0026rsquo;s Hugo project root directory and place the DeCap CMS frontend files:\nindex.html: The main interface file for DeCap CMS, containing the frontend logic and interface design.\n1 2 3 4 5 6 7 8 9 10 11 \u0026lt;!doctype html\u0026gt; \u0026lt;html\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;utf-8\u0026#34; /\u0026gt; \u0026lt;meta name=\u0026#34;viewport\u0026#34; content=\u0026#34;width=device-width, initial-scale=1\u0026#34; /\u0026gt; \u0026lt;title\u0026gt;Jin Li Misc Admin\u0026lt;/title\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;script src=\u0026#34;https://unpkg.com/decap-cms@^3.0.0/dist/decap-cms.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; config.yml: The configuration file for DeCap CMS, used to configure the authentication method, Git repository access tokens, and other information.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 backend: name: github repo: jin-li/blog branch: main base_url: https://decap.example.com auth_endpoint: /auth media_folder: \u0026#34;hugosite/static/images\u0026#34; public_folder: \u0026#34;/images\u0026#34; site_url: https://blog.example.com display_url: https://blog.example.com publish_mode: editorial_workflow collections: - name: \u0026#34;blog\u0026#34; label: \u0026#34;Blog Posts\u0026#34; folder: \u0026#34;blog/content/posts\u0026#34; create: true slug: \u0026#34;{{year}}-{{month}}-{{day}}-{{slug}}\u0026#34; fields: - { label: \u0026#34;Title\u0026#34;, name: \u0026#34;title\u0026#34;, widget: \u0026#34;string\u0026#34; } - { label: \u0026#34;Date\u0026#34;, name: \u0026#34;date\u0026#34;, widget: \u0026#34;datetime\u0026#34; } - { label: \u0026#34;Draft\u0026#34;, name: \u0026#34;draft\u0026#34;, widget: \u0026#34;boolean\u0026#34;, default: true } - { label: \u0026#34;Tags\u0026#34;, name: \u0026#34;tags\u0026#34;, widget: \u0026#34;list\u0026#34;, required: false } - { label: \u0026#34;Body\u0026#34;, name: \u0026#34;body\u0026#34;, widget: \u0026#34;markdown\u0026#34; } The backend section configures the backend service to use GitHub with the repository jin-li/blog on the main branch, and the backend service address is https://decap.example.com with the authentication endpoint at /auth. The media_folder and public_folder configure the storage and access paths for media files respectively. The site_url and display_url configure the website URLs. The publish_mode configures the publishing mode, using editorial workflow here. Finally, the collections section configures content collections, with one \u0026ldquo;blog\u0026rdquo; collection configured for managing blog articles.\nIn this way, when we visit http://yourwebsite.com/admin/, DeCap CMS will load the admin interface and display the GitHub OAuth login option.\nConfiguring GitHub OAuth Authentication # To use GitHub OAuth authentication, we need to create an OAuth application on GitHub and obtain the corresponding Client ID and Client Secret. The specific steps are:\nLog in to GitHub, click the avatar in the top right corner, and select \u0026ldquo;Settings\u0026rdquo;. Select \u0026ldquo;Developer settings\u0026rdquo; from the left menu. Select \u0026ldquo;OAuth Apps\u0026rdquo; from the left menu, then click \u0026ldquo;New OAuth App\u0026rdquo;. On the \u0026ldquo;Register a new OAuth application\u0026rdquo; page, fill in the application name, homepage, and callback URL. The homepage URL should be your blog\u0026rsquo;s address (for example, https://blog.example.com), and the callback URL should be the authentication endpoint address of the DeCap CMS backend service (for example, https://decap.example.com/callback). Click the \u0026ldquo;Register application\u0026rdquo; button to complete the application registration. After registration, you\u0026rsquo;ll see the application\u0026rsquo;s Client ID and Client Secret. Record these two values; you\u0026rsquo;ll need them when configuring DeCap CMS later. Configuring DeCap CMS Proxy # When we visit https://blog.example.com/admin/ and click the GitHub OAuth login button, DeCap CMS will send an authentication request to https://decap.example.com/auth. This request needs to be forwarded to GitHub\u0026rsquo;s OAuth authentication endpoint to complete the authentication flow. Therefore, we need to configure a proxy to handle this request.\nThere are Docker containers developed by community members available to handle DeCap CMS request forwarding, such as:\ndecap-cms-github-backend docker-decap-cms-standalone I attempted to deploy both containers, using Traefik and Cloudflare Tunnel for reverse proxying like other containers, but neither were successful.\nEventually, I discovered that someone had implemented this proxy functionality using Cloudflare Workers, as shown in the GitHub repository decap-proxy. Cloudflare Workers is a serverless computing platform provided by Cloudflare that allows us to run JavaScript code on Cloudflare\u0026rsquo;s edge network, thereby implementing request processing and forwarding.\nThe specific steps can be found in the GitHub repository\u0026rsquo;s documentation. The general process is as follows:\nClone the decap-proxy repository to your local machine:\n1 git clone https://github.com/sterlingwes/decap-proxy.git Enter the decap-proxy directory and find a wrangler.toml.sample file, which is the configuration file for Cloudflare Workers. We need to copy it and rename it to wrangler.toml:\n1 cp wrangler.toml.sample wrangler.toml Edit the wrangler.toml file to configure the relevant Cloudflare Workers information, for example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 #:schema node_modules/wrangler/config-schema.json # \u0026#34;compatibility_date\u0026#34; and \u0026#34;main\u0026#34; are values you are unlikely to need to change compatibility_date = \u0026#34;2025-11-17\u0026#34; # schema version main = \u0026#34;src/index.ts\u0026#34; # entry point for the worker # The \u0026#34;name\u0026#34; parameter defines the name of the worker in your Cloudflare # Dashboard. It also specifies the first element of the default URL that # will reach your worker. name = \u0026#34;decap-proxy\u0026#34; # optional: uncomment and alter the following lines if using a custom domain. # route = { pattern = \u0026#34;decap.example.com\u0026#34;, zone_name = \u0026#34;example.com\u0026#34;, custom_domain = true } # optional: uncomment the following line if you don\u0026#39;t want wrangler to set up # the worker to be available at the default workers.dev url of # \u0026lt;worker-name\u0026gt;.\u0026lt;account-name\u0026gt;.workers.dev # where \u0026lt;worker-name\u0026gt; is the \u0026#34;name\u0026#34; parameter configured above. # workers_dev = false # optional: this worker template uses Web Crypto API natively and doesn\u0026#39;t require # nodejs_compat you can add this flag if you need Node.js polyfills, # but please be aware that those polyfills may have vulnerabilities # compatibility_flags = [\u0026#34;nodejs_compat\u0026#34;] # Variable bindings. These are arbitrary, plaintext strings (similar to environment variables) # Docs: # - https://developers.cloudflare.com/workers/wrangler/configuration/#environment-variables # Note: Use secrets to store sensitive data. # - https://developers.cloudflare.com/workers/configuration/secrets/ [vars] GITHUB_REPO_PRIVATE = \u0026#34;1\u0026#34; # Should be set to \u0026#34;1\u0026#34;, if your website repo is private The comments in the file explain things clearly. Mainly, we need to modify the route to configure it to the address of our DeCap CMS backend service, for example, decap.example.com. If you don\u0026rsquo;t want Cloudflare to automatically set up the default workers.dev domain, you can set workers_dev to false. Finally, we need to configure a variable GITHUB_REPO_PRIVATE in the [vars] section; if your blog\u0026rsquo;s GitHub repository is private, set it to 1, otherwise you can leave it unset.\nAfter configuring the wrangler.toml file, we can use the Cloudflare Workers CLI tool wrangler to deploy our Worker. The GitHub repository mentions that you can set your Cloudflare account and token as environment variables to log in directly from the command line, but I tried this without success. So I used the unset command to clear the environment variables, then ran the npx wrangler login command to log in.\nIf you haven\u0026rsquo;t installed wrangler before, the first time you run npx wrangler login, the system will prompt you to install wrangler. You can follow the prompts to install it. After installation is complete, run npx wrangler login again, and it will display a URL in the command line, prompting you to open this URL to complete the login.\nNote: You need to open this URL on the same computer where you ran the npx wrangler login command, because after you complete verification with your Cloudflare account, the URL that wrangler redirects to is a local address, such as http://localhost:8787/callback?code=xxx\u0026amp;state=yyy. This URL will be intercepted by wrangler, which will extract authentication information from it to complete the login. If you open this URL on another computer, although the verification completes, wrangler cannot intercept the URL, so the login will fail.\nAfter successful verification, you\u0026rsquo;ll see the following browser interface:\nAfter completion of verification, wrangler will indicate successful login. Next, we set the GitHub OAuth application\u0026rsquo;s Client ID and Client Secret we obtained earlier as environment variables for the Cloudflare Worker:\n1 2 npx wrangler secret put GITHUB_CLIENT_ID npx wrangler secret put GITHUB_CLIENT_SECRET After running the above commands, the system will prompt you to enter the corresponding values. After entering them, wrangler will store these values as secret environment variables in Cloudflare.\nFinally, we can deploy our Worker:\n1 npx wrangler publish After deployment is complete, you can open a browser and visit https://decap.example.com. If you see the page displaying Hello 👋, it means our Cloudflare Worker has been successfully deployed and is running.\nTesting DeCap CMS Access # After completing the above installation and configuration, our DeCap CMS should be ready to work normally. We can test accessing the DeCap CMS login interface by visiting https://blog.example.com/admin/:\nIf you can see the GitHub OAuth login option and successfully log in, you\u0026rsquo;ll see the DeCap CMS content editing interface:\nIn the editing interface, we can create new blog articles, edit existing blog articles, and preview the editing effects. When we finish editing, we click the publish button, and DeCap CMS will commit the changes to the GitHub repository, trigger Hugo\u0026rsquo;s build process, and finally publish the updated blog content to the website.\n","date":"9 February 2026","externalUrl":null,"permalink":"/en/p/building-a-personal-website-part-5-choosing-and-configuring-a-content-management-system-cms-for-hugo/","section":"Posts","summary":" Motivation # This website has been online for five years, consistently using the Hugo framework. Hugo is an excellent framework, but after years of use, I’ve encountered some inconveniences.\nSpecifically, since my website is deployed using Docker containers and the blog content is managed through a GitHub repository, every time I need to update the blog content, I have to modify markdown files on my local computer, commit to the GitHub repository, and then pull the updates on the server, rebuild the container, and restart it.\n","title":"Building a Personal Website (Part 5): Choosing and Configuring a Content Management System (CMS) for Hugo","type":"post"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/en/tags/cms/","section":"Tags","summary":"","title":"CMS","type":"tags"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/en/tags/content-management-system/","section":"Tags","summary":"","title":"Content Management System","type":"tags"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/en/tags/decap-cms/","section":"Tags","summary":"","title":"DeCap CMS","type":"tags"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/en/tags/hugo/","section":"Tags","summary":"","title":"Hugo","type":"tags"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/en/tags/personal-website/","section":"Tags","summary":"","title":"Personal Website","type":"tags"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/en/series/personal-website-building-series/","section":"Series","summary":"","title":"Personal Website Building Series","type":"series"},{"content":"","date":"9 February 2026","externalUrl":null,"permalink":"/en/categories/website/","section":"Categories","summary":"","title":"Website","type":"categories"},{"content":"","date":"2026年2月9日","externalUrl":null,"permalink":"/tags/%E4%B8%AA%E4%BA%BA%E7%BD%91%E7%AB%99/","section":"Tags","summary":"","title":"个人网站","type":"tags"},{"content":"","date":"2026年2月9日","externalUrl":null,"permalink":"/series/%E4%B8%AA%E4%BA%BA%E7%BD%91%E7%AB%99%E6%90%AD%E5%BB%BA%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"个人网站搭建系列","type":"series"},{"content":"","date":"2026年2月9日","externalUrl":null,"permalink":"/tags/%E5%86%85%E5%AE%B9%E7%AE%A1%E7%90%86%E7%B3%BB%E7%BB%9F/","section":"Tags","summary":"","title":"内容管理系统","type":"tags"},{"content":"","date":"2026年2月9日","externalUrl":null,"permalink":"/categories/%E7%BD%91%E7%AB%99/","section":"Categories","summary":"","title":"网站","type":"categories"},{"content":"","date":"14 November 2025","externalUrl":null,"permalink":"/en/tags/conky/","section":"Tags","summary":"","title":"Conky","type":"tags"},{"content":"","date":"14 November 2025","externalUrl":null,"permalink":"/en/tags/linux/","section":"Tags","summary":"","title":"Linux","type":"tags"},{"content":" Motivation # When using Linux systems, it is often necessary to monitor the system status in real-time, such as monitoring CPU and GPU usage and temperature while gaming, checking VRAM usage when training machine learning models, or tracking network speed when downloading large files. Although there are graphical tools like System Monitor and command-line tools like htop and nvidia-smi available for monitoring system status, these tools usually require manual opening and do not display information directly on the desktop, which can be inconvenient at times.\nConky is a lightweight system monitoring software that can display various system status information directly on the desktop in real-time, allowing users to easily check their system status at any time. I have been using it for a long time and find it quite useful. However, I previously used a heavily modified theme that met my needs but had poor portability. Recently, after changing my computer, I decided to create a new elegant, user-friendly, and highly portable Conky theme and share the installation and usage methods here.\nInstalling Conky # Conky Versions # Most Linux distributions provide Conky packages in their software repositories, but the versions are often outdated. Many Conky packages lack Lua support, do not support NVIDIA GPU monitoring, or have poor support for Wayland. If you install Conky using package managers like apt (for Ubuntu, Debian, etc.), dnf (for Fedora, etc.), or pacman (for Arch Linux, etc.), it is recommended to check whether the installed Conky package supports the features you need.\nIn addition to distribution repositories, the Conky GitHub repository provides Conky binary packages in AppImage format, which are more up-to-date and include Lua support and NVIDIA GPU monitoring support. This version is recommended. You can download it from the Releases page.\nInstallation Steps # Installing via Package Manager # You can refer to the Installation Guide in the GitHub repository. Below are some common installation methods for popular Linux distributions.\nDebian/Ubuntu # In Debian and Ubuntu, you can search for available Conky packages using the following command:\n1 apt search conky You will typically see multiple Conky-related packages, such as conky-all, conky-lua, etc. You can check the description of the conky-all package to confirm its version and supported features:\n1 apt show conky-all Finally, you can install Conky using the following command:\n1 sudo apt install conky-all Fedora # In Fedora, you can search for available Conky packages using the following command:\n1 dnf search conky Then, check the description of the conky package using the following command:\n1 dnf info conky Finally, you can install Conky using the following command:\n1 sudo dnf install conky Arch Linux # If you are using Arch Linux, you can search for available Conky packages using the following command:\n1 pacman -Ss conky Then, check the description of the conky package using the following command:\n1 pacman -Si conky Finally, you can install Conky using the following command:\n1 sudo pacman -S conky Installing via AppImage # You can refer to the installation instructions in the GitHub repository. Below are the brief installation steps:\nDownload the latest Conky AppImage package from the Releases page, for example, conky-*.AppImage.\nMake the downloaded AppImage package executable:\n1 chmod +x conky-*.AppImage Run the Conky AppImage package:\n1 ./conky-*.AppImage (Optional) Move the Conky AppImage package to a system directory, such as /usr/local/bin/, so that you can run Conky from anywhere in the command line:\n1 sudo mv conky-*.AppImage /usr/local/bin/conky (Optional) Add the Conky AppImage package to the start menu for easier launching in the future. You can create a .desktop file with the following content:\n1 2 3 4 5 6 [Desktop Entry] Name=Conky Exec=/path/to/conky-*.AppImage Icon=conky Type=Application Categories=Utility;System; Save the above content as ~/.local/share/applications/conky.desktop, and replace /path/to/conky-*.AppImage with the actual path to the AppImage package.\nRunning Conky will display system status information on the desktop, as shown in the image below:\nConfiguring Conky # Configuration File # Conky\u0026rsquo;s configuration file is usually located at ~/.config/conky/conky.conf. If this file does not exist, you can generate a default configuration file by running the following command:\n1 conky -C \u0026gt; ~/.config/conky/conky.conf Here, the generated default configuration file is relatively simple and can be modified and expanded as needed. Conky\u0026rsquo;s configuration file uses a syntax similar to the INI format, allowing you to set various options and display content. For specific configuration options, you can refer to the Conky official documentation.\nI recommend using Lua scripts to write Conky configuration files, as this allows for more flexible control over display content and styles. Below is a simple Lua configuration file example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 conky.config = { alignment = \u0026#39;top_right\u0026#39;, background = false, border_width = 1, cpu_avg_samples = 2, default_color = \u0026#39;white\u0026#39;, double_buffer = true, draw_borders = false, draw_graph_borders = true, gap_x = 10, gap_y = 10, minimum_width = 200, net_avg_samples = 2, no_buffers = true, out_to_console = false, out_to_stderr = false, own_window = true, own_window_class = \u0026#39;Conky\u0026#39;, own_window_type = \u0026#39;desktop\u0026#39;, update_interval = 1.0, use_xft = true, xftalpha = 0.8, }; conky.text = [[ ${font Arial:bold:size=12}System Monitor${font} Uptime: ${uptime} CPU: ${cpu cpu0}% ${cpubar cpu0} RAM: ${mem} / ${memmax} ${membar} Disk: ${fs_used /} / ${fs_size /} ${fs_bar /} GPU: ${nvidia gpu0}°C ${nvidia gpu0 usage}% Net: ${downspeed eth0} ↓↑ ${upspeed eth0} ]]; Using Conky Themes # Creating a Conky configuration from scratch can be cumbersome, so I recommend using existing Conky themes. Many Conky themes can be found online, such as:\nThe Conky GitHub repository contains some user-shared themes Conky Themes website has many user-shared Conky themes. DeviantArt also has many Conky themes. Here, I recommend a Conky theme I developed myself, called jinli-conky.\nThe effect is shown in the image below:\nThis theme supports monitoring various system statuses such as CPU, memory, disk, network, and NVIDIA GPU. It features a simple and elegant style, is written purely in Lua, and is easy to modify and extend.\njinli-conky Theme # Installing the jinli-conky Theme # Installation instructions can be found on the jinli-conky GitHub repository. The main steps are as follows:\nClone the jinli-conky repository:\n1 git clone https://github.com/jin-li/jinli-conky.git It is best to clone it into the ~/.config/conky/ directory.\nInstall fonts (optional)\nThe jinli-conky theme uses some custom fonts to display certain icons. If you want to display these icons, you need to install these fonts first. It is okay if you do not install the fonts; the icons will be displayed as text.\nNavigate to the jinli-conky/fonts/ directory, where you will find a font file named subset-SymbolNF.ttf. You can double-click to install it. Alternatively, you can install it via the command line:\n1 2 3 cd jinli-conky/fonts/ cp subset-SymbolsNF.ttf ~/.local/share/fonts/ fc-cache -fv Navigate to the jinli-conky directory and run the start.sh script to install the theme:\n1 2 cd jinli-conky ./start.sh After running the script, the Conky theme will automatically start and display on the desktop.\nAuto-start on boot (optional)\nRunning the startup script in the previous step will generate an auto-start entry file named conky-start.desktop. If you want Conky to automatically start each time you log in, you can copy it to the ~/.config/autostart/ directory. First, check if the ~/.config/autostart/ directory exists; if not, you can create it:\n1 mkdir -p ~/.config/autostart/ Then copy the auto-start entry file:\n1 cp conky-start.desktop ~/.config/autostart/ Configuring the jinli-conky Theme # After running the start.sh script, a configuration file named jinli-config.lua will be generated in the theme directory. This configuration file is a copy of the theme\u0026rsquo;s default jinli-config.example.lua file. You can edit this configuration file to modify various options of the theme, such as display content and styles.\nSome commonly used configuration options include:\nScaling Factor: The default screen resolution for the theme is 1920x1080. If your screen resolution is different, you can modify the scaling option to adjust the size of the theme. For example, if your screen resolution is 3840x2160, you can change the scaling option to scaling = 2.0 (3840/1920). Overall Parameters: You can modify overall display width, height, position, etc. You can also modify the refresh interval (default is 1 second), transparency, change fonts, and adjust the distance from the screen edges. Individual Components: The theme includes components such as Clock, System Info, CPU, Memory, Disk, Network, and GPU. You can enable or disable individual components, or modify the display style and parameters of specific components, such as displaying the gauge on the right side or changing the display order of components. After modifying the configuration file, save the file, and the changes will take effect immediately without needing to restart Conky.\n","date":"14 November 2025","externalUrl":null,"permalink":"/en/p/linux-system-monitor-conky/","section":"Posts","summary":" Motivation # When using Linux systems, it is often necessary to monitor the system status in real-time, such as monitoring CPU and GPU usage and temperature while gaming, checking VRAM usage when training machine learning models, or tracking network speed when downloading large files. Although there are graphical tools like System Monitor and command-line tools like htop and nvidia-smi available for monitoring system status, these tools usually require manual opening and do not display information directly on the desktop, which can be inconvenient at times.\n","title":"Linux System Monitor: Conky","type":"post"},{"content":"","date":"14 November 2025","externalUrl":null,"permalink":"/en/tags/system-monitoring/","section":"Tags","summary":"","title":"System Monitoring","type":"tags"},{"content":"","date":"2025年11月14日","externalUrl":null,"permalink":"/tags/%E7%8A%B6%E6%80%81%E7%9B%91%E6%8E%A7/","section":"Tags","summary":"","title":"状态监控","type":"tags"},{"content":"","date":"7 October 2025","externalUrl":null,"permalink":"/en/tags/clipboard/","section":"Tags","summary":"","title":"Clipboard","type":"tags"},{"content":"","date":"7 October 2025","externalUrl":null,"permalink":"/en/tags/clipcascade/","section":"Tags","summary":"","title":"ClipCascade","type":"tags"},{"content":" Motivation # In a previous article, I introduced several software solutions for sharing keyboard and mouse across multiple computers, including Barrier, InputLeap, and Deskflow. These tools greatly enhance the experience of working with multiple computers by allowing seamless sharing of keyboard and mouse input between Windows, macOS, and Linux systems. However, I discovered that the clipboard sharing feature is not available in Wayland environments. Unfortunately, my primary computer runs Fedora 42, which uses Wayland as its default graphical interface, preventing me from using the clipboard sharing functionality of Barrier, InputLeap, and Deskflow on this machine.\nLater, I came across a mention of another cross-platform clipboard sharing software called ClipCascade in the Deskflow discussion forum. It was said that this third-party software could enable cross-system clipboard sharing and that it supports Wayland environments. I tried installing and using it, and I found that it indeed met my needs. Therefore, I would like to share the installation and usage methods for ClipCascade here.\nFor information on the installation and usage of Barrier/InputLeap/Deskflow, you can refer to my article: Installation and Usage of Free Cross-Platform Keyboard and Mouse Sharing Software Barrier/InputLeap/Deskflow.\nIntroduction # ClipCascade is a free and open-source cross-platform clipboard sharing software that supports sharing clipboard content between Windows, macOS, Linux, and Android systems. It supports Wayland environments, making it usable on newer Linux systems. Additionally, it allows users to set up their own servers, enhancing data transmission security.\nInstallation # Shared Clipboard Server # Before installing ClipCascade, we need to set up a shared clipboard server.\nClipCascade officially provides a public server that anyone can use, but the data will go through a third party. If you have high privacy requirements, it is recommended to set up your own server. Of course, when using the public server, you can also choose to encrypt the data transmission, so the security is still quite good. However, I have become accustomed to self-hosting various services, so I decided to set up my own ClipCascade server here.\nSetting Up ClipCascade Server with Docker # ClipCascade officially provides a Docker image, so you can easily set up the server using Docker. I prefer to use Docker Compose to manage Docker containers, so I will also use Docker Compose to set up the ClipCascade server here. The required docker-compose.yml file is also provided in the ClipCascade GitHub repository, which you can refer to here. Below is the content of the docker-compose.yml file that I used:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 services: clipcascade: image: sathvikrao/clipcascade:latest ports: - \u0026#34;8084:8080\u0026#34; # Expose the ClipCascade server on port 8080 restart: always # Automatically restart the container if it stops volumes: - ${DATA_DIR}/cc_users:/database # Persistent storage for user data environment: - TZ=${TZ} # Set the timezone from the .env file - CC_MAX_MESSAGE_SIZE_IN_MiB=10 # Maximum message size in MiB (ignored if P2P mode is enabled) - CC_P2P_ENABLED=false # Enables or disables peer-to-peer(P2P) mode # - CC_ALLOWED_ORIGINS=https://clipcascade.example.com # Defines allowed CORS origins for security # - CC_SIGNUP_ENABLED=false # Enables or disables user self-registration networks: - traefik-network networks: traefik-network: external: true Here, DATA_DIR and TZ are environment variables read from the .env file. DATA_DIR specifies the data storage directory for the ClipCascade server, and TZ specifies the time zone. You can modify these configurations according to your needs.\nIn addition, I specified the traefik-network network because I use Traefik as a reverse proxy server, and all services connect to this network. If you are not using Traefik, you can remove this part of the configuration.\nUsing Traefik to Reverse Proxy the ClipCascade Server # For information on reverse proxying Docker services, I have also introduced this in previous articles, which you can refer to:\n\u0026ldquo;Accessing Personal Websites from the Public Network - Nginx Reverse Proxy Configuration\u0026rdquo; \u0026ldquo;Accessing Personal Websites from the Public Network (2) - Traefik Reverse Proxy Configuration\u0026rdquo; These two articles respectively introduce how to use Nginx and Traefik to reverse proxy Docker services. Here I continue to use Traefik to reverse proxy the ClipCascade server.\nI use a dynamic configuration file to configure Traefik\u0026rsquo;s reverse proxy rules, creating a dynamic configuration file named clipcascade.yml, with the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 http: routers: clipcascade-router: entryPoints: - web rule: \u0026#34;Host(`clipcascade.example.com`)\u0026#34; service: clipcascade-service services: clipcascade-service: loadBalancer: servers: - url: \u0026#34;http://clipcascade:8080\u0026#34; Starting the Service and Creating Users # Start the Docker Container\nNavigate to the directory containing the docker-compose.yml file and run the following command to start the ClipCascade server:\n1 docker-compose up -d Reverse Proxy\nIf you are using the Traefik dynamic configuration file, Traefik will automatically load the configuration after the configuration file is created, and there is no need to restart Traefik.\nIf you are using Nginx as a reverse proxy, you need to restart the Nginx service:\n1 sudo systemctl restart nginx Login and Create Users\nOpen your browser and visit http://clipcascade.example.com (replace clipcascade.example.com with your domain), and you will see the ClipCascade login page. By default, the admin account is admin and the password is admin123. After logging in, you will see the following page:\nIt is recommended that you click the \u0026ldquo;Change Password\u0026rdquo; link at the top to change the admin password immediately.\nThen you can create a regular user account for daily use. Click the Add New User button in the Admin Panel, enter the username and password, and then click the Add User button to create a new user.\nClient Installation and Usage # ClipCascade supports multiple operating systems, including Windows, macOS, Linux, and Android. You can download and install the corresponding client for your system according to the documentation in the GitHub repository.\nIt is important to note that there are packaged installation files available for Windows, macOS, and Android systems, but for Linux systems, only the unbundled Python source code is available, and you need to resolve dependencies and run it yourself. Here, I will take Fedora 42 as an example to introduce the installation and usage of the ClipCascade client on Linux systems.\nInstalling ClipCascade Client on Fedora 42 # Python Environment\nClipCascade client is written in Python, so you need to install the Python environment first. Fedora 42 comes with Python 3.12 by default, but some Python libraries that the ClipCascade client depends on are no longer supported in Python 3.12, so you need to install Python 3.11. I use conda to manage Python environments, so I created a Python 3.11 virtual environment with conda. You can refer to my article on Python Environment Management Summary for the advantages of using conda.\n1 2 conda create -n clipcascade python=3.11 conda activate clipcascade Then install the Python dependencies:\n1 pip3 install -r requirements.txt Installing Other Dependencies\nIn addition to the Python dependencies, you also need to install some system dependencies. You can install these according to the instructions in the ClipCascade GitHub repository:\n1 2 3 4 5 sudo dnf check-update sudo dnf install -y python3 python3-pip python3-gobject xclip wl-clipboard dunst sudo dnf install -y libappindicator-gtk3 sudo dnf install -y python3-xxhash python3-pyperclip python3-requests python3-websocket-client python3-pycryptodomex python3-tkinter python3-pystray python3-pyfiglet python3-beautifulsoup4 sudo dnf install -y ffmpeg ffmpeg-devel It is important to note that the installation instructions for ClipCascade mention that you also need to install python3-plyer and python3-aiortc using DNF, but these two packages are no longer available in the official repository for Fedora 42. However, you don\u0026rsquo;t need to worry, because these two packages are already listed in requirements.txt, and they have been installed when you installed the Python dependencies with pip.\nThe above two issues are not pointed out in the installation instructions in the ClipCascade GitHub repository, and I have already raised an issue with the author, hoping that they can update the installation instructions.\nRunning the Client # Navigate to the source directory of the ClipCascade client and run the following command to start the client:\n1 python3 main.py After starting, a window will pop up prompting you to enter the server address, username, and password:\nOnce logged in successfully, a notification will pop up, and you can share clipboard content with other devices logged into the same ClipCascade account.\n(Bonus) Adding to the Start Menu # At this point, we can use the ClipCascade client on Fedora 42, but it\u0026rsquo;s still a bit cumbersome to open a terminal, navigate to the source directory, and run the python3 main.py command each time. We can add ClipCascade to the start menu so that it can be launched like other applications.\nLinux desktop environments use .desktop files to manage applications in the start menu. I am using the KDE desktop environment on Fedora 42, and the start menu\u0026rsquo;s .desktop files are stored in the ~/.local/share/applications/ directory. We can create a file named clipcascade.desktop in this directory with the following content:\n1 2 3 4 5 6 7 8 9 [Desktop Entry] Version=1.0 Type=Application Name=ClipCascade Comment=Cross-platform clipboard sharing tool Exec=/home/yourusername/miniconda3/envs/clipcascade/bin/python3 /path/to/ClipCascade/main.py Icon=/path/to/ClipCascade/icon.png Terminal=false Categories=Utility; You need to replace the paths in the Exec and Icon fields with your own paths. The Exec field specifies the command to start the ClipCascade client, and here you need to specify the path to the Python interpreter and the main.py file. The Icon field specifies the path to the ClipCascade icon, and you can download the ClipCascade icon file and place it in an appropriate location.\nIt is important to note that the Python interpreter path in the Exec field needs to be specified as the path to the Python interpreter in the Python environment you used to install the ClipCascade client. If you used a conda-created virtual environment, you can run the which python3 command in the terminal to find out the path to the Python interpreter.\nInstalling ClipCascade Client on macOS # The installation of the ClipCascade client on macOS is relatively simple. You can directly download the packaged installation file and follow the instructions in the ClipCascade GitHub repository to install it. I followed the instructions step by step and did not encounter any additional issues.\n","date":"7 October 2025","externalUrl":null,"permalink":"/en/p/installation-and-usage-of-clipcascade-a-free-cross-platform-clipboard-sharing-software/","section":"Posts","summary":" Motivation # In a previous article, I introduced several software solutions for sharing keyboard and mouse across multiple computers, including Barrier, InputLeap, and Deskflow. These tools greatly enhance the experience of working with multiple computers by allowing seamless sharing of keyboard and mouse input between Windows, macOS, and Linux systems. However, I discovered that the clipboard sharing feature is not available in Wayland environments. Unfortunately, my primary computer runs Fedora 42, which uses Wayland as its default graphical interface, preventing me from using the clipboard sharing functionality of Barrier, InputLeap, and Deskflow on this machine.\n","title":"Installation and Usage of ClipCascade: A Free Cross-Platform Clipboard Sharing Software","type":"post"},{"content":"","date":"7 October 2025","externalUrl":null,"permalink":"/en/tags/sharing/","section":"Tags","summary":"","title":"Sharing","type":"tags"},{"content":"","date":"2025年10月7日","externalUrl":null,"permalink":"/tags/%E5%85%B1%E4%BA%AB/","section":"Tags","summary":"","title":"共享","type":"tags"},{"content":"","date":"2025年10月7日","externalUrl":null,"permalink":"/tags/%E5%89%AA%E5%88%87%E6%9D%BF/","section":"Tags","summary":"","title":"剪切板","type":"tags"},{"content":"","date":"20 August 2025","externalUrl":null,"permalink":"/en/tags/cuda/","section":"Tags","summary":"","title":"CUDA","type":"tags"},{"content":"","date":"20 August 2025","externalUrl":null,"permalink":"/en/tags/deep-learning/","section":"Tags","summary":"","title":"Deep Learning","type":"tags"},{"content":"","date":"20 August 2025","externalUrl":null,"permalink":"/en/tags/gpu/","section":"Tags","summary":"","title":"GPU","type":"tags"},{"content":"","date":"20 August 2025","externalUrl":null,"permalink":"/en/tags/nvidia/","section":"Tags","summary":"","title":"Nvidia","type":"tags"},{"content":" Motivation # This is the fourth article in the \u0026ldquo;Nvidia GPU\u0026rdquo; series, which mainly records how to use Nvidia GPU, especially on Linux systems. In previous articles, I recorded the settings for using Nvidia GPU on Ubuntu, including games, CUDA programming, deep learning, Docker containers, etc. Later, I switched to Fedora system, thinking that configuring Nvidia GPU for CUDA programming and deep learning environment on Fedora would be similar to that on Ubuntu, and with the support of RPM Fusion, installation and configuration would be more convenient. However, I actually encountered unexpected difficulties, so I had to write another article to record these problems and solutions.\nSimply put, the problem encountered in the previous article was that the GCC version used by Fedora 42 system is incompatible with Nvidia\u0026rsquo;s CUDA version, resulting in the inability to use the CUDA compiler. For details, please refer to the last part of the previous article Nvidia GPU (3): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Fedora. To solve this problem, I decided to configure Nvidia GPU for CUDA programming and deep learning environment on Fedora using Docker containers.\nOther articles in this series can be found at:\nNvidia GPU (1): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Ubuntu Nvidia GPU (2): Accelerate Video Editing and Transcoding for FFmpeg Using GPU Nvidia GPU (3): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Fedora Container Solution # To bypass the incompatibility issue between Fedora 42 system and CUDA compiler, I decided to use container technology to create an Ubuntu-based environment using the image provided by Nvidia.\nBut I still took some detours in the process of creating it, mainly because I had always wanted to use Podman to replace Docker, so I tried to use Podman to create a CUDA programming environment this time. However, due to permission issues, Nvidia\u0026rsquo;s support issues, etc., I encountered some unsolvable problems. In the end, I gave up using Podman and continued to use Docker. I will write an article to introduce Podman when I have the chance. In the future, I will also gradually transfer container services to Podman, so I will not elaborate here. Here, I mainly record the process of setting up a CUDA programming and deep learning environment using Docker.\nBasic Framework # We have installed Nvidia GPU and drivers on the Fedora system. To allow container applications to use Nvidia GPU, we also need to install Nvidia\u0026rsquo;s Docker support. Therefore, the basic framework of the entire environment is as follows:\nAt the bottom is the Nvidia GPU that supports CUDA. Next is the host machine with Nvidia drivers installed. Then is the Nvidia Container Toolkit that allows container applications to call the host machine\u0026rsquo;s GPU. At the top is the container, which contains the CUDA toolkit for programming and calling the GPU. Setting Up CUDA Programming and Deep Learning Environment Container # Preparing the Host Machine # Before creating and deploying the container, we need to ensure that Nvidia drivers, Docker, and Nvidia Container Toolkit are installed on the host machine.\nNvidia Drivers: Please refer to the previous article Nvidia GPU (3): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Fedora for installing Nvidia drivers on Fedora system through RPM Fusion. You can confirm whether the Nvidia drivers are installed successfully by running the following command:\n1 nvidia-smi If running the above command correctly outputs the current status of the GPU as shown below, it means the drivers are installed successfully:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 575.64.05 Driver Version: 575.64.05 CUDA Version: 12.9 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 5070 Ti Off | 00000000:02:00.0 On | N/A | | 0% 42C P8 18W / 300W | 2837MiB / 16303MiB | 1% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2540 G /usr/bin/ksecretd 3MiB | Then record the CUDA version number corresponding to the current driver, for example, the above shows support for CUDA 12.9.\nDocker: Please refer to the Docker official documentation, the main steps are as follows:\n1 2 3 4 sudo dnf -y install dnf-plugins-core sudo dnf-3 config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin sudo systemctl enable --now docker You can run the following command to verify whether the installation was successful:\n1 sudo docker run hello-world If you don\u0026rsquo;t want to use sudo to run Docker, you need to add the current user to the docker group:\n1 2 sudo usermod -aG docker $USER newgrp docker Nvidia Container Toolkit: Please refer to the Nvidia official documentation for installation, the main steps are as follows:\n1 2 3 4 5 6 7 curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \\ export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1 sudo dnf install -y \\ nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \\ nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \\ libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \\ libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION} After installation, you can run the following command to verify whether the Nvidia Container Toolkit is installed successfully (note that the CUDA version number here should be consistent with your own version number):\n1 docker run --rm --gpus all nvidia/cuda:12.9.1-base-ubuntu24.04 nvidia-smi If you can correctly output the status information of the Nvidia GPU as when running the nvidia-smi command on the host machine, it means that the Nvidia Container Toolkit is installed successfully.\nCreating and Deploying the Container # Choosing the Image\nNvidia provides a variety of container images for CUDA programming and deep learning. We can choose the appropriate container to create and deploy based on our needs on their Docker image site. There are many types of containers, which can be roughly divided into the following two categories:\nGeneral Images: Suitable for defining some more general CUDA programming and deep learning environments. There are different types inside:\nBase Image: Such as nvidia/cuda:12.9.1-base-ubuntu24.04, this is a basic CUDA image that contains the basic environment of Ubuntu 24.04 and CUDA 12.9.1, suitable for users who need to customize the development environment. Deep Learning Framework Image: Such as nvidia/cuda:12.9.1-devel-ubuntu24.04, this is an image that contains the CUDA development environment, users do not need to install the CUDA development toolkit again, suitable for users who need to do CUDA programming. Runtime Image: Such as nvidia/cuda:12.9.1-runtime-ubuntu24.04, this is an image that only contains the runtime environment, suitable for users who only need to run CUDA applications. Specialized Images: Suitable for some specific deep learning frameworks and applications. There are also different types inside:\nTensorFlow Image: Such as nvidia/13.0.0-tensorrt-devel-ubuntu24.04, this is an image that contains TensorFlow and CUDA, suitable for users who need to use TensorFlow for deep learning. cudnn Image: Such as nvidia/13.0.0-cudnn-devel-ubuntu24.04, this is an image that contains cuDNN and CUDA, suitable for users who need to use cuDNN for deep learning. PyTorch Image: Such as nvidia/pytorch:23.08-py3, this is an image that contains PyTorch and CUDA, suitable for users who need to use PyTorch for deep learning. Creating a Custom Image\nWe can directly use the images provided by Nvidia to create and deploy containers, but these images are relatively basic and may not meet our needs. Therefore, we can create a custom image based on these images by writing a Dockerfile to define the environment we need. Here is an example Dockerfile that creates a CUDA programming and deep learning environment based on the nvidia/cuda:12.9.1-devel-ubuntu24.04 image:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 # Base: CUDA 12.9 + Ubuntu 22.04 FROM nvidia/cuda:12.9.1-devel-ubuntu24.04 ENV DEBIAN_FRONTEND=noninteractive # Install system dependencies and sudo USER root RUN apt-get update \u0026amp;\u0026amp; apt-get install -y --no-install-recommends \\ build-essential \\ gcc-14 g++-14 \\ cmake \\ git \\ wget \\ curl \\ sudo \\ bzip2 \\ ca-certificates \\ zsh \\ \u0026amp;\u0026amp; rm -rf /var/lib/apt/lists/* # Update alternatives to use GCC 14 RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100 \\ \u0026amp;\u0026amp; update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-14 100 # Give ubuntu user passwordless sudo RUN echo \u0026#34;ubuntu ALL=(ALL) NOPASSWD:ALL\u0026#34; \u0026gt;\u0026gt; /etc/sudoers # Switch to ubuntu user USER ubuntu WORKDIR /workspace CMD [\u0026#34;/bin/bash\u0026#34;] Above Dockerfile creates a container based on the nvidia/cuda:12.9.1-devel-ubuntu24.04 image. Besides the basic CUDA development environment, I also did the following configurations:\nadded the default user ubuntu to the sudo group, so that the sudo command can be used inside the container. installed some commonly used tools and libraries, such as build-essential, git, wget, curl, vim, etc. installed zsh, as I am used to using oh-my-zsh as a shell tool. In addition, I am also used to customizing the terminal with oh-my-zsh and powerlevel10k, and using mamba as a development environment management tool. But these are more personalized configurations, just install and configure them inside the container as mentioned before.\nCreate docker-compose.yml\nTo manage and deploy containers conveniently, we can use docker-compose to create a docker-compose.yml file. This file can be placed in the same directory, such as ~/docker/cuda-dev/docker-compose.yml. Here is a simple example of docker-compose.yml:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 version: \u0026#34;3.9\u0026#34; services: cuda-dev: build: . container_name: cuda-dev runtime: nvidia deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] environment: - NVIDIA_VISIBLE_DEVICES=all - NVIDIA_DRIVER_CAPABILITIES=compute,utility volumes: - /home/${USER}/develop:/home/ubuntu - /run/media/${USER}/Disk1/devdata:/data tty: true This configuration file defines a service named cuda-dev that uses the Dockerfile we created earlier to build the image. I created a workspace directory in the home directory of the host machine to store development files and mounted it to the /home/ubuntu directory in the container. At the same time, I also mounted an entire disk from the host machine to the /data directory in the container, which makes it convenient to access files on the host machine within the container.\nThe reason for choosing this mounting method is to match the design above:\nMount the develop folder to the /home/ubuntu directory in the container, so if we install directly to /home/ubuntu or put development files in this directory within the container, all changes will be directly saved to the develop folder on the host machine. Mount the /run/media/${USER}/Disk1/devdata directory of the host machine to the /data directory in the container, so if we put the data needed for development on the devdata directory of the Disk1 hard drive, it can be directly accessed within the container. Build and Run the Container\nIn the terminal, navigate to the directory where the docker-compose.yml file is located, such as ~/docker/cuda-dev, and then run the following command to build the container:\n1 docker compose build After the build is complete, use the following command to run the container:\n1 docker compose up -d After running, you can check the status of the container with the following command:\n1 docker ps | grep cuda-dev Access and Test the Container\nAfter the container is running, you can enter the terminal of the container with the following command:\n1 docker exec -it cuda-dev bash Check CUDA Compiler\nAfter entering the container, you can test whether the CUDA programming environment is working properly, for example, run the following command to check the CUDA version:\n1 nvcc --version If the correct CUDA version information is output, it means that the CUDA compiler is working properly.\nCheck if the Container Can Call the Nvidia GPU\nRun the following command to check the status of the Nvidia GPU:\n1 nvidia-smi If the correct status information of the Nvidia GPU is output, it means that the container can call the Nvidia GPU.\nTest Compiling CUDA Programs\nNext, let\u0026rsquo;s try to compile the CUDA-Sample program that failed to compile on the host machine. Go to the CUDA-Sample GitHub repository and the Release page, choose the version corresponding to your CUDA version to download, such as CUDA Samples v12.9. Place the downloaded file in the ~/docker/cuda-dev/workspace directory of the host machine, or you can use the wget command to download it directly in the container:\n1 wget https://github.com/NVIDIA/cuda-samples/archive/refs/tags/v12.9.tar.gz Then decompress it in the container:\n1 2 tar -xzvf v12.9.tar.gz cd cuda-samples-12.9 Then it\u0026rsquo;s the normal compilation and running process:\n1 2 3 mkdir build \u0026amp;\u0026amp; cd build cmake .. make -j4 After the compilation is complete, you can run some sample programs to test whether the CUDA programming environment is working properly, for example:\n1 2 cd 1_Utilities/deviceQuery ./deviceQuery If the relevant information of the Nvidia GPU is output, it means that everything is done!\nTest if Python Can Call the GPU\nAfter entering the container, you can test whether Python can call the GPU, for example, run the following Python code:\n1 2 3 import torch print(torch.cuda.is_available()) print(torch.cuda.get_device_name(0)) If True and the name of your Nvidia GPU are output, it means that Python can call the GPU.\nUsing VS Code for Remote Development\nIf you want to use this container for remote development in VS Code, you can install the Remote - Containers extension.\nAfter opening VS Code on the host machine, click the remote connection icon in the lower left corner, select \u0026ldquo;Attach to Running Container\u0026rdquo;, and then select the cuda-dev container to connect to it.\nNext, you can directly enter the container command line through the terminal of VS Code, or you can directly open the working directory in the container with VS Code for development.\nIssues # When using YOLO to train models in the CUDA development container, I encountered an error:\n1 ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). The reason for this error is that the default shared memory size of the Docker container is only 64MB, which is too small to meet the requirements when training YOLO models. This issue can be resolved by increasing the shared memory size of the Docker container. You can do this by adding the shm_size parameter in the docker-compose.yml file, for example:\n1 2 3 services: cuda-dev: shm_size: \u0026#39;8gb\u0026#39; Here, the shm_size can be adjusted according to your needs, such as 8gb, 16gb, etc. After adding it, rebuild and run the container again.\n","date":"20 August 2025","externalUrl":null,"permalink":"/en/p/nvidia-gpu-4-containerized-configuration-of-nvidia-gpu-for-cuda-programming-and-deep-learning-environment/","section":"Posts","summary":" Motivation # This is the fourth article in the “Nvidia GPU” series, which mainly records how to use Nvidia GPU, especially on Linux systems. In previous articles, I recorded the settings for using Nvidia GPU on Ubuntu, including games, CUDA programming, deep learning, Docker containers, etc. Later, I switched to Fedora system, thinking that configuring Nvidia GPU for CUDA programming and deep learning environment on Fedora would be similar to that on Ubuntu, and with the support of RPM Fusion, installation and configuration would be more convenient. However, I actually encountered unexpected difficulties, so I had to write another article to record these problems and solutions.\n","title":"Nvidia GPU (4): Containerized Configuration of Nvidia GPU for CUDA Programming and Deep Learning Environment","type":"post"},{"content":"","date":"20 August 2025","externalUrl":null,"permalink":"/en/series/nvidia-gpu-series/","section":"Series","summary":"","title":"Nvidia GPU Series","type":"series"},{"content":"","date":"2025年8月19日","externalUrl":null,"permalink":"/tags/linux%E7%B3%BB%E7%BB%9F/","section":"Tags","summary":"","title":"Linux系统","type":"tags"},{"content":" 缘起 # Linux的桌面端近年来发展非常迅速，目前在易用性上Linux的桌面端已经做得非常不错了。然而由于其生态繁杂，各种工具和配置项五花八门，对于中文的支持一直都比较差。使用中文的新用户光折腾输入法可能都摸不着头脑，我曾经也在配置中文输入法、中文字体上花了不少精力、浪费了不少时间。而且就在最近，我又遇到了一些问题，决定把这些经验整理一下，帮助更多的用户顺利使用Linux系统中的中文环境。\n输入法配置 # Linux输入法基本概念 # 输入法框架：Linux下的输入法框架主要有IBus、fcitx和scim等，其中IBus是最常用的输入法框架。输入法框架负责管理输入法引擎和输入法前端，提供输入法的切换、配置等功能。 IBus：一个简单易用的输入法框架，支持多种输入法引擎，是很多Linux发行版的默认输入法框架。 fcitx：一个功能强大的输入法框架，支持多种输入法引擎，适合有一定经验的用户使用。fcitx目前更新到了第5版，也就是fcitx5。 scim：一个较老的输入法框架，功能较为简单，现已逐渐被IBus和fcitx所取代。 如果你没有中文输入的需求，那么系统默认的IBus一般就够用了，但是如果你需要中文输入法，那么我推荐卸载掉IBus，安装fcitx5并配置相应的输入法。\n输入法引擎：输入法引擎是实现具体输入法功能的组件，常见的中文输入法引擎有Pinyin、Wubi、Sogou等。不同的输入法引擎有不同的输入方式和词库，用户可以根据自己的习惯选择合适的输入法引擎。 这里我推荐中州韻（Rime）輸入法引擎。\noh-my-rime：Oh-My-Rime又叫“薄荷输入法”，是一款基于Rime输入法的输入法解决方案，在多个平台都适用。我个人非常推荐使用这个输入法解决方案，它提供了非常友好的配置界面和丰富的功能，能够满足大部分用户的需求。 Linux中文输入法安装 # 我一开始在Fedora系统上使用默认的IBus输入法框架，搭配Rime输入法引擎。这个方案在大部分情况下都能正常工作，但在VS Code里切换为中文时，一输入就会在跳出候选词窗口的瞬间卡住。在网上搜索一番之后发现是因为VS Code使用了Electron框架，而IBus在Wayland下与Electron的兼容性较差。解决方法是切换到fcitx5输入法框架。\n这里以Fedora 42系统为例，简单讲一下如何安装和配置IBus输入法和fcitx5输入法。\nIBus + Rime （不太推荐） # 虽然我个人不太推荐使用IBus，因为它和Wayland以及Electron的兼容性不好。但是如果你想坚持系统自带的输入法，也不怎么用基于Electron的软件，那么这套方案也未尝不可。\n安装Rime：因为系统已经自带了IBus框架，这里我们只需要安装Rime就行了：\n1 sudo dnf install ibus-rime 配置Rime：接下来就可以在设置中将Rime输入法添加进来。具体方法是通过设置-\u0026gt;Keyboard-\u0026gt;Virtual Keyboard，找到IBus，确认启用的是IBus框架。然后右击系统托盘里的输入法图标，选择Configure，就会看到下面的界面：\n点击Add添加输入法，选择Chinese里面的Rime，然后点击Add按钮。添加完成后就可以在输入法列表中看到Rime了。\n这时你就已经可以点击任务栏里的输入法图标或者使用快捷组合键，选择Rime输入法进行使用了。\n使用oh-my-rime：oh-my-rime的使用也非常简单，只要下载整个项目文件夹并放到输入法的配置里面就行了。在Fedora系统中，IBus的配置文件一般在~/.config/ibus/rime目录下。你可以通过以下命令下载oh-my-rime：\n1 git clone https://github.com/Mintimate/oh-my-rime.git 然后将整个oh-my-rime文件夹复制到~/.config/ibus/rime目录下。\n重启或者注销后重新登录，oh-my-rime就会生效。右击系统托盘里的输入法图标，就可以选择“薄荷拼音”作为输入方案了：\n额外配置 oh-my-rime还提供了很多可以配置的选项，你可以参照官方文档进行配置。我觉得默认配置就已经挺好用了，所以暂时没做额外的配置。\nfcitx5 + Rime (推荐) # 安装fcitx5：首先安装fcitx5及其相关组件：\n1 sudo dnf install fcitx5 fcitx5-chinese-addons 其中fcitx5是输入法框架，fcitx5-chinese-addons是中文输入法插件。\n安装Rime：可以通过以下命令安装Rime输入法：\n1 sudo dnf install fcitx5-rime librime-lua 其中fcitx5-rime就是Rime（中州韻）输入法的fcitx5版本；librime-lua是Rime的Lua脚本支持库，支持使用Lua进行输入法的定制。\n配置fcitx5：安装完成后将系统的输入法框架换为fcitx5，可以通过设置-\u0026gt;Keyboard-\u0026gt;Virtual Keyboard，可以看到下面的界面：\n点击fcitx5，再点击Apply。你就会发现任务栏里输入法的图标会从原来的IBus的变成一个小键盘，右击这个小键盘，选择Configure，会弹出下面的窗口：\n首先点击右上方位置的Update按钮来确保刚才安装的Rime输入法被正确识别。\n然后取消勾选右侧下方的Only Show Current Language选项，因为Rime是中文输入法，如果你的系统语言是英文，不取消勾选会找不到Rime输入法。取消勾选后你就可以找到Rime输入法了，选中后点击向左的箭头把Rime加入到输入法列表中。\n再等待一会儿等Rime自己配置完成就可以点击任务栏里的输入法图标或者使用快捷组合键，选择Rime输入法进行使用了：\noh-my-rime：oh-my-rime的使用和配置方法与IBus下的使用方法类似。不同的是fcitx的配置文件一般在~/.local/share/fcitx5目录下。你可以通过以下命令下载oh-my-rime：\n1 git clone https://github.com/Mintimate/oh-my-rime.git 然后把文件夹命名为rime放到~/.local/share/fcitx5目录下（如果里面已经有了rime文件夹，可以备份原来的再放过去）。\n重启或者注销后重新登录，oh-my-rime就会生效。右击系统托盘里的输入法图标，就可以选择“薄荷拼音”作为输入方案了：\nVS Code：完成上述配置后，大多数情况下都可以直接使用Rime输入法。但VS Code中还是无法输入中文，网上有建议将下述内容加入到环境变量中：\n1 2 3 GTK_IM_MODULE=fcitx QT_IM_MODULE=fcitx XMODIFIERS=@im=fcitx 但是这样偶尔在启动进入桌面时有警告信息。我觉得一个更好的方法是在VS Code的快捷方式中加入参数--enable-features=UseOzonePlatform,WaylandWindowDecorations '--ozone-platform=wayland %F'。具体方法是右击开始菜单的VS Code图标：\n点击Edit Application，在命令行参数里添加上面的参数：\n这样再打开VS Code就可以用Rime输入法在里面输入中文了！\n额外配置 同样地，oh-my-rime还提供了很多可以配置的选项，你可以参照官方文档进行配置。我觉得默认配置就已经挺好用了，所以暂时没做额外的配置。\n（可选）卸载IBus：首先需要卸载掉系统默认的IBus输入法框架，可以使用以下命令：\n1 sudo dnf remove ibus 通过以上步骤，就可以在Fedora 42系统中顺利安装和配置fcitx5中文输入法了。\n","date":"2025年8月19日","externalUrl":null,"permalink":"/p/linux%E7%B3%BB%E7%BB%9F%E4%B8%AD%E6%96%87%E8%BE%93%E5%85%A5%E6%B3%95%E4%B8%AD%E5%B7%9E%E9%9F%BBrime%E5%92%8C%E8%96%84%E8%8D%B7%E8%BE%93%E5%85%A5%E6%B3%95oh-my-rime/","section":"Posts","summary":" 缘起 # Linux的桌面端近年来发展非常迅速，目前在易用性上Linux的桌面端已经做得非常不错了。然而由于其生态繁杂，各种工具和配置项五花八门，对于中文的支持一直都比较差。使用中文的新用户光折腾输入法可能都摸不着头脑，我曾经也在配置中文输入法、中文字体上花了不少精力、浪费了不少时间。而且就在最近，我又遇到了一些问题，决定把这些经验整理一下，帮助更多的用户顺利使用Linux系统中的中文环境。\n","title":"Linux系统中文输入法：中州韻Rime和薄荷输入法（oh-my-rime）","type":"post"},{"content":"","date":"2025年8月19日","externalUrl":null,"permalink":"/categories/%E6%93%8D%E4%BD%9C%E7%B3%BB%E7%BB%9F/","section":"Categories","summary":"","title":"操作系统","type":"categories"},{"content":"","date":"2025年8月19日","externalUrl":null,"permalink":"/tags/%E8%BE%93%E5%85%A5%E6%B3%95/","section":"Tags","summary":"","title":"输入法","type":"tags"},{"content":"","date":"16 August 2025","externalUrl":null,"permalink":"/en/tags/boot/","section":"Tags","summary":"","title":"Boot","type":"tags"},{"content":" Introduction # This is the second article in the \u0026ldquo;Boot Configuration\u0026rdquo; series. I never expected such a simple topic could turn into a series, because I thought I had already explained the relevant concepts of boot configuration clearly in the first article, and I had successfully configured dual boot many times. However, recently I encountered a huge pitfall while setting up a dual boot with Windows and Linux on separate drives, so I felt it was necessary to write another article to document it.\nThe other article in this series goes into more detail about the basic concepts of boot configuration and how to configure multi-system boot entries, so I recommend reading it first:\nBoot Configuration (I): Multi-System Boot Entry Configuration Pitfalls of Dual Booting Windows/Linux on Separate Drives # I originally thought that after the above exploration, I would be very familiar with multi-system boot entries. So when installing a dual boot of Windows and Linux on a dual-drive computer, I was full of confidence, thinking that it would be easier to install the two systems separately since I had already managed to install multiple systems on the same drive. However, reality slapped me in the face, and in the end, I couldn\u0026rsquo;t even solve it gracefully.\nBackground # Let me explain my basic situation: my computer has two solid-state drives, A (1TB) and B (500GB), both installed in the M.2 slots on the motherboard. The M.2 slot for drive A is closer to the CPU, while drive B is slightly farther away. Does the distance from the CPU matter? I originally thought it didn\u0026rsquo;t, but I eventually found out that it seemed to be a huge influencing factor.\nI chose to use Fedora 42 as my main operating system, so I installed it on the 1TB solid-state drive A. Windows 11 is just an auxiliary system, installed on the 500GB drive B. According to convention, when installing a dual boot of Windows and Linux, it is recommended to install Windows first, because if Linux is installed afterward, it will overwrite the Linux bootloader, making Linux unbootable. However, when I powered on the machine, I had already installed Fedora 42 on drive A and had been using it for a few days, so I was too lazy to clean it up. Moreover, I thought I had a good grasp of boot configuration, and even if I installed Linux first and then Windows, it would just be a matter of adjusting the boot configuration. Thus, I had my first failed attempt.\nFirst Failed Attempt # The failure process can be summarized as follows:\nFirst, I installed Fedora 42 on drive A and successfully booted into the system.\nNext, I attempted to install Windows 11 on drive B. The safest approach would have been to remove drive A and leave only drive B for the Windows 11 installation. This way, Windows 11 would be completely installed on drive B without affecting Fedora 42 on drive A. However, I was too lazy to open the case, and the M.2 slot for drive A was blocked by the graphics card, making it even more troublesome to remove. So I thought, could I hide drive A during the Windows 11 installation?\nTo hide drive A, my first thought was to use the BIOS settings to disable drive A, but I found that my MSI motherboard didn\u0026rsquo;t have this feature. My Z890 is considered a mid-to-high-end motherboard, yet it lacks such a basic function. So I had to find other ways to hide it. After some exploration, I discovered that I could use the Diskpart tool during the Windows installation to hide drive A. The specific steps are as follows:\nStart the Windows installation program, and when you enter the first installation interface (the one where you select the language), press Shift + F10 to open the command prompt.\nIn the command prompt, type the following command to start the Diskpart tool:\n1 diskpart Then enter the following command to list all disks:\n1 list disk Find the disk number of drive A (for example, Disk 0), and then enter the following command to select drive A:\n1 select disk 0 Finally, enter the following command to take drive A offline:\n1 offline disk This way, during the Windows installation process, the installer will not be able to see drive A.\nExit the Diskpart tool:\n1 exit Close the command prompt and continue with the Windows 11 installation.\nAfter hiding drive A, I continued with the installation program. When the installation program reached the end, it needed to create the boot entry and then restart. At this point, a problem arose. Although the Windows 11 boot program could not see drive A, it seemed to realize that the drive B it was on was not the first boot drive because the M.2 interface for drive A was closer to the CPU. Therefore, it still tried to write boot information to the drive that was higher in the boot order. However, drive A had already been taken offline. As a result, the Windows installation program stubbornly refused to write the boot information to the lower-priority drive B and reported an error saying\n1 Windows could not prepare the computer to boot into the next phase of installation. To install Windows, restart the installation. Thus, this attempt ended in failure.\nSecond Failed Attempt # With the lessons learned from the first attempt, I didn\u0026rsquo;t bother with the Fedora system on drive A anymore. After all, I had only used it for a few days and hadn\u0026rsquo;t made many configurations. So I followed the traditional approach: first install Windows, and it didn\u0026rsquo;t matter if the Fedora boot entry was damaged; I could just reinstall Fedora afterward. This led to the following operations:\nInstall Windows 11 on drive B without taking drive A offline. This went smoothly, and I was able to install and boot into the Windows system normally.\nThe Windows 11 installation damaged the Fedora boot entry, and I was too lazy to fix it, so I just reinstalled Fedora. After the reinstallation, Fedora could boot into the system normally.\nHowever, when I tried to regenerate the GRUB configuration in Fedora using the grub2-mkconfig command, I found that it did not automatically detect the Windows 11 boot entry. It was at this point that I realized the Windows 11 boot program had not written the boot information to drive B, but was still trying to write it to drive A. And Fedora had overwritten it during installation!\nFine, I just reinstalled Fedora because I was too lazy to fix its boot entry, but now I had to fix the Windows boot entry. But since I was familiar with this task, I began the repair:\nAdjust the partition on drive B in the Fedora system, create a new 300M FAT32 partition in it, and name it EFI.\nRestart the Windows 11 installation program from the USB drive, and press Shift + F10 to open the command prompt when entering the first installation interface.\nIn the command prompt, enter the following command:\n1 2 diskpart list disk Find the disk number of drive B (for example, Disk 1), and then enter the following command to select drive B:\n1 2 select disk 1 list partition Find the partition number of the newly created EFI partition, as well as the partition number and drive letter of the partition where Windows 11 is installed on drive B. You can determine this by the size of the partitions: the one that is 300M is the EFI partition, and the one that is over 400G is the Windows system partition. It turns out that the EFI partition number is 4, the Windows system partition number is 3, and no drive letters have been assigned. So we assign the drive letter S to the EFI partition and the drive letter C to the Windows system partition.\n1 2 3 4 5 select partition 4 assign letter=S select partition 3 assign letter=C exit Then, use the bcdboot command to copy the boot files to the EFI partition and create a new boot entry:\n1 bcdboot C:\\Windows /s S: /f UEFI This completes the repair of the Windows bootloader.\nAfter the repair is complete, restart the computer and enter the BIOS settings to set drive B as the first boot option. This allows Windows 11 to boot normally. However, I primarily use Fedora, so I wanted to add the Windows boot entry to Fedora\u0026rsquo;s GRUB. I then set drive A as the first boot option, so it would default to Fedora, and when I wanted to boot into Windows, I could just select it from the GRUB menu.\nSo I entered the Fedora system and ran sudo grub2-mkconfig -o /boot/grub2/grub.cfg. However, Fedora did not detect the Windows boot entry we had just repaired. Running sudo os-prober also produced no output, indicating that Fedora could not see the Windows boot entry. I was puzzled; this EFI partition was clearly on drive B, and I had mounted it, so why couldn\u0026rsquo;t it be seen?\nI found that this EFI partition was somewhat different from the EFI partition created by the installer during a normal system installation. Specifically, the EFI partition created by the installer and the system partition are displayed as a single disk in the file manager of other systems, but now this EFI partition is separate from the system partition on the same disk. I don\u0026rsquo;t know why the partition I manually created is so out of place. Of course, there is a workaround: manually modify the GRUB configuration to forcefully add the Windows boot entry. But that would not be elegant.\nSo, this attempt also ended in failure.\nFinal Solution # I realized that Windows is so rogue that it insists on creating the boot partition on the first bootable disk it detects. So, to make it obediently place both the system and boot entries on drive B, I could only move drive B to the first M.2 slot or remove the drive in the first M.2 slot during the Windows installation.\nIn short, I still had to open the case.\nSince the Fedora system itself was fine, I only needed to remove drive A. Then I could install Windows 11 on drive B. After the installation was complete, I could put drive A back into the motherboard.\nAfter booting into the Fedora system, I ran os-prober and grub2-mkconfig again, and this time it could detect the Windows 11 boot entry.\nSummary # I really didn\u0026rsquo;t expect there to be so many pitfalls when installing a dual-boot system with Windows and Linux on two hard drives. The lessons learned here are:\nIf you want to install a dual-boot system on two hard drives, it\u0026rsquo;s best to insert the hard drive you want to install Windows on into the M.2 slot with a higher boot priority. If you must insert the hard drive you want to install Linux on into the M.2 slot with a higher boot priority, then either remove the first hard drive during the Windows installation or use another computer to install Windows on drive B before inserting it into the motherboard. ","date":"16 August 2025","externalUrl":null,"permalink":"/en/p/boot-configuration-ii-dual-boot-setup-for-windows/linux-on-dual-drives/","section":"Posts","summary":" Introduction # This is the second article in the “Boot Configuration” series. I never expected such a simple topic could turn into a series, because I thought I had already explained the relevant concepts of boot configuration clearly in the first article, and I had successfully configured dual boot many times. However, recently I encountered a huge pitfall while setting up a dual boot with Windows and Linux on separate drives, so I felt it was necessary to write another article to document it.\n","title":"Boot Configuration (II): Dual Boot Setup for Windows/Linux on Dual Drives","type":"post"},{"content":"","date":"16 August 2025","externalUrl":null,"permalink":"/en/series/boot-loader-series/","section":"Series","summary":"","title":"Boot Loader Series","type":"series"},{"content":"","date":"16 August 2025","externalUrl":null,"permalink":"/en/tags/bootloader/","section":"Tags","summary":"","title":"Bootloader","type":"tags"},{"content":"","date":"16 August 2025","externalUrl":null,"permalink":"/en/tags/grub/","section":"Tags","summary":"","title":"Grub","type":"tags"},{"content":"","date":"16 August 2025","externalUrl":null,"permalink":"/en/tags/windows/","section":"Tags","summary":"","title":"Windows","type":"tags"},{"content":"","date":"16 August 2025","externalUrl":null,"permalink":"/en/tags/windows/linux-dual-boot/","section":"Tags","summary":"","title":"Windows/Linux Dual Boot","type":"tags"},{"content":"","date":"2025年8月16日","externalUrl":null,"permalink":"/tags/windows/linux%E5%8F%8C%E7%B3%BB%E7%BB%9F/","section":"Tags","summary":"","title":"Windows/Linux双系统","type":"tags"},{"content":"","date":"2025年8月16日","externalUrl":null,"permalink":"/series/%E5%90%AF%E5%8A%A8%E5%BC%95%E5%AF%BC%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"启动引导系列","type":"series"},{"content":"","date":"15 August 2025","externalUrl":null,"permalink":"/en/tags/fedora/","section":"Tags","summary":"","title":"Fedora","type":"tags"},{"content":" Motivation # This is the third article in the \u0026ldquo;Nvidia GPU\u0026rdquo; series, which mainly documents how to use Nvidia GPUs, especially on Linux systems. In the previous articles, I documented the relevant settings for using Nvidia GPUs on Ubuntu, including gaming, CUDA programming, deep learning, Docker containers, etc. Now I have migrated from Ubuntu to Fedora, and the settings for using Nvidia GPUs on Fedora are different from those on Ubuntu, so I am documenting them here.\nOther articles in this series can be found at:\nNvidia GPU (1): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Ubuntu Nvidia GPU (2): Accelerate Video Editing and Transcoding for FFmpeg Using GPU Nvidia GPU (4): Containerized Configuration of Nvidia GPU for CUDA Programming and Deep Learning Environment Fedora Installation of Nvidia GPU Drivers # Check GPU Information # First, we need to check our GPU information. Open the terminal and enter the following command:\n1 lspci | grep VGA If you have an Nvidia GPU installed on your computer, you will see output similar to the following:\n1 01:00.0 VGA compatible controller: NVIDIA Corporation Device 2803 (rev a1) For some reason, my computer shows NVIDIA Corporation Device 2803 instead of RTX 4060 Ti, but it doesn\u0026rsquo;t matter. We just need to know that it\u0026rsquo;s an Nvidia GPU.\nInstall Nvidia GPU Drivers # In Ubuntu, we used the graphics-drivers package from the apt repository to install Nvidia GPU drivers. In Fedora, we need to use the rpmfusion repository to install Nvidia GPU drivers.\nCompared to downloading and installing the driver directly from the Nvidia official website, using the rpmfusion repository to install Nvidia GPU drivers is simpler and more convenient. Additionally, since the drivers in rpmfusion have been tested on Fedora, they are more stable. The main steps are as follows:\nAdd the rpmfusion repository:\n1 2 sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm sudo dnf install https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm You can also download the rpmfusion repository installation package corresponding to your Fedora version from the RPM Fusion official website. In the Fedora system, simply double-click the downloaded installation package, and Fedora will automatically open the Discover software center. Click the \u0026ldquo;Install\u0026rdquo; button in the title bar to proceed. Generally, we need to install two repositories: rpmfusion-free and rpmfusion-nonfree. The former contains open-source software, while the latter contains non-open-source software.\nAfter installation, you can update the software repository:\n1 sudo dnf update Install Nvidia GPU drivers:\nYou can refer to the instructions on the RPM Fusion official website and use the following commands to install Nvidia GPU drivers:\n1 2 sudo dnf install akmod-nvidia # rhel/centos users can use kmod-nvidia instead sudo dnf install xorg-x11-drv-nvidia-cuda #optional for cuda/nvdec/nvenc support Here, akmod-nvidia is the package name for Nvidia GPU drivers in Fedora. After installation, it will automatically compile the kernel module.\n(Optional) Secure Boot:\nWhat is Secure Boot # Secure Boot is a security standard designed to ensure that a computer only loads verified operating systems and software during startup. It can prevent malicious software from loading during system startup, thereby enhancing system security. In simple terms, Secure Boot ensures that only trusted software can run when the system starts.\nHow Secure Boot Works # The UEFI firmware contains a Secure Boot database that stores all trusted public keys and certificates. Generally, trusted public keys include those from operating system vendors, hardware vendors, and user-generated keys. When the computer starts, the UEFI firmware verifies whether each loaded component (such as the operating system kernel, drivers, etc.) is trusted. Only components registered in the Secure Boot database can be loaded; unauthorized components will be blocked. Linux kernel modules (such as Nvidia GPU drivers) need to be signed to be loaded on systems with Secure Boot enabled. If the kernel module is not signed or the signature is not trusted, these modules will not load on systems with Secure Boot enabled, causing the GPU driver to fail. This is why, after installing Nvidia GPU drivers, you need to generate a signature for the Linux kernel module and register it in the Secure Boot database.\nDo You Need Secure Boot # For ordinary users (i.e., users with limited knowledge of computers and who do not understand the computer startup process), my recommendation is to enable Secure Boot; for advanced users (who know what they are doing, especially those who understand what their system does during startup or know what they are making their system do during startup), my recommendation is that they can disable Secure Boot.\nIf Your Computer Has Secure Boot Enabled # If your computer has Secure Boot enabled, it may cause Nvidia GPU drivers to fail to load. You can disable Secure Boot in the BIOS.\nIf you do not want to disable Secure Boot, you can refer to the instructions on the RPM Fusion official website to register a key and add it to your system to enable Secure Boot. The specific steps are as follows:\nInstall Related Tools\n1 sudo dnf install kmodtool akmods mokutil openssl Generate Key\n1 sudo kmodgenca -a Register Key\n1 sudo mokutil --import /etc/pki/akmods/certs/public_key.der You will be prompted to set a password, which you will need to enter after rebooting.\nNote: Since the key is generated for the current kernel version, you may need to regenerate and register the key again after updating the kernel version (i.e., repeat the Generate Key and Register Key steps).\nReboot the computer\nIf you set up Secure Boot in the previous step, after rebooting, enter the MOK management interface, select \u0026ldquo;Enroll MOK\u0026rdquo;, enter the password you set, and continue to start the system. If you disabled Secure Boot in the previous step, simply reboot the computer. Confirm whether the GPU driver is installed successfully:\n1 nvidia-smi If you see output similar to the following, congratulations, your Nvidia GPU driver has been successfully installed:\n1 2 3 4 5 6 7 8 9 10 11 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 12.5 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA RTX 4060 Ti Off | 00000000:01:00.0 Off | N/A | | N/A 41C P8 10W / N/A | 0MiB / 7611MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ Test the GPU # We can use glmark2 to test the GPU performance.\nInstall glmark2: 1 sudo dnf install glmark2 Run glmark2: 1 glmark2 If you see output similar to the following in the terminal, it means the GPU performance test was successful:\n1 2 3 4 5 6 7 8 ======================================================= glmark2 2021.02 ======================================================= OpenGL Information GL_VENDOR: NVIDIA Corporation GL_RENDERER: NVIDIA GeForce RTX 4060 Ti/PCIe/SSE2 GL_VERSION: 4.6.0 NVIDIA 555.58.02 ======================================================= And there will be a window popping up, showing the content being tested. After the test is completed, the terminal will display the test score.\nGaming # Linux systems are not originally very suitable for gaming, but with the promotion of Steam, more and more games can run on Linux through Proton. Proton is a tool developed by Valve based on Wine that allows Windows games to run on Linux.\nIn Ubuntu, installing Steam requires installing some additional dependencies and manually installing Proton. In Fedora, installing Steam is very simple, and it includes Proton by default.\nInstall Steam # Install Steam: 1 sudo dnf install steam Log in to your Steam account In the start menu, find Steam and click to open it, then log in to your Steam account.\nInstall Games # If this is your first time using Steam and you haven\u0026rsquo;t purchased any games yet, you can choose some free games to test, such as \u0026ldquo;Dota 2\u0026rdquo;, \u0026ldquo;Counter-Strike: Global Offensive\u0026rdquo;, etc.\nLaunch Games # After installation, you can click \u0026ldquo;Launch\u0026rdquo; to start the game, but it often won\u0026rsquo;t succeed the first time and may require three attempts.\nThe first launch may get stuck downloading a component. Opening the download manager shows it\u0026rsquo;s trying to download Steam Linux Runtime 2.0. However, for some reason, it cannot start the download, even though Steam Linux Runtime 3.0 is already available. I had to cancel the launch. After canceling the first launch, trying to launch the game again allows Steam Linux Runtime 2.0 to be successfully downloaded, and the game enters the loading screen. However, it gets stuck on a full-screen black screen. I could only click to stop the game. On the third attempt to launch the game, it successfully loads and enters the shader compilation phase. After waiting for the shaders to compile, the game starts normally. Game Controllers # If you are using a game controller, some additional configuration is needed. If you connect it using a USB cable, it should work directly.\nIf you are using an Xbox controller, you can connect it via Bluetooth. However, even if it connects, you may find that the controller still cannot control the game. This is because the relevant drivers are missing. We can install the xpadneo package to solve this problem:\n1 sudo dnf install xpadneo Then reconnect via Bluetooth. When connected, the controller will vibrate slightly, whereas previously it did not vibrate when connected. This indicates that the controller has been successfully recognized and can be used to play games.\nCUDA Programming # CUDA is a parallel computing platform and programming model developed by Nvidia that leverages the parallel computing power of GPUs to accelerate compute-intensive applications. CUDA programming requires installing Nvidia GPU drivers and the CUDA toolkit. Additionally, there is a certain compatibility relationship between CUDA versions and Nvidia GPU driver versions, so you need to choose the appropriate CUDA version based on your GPU driver version.\nUpdate in August 2025: After installation, I found that this method still doesn\u0026rsquo;t work, mainly due to compatibility issues between Fedora 42 and CUDA 12.9. Please refer to the \u0026ldquo;Issues\u0026rdquo; section at the end for details.\nInstall CUDA # Check the required CUDA version for your Nvidia GPU driver: 1 nvidia-smi In the CUDA Version line, you can see the required CUDA version for your Nvidia GPU driver, for example, CUDA Version: 12.9. This means we need to install CUDA 12.9.\nInstall CUDA: When installing CUDA in Ubuntu, we directly followed the installation guide on the Nvidia official website.\nIn Fedora, since we used the RPM Fusion repository to install the drivers earlier, directly following the Nvidia official website may lead to version mismatch issues. For example, in the previous step, we found that we needed to install CUDA 12.9, but the Nvidia official website may only provide installation packages for CUDA 13.0. If you follow the instructions on the Nvidia official website directly, it will cause version mismatch issues. This is because RPM Fusion tests the drivers after they are released by Nvidia and only pushes them to the RPM repository after they are stable. Therefore, we still need to follow the instructions on RPM Fusion to install CUDA.\nThe commands are as follows:\n1 2 3 4 5 sudo dnf config-manager addrepo --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora41/$(uname -m)/cuda-fedora41.repo sudo dnf clean all sudo dnf module disable nvidia-driver sudo dnf config-manager setopt cuda-fedora41-$(uname -m).exclude=nvidia-driver,nvidia-modprobe,nvidia-persistenced,nvidia-settings,nvidia-libXNVCtrl,nvidia-xconfig sudo dnf -y install cuda-toolkit Note that in the above commands, the installed CUDA version is for Fedora 41, while our Fedora version is 42. However, don\u0026rsquo;t worry. As mentioned in the RPM Fusion instructions, the above commands are applicable to versions after Fedora 41.\nConfigure environment variables: If the installation is successful, you will see the CUDA installation files in the /usr/local/cuda-12.9 directory. At the same time, you will see a symbolic link pointing to /usr/local/cuda-12.9 in the /usr/local/cuda directory. If you upgrade the CUDA version later, /usr/local/cuda will point to the new CUDA version directory.\nWe need to configure environment variables so that CUDA can be found. When adding the CUDA bin directory and lib64 directory to the PATH and LD_LIBRARY_PATH environment variables, we use the /usr/local/cuda directory instead of the /usr/local/cuda-12.9 directory, because this way, we don\u0026rsquo;t need to modify the environment variables when upgrading the CUDA version in the future.\nIf you are using bash, you can add the following content to the ~/.bashrc file:\n1 2 3 echo \u0026#39;export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}\u0026#39; \u0026gt;\u0026gt; ~/.bashrc echo \u0026#39;export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}\u0026#39; \u0026gt;\u0026gt; ~/.bashrc source ~/.bashrc If you are using another shell, you can add the above content to the corresponding configuration file of that shell.\nTest CUDA: 1 nvcc --version If you see output similar to the following, it means CUDA has been successfully installed:\n1 2 3 4 5 nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Jun__6_02:18:23_PDT_2024 Cuda compilation tools, release 12.9, V12.9.82 Build cuda_12.9.r12.9/compiler.34385749_0 Issues # After going through the above steps, it seems that we have successfully installed CUDA, but in fact, the compiler above cannot be used in Fedora 42. If you use it directly for compilation, it will report a GCC version mismatch error. This is because the installed CUDA version is 12.9, which only matches GCC 14 and below, while Fedora 42 has GCC 15 installed by default.\nBesides, this issue cannot be resolved by installing GCC 14, because even if GCC 14 is installed, the libraries, header files, and other components in Fedora 42 are still of GCC 15 version, so compilation will still report errors.\nThis issue is difficult to resolve. One solution is to stop using RPM Fusion to install Nvidia drivers and CUDA, and instead download and install CUDA 13.0 and the corresponding drivers directly from the Nvidia official website. CUDA 13.0 supports GCC 15, but there are still two problems:\nYou cannot use Fedora\u0026rsquo;s package management system to manage the installation and upgrade of Nvidia drivers and CUDA. If there are dependencies related to them managed by Fedora, then any update may fail. In severe cases, it may even affect system stability. As of August 2025, PyTorch, commonly used in deep learning, has not yet adapted to CUDA 13.0, so installing CUDA 13.0 still cannot support many deep learning tasks. If you insist on using the Fedora system, I currently (as of August 2025) have two solutions:\nOne is to give up Fedora 42 and switch to Fedora 41. But this probably cannot be achieved through a simple system version downgrade, as it involves too many components. A system reinstallation is likely required. The other is to use a virtualized container for configuration. Perhaps when RPM Fusion and PyTorch both support CUDA 13.0 in the future, the container can be discarded. I have thought it over and chosen the second option. I decided to keep the installed Nvidia drivers and CUDA here, and set up a separate Docker/Podman container for the CUDA programming and deep learning development environment. The main idea is to run Ubuntu 24.04 in the container and install compatible CUDA and Nvidia drivers. Since PyTorch does not yet support CUDA 13.0, I will use CUDA 12.9 in the container. The specific configuration process will be documented in a separate article, please refer to:\nNvidia GPU (4): Containerized Configuration of Nvidia GPU for CUDA Programming and Deep Learning Environment ","date":"15 August 2025","externalUrl":null,"permalink":"/en/p/nvidia-gpu-3-gaming-cuda-deep-learning-docker-etc.-on-fedora/","section":"Posts","summary":" Motivation # This is the third article in the “Nvidia GPU” series, which mainly documents how to use Nvidia GPUs, especially on Linux systems. In the previous articles, I documented the relevant settings for using Nvidia GPUs on Ubuntu, including gaming, CUDA programming, deep learning, Docker containers, etc. Now I have migrated from Ubuntu to Fedora, and the settings for using Nvidia GPUs on Fedora are different from those on Ubuntu, so I am documenting them here.\n","title":"Nvidia GPU (3): Gaming, CUDA, Deep Learning, Docker, etc. on Fedora","type":"post"},{"content":"","date":"11 August 2025","externalUrl":null,"permalink":"/en/tags/remote-development/","section":"Tags","summary":"","title":"Remote Development","type":"tags"},{"content":"","date":"11 August 2025","externalUrl":null,"permalink":"/en/tags/ssh/","section":"Tags","summary":"","title":"Ssh","type":"tags"},{"content":" Motivation # SSH (Secure Shell) is a widely used remote login protocol. I have used it extensively, and setting it up is not difficult, with many configurations being one-time setups. However, most of these configurations are only done when changing computers or reinstalling the system, leading to a low usage frequency. As a result, I often have to look up related information each time.\nHere, I have compiled commonly used SSH-related content for easy reference in the future.\nSSH Installation # First, SSH consists of a client and a server. The client is used to connect to other computers, while the server allows other computers to connect to yours.\nAlmost all mainstream operating systems (Windows/macOS/Linux) come with an SSH client, so you can usually open a command-line tool on any computer and use the ssh command.\nHowever, the SSH server is different. Most operating systems do not install the SSH server by default, or even if it is installed, it is generally not enabled by default. Therefore, this section mainly discusses how to install and set up the SSH server to start automatically.\nWindows # Earlier versions of Windows (before Windows 10 1809) did not come with an SSH server, but starting from Windows 10 1809, Windows includes the OpenSSH server. However, it is not installed by default and needs to be enabled by the user.\nOpen the \u0026ldquo;Settings\u0026rdquo; app and click on \u0026ldquo;Apps\u0026rdquo;. On the \u0026ldquo;Apps \u0026amp; features\u0026rdquo; page, click on \u0026ldquo;Optional features\u0026rdquo; on the right side. On the \u0026ldquo;Optional features\u0026rdquo; page, click on \u0026ldquo;Add a feature\u0026rdquo;. On the \u0026ldquo;Add a feature\u0026rdquo; page, find \u0026ldquo;OpenSSH Server\u0026rdquo; and click to install it. After installation, open the \u0026ldquo;Services\u0026rdquo; app (you can search for \u0026ldquo;Services\u0026rdquo; in the Start menu), find \u0026ldquo;OpenSSH SSH Server\u0026rdquo;, right-click it, and select \u0026ldquo;Properties\u0026rdquo;. In the \u0026ldquo;Properties\u0026rdquo; window, set the \u0026ldquo;Startup type\u0026rdquo; to \u0026ldquo;Automatic\u0026rdquo;, then click the \u0026ldquo;Start\u0026rdquo; button to start the service. Click \u0026ldquo;Apply\u0026rdquo; and \u0026ldquo;OK\u0026rdquo; to save the settings. Linux # Most Linux distributions come with an SSH server, but if not, you can install it using the package manager.\nUbuntu/Debian systems: 1 2 sudo apt update sudo apt install openssh-server Fedora/RHEL/CentOS systems: 1 sudo dnf install openssh-server Arch Linux: 1 sudo pacman -S openssh openSUSE: 1 sudo zypper install openssh After installation, you can start the SSH service and set it to start on boot with the following commands:\n1 2 sudo systemctl start sshd sudo systemctl enable sshd macOS # macOS comes with an SSH server, but it is disabled by default. You can enable it by following these steps:\nOpen \u0026ldquo;System Preferences\u0026rdquo; and click on \u0026ldquo;Sharing\u0026rdquo;. On the \u0026ldquo;Sharing\u0026rdquo; page, check the box for \u0026ldquo;Remote Login\u0026rdquo;. This will automatically enable the SSH server and display the SSH connection information. SSH Passwordless Login # SSH passwordless login is achieved through SSH key pairs, allowing you to connect to a remote server without entering a password.\nThe principle is to generate a key pair (public and private keys) on your local computer A, then copy the public key to the remote server B. After that, when connecting from computer A to server B, SSH will use the key pair for authentication without requiring a password.\nGenerate an SSH key pair on your local computer A:\n1 ssh-keygen -t ed25519 Here, -t ed25519 specifies using the ed25519 algorithm to generate the key pair. You can also use -t rsa to generate an RSA key pair.\nFollow the prompts and press Enter to accept the defaults. The private and public keys will be saved in ~/.ssh/id_ed25519 and ~/.ssh/id_ed25519.pub, respectively.\nCopy the public key to the remote server B:\nIf your local computer A is Linux or macOS, you can use the following command to copy the public key to the remote server B:\n1 ssh-copy-id user@remote_server Here, user is the username on the remote server B, and remote_server is the IP address or domain name of the remote server B. You will need to enter the password for the remote server B.\nIf your local computer A is Windows, you can manually copy the public key content to the ~/.ssh/authorized_keys file on the remote server B.\nSpecifically, first view the public key content on your local computer A:\n1 Get-Content $env:USERPROFILE\\.ssh\\id_ed25519.pub Or open the C:\\Users\\\u0026lt;YourUsername\u0026gt;\\.ssh\\id_ed25519.pub file directly with Notepad and copy its content.\nThen, on the remote server B, create or edit the ~/.ssh/authorized_keys file and paste the public key content into it.\nFinally, for security reasons, ensure that the permissions of the ~/.ssh/authorized_keys file are set correctly:\n1 chmod 600 ~/.ssh/authorized_keys Test passwordless login: Now you can try connecting from your local computer A to the remote server B by running:\n1 ssh user@remote_server If everything is set up correctly, you should be able to log in to the remote server B directly without entering a password.\nSSH Configuration Files # SSH configuration files can be used to simplify SSH connection commands. User-level configurations are generally located in ~/.ssh/ (Linux and macOS) or C:\\Users\\\u0026lt;YourUsername\u0026gt;\\.ssh\\ (Windows), while system-level configurations are generally located in /etc/ssh/ (Linux and macOS) or C:\\ProgramData\\ssh\\ (Windows). System-level configuration files affect all users and require administrator or sudo privileges to modify; user-level configuration files only affect the current user.\nSSH configuration files are divided into client configuration files and server configuration files.\nClient Configuration Files # System-level client configuration files are located at /etc/ssh/ssh_config (Linux and macOS) or C:\\ProgramData\\ssh\\ssh_config (Windows), while user-level client configuration files are located at ~/.ssh/config (Linux and macOS) or C:\\Users\\\u0026lt;YourUsername\u0026gt;\\.ssh\\config (Windows). If there are many configurations, you can also split the configuration files into multiple files. System-level configuration files can be placed in the /etc/ssh/ssh_config.d/ directory, and user-level configuration files can be placed in the ~/.ssh/config.d/ directory.\nClient configuration files commonly include the following options:\nHost: Specifies the host alias, which can use the wildcard * to match all hosts. HostName: Specifies the hostname or IP address. User: Specifies the login username. Port: Specifies the SSH connection port, default is 22. IdentityFile: Specifies the private key file path. ForwardAgent: Whether to enable SSH agent forwarding, default is no. ServerAliveInterval: Sets the interval for server alive checks in seconds, default is 0 (no checks). ServerAliveCountMax: Sets the maximum number of server alive checks, default is 3. ControlMaster: Whether to enable SSH connection multiplexing, default is no. ControlPath: Specifies the socket file path for SSH connection multiplexing, default is ~/.ssh/sockets/%r@%h:%p. ControlPersist: Whether to enable persistent SSH connection multiplexing, default is no. Below is an example of a user-level SSH client configuration file ~/.ssh/config:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Default configuration Host * User your_username Port 22 IdentityFile ~/.ssh/id_ed25519 ForwardAgent no ServerAliveInterval 60 ServerAliveCountMax 3 # Specific host configuration Host example HostName example.com User example_user Port 2222 IdentityFile ~/.ssh/id_ed25519_example ForwardAgent yes ControlMaster auto ControlPath ~/.ssh/sockets/%r@%h:%p ControlPersist yes # Use wildcard to match multiple hosts Host *.example.com User wildcard_user IdentityFile ~/.ssh/id_ed25519_wildcard Server Configuration Files # Server configuration files are located at /etc/ssh/sshd_config (Linux and macOS) or C:\\ProgramData\\ssh\\sshd_config (Windows). Common options in server configuration files include:\nPort: Specifies the SSH server port, default is 22. ListenAddress: Specifies the IP address the SSH server listens on, default is all addresses. PermitRootLogin: Whether to allow root user login, default is prohibit-password (disallow password login but allow key-based login). PasswordAuthentication: Whether to allow password login, default is yes. PubkeyAuthentication: Whether to allow public key authentication, default is yes. ChallengeResponseAuthentication: Whether to enable challenge-response authentication, default is no. UsePAM: Whether to enable PAM authentication, default is yes. AllowUsers: Specifies a list of users allowed to log in, can use the wildcard *, default is all users. DenyUsers: Specifies a list of users denied login, can use the wildcard *, default is no denied users. AllowGroups: Specifies a list of groups allowed to log in, can use the wildcard *, default is all groups. DenyGroups: Specifies a list of groups denied login, can use the wildcard *, default is no denied groups. MaxAuthTries: Sets the maximum number of authentication attempts, default is 6. MaxSessions: Sets the maximum number of sessions, default is 10. ClientAliveInterval: Sets the interval for client alive checks in seconds, default is 0 (no checks). ClientAliveCountMax: Sets the maximum number of client alive checks, default is 3. PermitTunnel: Whether to allow SSH tunneling, default is no. X11Forwarding: Whether to allow X11 forwarding, default is no. Subsystem: Specifies the subsystem configuration, default is subsystem sftp /usr/lib/openssh/sftp-server. Below is an example of an SSH server configuration file /etc/ssh/sshd_config: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # SSH server configuration Port 22 ListenAddress * PermitRootLogin prohibit-password PasswordAuthentication yes PubkeyAuthentication yes ChallengeResponseAuthentication no UsePAM yes AllowUsers user1 user2 DenyUsers user3 AllowGroups group1 group2 DenyGroups group3 MaxAuthTries 3 MaxSessions 10 ClientAliveInterval 60 ClientAliveCountMax 3 PermitTunnel no X11Forwarding yes Subsystem sftp /usr/lib/openssh/sftp-server Some Common SSH Features # Usually, we use SSH to remotely connect to a computer (server) and then use the server\u0026rsquo;s command-line tools to execute commands. However, SSH has many other features, such as port forwarding, X11 forwarding, etc.\nPort Forwarding # SSH can forward local ports to remote server ports or forward remote server ports to local ports. This is very useful for accessing services behind a firewall.\nLocal Port Forwarding:\n1 ssh -L local_port:remote_host:remote_port user@remote_server With local port forwarding, you can access services on the remote server from your local machine. For example, if there is a web service running on port 8080 on the remote server, you can forward it to your local port 8080 with the following command:\n```bash ssh -L 8080:localhost:8080 user@remote_server ``` Remote Port Forwarding:\n1 ssh -R remote_port:local_host:local_port user@remote_server With remote port forwarding, you can allow the remote server to access services on your local machine. For example, if you are running a web service locally, you can forward it to port 8080 on the remote server with the following command:\n```bash ssh -R 8080:localhost:8080 user@remote_server ``` Dynamic Port Forwarding (similar to a SOCKS proxy):\n1 ssh -D local_port user@remote_server With dynamic port forwarding, you can create a SOCKS proxy server on your local machine, allowing other applications to connect to the remote server via SSH. For example, you can create a SOCKS proxy server with the following command:\n```bash ssh -D 1080 user@remote_server ``` Then, configure your browser or other applications to use the SOCKS proxy with localhost:1080 as the proxy server address. This way, the traffic from your browser or other applications configured with the SOCKS proxy will go through the remote server\u0026rsquo;s network. A useful scenario is when your local computer is in country A and the remote server is in country B; you can use SSH\u0026rsquo;s dynamic port forwarding feature to route your local traffic through the remote server to access resources in country B.\nX11 Forwarding # SSH can forward graphical applications from the remote server to your local computer through an SSH tunnel, allowing you to run graphical applications from the remote server on your local machine.\nNote: There is a pair of easily confused concepts here: X server and X client. In the X11 protocol, the X server refers to the computer with a graphical interface that can serve as a display, while the X client refers to the computer that wants to run a graphical program. Typically, we run the X server on our local computer (such as Xming or VcXsrv on Windows, and XQuartz on macOS), while running the X client (graphical applications) on the remote server. This allows us to display programs running on the remote server on our local machine. This is opposite to the SSH server and client concepts.\nTo enable X11 forwarding, you need to configure both the SSH client and server.\nOn the SSH server\nEnsure that the sshd_config file has the following configuration:\n1 2 X11Forwarding yes X11DisplayOffset 10 On the SSH client\nPrerequisites:\nEnsure that you have X11 server software installed on your local computer. Linux usually comes with an X11 server, but on Windows and macOS, you need to install additional software, such as Xming or VcXsrv on Windows, and XQuartz on macOS.\nEnsure that your SSH client supports X11 forwarding. Most SSH clients on Linux and macOS support X11 forwarding, but on Windows, OpenSSH needs to be version 8.0 or higher (see PowerShell Issue #1515 on GitHub). You can run the ssh -V command to check the current version.\nEnsure that the DISPLAY environment variable is set on your local computer.\nWindows:\nIf using PowerShell, you can run the following command:\n1 $env:DISPLAY = \u0026#34;localhost:0.0\u0026#34; If using cmd, you can run the following command:\n1 set DISPLAY=localhost:0.0 Linux/macOS:\nOn Linux and macOS, you usually do not need to manually set the DISPLAY environment variable, as the SSH client will automatically set it.\nHowever, if you need to set it manually, you can run the following command in the terminal:\n1 export DISPLAY=localhost:0.0 Connect to the remote server:\nWhen connecting to the remote server with the SSH client, use the -X or -Y option to enable X11 forwarding:\n1 ssh -X user@remote_server or\n1 ssh -Y user@remote_server The -X option enables secure X11 forwarding, while the -Y option enables insecure X11 forwarding (allowing higher privilege operations).\nTest X11 forwarding:\nAfter enabling X11 forwarding, you can run graphical applications on the remote server, and their windows will be displayed on your local computer. For example, you can run the xclock command on the remote server to test X11 forwarding:\n1 xclock If everything is set up correctly, you should see a clock window on your local computer.\nRemote Development with VSCode # When VS Code was first released, it was just one of many editors. Compared to text editors like Notepad++ and Sublime Text, it wasn\u0026rsquo;t significantly better; compared to IDEs like Visual Studio and IntelliJ IDEA, it wasn\u0026rsquo;t as powerful. Its popularity mainly stemmed from two killer features: extensions and remote development.\nWhen VS Code was first released, I was also using Sublime Text, Visual Studio Community Edition, PyCharm, and other editors and IDEs. However, after VS Code supported remote development, I completely abandoned other editors and IDEs and fully switched to VS Code. Although I used IntelliJ IDEA for a while when learning Java Web development, in most cases, VS Code has been sufficient.\nI digress, but in my opinion, the most revolutionary and attractive feature of VS Code is its native support for remote development, which is also very convenient and easy to use.\nVS Code comes with the Remote Development Extension Pack pre-installed. Using it is also very simple:\nOpen VS Code on your local computer and click the \u0026ldquo;Remote Explorer\u0026rdquo; icon (a computer with an arrow) in the left activity bar. Click the \u0026ldquo;Connect to Host\u0026hellip;\u0026rdquo; button in the upper left corner, enter the SSH connection information for the remote server (in the format user@remote_server), and press Enter. If it\u0026rsquo;s your first time connecting, you will be prompted to choose an SSH configuration file. Select the default user-level configuration file (either ~/.ssh/config or C:\\Users\\\u0026lt;YourUsername\u0026gt;\\.ssh\\config). This configuration file is the same as the SSH client configuration file mentioned earlier, and the previous configurations will also apply here. After a successful connection, VS Code will install a VS Code server component on the remote server, and then you can use VS Code to edit files on the remote server just like you would locally. Acknowledgments # This article\u0026rsquo;s cover image is from IPXO.\n","date":"11 August 2025","externalUrl":null,"permalink":"/en/p/ssh-ssh-to-windows-ssh-configuration-passwordless-login-vscode-remote-development-etc./","section":"Posts","summary":" Motivation # SSH (Secure Shell) is a widely used remote login protocol. I have used it extensively, and setting it up is not difficult, with many configurations being one-time setups. However, most of these configurations are only done when changing computers or reinstalling the system, leading to a low usage frequency. As a result, I often have to look up related information each time.\n","title":"SSH: SSH to Windows, SSH Configuration, Passwordless Login, VSCode Remote Development, etc.","type":"post"},{"content":"","date":"11 August 2025","externalUrl":null,"permalink":"/en/tags/vscode/","section":"Tags","summary":"","title":"VSCode","type":"tags"},{"content":"","date":"2025年8月11日","externalUrl":null,"permalink":"/tags/%E8%BF%9C%E7%A8%8B%E5%BC%80%E5%8F%91/","section":"Tags","summary":"","title":"远程开发","type":"tags"},{"content":"","date":"7 August 2025","externalUrl":null,"permalink":"/en/tags/headscale/","section":"Tags","summary":"","title":"Headscale","type":"tags"},{"content":"","date":"7 August 2025","externalUrl":null,"permalink":"/en/tags/network/","section":"Tags","summary":"","title":"Network","type":"tags"},{"content":"","date":"7 August 2025","externalUrl":null,"permalink":"/en/series/private-virtual-network-vpn-series/","section":"Series","summary":"","title":"Private Virtual Network (VPN) Series","type":"series"},{"content":"","date":"7 August 2025","externalUrl":null,"permalink":"/en/tags/tailscale/","section":"Tags","summary":"","title":"Tailscale","type":"tags"},{"content":" Motivation # This is the third article in the \u0026ldquo;Virtual Private Network VPN\u0026rdquo; series. In the first two articles, I introduced the basic concepts and use cases of VPNs, as well as how to set up a private virtual network using Tailscale. Other articles in this series can be found at:\nPrivate Virtual Network VPN (1): Setting up a VPN with Shadowsocks on a Cloud Server Private Virtual Network VPN (2): Basic Concepts and Guide to Setting up with Tailscale In the previous article, I explained how to set up a private virtual network using Tailscale. Tailscale is a VPN solution based on WireGuard that allows us to quickly create a private virtual network. However, the official version of Tailscale uses Tailscale\u0026rsquo;s control servers, meaning our data and network traffic pass through Tailscale\u0026rsquo;s servers. Although this data and traffic are encrypted, if we want complete control over our network environment or have privacy and security concerns, we can set up our own Tailscale control server. The tool to achieve this is Headscale.\nPrerequisites # Basic understanding of containers and Docker Basic understanding of reverse proxy concepts A static IP address (this is very important; as mentioned in previous articles, if you don\u0026rsquo;t have a static IP, you can use Cloudflare Tunnel for reverse proxy, but Cloudflare Tunnel cannot proxy Headscale services. This issue will be discussed later, but I don\u0026rsquo;t have a solution for it.) Headscale # Introduction to Headscale # Headscale is an open-source implementation of a Tailscale control server that allows users to host the Tailscale control plane on their own infrastructure. By using Headscale, users can have complete control over their VPN environment without relying on Tailscale\u0026rsquo;s official servers. Its GitHub repository can be found at: headscale。\nAdditionally, to facilitate user management of Headscale, developers have created several web interface tools that integrate with Headscale. More details can be found at: headscale-web. I have only tried headplane and headscale-ui。headscale-ui did not run successfully, while headplane deployed successfully. Their functionalities should be similar, and I am too lazy to explore why headscale-ui didn\u0026rsquo;t work, so I just used headplane.\nHow Headscale Works # Headscale acts as an intermediary between Tailscale clients and servers by implementing the Tailscale control protocol. Users can configure Tailscale clients to use Headscale as the control server, allowing for complete control over VPN connections. Headscale supports all major features of Tailscale, including authentication, device registration, and network configuration.\nDeploying Headscale and Headplane with Docker # Here, we will deploy both Headscale and Headplane services in the same Docker container. Both Headscale and Headplane require configuration files, so we will create directories to store these configuration files. The directory structure for the entire project is as follows:\n1 2 3 4 5 6 7 headscale/ ├── headplane/ │ └── config.yaml ├── headscale/ │ └── config.yaml └── docker-compose.yaml └── .env Next, we will create these files.\n1. Docker Compose File # Create the docker-compose.yaml file with the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 version: \u0026#39;3.5\u0026#39; services: headscale: image: headscale/headscale:v0.26 container_name: headscale volumes: - ./headscale/config:/etc/headscale - ${DATA_DIR}/headscale/data:/var/lib/headscale - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro command: serve restart: unless-stopped headplane: container_name: headplane image: ghcr.io/tale/headplane:0.6.0 restart: unless-stopped volumes: - \u0026#39;./headplane/config/config.yaml:/etc/headplane/config.yaml\u0026#39; - \u0026#39;./headscale/config/config.yaml:/etc/headscale/config.yaml\u0026#39; - \u0026#39;${DATA_DIR}/headplane/data:/var/lib/headplane\u0026#39; - \u0026#39;/var/run/docker.sock:/var/run/docker.sock:ro\u0026#39; Note:\nThe ${DATA_DIR} is an environment variable that you can define in the .env file. This variable is used to specify the directory for data storage. Here is an example of the content for the .env file:\n1 DATA_DIR=/path/to/your/data The headscale service only needs to read the headscale configuration file, while the headplane service needs to read both the headscale and headplane configuration files.\n2. Headscale Configuration File # The configuration file for Headscale is located at headscale/config/config.yaml. Following is an example configuration and you can modify it according to your needs:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 --- # headscale will look for a configuration file named `config.yaml` (or `config.json`) in the following order: # # - `/etc/headscale` # - `~/.headscale` # - current working directory # The url clients will connect to. # Typically this will be a domain like: # # https://myheadscale.example.com:443 # server_url: https://\u0026lt;headscale.example.com\u0026gt; # Address to listen to / bind to on the server # # For production: listen_addr: 0.0.0.0:8080 #listen_addr: 127.0.0.1:8080 # Address to listen to /metrics, you may want # to keep this endpoint private to your internal # network # metrics_listen_addr: 127.0.0.1:9090 # Address to listen for gRPC. # gRPC is used for controlling a headscale server # remotely with the CLI # Note: Remote access _only_ works if you have # valid certificates. # # For production: grpc_listen_addr: 0.0.0.0:50443 #grpc_listen_addr: 127.0.0.1:50443 # Allow the gRPC admin interface to run in INSECURE # mode. This is not recommended as the traffic will # be unencrypted. Only enable if you know what you # are doing. grpc_allow_insecure: false # The Noise section includes specific configuration for the # TS2021 Noise protocol noise: # The Noise private key is used to encrypt the # traffic between headscale and Tailscale clients when # using the new Noise-based protocol. private_key_path: /var/lib/headscale/noise_private.key # List of IP prefixes to allocate tailaddresses from. # Each prefix consists of either an IPv4 or IPv6 address, # and the associated prefix length, delimited by a slash. # It must be within IP ranges supported by the Tailscale # client - i.e., subnets of 100.64.0.0/10 and fd7a:115c:a1e0::/48. # See below: # IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71 # IPv4: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#L33 # Any other range is NOT supported, and it will cause unexpected issues. prefixes: v4: 100.64.0.0/10 v6: fd7a:115c:a1e0::/48 # Strategy used for allocation of IPs to nodes, available options: # - sequential (default): assigns the next free IP from the previous given IP. # - random: assigns the next free IP from a pseudo-random IP generator (crypto/rand). allocation: sequential # DERP is a relay system that Tailscale uses when a direct # connection cannot be established. # https://tailscale.com/blog/how-tailscale-works/#encrypted-tcp-relays-derp # # headscale needs a list of DERP servers that can be presented # to the clients. derp: server: # If enabled, runs the embedded DERP server and merges it into the rest of the DERP config # The Headscale server_url defined above MUST be using https, DERP requires TLS to be in place enabled: false # Region ID to use for the embedded DERP server. # The local DERP prevails if the region ID collides with other region ID coming from # the regular DERP config. region_id: 999 # Region code and name are displayed in the Tailscale UI to identify a DERP region region_code: \u0026#34;headscale\u0026#34; region_name: \u0026#34;Headscale Embedded DERP\u0026#34; # Listens over UDP at the configured address for STUN connections - to help with NAT traversal. # When the embedded DERP server is enabled stun_listen_addr MUST be defined. # # For more details on how this works, check this great article: https://tailscale.com/blog/how-tailscale-works/ stun_listen_addr: \u0026#34;0.0.0.0:3478\u0026#34; # Private key used to encrypt the traffic between headscale DERP # and Tailscale clients. # The private key file will be autogenerated if it\u0026#39;s missing. # private_key_path: /var/lib/headscale/derp_server_private.key # This flag can be used, so the DERP map entry for the embedded DERP server is not written automatically, # it enables the creation of your very own DERP map entry using a locally available file with the parameter DERP.paths # If you enable the DERP server and set this to false, it is required to add the DERP server to the DERP map using DERP.paths automatically_add_embedded_derp_region: true # For better connection stability (especially when using an Exit-Node and DNS is not working), # it is possible to optionally add the public IPv4 and IPv6 address to the Derp-Map using: ipv4: 1.2.3.4 ipv6: 2001:db8::1 # List of externally available DERP maps encoded in JSON urls: - https://controlplane.tailscale.com/derpmap/default # Locally available DERP map files encoded in YAML # # This option is mostly interesting for people hosting # their own DERP servers: # https://tailscale.com/kb/1118/custom-derp-servers/ # # paths: # - /etc/headscale/derp-example.yaml paths: [] # If enabled, a worker will be set up to periodically # refresh the given sources and update the derpmap # will be set up. auto_update_enabled: true # How often should we check for DERP updates? update_frequency: 24h # Disables the automatic check for headscale updates on startup disable_check_updates: false # Time before an inactive ephemeral node is deleted? ephemeral_node_inactivity_timeout: 30m database: # Database type. Available options: sqlite, postgres # Please note that using Postgres is highly discouraged as it is only supported for legacy reasons. # All new development, testing and optimisations are done with SQLite in mind. type: sqlite # Enable debug mode. This setting requires the log.level to be set to \u0026#34;debug\u0026#34; or \u0026#34;trace\u0026#34;. debug: false # GORM configuration settings. gorm: # Enable prepared statements. prepare_stmt: true # Enable parameterized queries. parameterized_queries: true # Skip logging \u0026#34;record not found\u0026#34; errors. skip_err_record_not_found: true # Threshold for slow queries in milliseconds. slow_threshold: 1000 # SQLite config sqlite: path: /var/lib/headscale/db.sqlite # Enable WAL mode for SQLite. This is recommended for production environments. # https://www.sqlite.org/wal.html write_ahead_log: true # Maximum number of WAL file frames before the WAL file is automatically checkpointed. # https://www.sqlite.org/c3ref/wal_autocheckpoint.html # Set to 0 to disable automatic checkpointing. wal_autocheckpoint: 1000 # # Postgres config # Please note that using Postgres is highly discouraged as it is only supported for legacy reasons. # See database.type for more information. # postgres: # # If using a Unix socket to connect to Postgres, set the socket path in the \u0026#39;host\u0026#39; field and leave \u0026#39;port\u0026#39; blank. # host: localhost # port: 5432 # name: headscale # user: foo # pass: bar # max_open_conns: 10 # max_idle_conns: 10 # conn_max_idle_time_secs: 3600 # # If other \u0026#39;sslmode\u0026#39; is required instead of \u0026#39;require(true)\u0026#39; and \u0026#39;disabled(false)\u0026#39;, set the \u0026#39;sslmode\u0026#39; you need # # in the \u0026#39;ssl\u0026#39; field. Refers to https://www.postgresql.org/docs/current/libpq-ssl.html Table 34.1. # ssl: false ### TLS configuration # ## Let\u0026#39;s encrypt / ACME # # headscale supports automatically requesting and setting up # TLS for a domain with Let\u0026#39;s Encrypt. # # URL to ACME directory acme_url: https://acme-v02.api.letsencrypt.org/directory # Email to register with ACME provider acme_email: \u0026#34;\u0026#34; # Domain name to request a TLS certificate for: tls_letsencrypt_hostname: \u0026#34;\u0026#34; # Path to store certificates and metadata needed by # letsencrypt # For production: tls_letsencrypt_cache_dir: /var/lib/headscale/cache # Type of ACME challenge to use, currently supported types: # HTTP-01 or TLS-ALPN-01 # See: docs/ref/tls.md for more information tls_letsencrypt_challenge_type: HTTP-01 # When HTTP-01 challenge is chosen, letsencrypt must set up a # verification endpoint, and it will be listening on: # :http = port 80 tls_letsencrypt_listen: \u0026#34;:http\u0026#34; ## Use already defined certificates: tls_cert_path: \u0026#34;\u0026#34; tls_key_path: \u0026#34;\u0026#34; log: # Output formatting for logs: text or json format: text level: info ## Policy # headscale supports Tailscale\u0026#39;s ACL policies. # Please have a look to their KB to better # understand the concepts: https://tailscale.com/kb/1018/acls/ policy: # The mode can be \u0026#34;file\u0026#34; or \u0026#34;database\u0026#34; that defines # where the ACL policies are stored and read from. mode: file # If the mode is set to \u0026#34;file\u0026#34;, the path to a # HuJSON file containing ACL policies. path: \u0026#34;\u0026#34; ## DNS # # headscale supports Tailscale\u0026#39;s DNS configuration and MagicDNS. # Please have a look to their KB to better understand the concepts: # # - https://tailscale.com/kb/1054/dns/ # - https://tailscale.com/kb/1081/magicdns/ # - https://tailscale.com/blog/2021-09-private-dns-with-magicdns/ # # Please note that for the DNS configuration to have any effect, # clients must have the `--accept-dns=true` option enabled. This is the # default for the Tailscale client. This option is enabled by default # in the Tailscale client. # # Setting _any_ of the configuration and `--accept-dns=true` on the # clients will integrate with the DNS manager on the client or # overwrite /etc/resolv.conf. # https://tailscale.com/kb/1235/resolv-conf # # If you want stop Headscale from managing the DNS configuration # all the fields under `dns` should be set to empty values. dns: # Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/). magic_dns: true # Defines the base domain to create the hostnames for MagicDNS. # This domain _must_ be different from the server_url domain. # `base_domain` must be a FQDN, without the trailing dot. # The FQDN of the hosts will be # `hostname.base_domain` (e.g., _myhost.example.com_). base_domain: \u0026lt;hs.example.com\u0026gt; # List of DNS servers to expose to clients. nameservers: global: - 1.1.1.1 - 1.0.0.1 - 2606:4700:4700::1111 - 2606:4700:4700::1001 # NextDNS (see https://tailscale.com/kb/1218/nextdns/). # \u0026#34;abc123\u0026#34; is example NextDNS ID, replace with yours. # - https://dns.nextdns.io/abc123 # Split DNS (see https://tailscale.com/kb/1054/dns/), # a map of domains and which DNS server to use for each. split: {} # foo.bar.com: # - 1.1.1.1 # darp.headscale.net: # - 1.1.1.1 # - 8.8.8.8 # Set custom DNS search domains. With MagicDNS enabled, # your tailnet base_domain is always the first search domain. search_domains: [] # Extra DNS records # so far only A and AAAA records are supported (on the tailscale side) # See: docs/ref/dns.md extra_records: [] # - name: \u0026#34;grafana.myvpn.example.com\u0026#34; # type: \u0026#34;A\u0026#34; # value: \u0026#34;100.64.0.3\u0026#34; # # # you can also put it in one line # - { name: \u0026#34;prometheus.myvpn.example.com\u0026#34;, type: \u0026#34;A\u0026#34;, value: \u0026#34;100.64.0.3\u0026#34; } # # Alternatively, extra DNS records can be loaded from a JSON file. # Headscale processes this file on each change. # extra_records_path: /var/lib/headscale/extra-records.json # Unix socket used for the CLI to connect without authentication # Note: for production you will want to set this to something like: unix_socket: /var/run/headscale/headscale.sock unix_socket_permission: \u0026#34;0770\u0026#34; # # headscale supports experimental OpenID connect support, # it is still being tested and might have some bugs, please # help us test it. # OpenID Connect oidc: only_start_if_oidc_is_available: true issuer: \u0026#34;https://your-oidc.issuer.com/path\u0026#34; client_id: \u0026#34;your-oidc-client-id\u0026#34; client_secret: \u0026#34;your-oidc-client-secret\u0026#34; # Alternatively, set `client_secret_path` to read the secret from the file. # It resolves environment variables, making integration to systemd\u0026#39;s # `LoadCredential` straightforward: # client_secret_path: \u0026#34;${CREDENTIALS_DIRECTORY}/oidc_client_secret\u0026#34; # # client_secret and client_secret_path are mutually exclusive. # The amount of time from a node is authenticated with OpenID until it # expires and needs to reauthenticate. # Setting the value to \u0026#34;0\u0026#34; will mean no expiry. expiry: 180d # Use the expiry from the token received from OpenID when the user logged # in, this will typically lead to frequent need to reauthenticate and should # only been enabled if you know what you are doing. # Note: enabling this will cause `oidc.expiry` to be ignored. use_expiry_from_token: false # Customize the scopes used in the OIDC flow, defaults to \u0026#34;openid\u0026#34;, \u0026#34;profile\u0026#34; and \u0026#34;email\u0026#34; and add custom query # parameters to the Authorize Endpoint request. Scopes default to \u0026#34;openid\u0026#34;, \u0026#34;profile\u0026#34; and \u0026#34;email\u0026#34;. scope: [\u0026#34;openid\u0026#34;, \u0026#34;profile\u0026#34;, \u0026#34;email\u0026#34;] # extra_params: # domain_hint: example.com # # # List allowed principal domains and/or users. If an authenticated user\u0026#39;s domain is not in this list, the # # authentication request will be rejected. # # allowed_domains: # - example.com # # Note: Groups from keycloak have a leading \u0026#39;/\u0026#39; # allowed_groups: # - /headscale # allowed_users: # - alice@example.com # # # Optional: PKCE (Proof Key for Code Exchange) configuration # # PKCE adds an additional layer of security to the OAuth 2.0 authorization code flow # # by preventing authorization code interception attacks # # See https://datatracker.ietf.org/doc/html/rfc7636 # pkce: # # Enable or disable PKCE support (default: false) # enabled: false # # PKCE method to use: # # - plain: Use plain code verifier # # - S256: Use SHA256 hashed code verifier (default, recommended) # method: S256 # # # Map legacy users from pre-0.24.0 versions of headscale to the new OIDC users # # by taking the username from the legacy user and matching it with the username # # provided by the OIDC. This is useful when migrating from legacy users to OIDC # # to force them using the unique identifier from the OIDC and to give them a # # proper display name and picture if available. # # Note that this will only work if the username from the legacy user is the same # # and there is a possibility for account takeover should a username have changed # # with the provider. # # When this feature is disabled, it will cause all new logins to be created as new users. # # Note this option will be removed in the future and should be set to false # # on all new installations, or when all users have logged in with OIDC once. # map_legacy_users: false # Logtail configuration # Logtail is Tailscales logging and auditing infrastructure, it allows the control panel # to instruct tailscale nodes to log their activity to a remote server. logtail: # Enable logtail for this headscales clients. # As there is currently no support for overriding the log server in headscale, this is # disabled by default. Enabling this will make your clients send logs to Tailscale Inc. enabled: false # Enabling this option makes devices prefer a random port for WireGuard traffic over the # default static port 41641. This option is intended as a workaround for some buggy # firewall devices. See https://tailscale.com/kb/1181/firewalls/ for more information. randomize_client_port: false 3. Headplane的配置 # Headplane的配置文件位于./headplane/config/config.yaml，你可以根据需要进行修改。以下是一个示例配置：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 server: host: \u0026#34;0.0.0.0\u0026#34; port: 3000 cookie_secret: \u0026#34;32-character-long-random-string\u0026#34; cookie_secure: false headscale: url: \u0026#34;https://\u0026lt;headscale.example.com\u0026gt;\u0026#34; config_path: \u0026#34;/etc/headscale/config.yaml\u0026#34; config_strict: true integration: docker: enabled: true container_name: \u0026#34;headscale\u0026#34; socket: \u0026#34;unix:///var/run/docker.sock\u0026#34; oidc: issuer: \u0026#34;https://\u0026lt;authelia.example.com\u0026gt;\u0026#34; client_id: \u0026#34;headplane\u0026#34; # The client secret for the OIDC client # Either this or `client_secret_path` must be set for OIDC to work client_secret: \u0026#34;some-random-client-secret\u0026#34; # You can alternatively set `client_secret_path` to read the secret from disk. # The path specified can resolve environment variables, making integration # with systemd\u0026#39;s `LoadCredential` straightforward: # client_secret_path: \u0026#34;${CREDENTIALS_DIRECTORY}/oidc_client_secret\u0026#34; disable_api_key_login: false token_endpoint_auth_method: \u0026#34;client_secret_post\u0026#34; # If you are using OIDC, you need to generate an API key # that can be used to authenticate other sessions when signing in. # # This can be done with `headscale apikeys create --expiration 999d` headscale_api_key: \u0026#34;your-headscale-api-key\u0026#34; # Optional, but highly recommended otherwise Headplane # will attempt to automatically guess this from the issuer # # This should point to your publicly accessibly URL # for your Headplane instance with /admin/oidc/callback redirect_uri: \u0026#39;https://\u0026lt;url-to-your-headplane\u0026gt;/admin/oidc/callback\u0026#39; # Stores the users and their permissions for Headplane # This is a path to a JSON file, default is specified below. user_storage_file: \u0026#34;/var/lib/headplane/users.json\u0026#34; Integration with Authelia for Authentication # In the above configuration, Headplane uses OIDC (OpenID Connect) for authentication. You can use Authelia as the OIDC provider.\nIf you haven\u0026rsquo;t set up Authelia yet, you can refer to my previous article on [Authentication: Deploying Authelia for Private Identity Service]({{ \u0026lt;relref \u0026ldquo;../authelia/index.en.md\u0026rdquo;\u0026gt; }}) for a detailed guide on how to set it up.\nThe configuration file for Authelia is usually located at /etc/authelia/configuration.yml, and you need to add an OIDC client configuration in it. Below is an example configuration:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 identity_providers: oidc: clients: - client_id: \u0026#39;headplane\u0026#39; client_name: \u0026#39;Headplane OIDC Client\u0026#39; client_secret: \u0026#39;client-secret\u0026#39; public: false authorization_policy: one_factor redirect_uris: - \u0026#39;https://example.com/oidc/callback\u0026#39; scopes: - openid - profile - email grant_types: - authorization_code response_types: - code token_endpoint_auth_method: client_secret_post Make sure that the client_id, client_secret, and redirect_uris match the settings in your Headplane configuration.\n启动服务 # After creating the above files, you can start the Headscale and Headplane services using the following command:\n1 docker-compose up -d If everything is set up correctly, you should be able to access the Headplane web interface through your browser at http://\u0026lt;your-server-ip\u0026gt;:3000/admin/. Note that the /admin/ path is very important; if you don\u0026rsquo;t include it, you will see a 404 error page. You should see a login interface like this:\nIf you haven\u0026rsquo;t configured OIDC authentication, you won\u0026rsquo;t see the Single Sign On option on the above page. You can log in directly using the Headscale API key. You can create an API key through the command line interface of the Headscale container with the following command:\n1 docker exec -it headscale headscale apikeys create --expiration 999d This will generate an API key that you can use to log in to the Headplane interface.\nIf you have configured OIDC authentication, you can select the Single Sign On option on the login page, which will redirect you to the OIDC provider\u0026rsquo;s login page (e.g., Authelia):\nThen click the confirm authorization button:\nYou will then see the Headplane management interface:\nThis interface is very similar to the Tailscale web interface, where you can manage your Headscale server, view connected devices, create pre-authentication keys, and more.\nUsing Tailscale to Connect to Headscale # Tailscale clients by default connect to Tailscale\u0026rsquo;s official control servers, but we can configure them to connect to our own Headscale server. On macOS or Android, you can find the \u0026ldquo;Control Server\u0026rdquo; option in Tailscale settings and set it to your Headscale server address, such as https://\u0026lt;headscale.example.com\u0026gt;.\nHowever, on Windows, I couldn\u0026rsquo;t find this option. We can set the Tailscale control server through the command line. Open Command Prompt or PowerShell and run the following command:\n1 tailscale up --login-server https://\u0026lt;headscale.example.com\u0026gt; If your Headscale service is running correctly and authentication is deployed, running the above command will output a URL link. Open this link in your browser, and you will see the authentication login interface. After logging in, you will see an authorization page; click the \u0026ldquo;Authorize\u0026rdquo; button, and the Tailscale client will connect to your Headscale server.\nIf you haven\u0026rsquo;t configured OIDC authentication, you can log in using a pre-authentication key. First, create a pre-authentication key in the Headscale container with the following command:\n1 docker exec -it headscale headscale preauthkeys create --reusable --user your_user --expiration=72h This will generate a pre-authentication key that you can use to log in to the Tailscale client. Run the following command:\n1 tailscale up --authkey \u0026lt;your_preauth_key\u0026gt; Troubleshooting # 1. Issues with Using Nginx as a Reverse Proxy # When using Nginx as a reverse proxy, you can check the Tailscale logs with the following command:\n1 journalctl -u tailscaled -f You might see output like this:\n1 2 3 4 5 6 7 Apr 30 20:34:36 fedora tailscaled[1058]: control: LoginInteractive -\u0026gt; regen=true Apr 30 20:34:36 fedora tailscaled[1058]: control: doLogin(regen=true, hasUrl=false) Apr 30 20:34:36 fedora tailscaled[1058]: control: control server key from https://\u0026lt;headscale domain name\u0026gt;: ts2021=[lx/4O], legacy= Apr 30 20:34:36 fedora tailscaled[1058]: control: Generating a new nodekey. Apr 30 20:34:36 fedora tailscaled[1058]: control: RegisterReq: onode= node=[W5uIA] fup=false nks=false Apr 30 20:34:36 fedora tailscaled[1058]: control: controlhttp: forcing port 443 dial due to recent noise dial Apr 30 20:34:46 fedora tailscaled[1058]: Received error: register request: Post \u0026#34;https://\u0026lt;headscale domain name\u0026gt;/machine/register\u0026#34;: connection attempts aborted by context: context deadline exceeded This indicates that Tailscale is timing out while trying to connect to Headscale. Then we check the logs of the Headscale Docker container. You can view the logs in Portainer or use the following command:\n1 docker logs \u0026lt;headscale_container_name\u0026gt; You might see the following warning message:\n1 2025-04-30T21:45:45-04:00 WRN home/runner/work/headscale/headscale/hscontrol/noise.go:66 \u0026gt; No Upgrade header in TS2021 request. If headscale is behind a reverse proxy, make sure it is configured to pass WebSockets through. Primarily, I thought this was just a warning. As we all know, programmers often overlook warnings, so I ignored this message for a long time. It wasn\u0026rsquo;t until I had checked all possible errors that I thought to search online for this warning. To my surprise, I found that this warning was the culprit preventing Tailscale from connecting to Headscale. I really wanted to curse the programmer who put such an important message in the warning category!\nAnyway, this warning means that if Headscale is deployed behind a reverse proxy, you need to ensure that the reverse proxy is configured correctly to allow communication via WebSockets. Specifically, you need to add the Upgrade header in the reverse proxy configuration. If you are using Nginx as a reverse proxy, you can add the following content to your Nginx configuration file:\n1 2 3 4 5 6 7 8 9 10 11 12 location / { proxy_pass http://\u0026lt;headscale_ip\u0026gt;:\u0026lt;headscale_port\u0026gt;; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection \u0026#34;upgrade\u0026#34;; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } Make sure to replace \u0026lt;headscale_ip\u0026gt; and \u0026lt;headscale_port\u0026gt; with the actual IP address and port of your Headscale service.\nIf you are still having issues, check the Nginx error logs for any errors related to the proxy configuration. You can find the logs in /var/log/nginx/error.log.\n2. Issues with Using Cloudflare Tunnel as a Reverse Proxy # Before considering the use of a fixed IP, I had been trying to set up my Docker services through Cloudflare Tunnel for reverse proxying. For more details, refer to the following articles:\n“Dynamic Domain Name Resolution with Cloudflare”. “Accessing Personal Websites from the Public Internet - Nginx Reverse Proxy Configuration”. “Accessing Personal Websites from the Public Internet (Part 2) - Traefik Reverse Proxy Configuration”. “Accessing Personal Websites from the Public Internet (Part 3) - Using Cloudflare Tunnel for Docker Container Reverse Proxy”. Therefore, when deploying Headscale, I also attempted to use Cloudflare Tunnel for reverse proxying. After configuring Cloudflare Tunnel as described in the previous articles, I found that the Headscale web interface (Headplane) could be accessed normally, but the Tailscale client could not connect to Headscale. The Tailscale login command would hang, and the Tailscale logs showed the following error:\n1 2 3 4 5 6 7 Jun 24 15:29:02 fedora.attlocal.net tailscaled[885490]: Received error: register request: Post \u0026#34;https://headscale.example.com/machine/register\u0026#34;: connection attempts aborted by context: context deadline exceeded Jun 24 15:29:02 fedora.attlocal.net tailscaled[885490]: control: LoginInteractive -\u0026gt; regen=true Jun 24 15:29:02 fedora.attlocal.net tailscaled[885490]: control: doLogin(regen=true, hasUrl=false) Jun 24 15:29:02 fedora.attlocal.net tailscaled[885490]: control: control server key from https://headscale.example.com: ts2021=[lx/4O], legacy= Jun 24 15:29:02 fedora.attlocal.net tailscaled[885490]: control: Generating a new nodekey. Jun 24 15:29:02 fedora.attlocal.net tailscaled[885490]: control: RegisterReq: onode= node=[yFWNe] fup=false nks=false Jun 24 15:29:02 fedora.attlocal.net tailscaled[885490]: control: controlhttp: forcing port 443 dial due to recent noise dial The Headscale container logs showed the following error:\n1 2025-06-24T17:53:20Z WRN home/runner/work/headscale/headscale/hscontrol/noise.go:66 \u0026gt; No Upgrade header in TS2021 request. If headscale is behind a reverse proxy, make sure it is configured to pass WebSockets through. I spent about a week or two trying various methods to resolve this issue, but ultimately I couldn\u0026rsquo;t find a solution.ChatGPT helped me analyze the cause of this issue, and it concluded that it was due to the limitations of the free version of Cloudflare Tunnel. The free version of Cloudflare Tunnel only supports proxying HTTP/1.1 (+WebSocket upgrades) and HTTP/2 + WebSocket traffic. However, the TS2021 Noise handshake used by Headscale is done over pure gRPC traffic in HTTP2, which cannot be proxied by the free version of Cloudflare Tunnel.\nChatGPT also mentioned that the paid version of Cloudflare Tunnel supports HTTP/3 and gRPC traffic, so if you have a paid Cloudflare account, you can try using the paid version of Tunnel for proxying. However, I ultimately decided to abandon this approach.\nThis issue has also been discussed in an issue on the Cloudflare Tunnel GitHub repository: General Upgrade header support #990. In summary, my current conclusion is that Cloudflare Tunnel cannot proxy Headscale services.\n","date":"7 August 2025","externalUrl":null,"permalink":"/en/p/virtual-private-network-vpn-3-using-headscale-to-build-your-own-tailscale-server/","section":"Posts","summary":" Motivation # This is the third article in the “Virtual Private Network VPN” series. In the first two articles, I introduced the basic concepts and use cases of VPNs, as well as how to set up a private virtual network using Tailscale. Other articles in this series can be found at:\nPrivate Virtual Network VPN (1): Setting up a VPN with Shadowsocks on a Cloud Server Private Virtual Network VPN (2): Basic Concepts and Guide to Setting up with Tailscale In the previous article, I explained how to set up a private virtual network using Tailscale. Tailscale is a VPN solution based on WireGuard that allows us to quickly create a private virtual network. However, the official version of Tailscale uses Tailscale’s control servers, meaning our data and network traffic pass through Tailscale’s servers. Although this data and traffic are encrypted, if we want complete control over our network environment or have privacy and security concerns, we can set up our own Tailscale control server. The tool to achieve this is Headscale.\n","title":"Virtual Private Network VPN (3): Using Headscale to Build Your Own Tailscale Server","type":"post"},{"content":"","date":"7 August 2025","externalUrl":null,"permalink":"/en/tags/vpn/","section":"Tags","summary":"","title":"VPN","type":"tags"},{"content":"","date":"2025年8月7日","externalUrl":null,"permalink":"/series/%E7%A7%81%E6%9C%89%E8%99%9A%E6%8B%9F%E7%BD%91%E7%BB%9Cvpn%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"私有虚拟网络VPN系列","type":"series"},{"content":"","date":"2025年8月7日","externalUrl":null,"permalink":"/tags/%E7%BD%91%E7%BB%9C/","section":"Tags","summary":"","title":"网络","type":"tags"},{"content":"","date":"21 July 2025","externalUrl":null,"permalink":"/en/tags/notes/","section":"Tags","summary":"","title":"Notes","type":"tags"},{"content":"","date":"21 July 2025","externalUrl":null,"permalink":"/en/tags/obsidian/","section":"Tags","summary":"","title":"Obsidian","type":"tags"},{"content":" Motivation # In the previous post, I covered how to use Obsidian and Nextcloud to manage notes. The title of the previous article uses the word \u0026ldquo;all platform\u0026rdquo;, which means that Obsidian supports multiple platforms such as Windows, Linux, Mac, Android, etc. Generally speaking, this is enough, but it is still inconvenient in some scenarios.\nFor example, some work computers do not allow arbitrary software installation, making it impossible to use Obsidian on those machines. To truly achieve cross-platform compatibility, I believe a web version is necessary. After some online searching, I found that someone had already implemented a web version of Obsidian, so I decided to give it a try. Objectively speaking, the web version of Obsidian is still not very mature and feels a bit rough around the edges, but the basic functionality is there. Therefore, I deployed it on my own server for occasional use.\nFor other articles in this series about Obsidian, see:\nObsidian (1): All-platform Notes Management Software Prerequisites # The basic prerequisites for deploying various container services are as follows:\nA computer that can serve as a server Docker and Docker Compose Obsidian Docker # Obsidian Docker Overview # The web version of Obsidian is an open-source project developed by linuxserver.io, which is based on the core functionality of Obsidian and provides a web interface. It can run through a Docker container, making it very suitable for deployment on servers. Its GitHub address is: docker-obsidian.\ndocker-obsidian uses KasmVNC as the web interface. KasmVNC is a web desktop solution based on VNC. It allows users to access the Obsidian desktop environment through a browser.\ndocker-obsidian includes some customizable options, such as support for HTTP basic authentication, allowing users to set a username and password to protect access to Obsidian when deploying the container.\nDeploying Obsidian Docker # Project Structure\nSimilar to the previous container deployment, we use docker compose to deploy Obsidian. First, create a directory to store the configuration files and data:\n1 2 mkdir -p ~/docker/obsidian cd ~/docker/obsidian In this directory, we create two files: docker-compose.yml and .env.\ndocker-compose.yml\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 services: obsidian: image: lscr.io/linuxserver/obsidian:latest container_name: obsidian security_opt: - seccomp:unconfined #optional environment: - PUID=1000 - PGID=1000 - TZ=America/New_York - DOCKER_MODS=linuxserver/mods:universal-package-install - CUSTOM_USER=${CUSTOM_USER} - PASSWORD=${PASSWORD} volumes: - ${APP_DIR}/config:/config shm_size: \u0026#34;1gb\u0026#34; restart: unless-stopped networks: traefik-net: aliases: - obsidian .env\n1 2 3 APP_DIR=/media/user/docker_data/obsidian CUSTOM_USER=username PASSWORD=\u0026#39;your_password_here\u0026#39; Here, APP_DIR is the directory for Obsidian\u0026rsquo;s configuration files and data storage, CUSTOM_USER is the username you want to use, and PASSWORD is the password for accessing Obsidian.\nStart the Container\n1 docker compose up -d After running the above command, Docker will automatically download the Obsidian image and start the container. The default port used by docker-obsidian is 3000, and you can access Obsidian through your browser at http://your_server_ip:3000.\nReverse Proxy\nThere are various ways to set up a reverse proxy for the container to access it via a domain name. I have previously written articles on using Nginx, Traefik, and Cloudflare Tunnel for reverse proxying. You can refer to the following articles for details:\n\u0026ldquo;Access Personal Website from Internet - Nginx Reverse Proxy Configuration\u0026rdquo;. \u0026ldquo;Access Personal Website from Internet (2) - Traefik Reverse Proxy Configuration\u0026rdquo;. \u0026ldquo;Access Personal Website from Internet (3): Using Cloudflare Tunnel for Reverse Proxy\u0026rdquo; Initial Configuration of docker-obsidian # If the above steps are successful, you should be able to access the Obsidian web interface through your browser.\nOn the first visit, you will see a login authentication pop-up.\nEnter the username and password you set in the .env file.\nAfter logging in, you will see the Obsidian web interface:\nThis interface is similar to the desktop version of Obsidian, but you will also see the KasmVNC toolbar at the top and side.\nAt the same time, a pop-up will appear in the browser, prompting you to authorize access to the clipboard:\nClick \u0026ldquo;Allow\u0026rdquo; to enable clipboard functionality in Obsidian. Otherwise, content copied elsewhere will not be able to be pasted into the Obsidian web version, and vice versa.\nThen initialize Obsidian, making sure that the name of the note repository is consistent with the name you used in other Obsidian clients so that synchronization can occur. Other settings, such as enabling plugins and setting up cloud synchronization, are the same as in the desktop version of Obsidian. You can refer to my previous article: Obsidian (1): All-platform Notes Management Software.\nOnce the setup is complete, you can start using Obsidian:\nYou can use all the features of Obsidian in your browser, including editing notes and using plugins.\nPasting content copied elsewhere seems to not work directly in Obsidian; you need to use the KasmVNC clipboard feature to achieve this:\nSmall Issues and Solutions # I found that the web version of Obsidian cannot display Chinese characters:\nThe solution is to install fonts that can display Chinese characters when the container starts and set the LC_ALL environment variable.\nTo do this, add the following environment variables to your docker-compose.yml:\n1 2 3 environment: - INSTALL_PACKAGES=fonts-noto-cjk - LC_ALL=zh_CN.UTF-8 Then restart the container:\n1 2 docker compose down docker compose up -d Now the obsidian web version should be able to display Chinese characters normally:\nConclusion # The web version of Obsidian, while not yet fully mature, already provides basic functionality sufficient for daily use. Deploying the web version of Obsidian through a Docker container allows for easy operation on a server and access via a browser. This truly achieves cross-platform note management.\n","date":"21 July 2025","externalUrl":null,"permalink":"/en/p/obsidian-2-deployment-and-use-of-web-based-obsidian/","section":"Posts","summary":" Motivation # In the previous post, I covered how to use Obsidian and Nextcloud to manage notes. The title of the previous article uses the word “all platform”, which means that Obsidian supports multiple platforms such as Windows, Linux, Mac, Android, etc. Generally speaking, this is enough, but it is still inconvenient in some scenarios.\n","title":"Obsidian (2): Deployment and Use of Web-based Obsidian","type":"post"},{"content":"","date":"21 July 2025","externalUrl":null,"permalink":"/en/series/obsidian-series/","section":"Series","summary":"","title":"Obsidian Series","type":"series"},{"content":"","date":"2025年7月21日","externalUrl":null,"permalink":"/series/obsidian%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"Obsidian系列","type":"series"},{"content":"","date":"21 July 2025","externalUrl":null,"permalink":"/en/tags/web/","section":"Tags","summary":"","title":"Web","type":"tags"},{"content":"","date":"2025年7月21日","externalUrl":null,"permalink":"/tags/%E7%AC%94%E8%AE%B0/","section":"Tags","summary":"","title":"笔记","type":"tags"},{"content":" Motivation # In order to access the websites and services I have set up on my personal server from the internet, my previous approach was:\nUse a Dynamic DNS (DDNS) service to bind my public IP address to a domain name. Then configure Nginx as a reverse proxy to point the domain name to the services on my server. Finally, use acme to obtain SSL certificates for HTTPS access. Later, to containerize the reverse proxy, I switched to Traefik, which combined steps 2 and 3 into a more convenient solution. You can refer to my previous articles for these solutions:\n\u0026ldquo;Dynamic DNS - Cloudflare\u0026rdquo;. \u0026ldquo;Access Personal Website from Internet - Nginx Reverse Proxy Configuration\u0026rdquo;. \u0026ldquo;Access Personal Website from Internet (2) - Traefik Reverse Proxy Configuration\u0026rdquo;. These solutions were already quite effective, but they still relied on the stability of the public IP address and the openness of ports. If deployed in a corporate or school network environment, you might encounter issues with unstable IP addresses or blocked ports. For a long time, I thought this problem was difficult to solve.\nRecently, I finally discovered a better solution: using Cloudflare Tunnel. This solution allows you to access your services from the internet without a public IP address, even if ports are blocked or your network is restricted by a firewall.\nPrerequisites # A computer running a Linux system (as a personal server). A private domain name (for detailed operations, please refer to my previous post \u0026ldquo;Establishing a Personal Website (1): Purchasing a Personal Domain Name and Configuring Dynamic DNS\u0026rdquo;). Basic understanding of containerization concepts and experience with deploying services using Docker or Kubernetes (K8s or K3s). Basic understanding of reverse proxy concepts and usage. You can refer to \u0026ldquo;Access Personal Website from Internet - Nginx Reverse Proxy Configuration\u0026rdquo; or \u0026ldquo;Access Personal Website from Internet (2) - Traefik Reverse Proxy Configuration\u0026rdquo;, which provide detailed introductions. A Cloudflare account with your domain added to Cloudflare. If your domain was previously managed by another DNS service provider, you need to switch the domain\u0026rsquo;s DNS servers to those provided by Cloudflare. You can refer to \u0026ldquo;Migrating Domain Service Provider from Hostinger to Cloudflare\u0026rdquo; to learn how to migrate your domain to Cloudflare. Basic Concepts of Cloudflare Tunnel # Introduction # Cloudflare Tunnel (formerly known as Argo Tunnel) is a service provided by Cloudflare that allows you to expose your services to the internet without a public IP address. It works by running a lightweight proxy program on your server that forwards your services through Cloudflare\u0026rsquo;s network, enabling access from the internet.\nHow It Works # The working principle of Cloudflare Tunnel can be found in the official documentation. In simple terms, as long as your computer can access Cloudflare\u0026rsquo;s DNS servers, you can establish a tunnel from Cloudflare to your computer. After configuring your domain to resolve to this tunnel, Cloudflare will forward all requests for your domain through this tunnel to the computer connected to it. Similarly, your services on the computer can send responses back through this tunnel. This allows you to access your services from the internet without needing a public IP address or open ports, regardless of whether your network is restricted by a firewall.\nWhen a user accesses a service on the internet, the process is as follows:\nThe user enters your domain in their browser, which resolves to Cloudflare\u0026rsquo;s DNS servers. Cloudflare\u0026rsquo;s servers forward the request to the Cloudflare Tunnel you established. This tunnel is maintained by the cloudflared program running on your computer. cloudflared forwards the request to your configured reverse proxy service (such as Traefik or Nginx). The reverse proxy service forwards the request to the specific service running on your computer (such as NextCloud, Jellyfin, etc.). The specific service processes the request and sends the response back through the reverse process to the user. Using Cloudflare Tunnel # Enabling Cloudflare Zero Trust Service # Cloudflare Tunnel is part of the Cloudflare Zero Trust service, so you need to enable Cloudflare Zero Trust first.\nLog in to your Cloudflare account, find \u0026ldquo;Zero Trust\u0026rdquo; in the left sidebar, and click to enter.\nThe first time you enable it, you will be prompted to choose a plan. The free plan is sufficient for personal users. Fill in your credit card information to enable it.\nCreating a Tunnel # In the Zero Trust page, expand the left sidebar to find the Networks option, then click on Tunnels to open the Tunnels management page:\nYou can click the Create a tunnel button to create a new Tunnel, but it is not recommended to create it this way because you will need to transfer the configuration to your server later. Instead, we will create it directly on the server. The steps are as follows:\n1. Install cloudflared on the Server # First, we need to install the cloudflared program on the server. cloudflared is a command-line tool provided by Cloudflare for creating and managing Cloudflare Tunnels. For specific installation methods, refer to the official documentation. Here are the basic installation commands:\n1 2 3 4 5 6 7 8 # For Debian/Ubuntu sudo apt-get install cloudflared # For CentOS/RHEL sudo yum install cloudflared # For macOS brew install cloudflared 2. Create a Tunnel on the Server # After installing cloudflared, we first need to log in to your Cloudflare account on the server by running:\n1 cloudflared tunnel login This will output a link that you need to copy and open in your browser to log in to your Cloudflare account. After logging in, you will be prompted to select the domain you want to bind to the Tunnel.\nThen, run the following command on the server to create a new Tunnel:\n1 cloudflared tunnel create my-tunnel Replace my-tunnel with the name you want to give to your Tunnel. After running this command, it will output a Tunnel ID, which you should note down.\nAt the same time, cloudflared will create a JSON file on your server that records this Tunnel, usually located in the ~/.cloudflared/ directory, with the filename being the Tunnel ID. This file is important as it contains the credentials needed to connect to this Tunnel.\nYou can log in to the Cloudflare web interface to check if the Tunnel was created successfully. In the Zero Trust -\u0026gt; Networks -\u0026gt; Tunnels page, you should see the Tunnel you just created:\nNote that at this point, we have only created the Tunnel but not connected it, so its status is inactive.\n3. Connect the Tunnel # Here, I will explain how to connect the Tunnel using both Docker and K3s as examples.\nDocker # Prerequisites:\nThis assumes that you have already installed Docker and Docker Compose, are using Traefik as a reverse proxy, and have created a shared network for all containers named traefik-net. For the specific process, refer to \u0026ldquo;Access Personal Website from Internet (2) - Traefik Reverse Proxy Configuration\u0026rdquo;.\nCreate a cloudflared Container\nCreate a new directory for the cloudflared configuration files:\n1 mkdir -p ~/cloudflared The directory structure should look like this:\n1 2 3 4 cloudflared/ ├── docker-compose.yml ├── .env └── config.yml The docker-compose.yml in this directory is:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 version: \u0026#39;3.8\u0026#39; services: cloudflared: image: cloudflare/cloudflared:2025.5.0 container_name: cloudflared command: tunnel --no-autoupdate run my-tunnel volumes: - ./config.yaml:/etc/cloudflared/config.yaml:ro - ${CERT_FILE}:/etc/cloudflared/cert.pem - ${CRED_FILE}:/etc/cloudflared/creds/credentials.json restart: unless-stopped networks: - traefik-net networks: traefik-net: external: true Replace my-tunnel with the name of your Tunnel. This configuration will create a cloudflared container that runs your Tunnel and connects it to the traefik-net network.\nThe CERT_FILE and CRED_FILE environment variables should point to the certificate and credential files you created earlier. You can define these variables in a .env file in the ~/cloudflared directory:\n1 2 CERT_FILE=/home/user/.cloudflared/cert.pem CRED_FILE=/home/user/.cloudflared/\u0026lt;TUNNEL ID\u0026gt;.json Note: In some cases, the cert.pem file and \u0026lt;TUNNEL ID\u0026gt;.json file may not have read permissions, and you may need to manually modify the permissions to allow Docker to read them:\n1 2 chmod +r /home/user/.cloudflared/cert.pem chmod +r /home/user/.cloudflared/\u0026lt;TUNNEL ID\u0026gt;.json Start the cloudflared Container\nIn the ~/cloudflared directory, run the following command to start the cloudflared container:\n1 docker-compose up -d At this point, you can go back to the Zero Trust -\u0026gt; Networks -\u0026gt; Tunnels page and see that the status of the Tunnel has changed to active, indicating that the Tunnel is now connected:\nConfigure DNS Records\nNow that your Tunnel is active, you need to configure your DNS records to point to the Tunnel. Go back to the Cloudflare dashboard, and in the Zero Trust -\u0026gt; Networks -\u0026gt; Tunnels page, click on the Tunnel you just created.\nIn the Tunnel details page, you will see a section for DNS. Here, you can add a new DNS record that points to your Tunnel. For example, you can create a CNAME record for myapp.example.com that points to my-tunnel.cloudflare.com.\nAfter adding the DNS record, it may take a few minutes for the changes to propagate.\nTest Access\nNow you can access your service through the domain name you configured. For example, if you set up a service at myapp.example.com, you can open this URL in your browser to access it.\nK3s # Prerequisites:\nThis assumes that you have already installed K3s and are using Traefik as the Ingress Controller. For the specific process, refer to \u0026ldquo;Homelab (1): Building a Home Cluster with Kubernetes (K8s) or K3s\u0026rdquo;.\nConvert Tunnel Credentials to Secret\nWhen you created the Tunnel, Cloudflare generated a credentials file saved at ~/.cloudflared/\u0026lt;Tunnel ID\u0026gt;.json. We need to convert this file into a Kubernetes Secret.\nRun the following command on your server:\n1 kubectl create secret generic cloudflared-credentials --from-file=credentials.json=\u0026lt;User Home Path\u0026gt;/.cloudflared/\u0026lt;Tunnel ID\u0026gt;.json Replace \u0026lt;User Home Path\u0026gt; with your user home directory path and \u0026lt;Tunnel ID\u0026gt; with the ID of the Tunnel you created earlier.\nCreate cloudflared Deployment and Service\nThe required manifest file is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 apiVersion: apps/v1 kind: Deployment metadata: name: cloudflared spec: selector: matchLabels: app: cloudflared replicas: 1 template: metadata: labels: app: cloudflared spec: containers: - name: cloudflared image: cloudflare/cloudflared:2025.5.0 args: - tunnel - --config - /etc/cloudflared/config/config.yaml - run livenessProbe: httpGet: path: /ready port: 2000 failureThreshold: 1 initialDelaySeconds: 10 periodSeconds: 10 volumeMounts: - name: config mountPath: /etc/cloudflared/config readOnly: true - name: creds mountPath: /etc/cloudflared/creds readOnly: true volumes: - name: creds secret: secretName: li-tunnel-credentials - name: config configMap: name: cloudflared items: - key: config.yaml path: config.yaml --- apiVersion: v1 kind: ConfigMap metadata: name: cloudflared data: config.yaml: | tunnel: li-tunnel credentials-file: /etc/cloudflared/creds/credentials.json metrics: 0.0.0.0:2000 no-autoupdate: true ingress: - hostname: \u0026#34;*.jinli.li\u0026#34; service: http://traefik.kube-system.svc.cluster.local:80 - service: http_status:404 The configurations in this manifest file are similar to those in the Docker Compose file, so I won\u0026rsquo;t elaborate on them here.\nDeploy cloudflared\nSave the above manifest file as cloudflared.yaml, and then run the following command on your server to deploy cloudflared:\n1 kubectl apply -f cloudflared.yaml At this point, you can go to the Zero Trust -\u0026gt; Networks -\u0026gt; Tunnels page and see that the status of the Tunnel has changed to active, indicating that the Tunnel is now connected successfully.\nIf it doesn\u0026rsquo;t show as active, you need to check whether cloudflared was deployed successfully.\nConfigure DNS Records and Test Access\nThese two steps are the same as in the Docker method (steps 3 and 4 above).\n","date":"2 June 2025","externalUrl":null,"permalink":"/en/p/access-personal-website-from-internet-3-using-cloudflare-tunnel-for-reverse-proxy/","section":"Posts","summary":" Motivation # In order to access the websites and services I have set up on my personal server from the internet, my previous approach was:\nUse a Dynamic DNS (DDNS) service to bind my public IP address to a domain name. Then configure Nginx as a reverse proxy to point the domain name to the services on my server. Finally, use acme to obtain SSL certificates for HTTPS access. Later, to containerize the reverse proxy, I switched to Traefik, which combined steps 2 and 3 into a more convenient solution. You can refer to my previous articles for these solutions:\n","title":"Access Personal Website from Internet (3): Using Cloudflare Tunnel for Reverse Proxy","type":"post"},{"content":"","date":"2 June 2025","externalUrl":null,"permalink":"/en/tags/cloudflare/","section":"Tags","summary":"","title":"Cloudflare","type":"tags"},{"content":"","date":"2 June 2025","externalUrl":null,"permalink":"/en/tags/cloudflare-tunnel/","section":"Tags","summary":"","title":"Cloudflare Tunnel","type":"tags"},{"content":"","date":"2 June 2025","externalUrl":null,"permalink":"/en/tags/domain/","section":"Tags","summary":"","title":"Domain","type":"tags"},{"content":"","date":"2 June 2025","externalUrl":null,"permalink":"/en/tags/reverse-proxy/","section":"Tags","summary":"","title":"Reverse Proxy","type":"tags"},{"content":"","date":"2025年6月2日","externalUrl":null,"permalink":"/tags/%E5%8F%8D%E5%90%91%E4%BB%A3%E7%90%86/","section":"Tags","summary":"","title":"反向代理","type":"tags"},{"content":"","date":"2025年6月2日","externalUrl":null,"permalink":"/tags/%E5%9F%9F%E5%90%8D/","section":"Tags","summary":"","title":"域名","type":"tags"},{"content":"","date":"1 June 2025","externalUrl":null,"permalink":"/en/tags/homelab/","section":"Tags","summary":"","title":"Homelab","type":"tags"},{"content":" Motivation # After about 5 years of building my own home server and using container technology to deploy various private services, I have become familiar with a solution that manages all containers using Docker Compose. This solution is elegant and useful, but it is limited to a single computer. Recently, I suddenly had the need to manage multiple computers, and I finally began to feel the necessity of transitioning from a single server to a server cluster, or what is known as a Homelab.\nSpecifically, the computer I previously used as a server had a relatively large amount of memory and hard disk space but did not have a graphics card, which was sufficient for many of the services I had deployed before. Now I have a computer with a graphics card and want to run some large language models. Of course, I could deploy them individually on this computer, but that would create a disconnect from my previous solution. I hope to find a solution that can manage both computers simultaneously, making it easier to expand when I add more computers in the future.\nOf course, before transitioning to Kubernetes, it is best to have a good understanding of container concepts. I have previously written a series of articles on containers (mainly Docker) for reference:\nContainer (2): Docker Best Practices Guide - Docker Compose and Portainer Container (3): Docker Best Practices Guide - Managing Data Volumes Container (4): Docker Best Practices Guide - Container Updates, Upgrades, and Migrations Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What\u0026rsquo;s Up Docker) Container (6): Common Misconceptions, Bad Habits, and Issues When Using Docker Homelab # What is a Homelab # On the surface, a Homelab refers to setting up a laboratory at home for learning, experimentation, and development. A Homelab typically includes one or more servers, networking equipment, storage devices, etc., which can be used to run various services and applications.\nThe scale of a Homelab can vary greatly. If you have sufficient financial resources, you can purchase multiple high-performance servers to build a large home lab. If your budget is limited, you can also use a regular computer or even a Raspberry Pi to set up a small home lab.\nWhy Do You Need a Homelab # Server clusters or Homelabs may seem distant from individual users, but this is not the case. I even believe that many people who play with private servers may gradually move towards a Homelab. Even if you don\u0026rsquo;t have multiple computers or servers, you can still try using Kubernetes (K8s) or K3s to manage your containerized applications. Later, if you acquire more computers or servers, you can easily expand your existing Kubernetes cluster to multiple machines.\nIn summary, a Homelab offers high flexibility and scalability to meet various needs of individual users. Even if you only have simple requirements and basic hardware, you can learn a lot of new technologies and knowledge by building a Homelab.\nKubernetes (K8s) and K3s # Kubernetes (K8s) is an open-source container orchestration platform for automating the deployment, scaling, and management of containerized applications. It is likely the most widely used solution for building a Homelab.\nOf course, the knowledge within Kubernetes is quite complex, involving many concepts and components such as Pods, Services, Deployments, Ingress, etc. For beginners, it may seem overly complicated and somewhat difficult to understand. K3s is a lightweight version of Kubernetes, designed for resource-constrained environments. It removes some unnecessary components and features, making K3s easier to install and manage. K3s is well-suited for use in home labs or small clusters.\nIn this series of articles, we will start with a very simple application and use K3s to build a Kubernetes cluster from scratch, gradually expanding to more complex applications and multi-node setups.\nK3s Basics # Kubernetes (K8s) is a complex system with many components. K3s simplifies this by retaining only the most essential components. Here, we will introduce only the core and fundamental concepts to provide a basic understanding of how K3s operates, with other components and concepts introduced as needed later.\nThe following diagram shows a basic framework example of K3s running on a single-node server:\nWhen a user makes a request to a service, the process is as follows:\nWhen an external user\u0026rsquo;s request reaches the K3s cluster, it is first handled by the Ingress Controller (usually Traefik). The Ingress Controller forwards the request to the appropriate Service based on the routing rules defined in the Ingress resource. The Service forwards the request to the corresponding Pod, where the actual application container is running. Additionally, administrators can manage the components and resources within the K3s cluster using the command-line tool kubectl.\nNext, let\u0026rsquo;s briefly introduce these concepts.\nPod # Pod is the smallest deployment unit in Kubernetes and can contain one or more containers. The containers in a Pod share network and storage resources. Pods are typically used to run a single application or service.\nService # Service is an abstract concept in Kubernetes used to define a set of access policies for Pods. A Service can be accessed through a fixed IP address and port, regardless of how the actual IP addresses of the Pods change. Services can be divided into ClusterIP, NodePort, LoadBalancer, and other types.\nClusterIP: The default type, the Service can only be accessed from within the cluster. NodePort: The Service can be accessed from outside the cluster through a specified port. LoadBalancer: The Service can be accessed through a cloud provider\u0026rsquo;s load balancer. Ingress and Ingress Controller # Ingress is a resource in Kubernetes used to manage routing rules for external access to services within the cluster. Ingress can route requests to different Services based on domain names or paths. Ingress is typically used in conjunction with an Ingress Controller, which is responsible for implementing the routing rules defined in the Ingress resource.\nDeployment # Deployment is a controller in Kubernetes used to manage the deployment and updating of Pods. A Deployment can define the number of replicas of a Pod, update strategies, and more. With a Deployment, we can easily scale the number of Pods up or down, and perform rolling updates without affecting the availability of the service.\nK3s Installation and Configuration # Next, let\u0026rsquo;s take a single-node cluster as an example to illustrate how to use K3s. We will then discuss how to expand the cluster when more nodes are added.\nHere, we choose to deploy the simplest whoami service.\n1. Install K3s # Installing K3s on a Linux server is very simple, just run the following command:\n1 curl -sfL https://get.k3s.io | sudo sh - After it finishes running, K3s should be installed successfully and you should see output similar to the following:\n1 2 3 4 5 6 7 8 9 10 [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit Created symlink \u0026#39;/etc/systemd/system/multi-user.target.wants/k3s.service\u0026#39; → \u0026#39;/etc/systemd/system/k3s.service\u0026#39;. [INFO] systemd: Starting k3s This indicates that the command has done the following:\nDownloaded and installed K3s. Created symlinks for kubectl, crictl, and ctr, which are tools used to manage the K3s cluster. Among them: kubectl is the command-line tool for Kubernetes, used to manage Kubernetes clusters. crictl is the command-line tool for the Container Runtime Interface (CRI), used to manage containers. ctr is the command-line tool for the container runtime, used to interact directly with the container runtime. Created k3s-killall.sh and k3s-uninstall.sh scripts for stopping and uninstalling K3s. Running the command sudo /usr/local/bin/k3s-killall.sh can stop the K3s service. Running the command sudo /usr/local/bin/k3s-uninstall.sh can uninstall K3s. Created an environment file /etc/systemd/system/k3s.service.env for configuring the K3s service. Created the K3s systemd service file /etc/systemd/system/k3s.service and enabled the service. After that, the K3s service will start automatically every time the computer boots. Started the K3s service. 2. Verify K3s Installation # After the installation is complete, K3s creates a configuration file named k3s.yaml in the /etc/rancher/k3s directory. This file contains the configuration information for the K3s cluster, including the API server address, authentication information, and more.\nAs mentioned above, K3s also installs a kubectl as the command-line tool for managing K3s. When kubectl runs, it needs to know which cluster it is managing based on the API server address and authentication information. By default, kubectl reads the configuration information from the /etc/rancher/k3s/k3s.yaml file. If you are not running kubectl as root, you may encounter permission issues when trying to read the /etc/rancher/k3s/k3s.yaml file. In this case, you can copy the /etc/rancher/k3s/k3s.yaml file to the ~/.kube directory and change the file permissions to allow kubectl to read the cluster information:\n1 2 3 mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown $(id -u):$(id -g) ~/.kube/config Now you can use kubectl to manage the K3s cluster. You can run the following command to verify that K3s is installed successfully:\n1 kubectl get nodes If everything is working correctly, you should see output similar to the following:\n1 2 NAME STATUS ROLES AGE VERSION fedora.attlocal.net Ready control-plane,master 15s v1.32.5+k3s1 You can check the status of all Pods in the cluster by running:\n1 kubectl get pods -A If everything is working correctly, you should see output similar to the following:\n1 2 3 4 5 6 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-697968c856-scmft 0/1 ContainerCreating 0 15s \u0026lt;none\u0026gt; fedora.attlocal.net \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; helm-install-traefik-crd-7dkch 0/1 ContainerCreating 0 15s \u0026lt;none\u0026gt; fedora.attlocal.net \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; helm-install-traefik-qkl97 0/1 ContainerCreating 0 15s \u0026lt;none\u0026gt; fedora.attlocal.net \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; local-path-provisioner-774c6665dc-jrbrj 0/1 ContainerCreating 0 15s \u0026lt;none\u0026gt; fedora.attlocal.net \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; metrics-server-6f4c6675d5-v97zv 0/1 ContainerCreating 0 15s \u0026lt;none\u0026gt; fedora.attlocal.net \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; This output shows that K3s has successfully started several Pods, including the DNS service (coredns), Traefik Ingress Controller (helm-install-traefik-crd and helm-install-traefik-qkl97), the default storage class for K3s (local-path-provisioner), and the Kubernetes metrics server (metrics-server).\n3. Deploy a Simple Application (whoami) # Now we can deploy a simple whoami service to test the K3s cluster. whoami is a very simple HTTP service that returns the request\u0026rsquo;s IP address, request headers, and other information, making it ideal for testing the K3s cluster.\nCreate the whoami Application Manifest # K3s uses a YAML file called a manifest to define the deployment of applications. We can create a file named whoami.yaml with the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 apiVersion: apps/v1 kind: Deployment metadata: name: whoami labels: app: whoami spec: replicas: 1 selector: matchLabels: app: whoami template: metadata: labels: app: whoami spec: containers: - name: whoami image: traefik/whoami ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: whoami labels: app: whoami spec: type: ClusterIP ports: - port: 80 targetPort: 80 selector: app: whoami --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: whoami-ingress spec: ingressClassName: traefik rules: - host: whoami.example.com http: paths: - path: / pathType: Prefix backend: service: name: whoami port: number: 80 Although this is a YAML file, it actually contains three parts. For simplicity, we have placed these three parts in the same file. In later articles, we will explain how to better organize and manage applications in K3s by structuring files.\nThese three parts are separated by --- and define:\nDeployment: This defines a Deployment named whoami, indicating that we want to deploy an application called whoami. This application has 1 replica and uses the official Traefik whoami image, listening on port 80. Service: This defines a Service named whoami, indicating that we want to create a service called whoami. This service is of type ClusterIP, meaning it can only be accessed from within the cluster. It listens on port 80 and forwards requests to the whoami container\u0026rsquo;s port 80 in the Pod. Ingress: This defines an Ingress named whoami-ingress, indicating that we want to create an Ingress resource called whoami-ingress. This Ingress uses Traefik as the Ingress Controller and routes requests to the Service named whoami. Apply the whoami Service # Now we can use kubectl to apply this manifest file. Run the following command:\n1 kubectl apply -f whoami.yaml If everything is working correctly, you should see output similar to the following:\n1 2 3 deployment.apps/whoami created service/whoami created ingress.networking.k8s.io/whoami-ingress created Verify the whoami Service # First, let\u0026rsquo;s confirm that the whoami service has been successfully deployed. Run the following command:\n1 kubectl get deployments If everything is working correctly, you should see output similar to the following:\n1 2 NAME READY UP-TO-DATE AVAILABLE AGE whoami 1/1 1 1 2m Then, let\u0026rsquo;s check the status of the Pods in the cluster. Run the following command:\n1 kubectl get pods If everything is working correctly, you should see output similar to the following:\n1 2 NAME READY STATUS RESTARTS AGE whoami-5b6c7f8d9f-2j4k5 1/1 Running 0 2m Next, let\u0026rsquo;s confirm that the whoami service\u0026rsquo;s Service has been successfully created. Run the following command:\n1 kubectl get svc If everything is working correctly, you should see output similar to the following:\n1 2 3 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE whoami ClusterIP 10.43.42.148 \u0026lt;none\u0026gt; 8080/TCP 20s kubernetes ClusterIP 10.43.0.1 \u0026lt;none\u0026gt; 443/TCP 3m15s Access the whoami Service via Ingress # Now we can access the whoami service via Ingress. First, we can use the local address or the address within the same local area network to access the whoami service from the computer where whoami is deployed. If you are accessing it on your local machine, you can use the following command:\n1 curl \u0026#34;Host: whoami.example.com\u0026#34; http://localhost If you are accessing it from another computer on the same local area network, you can use the following command (replace 192.168.1.233 with your K3s server\u0026rsquo;s IP address):\n1 curl \u0026#34;Host: whoami.example.com\u0026#34; http://192.168.1.233 Here, we added the Host header because Ingress needs this header to route the request to the corresponding Service. Otherwise, how would Ingress know which Service you want to access?\nIf everything is working correctly, you should see output similar to the following:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Hostname: whoami-64f6cf779d-zxsm4 IP: 127.0.0.1 IP: ::1 IP: 10.42.0.9 IP: fe80::b43d:c0ff:fe52:80bd RemoteAddr: 10.42.0.8:39990 GET / HTTP/1.1 Host: whoami.example.com User-Agent: curl/8.12.0 Accept: */* Accept-Encoding: gzip X-Forwarded-For: 10.42.0.1 X-Forwarded-Host: whoami.jinli.li X-Forwarded-Port: 80 X-Forwarded-Proto: http X-Forwarded-Server: traefik-c98fdf6fb-5q6m6 X-Real-Ip: 10.42.0.1 If you can see output similar to the above, it means the whoami service is running successfully. However, note that the values of X-Forwarded-For and X-Real-Ip are 10.42.0.1 and 10.42.0.1, respectively. This is the internal IP address of the K3s cluster, not the IP address of the computer you are accessing from. This is because the Ingress Controller (Traefik) uses the internal IP address of the K3s cluster when forwarding requests to the whoami service.\nTODO: How to fix this problem?\nIf you want to access the whoami service from outside the K3s cluster, you need to resolve the domain name whoami.example.com to the public IP address of your K3s server. Then you can access the whoami service through a browser or other HTTP clients.\nIf everything is working correctly, you should see something like the following in your browser:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Hostname: whoami-64f6cf779d-ktwm9 IP: 127.0.0.1 IP: ::1 IP: 10.42.0.18 IP: fe80::1453:3cff:fe59:5835 RemoteAddr: 10.42.0.8:50132 GET / HTTP/1.1 Host: whoami.jinli.li User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Accept-Encoding: gzip, br Accept-Language: en Cache-Control: max-age=0 Priority: u=0, i Sec-Ch-Ua: \u0026#34;Chromium\u0026#34;;v=\u0026#34;136\u0026#34;, \u0026#34;Google Chrome\u0026#34;;v=\u0026#34;136\u0026#34;, \u0026#34;Not.A/Brand\u0026#34;;v=\u0026#34;99\u0026#34; Sec-Ch-Ua-Mobile: ?0 Sec-Ch-Ua-Platform: \u0026#34;macOS\u0026#34; Sec-Fetch-Dest: document Sec-Fetch-Mode: navigate Sec-Fetch-Site: none Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 X-Forwarded-For: 10.42.0.11 X-Forwarded-Host: whoami.jinli.li X-Forwarded-Port: 80 X-Forwarded-Proto: http X-Forwarded-Server: traefik-c98fdf6fb-5q6m6 X-Real-Ip: 10.42.0.11 Conclusion # In this article, we introduced the basics of K3s and demonstrated how to deploy a simple whoami service on a single-node K3s cluster.\nIn the upcoming articles, we will continue to expand the K3s cluster, covering how to deploy more complex applications, manage multi-node clusters, and utilize other features of K3s to meet various needs.\nAcknowledgments # I referenced the video From Zero to Hero: K3s, Traefik \u0026amp; Cloudflare Your Home Lab Powerhouse by YouTuber LinuxCloudHacks while learning how to use K3s.\n","date":"1 June 2025","externalUrl":null,"permalink":"/en/p/homelab1using-kubernetes-k8s-or-k3s-to-build-a-home-lab/","section":"Posts","summary":" Motivation # After about 5 years of building my own home server and using container technology to deploy various private services, I have become familiar with a solution that manages all containers using Docker Compose. This solution is elegant and useful, but it is limited to a single computer. Recently, I suddenly had the need to manage multiple computers, and I finally began to feel the necessity of transitioning from a single server to a server cluster, or what is known as a Homelab.\n","title":"Homelab（1）：Using Kubernetes (K8s) or K3s to Build a Home Lab","type":"post"},{"content":"","date":"1 June 2025","externalUrl":null,"permalink":"/en/tags/k3s/","section":"Tags","summary":"","title":"K3s","type":"tags"},{"content":"","date":"1 June 2025","externalUrl":null,"permalink":"/en/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":" Motivation # I previously used Nginx as a reverse proxy to access websites on my private server. At that time, Nginx was installed directly on the computer without containerization, which meant that when I wanted to migrate these services to other computers later, I had to reconfigure Nginx and reconfigure SSL certificates, etc.\nSo I decided to containerize the reverse proxy service as well, to facilitate future management and migration. I saw many examples of using Traefik in Docker online, so I decided to migrate from Nginx to Traefik.\nPrerequisites # A computer running Linux (as a private server) A private domain name (for detailed operations, please refer to the previous post \u0026ldquo;The Process of Building a Personal Website (1): Purchasing a Personal Domain Name and Configuring Dynamic Domain Name Resolution\u0026rdquo;) Docker and docker-compose installed, and a basic understanding of Docker concepts and usage. If you haven\u0026rsquo;t installed Docker and docker-compose, or if you don\u0026rsquo;t understand the basic concepts and usage of Docker and containerization, you can refer to \u0026ldquo;Containers (1): Introduction to Container-Related Knowledge - Containerization, Docker, Docker-Compose, Kubernetes / K8s, etc.\u0026rdquo; for a more detailed introduction. A basic understanding of Nginx concepts and usage. If you haven\u0026rsquo;t used or don\u0026rsquo;t understand Nginx, you can refer to \u0026ldquo;Accessing Personal Websites from the Public Network - Nginx Reverse Proxy Configuration\u0026rdquo; for a more detailed introduction. Traefik Overview # Traefik is a modern reverse proxy and load balancer that can automatically discover and configure backend services. Traefik supports various backend services such as Docker, Kubernetes, Consul, etc., and can automatically obtain SSL certificates. Traefik\u0026rsquo;s configuration is very flexible and can be done through labels, files, or APIs.\nInstalling and Configuring Traefik # Overall Architecture # Traefik can be configured using labels, APIs, or files. Since I previously used file-based configuration with Nginx, where each service had its own configuration file, I found this approach very clear and manageable. Therefore, I will also use file-based configuration with Traefik.\nThe Traefik container itself is independent and managed using a docker-compose.yml file like other Docker containers. We will create a new network in Docker called traefik-net, and all containers, including Traefik, will use this network. Within this network, we will use Traefik for reverse proxying.\nIn this article, we will take the deployment of a Whoami service as an example to introduce the use of Traefik. Whoami is a simple HTTP server that returns the request\u0026rsquo;s IP address, request headers, and other information, making it ideal for testing reverse proxies.\nProject Directory Structure # traefik Project # Based on the above architecture design, we create a new traefik directory under the directory where all Docker projects are stored, to hold the relevant files for the Traefik project. The directory structure is as follows:\n1 2 3 4 5 6 7 8 9 traefik/ ├── docker-compose.yml ├── .env ├── traefik.yml ├── acme.json └── dynamic/ ├── app1.yml ├── app2.yml └── app3.yml where:\ndocker-compose.yml: The Docker Compose configuration file for Traefik. .env: The environment variable file used to store environment variables for the Traefik project. traefik.yml: The main configuration file for Traefik. acme.json: The file used to store SSL certificates. dynamic: The directory for storing dynamic configuration files, with each service having its own configuration file. whoami Project # Since we are taking a simple Whoami service as an example, we need to create a Whoami project in addition to the Traefik project. The Whoami project is very simple and also placed in the directory where all Docker projects are stored, containing only a docker-compose.yml file.\n1 2 whoami/ └── docker-compose.yml Traefik Installation, Configuration, and Testing Process # 1. Create traefik-net Network in Docker # First, we create a public network in Docker called traefik-net (you can use a different name, but it must be consistent in subsequent configurations):\n1 docker network create traefik-net 2. Create the traefik Project Directory # Create a traefik directory under the directory where all Docker projects are stored, and create the above files and subdirectories in that directory.\ndocker-compose.yml # docker-compose.yml如下：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 version: \u0026#34;3.8\u0026#34; services: traefik: image: traefik:v3.4.0 container_name: traefik command: - \u0026#34;--configFile=/etc/traefik/traefik.yml\u0026#34; - \u0026#34;--entryPoints.web.address=:80\u0026#34; - \u0026#34;--entryPoints.websecure.address=:443\u0026#34; - \u0026#34;--providers.docker=true\u0026#34; - \u0026#34;--providers.docker.network=traefik-net\u0026#34; - \u0026#34;--providers.docker.exposedByDefault=false\u0026#34; - \u0026#34;--providers.file.directory=/etc/traefik/dynamic\u0026#34; - \u0026#34;--providers.file.watch=true\u0026#34; - \u0026#34;--certificatesResolvers.le.acme.email=1067848738@qq.com\u0026#34; - \u0026#34;--certificatesResolvers.le.acme.storage=/letsencrypt/acme.json\u0026#34; - \u0026#34;--certificatesResolvers.le.acme.httpChallenge.entryPoint=web\u0026#34; - \u0026#34;--api.dashboard=true\u0026#34; - \u0026#34;--log.level=INFO\u0026#34; ports: - \u0026#34;80:80\u0026#34; - \u0026#34;443:443\u0026#34; - \u0026#34;8080:8080\u0026#34; # comment out if you don’t need the dashboard volumes: - \u0026#34;/var/run/docker.sock:/var/run/docker.sock:ro\u0026#34; - \u0026#34;./traefik.yml:/etc/traefik/traefik.yml:ro\u0026#34; - \u0026#34;./dynamic:/etc/traefik/dynamic:ro\u0026#34; - \u0026#34;./acme.json:/letsencrypt/acme.json\u0026#34; environment: - CF_API_KEY=${CF_API_KEY} - CF_API_EMAIL=${CF_API_EMAIL} networks: - traefik-net restart: unless-stopped networks: traefik-net: external: true where traefik-net is the network we created earlier. The Traefik configuration file traefik.yml, dynamic configuration directory dynamic/, and the file for storing SSL certificates acme.json are mounted into the container.\n.env # In .env, we store the CloudFlare API key and email address used for automatically obtaining SSL certificates. You need to replace the following content with your own Cloudflare API key and email address. In theory, providing just the key should be enough, but I haven\u0026rsquo;t tested it. If you are using a DNS provider other than Cloudflare (for example, Alibaba Cloud), you need to replace the relevant API key and email address with those of your DNS provider. For details, please refer to the Traefik documentation.\n1 2 CF_API_KEY=your_cloudflare_api_key CF_API_EMAIL=your_cloudflare_email traefik.yml # traefik.yml is the main configuration file for Traefik. Below is a basic configuration example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 entryPoints: web: address: \u0026#34;:80\u0026#34; websecure: address: \u0026#34;:443\u0026#34; providers: file: directory: /etc/traefik/dynamic watch: true api: dashboard: true insecure: false certificatesResolvers: le: acme: email: your-email@example.com storage: /letsencrypt/acme.json dnsChallenge: provider: cloudflare resolvers: - \u0026#34;1.1.1.1:53\u0026#34; - \u0026#34;1.0.0.1:53\u0026#34; #log: # level: \u0026#34;DEBUG\u0026#34; Here we specify two entry points: web (HTTP) and websecure (HTTPS), and set the directory for dynamic configuration files to /etc/traefik/dynamic. We also enable the Traefik API dashboard and configure the ACME certificate resolver for Cloudflare DNS Challenge. If you are using a different DNS provider, you can replace provider: cloudflare with the name of your DNS provider and configure the corresponding API key and email address according to its documentation.\nacme.json # acme.json is the file used to store SSL certificates. You need to create this file first (an empty file is sufficient) and set its permissions to 600 to ensure that Traefik can write certificate information:\n1 2 touch acme.json chmod 600 acme.json dynamic Directory # In the dynamic directory, we can create a configuration file for each service. Below is a very simple configuration file for the Whoami service, and this article will use the deployment of a Whoami service as an example to introduce the use of Traefik.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 http: routers: whoami-router: entryPoints: - websecure rule: \u0026#34;Host(`whoami.example.com`)\u0026#34; tls: certResolver: le service: whoami-service services: whoami-service: loadBalancer: servers: - url: \u0026#34;http://whoami:80\u0026#34; 3. Create the whoami Project Directory # Create a whoami directory under the directory where all Docker projects are stored, and create a docker-compose.yml file in that directory. The content is also very simple:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 version: \u0026#39;3.8\u0026#39; services: whoami: image: traefik/whoami:v1.11.0 container_name: whoami restart: always networks: traefik-net: aliases: - whoami networks: traefik-net: external: true This docker-compose.yml file defines a Whoami service that uses the official Whoami image provided by Traefik and connects it to the previously created traefik-net network.\n4. Start the Whoami and Traefik Services # If you have previously deployed an Nginx service on this computer, you need to stop the Nginx service first to avoid port conflicts with Traefik:\n1 sudo systemctl stop nginx To prevent Nginx from starting again after the computer restarts, you can disable the Nginx service:\n1 sudo systemctl disable nginx Enter the whoami directory and use the following command to start the Whoami service:\n1 docker-compose up -d Enter the traefik directory and use the following command to start the Traefik service:\n1 docker-compose up -d Check if the Traefik and Whoami services are running properly:\n1 docker ps You should see that the containers for the Traefik and Whoami services are running.\n5. Test Traefik Reverse Proxy # First, make sure you have resolved your domain name (e.g., whoami.example.com) to your server\u0026rsquo;s IP address with your DNS provider.\nThen, you can test Traefik\u0026rsquo;s reverse proxy functionality by accessing https://whoami.example.com. You need to replace whoami.example.com with your own domain name and ensure that DNS resolution is correctly configured.\nIf everything is working properly, you should be able to access the Whoami service, and you should see a page containing request information, similar to the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Hostname: 5ded297e3bf3 IP: 127.0.0.1 IP: ::1 IP: 192.168.144.2 RemoteAddr: 192.168.144.4:33354 GET / HTTP/1.1 Host: whoami.example.com User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Accept-Encoding: gzip, br Accept-Language: en Cdn-Loop: cloudflare; loops=1 Cf-Connecting-Ip: 2600:1700:37a0:70e0:b8c8:f187:aaa1:1d07 Cf-Ipcountry: US Cf-Ray: 948a2cd22a1958ee-ORD Cf-Visitor: {\u0026#34;scheme\u0026#34;:\u0026#34;https\u0026#34;} Priority: u=0, i Sec-Ch-Ua: \u0026#34;Chromium\u0026#34;;v=\u0026#34;136\u0026#34;, \u0026#34;Google Chrome\u0026#34;;v=\u0026#34;136\u0026#34;, \u0026#34;Not.A/Brand\u0026#34;;v=\u0026#34;99\u0026#34; Sec-Ch-Ua-Mobile: ?0 Sec-Ch-Ua-Platform: \u0026#34;macOS\u0026#34; Sec-Fetch-Dest: document Sec-Fetch-Mode: navigate Sec-Fetch-Site: none Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 X-Forwarded-For: 172.69.17.115 X-Forwarded-Host: whoami.example.com X-Forwarded-Port: 443 X-Forwarded-Proto: https X-Forwarded-Server: bca3e42c8951 X-Real-Ip: 172.69.17.115 If you cannot access https://whoami.example.com, it means there is a problem with the configuration above. You can uncomment the log.level: \u0026quot;DEBUG\u0026quot; line in traefik.yml, so Traefik will output more detailed log information to help you troubleshoot the issue.\nIf you encounter SSL certificate errors when accessing the site, it may be because Traefik has not yet obtained the SSL certificate. You can wait for a while, and Traefik will automatically obtain the SSL certificate and configure HTTPS.\n","date":"31 May 2025","externalUrl":null,"permalink":"/en/p/accessing-personal-websites-from-the-public-network-2-traefik-reverse-proxy-configuration/","section":"Posts","summary":" Motivation # I previously used Nginx as a reverse proxy to access websites on my private server. At that time, Nginx was installed directly on the computer without containerization, which meant that when I wanted to migrate these services to other computers later, I had to reconfigure Nginx and reconfigure SSL certificates, etc.\nSo I decided to containerize the reverse proxy service as well, to facilitate future management and migration. I saw many examples of using Traefik in Docker online, so I decided to migrate from Nginx to Traefik.\n","title":"Accessing Personal Websites from the Public Network (2) - Traefik Reverse Proxy Configuration","type":"post"},{"content":" Introduction # This is the sixth article about Docker containers. The links to other articles in this series are as follows:\nContainer (1): Introduction to Container-Related Knowledge - Containerization, Docker, Docker Compose, Kubernetes / K8s, etc. Container (2): Docker Best Practices Guide - Docker Compose and Portainer Container (3): Docker Best Practices Guide - Managing Data Volumes Container (4): Docker Best Practices Guide - Container Updates, Upgrades, and Migrations Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What\u0026rsquo;s Up Docker) After using Docker for a while, through discussions with ChatGPT, I gradually realized some misconceptions and bad habits I had regarding Docker\u0026rsquo;s operation logic. To prevent others from making the same mistakes, I decided to document these misconceptions and bad habits for reference. I will continue to update this article whenever I encounter new issues.\nMisconceptions # 1. The Relationship Between Docker and Containers # Initially, I thought Docker and containers were the same concept, but in reality, containers are a broader concept, and Docker is just one tool for implementing containerization. It is the emergence of Docker that has made containerization so simple and popular, making it a representative of containerization.\nHowever, Docker is not the only containerization tool; there are other tools and platforms that can also achieve containerization, such as Podman, LXC, rkt, etc. Each has its own advantages and disadvantages and suitable scenarios, but Docker is undoubtedly the most popular and mature containerization tool at present. I had briefly used Podman when compiling Proton from source, but I did not delve into its usage methods and principles, so I will not elaborate on it here. If I use Podman or other containerization tools in depth in the future, I will publish related articles.\n2. Installing Docker # When I first installed Docker, I used the command apt install docker, which installs the version of Docker from the Ubuntu official repository, which may not be the latest version. Moreover, this installation method may lack some features or functionalities because it is maintained by the Ubuntu community rather than Docker\u0026rsquo;s official team.\nTo use the latest Docker version and features, the recommended installation method is to use the Docker official community repository. For details, refer to the official documentation. Generally, the installation process is as follows:\nUbuntu 22.04 # Uninstall old versions of Docker\n1 sudo apt-get remove docker docker-engine docker.io containerd runc Install dependencies\n1 2 3 4 5 6 sudo apt-get update sudo apt-get install \\ ca-certificates \\ curl \\ gnupg \\ lsb-release Add the Docker official community repository\n1 2 3 echo \u0026#34;deb [arch=$(dpkg --print-architecture) ] https://download.docker.com/linux/ubuntu \\ $(lsb_release -cs) \\ stable\u0026#34; | sudo tee /etc/apt/sources.list.d/docker.list \u0026gt; /dev/null Install Docker\n1 2 sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io Fedora 42 # First I followed the installation method described in the Fedora developer documentation for Docker, but after installation, I was unable to start the Docker service. Later, I discovered that it was an issue with iptables. So, I am documenting the entire process here.\nInstall dnf-plugins-core to manage dnf repositories:\n1 sudo dnf install dnf-plugins-core Add the docker-ce repository\n1 sudo dnf config-manager addrepo --from-repofile=\u0026#34;https://download.docker.com/linux/fedora/docker-ce.repo\u0026#34; Note that the above command only applies to systems after Fedora 41. For Fedora 40 and older systems, use the following command:\n1 sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo Install docker\n1 sudo dnf install docker-ce docker-ce-cli containerd.io You can try to start the docker service:\n1 sudo systemctl start docker If everything is normal, you can set it to start on boot:\n1 sudo systemctl enable docker If you encounter an error saying the startup failed, you can check the Docker logs for more information:\n1 sudo journalctl -u docker -n 50 --no-pager The error I encountered was failed to find iptables error=\u0026quot;exec: \\\u0026quot;iptables\\\u0026quot;: executable file not found in $PATH\u0026quot;, which means Docker couldn\u0026rsquo;t find the iptables command. iptables is usually installed with the system, but recent Fedora systems use iptables-nft as a replacement. You can confirm this with the following command:\n1 sudo dnf install iptables It will tell you Package \u0026quot;iptables-nft-1.8.11-8.fc42.x86_64\u0026quot; is already installed., indicating that iptables-nft is already installed. Then we can check where it is installed:\n1 which iptables-nft It will usually tell you it is installed at /usr/bin/iptables-nft. Now we need to link the iptables command to iptables-nft so that Docker can find it:\n1 sudo ln -s /usr/bin/iptables-nft /usr/bin/iptables Similarly, Docker also needs ip6tables, so we need to create a link for ip6tables:\n1 sudo ln -s /usr/bin/iptables-nft /usr/bin/ip6tables Now you should be able to start the docker service:\n1 sudo systemctl start docker And set it to start on boot:\n1 sudo systemctl enable docker 3. docker-compose vs docker compose # In the first article of this series, Container (1): Introduction to Container-Related Knowledge - Containerization, Docker, Docker Compose, Kubernetes / K8s, etc., I mentioned the difference between docker-compose and docker compose. At that time, my recommendation was to use docker-compose, which is an independent command-line tool. This was because I encountered issues when trying to create a container using docker compose, and I could only successfully create it using docker-compose. I don\u0026rsquo;t remember the specific reason for the issue, but I did face a problem.\nLater, I discovered that docker-compose is not maintained by Docker\u0026rsquo;s official team, while docker compose is maintained as a Docker plugin. Additionally, docker compose has a faster update speed and supports more features. Therefore, I now recommend using docker compose instead of docker-compose.\nBad Habits # 1. Docker Data Management # Previously, I did not understand the differences between various Docker data volumes, leading to messy data management. To avoid containers affecting each other when issues arise or to easily clean up unused containers, I created a separate partition for each container, storing all its data in that partition. The advantage of this approach is that it allows for easy cleanup of unused containers by simply deleting the partition. However, the downside is that it wastes a lot of space since each partition requires a certain amount of space for data, even if the actual data usage is minimal. For example, if a container is allocated 20GB but only uses 1GB, 19GB of space is wasted.\nLater, after understanding the different data volumes in Docker (details can be found in the third article of this series, Container (3): Docker Best Practices Guide - Managing Data Volumes), I started using a new approach:\nPlace all Docker configuration files (such as docker-compose.yml and various config.yml files) in a single directory for easy management with git. Store all persistent data required by containers on a large-capacity hard drive or partition, preventing excessive system space usage while avoiding space wastage. 2. Docker Image Management # In the past, while using Docker, I had developed the habit of using docker-compose.yml files to manage containers but habitually used the latest tag to pull the latest images. The downside of this approach is that over time, I would forget which version of the image I had pulled, and if a significant amount of time passed, updating could fail due to skipping multiple versions. Additionally, using the latest tag made it difficult to realize whether the image version had been updated.\nSo later, I started specifying the image version number in the docker-compose.yml file, for example:\n1 2 3 4 version: \u0026#39;3\u0026#39; services: nginx: image: nginx:1.23.3 This way, I can clearly see the current image version number being used. By comparing it with the official newly released image version number, I can easily determine how many versions behind I am. This allows for targeted updates when necessary.\nThe benefit of this approach is that it provides clarity on the current image version being used. By comparing it with the official newly released image version, I can easily see how many versions behind I am. This way, when updating, I can do so in a targeted manner.\nAdditionally, I have recently started using WUD (What\u0026rsquo;s Up Docker) to monitor container updates. For more details, please refer to the fifth article in this series, Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What\u0026rsquo;s Up Docker).\n3. Docker Permissions # When I first started using Docker, I used the version of Docker from the Ubuntu community repository, and I did not add my current user to the docker group during installation. As a result, I had to use the sudo command every time I wanted to run Docker commands. Later, when I started using docker-compose, I still needed to prepend sudo to run it correctly; otherwise, it would prompt for read/write permissions.\nEventually, I realized this issue and created a docker group, adding my current user to the docker group. This allowed me to run Docker commands without needing to use sudo.\n1 2 sudo groupadd docker sudo usermod -aG docker $USER Remember to log out and log back in, or run\n1 newgrp docker After this, I could run docker or docker compose commands without needing to prepend sudo.\n","date":"21 May 2025","externalUrl":null,"permalink":"/en/p/container-6-docker-issues-and-bad-habits/","section":"Posts","summary":" Introduction # This is the sixth article about Docker containers. The links to other articles in this series are as follows:\nContainer (1): Introduction to Container-Related Knowledge - Containerization, Docker, Docker Compose, Kubernetes / K8s, etc. Container (2): Docker Best Practices Guide - Docker Compose and Portainer Container (3): Docker Best Practices Guide - Managing Data Volumes Container (4): Docker Best Practices Guide - Container Updates, Upgrades, and Migrations Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What’s Up Docker) After using Docker for a while, through discussions with ChatGPT, I gradually realized some misconceptions and bad habits I had regarding Docker’s operation logic. To prevent others from making the same mistakes, I decided to document these misconceptions and bad habits for reference. I will continue to update this article whenever I encounter new issues.\n","title":"Container (6): Docker Issues and Bad Habits","type":"post"},{"content":"","date":"17 May 2025","externalUrl":null,"permalink":"/en/tags/authelia/","section":"Tags","summary":"","title":"Authelia","type":"tags"},{"content":"","date":"17 May 2025","externalUrl":null,"permalink":"/en/tags/authentication/","section":"Tags","summary":"","title":"Authentication","type":"tags"},{"content":" Motivation # I previously thought it was unnecessary to set up my own identity authentication service, as I could simply create users and passwords in the services I deployed. However, while preparing to set up a HeadScale (self-hosted version of Tailscale) server, I found that the available UI options for HeadScale do not support direct user and password creation.\nWithout an authentication service, I would have to enter an API key every time, which is a long and irregular string that is nearly impossible to remember. If I use an authentication service, I can utilize many existing services like Google, Apple, Microsoft, GitHub, etc. However, since I am already self-hosting various services, why not also set up my own identity authentication service to keep all data under my control?\nThe setup of identity authentication services can be done in many ways, such as using Authentik, Keycloak, or Authelia. Authentik and Keycloak are more complex and resource-intensive, while I need a simpler service that allows the creation of just a few users. Therefore, I chose Authelia.\nAuthelia Overview # Authelia is an open-source authentication and single sign-on (SSO) solution that supports various authentication methods, including username and password, two-factor authentication (2FA), WebAuthn, etc. It can integrate with multiple authentication backends such as LDAP, Active Directory, MySQL, PostgreSQL, etc. Authelia also supports various authentication protocols like OAuth2, OpenID Connect, SAML, etc.\nWhen using HeadScale, I primarily need to use the OpenID Connect (OIDC) protocol for authentication. Authelia supports the OpenID Connect protocol and can be integrated with HeadScale.\nAuthelia\u0026rsquo;s main features include:\nOpen Source: Authelia is an open-source project that can be freely used and modified. Multiple Authentication Methods: Supports username and password, two-factor authentication (2FA), WebAuthn, and other authentication methods. Multiple Authentication Backends: Supports LDAP, Active Directory, MySQL, PostgreSQL, and other authentication backends. Multiple Authentication Protocols: Supports OAuth2, OpenID Connect, SAML, and other authentication protocols. Single Sign-On (SSO): Supports single sign-on, allowing identity information to be shared across multiple applications. Multiple Authentication Policies: Supports various authentication policies, such as IP-based, time-based, etc. Authelia\u0026rsquo;s limitation is that it currently does not provide a graphical management interface; all configurations must be done through YAML files. However, Authelia\u0026rsquo;s official roadmap plans to introduce a management interface (including both graphical and command-line interfaces) in future versions, as detailed in the Roadmap. We look forward to this feature.\nHow Authelia Works # Authelia is merely an authentication service and does not provide any applications or services itself. Its working principle is to integrate with other applications or services to achieve authentication. When using Authelia, I primarily integrate it with Nginx. Nginx acts as a reverse proxy server, forwarding user requests to Authelia for authentication and then forwarding the requests to the actual application or service.\nTypically, when a user accesses a web application, the process is as follows:\nThat is:\nThe user enters the URL of the web application in the browser and accesses the server via HTTP or HTTPS. The server processes the request with Nginx, which forwards the request to the corresponding application. The application processes the request and returns a response to Nginx. Nginx returns the response to the user\u0026rsquo;s browser. If the web application has built-in authentication, a login window may pop up when the user accesses the application, prompting the user to enter a username and password. If authentication is successful, the application stores the user\u0026rsquo;s identity information in a session and allows access to other parts of the application.\nHowever, some web applications do not have built-in authentication. In such cases, if you want to restrict user access, you can add an authentication service layer. Authelia is such an authentication service. If we add Authelia authentication service to a web application, the user\u0026rsquo;s access process is as follows:\nThat is:\nThe user enters the URL of the web application in the browser and accesses the server via HTTP or HTTPS. The server processes the request with Nginx, which forwards the request to Authelia for authentication. Authelia verifies the user\u0026rsquo;s identity information: If authentication is successful, it forwards the request to Nginx. Nginx then forwards the request to the corresponding application. The application processes the request and returns a response to Nginx. Nginx returns the response to the user\u0026rsquo;s browser. If authentication fails, it returns a 401 Unauthorized response and does not forward the request to the application. The user cannot access the application\u0026rsquo;s services. Deploying Authelia with Docker # We can use docker compose to deploy Authelia. We need to prepare a docker-compose.yml file and two configuration files, with the following structure:\n1 2 3 4 5 6 authelia/ ├── config │ ├── configuration.yml │ └── users.yml ├── .env └── docker-compose.yml where:\ndocker-compose.yml is the Docker Compose configuration file that defines the Authelia service, networks, and volumes. The config directory contains the Authelia configuration file configuration.yml and the user configuration file users.yml. .env is the environment variable file used to define Authelia\u0026rsquo;s environment variables. docker-compose.yml # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 version: \u0026#39;3.8\u0026#39; services: authelia: image: authelia/authelia:4.39.3 container_name: authelia restart: always volumes: - ./config:/config:ro - ${DATA_DIR}/data:/data - /etc/localtime:/etc/localtime:ro - /etc/timezone:/etc/timezone:ro ports: - \u0026#34;9091:9091\u0026#34; networks: - authnet labels: - wud.tag.include=^(\\d+\\.\\d+)\\.\\d+$$ =\u0026gt; $$1 networks: authnet: driver: bridge where ${DATA_DIR}/data is an environment variable that specifies the directory where Authelia will store its data. We can define this variable in a .env file, for example:\n1 DATA_DIR=/path/to/data Configuration Files # Authelia\u0026rsquo;s configuration files include configuration.yml and users.yml. The configuration.yml file contains the main configuration for Authelia, while the users.yml file contains user information. Please refer to the official documentation for detailed configuration options. Below is a simple example of each configuration file.\nconfiguration.yml # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 ############################################################### # Authelia configuration # ############################################################### server: address: tcp://0.0.0.0:9091/ buffers: read: 4096 write: 4096 endpoints: enable_pprof: false enable_expvars: false disable_healthcheck: false # https://www.authelia.com/configuration/miscellaneous/logging/ log: level: info format: text file_path: /data/authelia.log keep_stdout: true # https://www.authelia.com/configuration/second-factor/time-based-one-time-password/ totp: issuer: authelia.com period: 30 skew: 1 # AUTHELIA_DUO_PLACEHOLDER # https://www.authelia.com/reference/guides/passwords/ authentication_backend: password_reset: disable: false refresh_interval: 5m file: path: /config/users.yml password: algorithm: argon2id iterations: 1 salt_length: 16 parallelism: 8 memory: 256 # blocks this much of the RAM # https://www.authelia.com/overview/authorization/access-control/ access_control: default_policy: deny rules: # - domain: # - \u0026#34;*.example.com\u0026#34; # - \u0026#34;example.com\u0026#34; # policy: bypass # networks: # bypass authentication for local networks # - 10.0.0.0/8 # - 192.168.0.0/16 # - 172.16.0.0/12 - domain: - \u0026#34;*.example.com\u0026#34; - \u0026#34;example.com\u0026#34; policy: one_factor # https://www.authelia.com/configuration/session/introduction/ session: name: authelia_session same_site: lax expiration: 7h inactivity: 5m remember_me: 1M cookies: - domain: \u0026#39;example.com\u0026#39; authelia_url: \u0026#39;https://authelia.example.com\u0026#39; default_redirection_url: \u0026#39;https://example.com\u0026#39; # AUTHELIA_REDIS_PLACEHOLDER # https://www.authelia.com/configuration/security/regulation/ regulation: max_retries: 3 find_time: 10m ban_time: 12h # https://www.authelia.com/configuration/storage/introduction/ storage: # For local storage, uncomment lines below and comment out mysql. https://docs.authelia.com/configuration/storage/sqlite.html # This is good for the beginning. If you have a busy site then switch to other databases. encryption_key: \u0026#39;some-secret-key-for-storage\u0026#39; local: path: /data/db.sqlite3 # https://www.authelia.com/configuration/notifications/introduction/ notifier: disable_startup_check: false # For testing purposes, notifications can be sent in a file. Be sure to map the volume in docker-compose. filesystem: filename: /data/notifications.txt # https://www.authelia.com/configuration/identity-validation/introduction/ identity_validation: reset_password: jwt_secret: \u0026#39;some-secret-key-for-reset-password\u0026#39; # https://www.authelia.com/configuration/identity-providers/openid-connect/provider/ identity_providers: oidc: hmac_secret: \u0026#39;hmac-secret-key\u0026#39; jwks: - key_id: \u0026#39;app-name\u0026#39; algorithm: \u0026#39;RS256\u0026#39; use: sig key: | -----BEGIN PRIVATE KEY----- Your private key here -----END PRIVATE KEY----- clients: - client_id: \u0026#39;client-id\u0026#39; client_name: \u0026#39;Description of the client\u0026#39; client_secret: \u0026#39;client-secret\u0026#39; public: false authorization_policy: one_factor redirect_uris: - \u0026#39;https://example.com/oidc/callback\u0026#39; scopes: - openid - profile - email grant_types: - authorization_code response_types: - code token_endpoint_auth_method: client_secret_post Here, identity_providers defines an OIDC identity provider. If other applications need to use OIDC authentication, they can use this provider. We can define multiple OIDC clients in clients, each with a client_id and client_secret for authentication. For detailed explanations of each configuration item, please refer to the comments in the configuration file.\nusers.yml # 1 2 3 4 5 6 7 users: user1: password: \u0026#34;hash-of-user1-password\u0026#34; displayname: \u0026#34;User One\u0026#34; email: user@example.com groups: - admins Here we define an Authelia user with the username user1. Note that the password is a hashed password, not a plaintext password. We can use Authelia\u0026rsquo;s command-line tool to generate a hashed password, for example:\n1 docker run authelia/authelia:4.39.3 authelia crypto hash generate bcrypt --password \u0026#39;your-user1-password\u0026#39; Replace the generated hashed password in the users.yml file. The email address is not important; it can be any valid email address. The groups field indicates the groups to which the user belongs. Authelia supports various groups, such as admins, users, etc. We can define these groups in the configuration file and assign different permissions to each group.\nStarting Authelia # After preparing the above, we can start Authelia. In the authelia directory, execute the following command:\n1 docker compose up -d This command will start the Authelia service in the background and bind it to port 9091. We can access Authelia\u0026rsquo;s web interface at http://localhost:9091.If everything is working correctly, we should see Authelia\u0026rsquo;s login interface.\nEnter the username and password configured above, which are user1 and your-user1-password. Note that here the password is in plaintext, not a hashed password.\nIf the login is successful, we will be redirected to the default redirection URL configured above, which is https://example.com.\nIntegrating Authelia with Other Services # As mentioned above, other applications or services can integrate with Authelia for authentication. Here, we will take a simple Whoami application as an example to demonstrate how to integrate with Authelia.\nWhoami Application # Whoami is a simple web application that returns information about the request, such as the IP address, request headers, request method, etc. We can use Whoami to test Authelia\u0026rsquo;s authentication functionality.\nDeploy Whoami Application\nWe can use Docker Compose to deploy the Whoami application. We need to prepare a docker-compose.yml file with the following content:\n1 2 3 4 5 6 7 8 version: \u0026#39;3.8\u0026#39; services: whoami: image: traefik/whoami:v1.11.0 container_name: whoami restart: always ports: - \u0026#34;2001:80\u0026#34; Start Whoami Application\nIn the whoami directory, execute the following command:\n1 docker compose up -d This command will start the Whoami service in the background and bind it to port 2001. We can access Whoami\u0026rsquo;s web interface at http://localhost:2001.\nIf everything is working correctly, we should see Whoami\u0026rsquo;s interface.\nConfigure Nginx Reverse Proxy\nWe need to configure Nginx as a reverse proxy to forward user requests to Authelia for authentication and then forward the requests to the Whoami application.\nWe need to prepare an Nginx configuration file with the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 upstream \u0026lt;whoami.example.com\u0026gt; { server 127.0.0.1:2001; } server { listen 80; server_name whoami.example.com; return 301 https://whoami.example.com$request_uri; } server { listen 443 ssl; server_name whoami.example.com; # ssl 配置 ssl_certificate \u0026lt;/path/to/fullchain.cer\u0026gt;; ssl_certificate_key \u0026lt;/path/to/whoami.example.com.key\u0026gt;; # Step 1: internal location that calls Authelia for authentication location /authelia { internal; proxy_pass http://localhost:9091/api/verify; proxy_set_header Content-Length \u0026#34;\u0026#34;; proxy_pass_request_body off; proxy_set_header X-Original-URL $scheme://$host$request_uri; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } # Step 2: protect Whoami via Authelia location / { # This sends an internal subrequest to Authelia for verification auth_request /authelia; # Redirect to Authelia portal if unauthorized error_page 401 =302 https://\u0026lt;authelia.example.com\u0026gt;/?rd=https://$host$request_uri; proxy_pass http://whoami.example.com; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Ssl on; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Frame-Options SAMEORIGIN; } } Here, the server_name needs to be replaced with your own domain or IP address. The ssl_certificate and ssl_certificate_key need to be replaced with the paths to your own SSL certificate and private key.\nThen reload the Nginx configuration:\n1 sudo systemctl reload nginx Based on the above Nginx configuration, we can see that the process of using Authelia to authenticate users for Whoami is as follows:\nThe user accesses https://whoami.example.com. Nginx will forward the request to /authelia, which is Authelia for authentication, based on the auth_request directive. This authentication request is then forwarded to Authelia\u0026rsquo;s /api/verify endpoint. Authelia verifies the user\u0026rsquo;s identity. If the verification is successful, it returns a 200 OK response, and Nginx will forward the request to the Whoami application. If the verification fails, it returns a 401 Unauthorized response, and Nginx will redirect the request to Authelia\u0026rsquo;s login page. Testing Authentication # Now we can test the authentication functionality by accessing http://example.com (replace with your own domain or IP address).\nIf everything is working correctly, we should see Authelia\u0026rsquo;s login interface: Enter the username and password, which are user1 and your-user1-password. Note that here the password is in plaintext, not a hashed password.\nIf the login is successful, we will be redirected to the Whoami application interface.\nIf the username or password is incorrect, it will prompt \u0026ldquo;Invalid username or password\u0026rdquo; and return to Authelia\u0026rsquo;s login interface.\nSummary # We have used Authelia to set up a simple authentication service and demonstrated how to integrate it with a simple Whoami application. If you want to integrate with other applications that have already been deployed, you only need to add the corresponding configuration in Nginx.\n","date":"17 May 2025","externalUrl":null,"permalink":"/en/p/authentication-deploying-authelia-for-private-identity-service/","section":"Posts","summary":" Motivation # I previously thought it was unnecessary to set up my own identity authentication service, as I could simply create users and passwords in the services I deployed. However, while preparing to set up a HeadScale (self-hosted version of Tailscale) server, I found that the available UI options for HeadScale do not support direct user and password creation.\n","title":"Authentication: Deploying Authelia for Private Identity Service","type":"post"},{"content":"","date":"17 May 2025","externalUrl":null,"permalink":"/en/tags/identity-service/","section":"Tags","summary":"","title":"Identity Service","type":"tags"},{"content":"","date":"17 May 2025","externalUrl":null,"permalink":"/en/tags/openid-connect/","section":"Tags","summary":"","title":"OpenID Connect","type":"tags"},{"content":"","date":"2025年5月17日","externalUrl":null,"permalink":"/tags/%E8%BA%AB%E4%BB%BD%E9%AA%8C%E8%AF%81/","section":"Tags","summary":"","title":"身份验证","type":"tags"},{"content":" Motivation # This is the second article in the \u0026ldquo;Private Virtual Network VPN\u0026rdquo; series. In the first article, I introduced how to set up a Shadowsocks service on a cloud server to achieve secure internet access from abroad based on my own needs. This article will cover the basic concepts of VPNs and focus on how to use Tailscale to set up a private virtual network. The other articles in this series are as follows:\nPrivate Virtual Network VPN (1): Setting Up a VPN with Shadowsocks on a Cloud Server VPN Basic Concepts # When we talk about VPNs, we often think of accessing websites that are blocked by firewalls or accessing corporate intranets and campus networks. In fact, the use cases for VPNs go far beyond that. VPNs can help us establish a secure tunnel between different networks, enabling data encryption and privacy protection. They can be used for remote work, cross-border access, data transmission, and more.\nFor example, I regularly use multiple computers, including those at home, in the office, and on cloud servers. These computers are located in different regions and networks; for instance, the office computer is on the company\u0026rsquo;s local area network, while the cloud servers are on Alibaba Cloud and Huawei Cloud, and they may even be distributed across different countries and regions. To facilitate file transfer and remote access between these computers, VPNs come into play.\nUse Cases # For me, the main use cases for VPNs are as follows:\nSSH Remote Login: Although it is also possible to log in via SSH using a public IP, the public IP is unstable. Of course, dynamic domain name resolution can be used, but dynamic domain name resolution updates are often not timely, and dynamic IP resolution services may also go down. After using a VPN, all computers are on the same virtual network and can log in via SSH using the internal IP. Remote Desktop: Similarly, after using a VPN, all computers are on the same virtual network and can log in via remote desktop using the internal IP. File Transfer: After using a VPN, all computers are on the same virtual network and can transfer files using the internal IP. Accessing Network Services on Other Computers: This can be divided into several scenarios: For example, when I am abroad, NetEase Cloud Music will determine that I am abroad based on my IP address, and thus I cannot listen to certain region-restricted music. After using a VPN, I can proxy the requests from NetEase Cloud Music to a computer located in China, pretending to be in China, and I can listen to music normally. For example, when I am in China, I cannot access websites such as Google and GitHub. After using a VPN, I can proxy the requests from Google and GitHub to a computer located abroad, pretending to be abroad, and I can access them normally. VPN Principles # The principle of VPN is to establish a virtual private network over the public internet, connecting different networks together. VPN uses encryption technology to encrypt data, ensuring the security and privacy of the data. The working principle of VPN can be divided into the following steps:\nEstablishing Connection: The client establishes a connection with the VPN server through VPN client software. This process is usually encrypted using protocols such as SSL/TLS. Authentication: The client sends authentication information, such as a username and password, to the VPN server. The VPN server verifies the identity to ensure the client is a legitimate user. Establishing Tunnel: After successful authentication, an encrypted tunnel is established between the VPN server and the client. This tunnel can be based on protocols such as IPsec, L2TP, or PPTP. Data Transmission: Data between the client and the VPN server is transmitted through the encrypted tunnel. The data is encrypted during transmission to ensure its security and privacy. Disconnecting: When the client no longer needs to use the VPN, it can disconnect from the VPN server through the VPN client software. At this point, the VPN tunnel will also be closed. Tailscale # Tailscale Overview # Tailscale is a VPN solution based on WireGuard that helps us quickly set up a private virtual network. Tailscale is characterized by its ease of use, cross-platform support, and automatic NAT traversal. It can be used on multiple platforms, including Windows, macOS, Linux, iOS, and Android.\nWireGuard is a modern VPN protocol that uses the latest encryption technology, offering high performance and low latency. The design goals of WireGuard are simplicity, security, and efficiency. Its codebase is very small, consisting of only a few thousand lines of code, making it much smaller than other VPN protocols (such as OpenVPN, IPsec, etc.). This makes WireGuard easier to audit and maintain.\nTailscale Working Principle # Tailscale\u0026rsquo;s working principle is to manage all clients through a centralized control server. Each client, when connecting to Tailscale, registers its IP address and public key with the control server. The control server distributes this information to other clients, enabling peer-to-peer connections.\nTailscale uses the WireGuard protocol to establish encrypted tunnels. Each client has a WireGuard private key and public key. Through the control server, clients can obtain the public keys of other clients, allowing them to establish peer-to-peer connections.\nTailscale also supports automatic NAT traversal and firewall penetration, making it usable in various network environments. It employs technologies such as STUN and TURN to achieve NAT traversal.\nTailscale also supports access control and authentication. It can integrate with identity providers such as Google, Microsoft, and GitHub to achieve single sign-on (SSO). This allows Tailscale to seamlessly integrate with existing identity management systems.\nTailscale Advantages and Disadvantages # Advantages of Tailscale:\nEasy to use: Tailscale\u0026rsquo;s installation and configuration are very simple, allowing you to set up a private virtual network in just a few minutes. Cross-platform support: Tailscale supports multiple platforms, including Windows, macOS, Linux, iOS, and Android. Automatic NAT traversal: Tailscale can automatically traverse NAT and firewalls without manual port forwarding configuration. High performance: Tailscale uses the WireGuard protocol, offering high performance and low latency. Security: Tailscale employs the latest encryption technologies to ensure data security and privacy. Access control: Tailscale supports access control and authentication, allowing integration with existing identity management systems. Low cost: Tailscale\u0026rsquo;s free version supports up to 20 devices, making it suitable for individuals and small teams. Open source: The core components of Tailscale are open source and can be found on GitHub. Community support: Tailscale has an active community that can provide technical support and assistance. Comprehensive documentation: Tailscale\u0026rsquo;s documentation is very detailed, providing a wealth of use cases and tutorials. Monitoring and logging: Tailscale provides monitoring and logging features to help troubleshoot issues and optimize performance. Disadvantages of Tailscale:\nCentralization: Tailscale uses a centralized control server to manage clients, which may lead to single points of failure and privacy concerns. Network dependency: Tailscale relies on network connectivity, and if the network is unstable, it may affect VPN performance. Tailscale Usage Methods # Tailscale has two usage methods: one is to use Tailscale\u0026rsquo;s official control server, and the other is to self-host a control server. For most users, using the official control server is sufficient. However, if you have higher requirements for privacy and security, or if you have special needs that require greater flexibility, you will need to self-host a control server.\nHere, we will briefly introduce the first usage method, which is to use Tailscale\u0026rsquo;s official control server. The second usage method, self-hosting a control server, will not be covered in detail here, but a separate article will be written to introduce it.\nRegister an account: Visit the Tailscale website to register an account. You can log in using your Google, Microsoft, GitHub, or other accounts.\nInstall Tailscale: Install Tailscale on the devices that need to connect. You can use the following command to install:\nWindows: Download the Windows installer and install it.\nmacOS: Download the macOS installer and install it.\nLinux: Use the following command to install:\n1 curl -fsSL https://tailscale.com/install.sh | sh iOS and Android: Search for \u0026ldquo;Tailscale\u0026rdquo; in the App Store or Google Play and install it.\nLog in: After installation, Tailscale will open a browser window prompting you to log in using your registered account or a third-party account like Google, Microsoft, or GitHub.\nConnect: Once logged in, Tailscale will automatically assign an IP address to the device. You can view the list of devices and their IP addresses in the Tailscale app:\nAccess other devices: Devices on the same Tailscale network can be accessed directly via their IP addresses. For example, you can use the ssh command to log in to the Fedora device mentioned above:\n1 ssh user@100.65.0.100 You can also access other devices via the DNS names provided by Tailscale, for example:\n1 ssh user@fedora.abcdefgh.ts.net Set up access control: You can set up access control rules in the Tailscale app, such as restricting access between certain devices.\nDisconnect: When you no longer need to use Tailscale, you can disconnect in the Tailscale app.\n","date":"4 May 2025","externalUrl":null,"permalink":"/en/p/private-virtual-network-vpn-2-basic-concepts-and-tailscale-setup-guide/","section":"Posts","summary":" Motivation # This is the second article in the “Private Virtual Network VPN” series. In the first article, I introduced how to set up a Shadowsocks service on a cloud server to achieve secure internet access from abroad based on my own needs. This article will cover the basic concepts of VPNs and focus on how to use Tailscale to set up a private virtual network. The other articles in this series are as follows:\n","title":"Private Virtual Network VPN (2): Basic Concepts and Tailscale Setup Guide","type":"post"},{"content":" Motivation # I used to manage my notes with Microsoft\u0026rsquo;s OneNote. OneNote is powerful, capable of tracking note revision history, and can sync notes using OneDrive. However, it has many drawbacks. The most unbearable one for me is its default font, Calibri, which cannot be changed (or even if changed, it never takes effect). I really don\u0026rsquo;t understand why such a simple thing as changing the default font has not been resolved for so many years. I have to tedious change the font to Times New Roman every time.\nAfter starting to self-host various services, I began to look for a self-hosted note management software. After trying Joplin, Trilium, SiYuan, Logseq, and other software, I finally chose Obsidian. The advantages of Obsidian are:\nSimple and clean interface, powerful features, and support for Markdown syntax Plugin support for extended functionality Cloud sync support, allowing the use of Nextcloud to sync notes Multi-platform support, available on Windows, Linux, Mac, Android, and more Note files are stored as plain text files, making backup and migration easy Support for custom themes and styles, allowing users to modify the interface according to their preferences These basically meet all my requirements for note management software.\nThis is the first article in the Obsidian series, mainly introducing the installation and configuration of Obsidian, as well as how to use Nextcloud to sync Obsidian notes. Other articles in this series include:\nObsidian (2): Deployment and Use of Web-based Obsidian Prerequisites # Docker and Docker Compose Nextcloud For installation and preparation of the above software and tools, please refer to \u0026ldquo;Using Nextcloud and Docker to Build a Personal Cloud Storage\u0026rdquo;\nObsidian # Obsidian Overview # Obsidian\u0026rsquo;s official description is that it is a Markdown editor and a knowledge base software. I don\u0026rsquo;t really need the features of a Markdown editor, and I basically just use it as a note-taking software. Obsidian\u0026rsquo;s note files are stored in Markdown format, and all note files are stored in a folder, with the file name being the title of the note. Obsidian supports bidirectional links, making it easy to establish links between different notes.\nObsidian supports cloud synchronization, and you can register for an Obsidian account and subscribe to the Obsidian sync service, starting at $4 per month. Since I have already self-hosted Nextcloud, I don\u0026rsquo;t need to spend this money and can directly use Nextcloud to sync Obsidian notes.\nObsidian Installation # Obsidian supports Windows, Linux, Mac, and Android platforms, and also provides a browser plugin for use in the browser. You can directly go to the Obsidian official website to download the installation package for your platform.\nObsidian Configuration # Create a Vault # After installation, open Obsidian, and it will prompt you to choose the location and name for your notes. We select a folder to store the notes, such as /home/user/Documents/Obsidian, and then click \u0026ldquo;Create New Vault\u0026rdquo; to create a new vault. My vault name here is \u0026ldquo;Jin Li Obsidian\u0026rdquo;:\nAfter creation, Obsidian will place all the notes we create into this folder. We can create new notes in this folder or copy existing notes into this folder.\nInstall and Configure Plugins # Since we want to use Nextcloud to sync Obsidian notes, we need to install a plugin to achieve this functionality. Obsidian\u0026rsquo;s plugins are divided into two categories:\nCore Plugins: Built-in plugins that can be enabled directly Community Plugins: Third-party developed plugins that need to be installed manually The plugin we need to install is \u0026ldquo;Remotely Save,\u0026rdquo; which is a community plugin. Community plugins are disabled by default in Obsidian, so we need to enable it manually first.\nClick the settings button in the lower left corner to open the settings interface. Click \u0026ldquo;Community Plugins,\u0026rdquo; then enable it, and you can search for and install community plugins. Here we search for the \u0026ldquo;Remotely Save\u0026rdquo; plugin: Click in and there will be an installation option. After the installation is complete, click enable.\nAfter successful installation, you will see the \u0026ldquo;Remotely Save\u0026rdquo; plugin in the \u0026ldquo;Installed Plugins\u0026rdquo; section: Click the gear icon to enter the settings interface:\nThe main options that need to be set are:\nChoose A Remote Service: Select \u0026ldquo;Webdav\u0026rdquo;\nServer Address: Fill in the WebDAV address of Nextcloud, such as https://\u0026lt;your-nextcloud-domain\u0026gt;/remote.php/dav/files/\u0026lt;username\u0026gt;/Obsidian/. You can log in to Nextcloud, open the folder where you want to store Obsidian notes, click the gear icon in the lower left corner to open \u0026ldquo;File Settings,\u0026rdquo; and then you can see the WebDAV address in \u0026ldquo;WebDAV\u0026rdquo;:\nCopy this address and paste it into Obsidian\u0026rsquo;s \u0026ldquo;Server Address.\u0026rdquo;\nUsername: Fill in the Nextcloud username\nPassword: Fill in the Nextcloud password\nAfter the settings are complete, you can sync the notes created in Obsidian to Nextcloud. We can create a new note in Obsidian and then click the \u0026ldquo;Remotely Save\u0026rdquo; plugin icon to save the note to Nextcloud: After clicking, a prompt will pop up in the upper right corner indicating that the sync was successful, and the status will also be displayed in the lower right corner.\nLog in to Nextcloud, open the folder where you store Obsidian notes, and you will see the notes you created in Obsidian: Multi-Device Sync # Obsidian is a cross-platform note-taking software, and you can also use it on other computers or mobile devices. When installing on other devices, you need to pay attention to the following points:\nAfter installing Obsidian on other devices, the first time you open it, you will be prompted to create a vault. The name of the vault you enter here needs to be the same as before, for example, \u0026ldquo;Jin Li Obsidian.\u0026rdquo;\nInstall and enable the \u0026ldquo;Remotely Save\u0026rdquo; plugin on other devices as well.\nWhen configuring the \u0026ldquo;Remotely Save\u0026rdquo; plugin on other devices, the WebDAV address, username, and password for Nextcloud need to be the same as before. If you don\u0026rsquo;t want to enter them one by one, the \u0026ldquo;Remotely Save\u0026rdquo; plugin provides the ability to export and import configurations. You can configure it on one device and then export the configuration file:\nThe exported configuration is a string that starts with \u0026ldquo;obsidian://remotely-save\u0026rdquo;. You can copy it, then open Obsidian on other devices, click the settings button for the \u0026ldquo;Remotely Save\u0026rdquo; plugin, click \u0026ldquo;Import Settings\u0026rdquo;, paste the copied string, and click confirm to import the configuration:\nFinally, close the plugin settings interface, return to Obsidian, and click the \u0026ldquo;Remotely Save\u0026rdquo; plugin icon in the left sidebar. Obsidian will sync the notes from Nextcloud to the new device.\n","date":"1 May 2025","externalUrl":null,"permalink":"/en/p/obsidian-1-all-platform-notes-management-software/","section":"Posts","summary":" Motivation # I used to manage my notes with Microsoft’s OneNote. OneNote is powerful, capable of tracking note revision history, and can sync notes using OneDrive. However, it has many drawbacks. The most unbearable one for me is its default font, Calibri, which cannot be changed (or even if changed, it never takes effect). I really don’t understand why such a simple thing as changing the default font has not been resolved for so many years. I have to tedious change the font to Times New Roman every time.\n","title":"Obsidian (1): All-platform Notes Management Software","type":"post"},{"content":" Background # This is the fifth article introducing Docker containers. Links to other articles in this series are as follows:\nContainer (1): Introduction to Container-related Knowledge - Containerization, Docker, Docker Compose, Kubernetes / K8s, etc. Container (2): Docker Best Practices Guide - Docker Compose and Portainer Container (3): Docker Best Practices Guide - Volume Management Container (4): Docker Best Practices Guide - Container Update, Upgrade, and Migration Container (6): Misconceptions, Bad Habits, and Issues When Using Docker The previous article mentioned how to update or upgrade containers. When I was updating Docker containers, I found that many of them had not been updated for two or three years, and the version gap was too large, which led to some containers being unable to update directly. Some containers, on the other hand, had not been updated much and could be updated directly. Since the update frequency of each container is different, and there are many containers created, I don\u0026rsquo;t want to manually check whether the container has updates available every few days. So I thought of a question: Is there a tool that can automatically monitor container updates?\nOf course, there are tools like WatchTower, but WatchTower is an automatic update tool for containers, and I don\u0026rsquo;t want it to update automatically. I just want to know if there are updates available for the container, so I haven\u0026rsquo;t used it. Recently, I found a tool called WUD (What\u0026rsquo;s Up Docker) that can monitor container updates and is easy to use, so I plan to give it a try.\nPrerequisites # Docker and docker-compose must be installed Understand the basic concepts and usage of Docker Use docker-compose to manage containers (if you do not use docker-compose to manage containers, or do not want to use docker-compose to manage containers, then this article is for reference only) WUD (What\u0026rsquo;s Up Docker) Introduction # WUD (What\u0026rsquo;s Up Docker) is an open-source Docker container update monitoring tool that can monitor whether container images on Docker Hub have updates and can notify users via email, Slack, and other methods.\nWUD Installation and Usage # Install WUD # The installation of WUD is very simple, just follow the instructions in the official documentation. Here we use docker-compose to install WUD, the docker-compose.yml file is as follows:\n1 2 3 4 5 6 7 8 services: whatsupdocker: image: getwud/wud container_name: wud volumes: - /var/run/docker.sock:/var/run/docker.sock ports: - 3000:3000 After starting, you can access it in your browser at http://localhost:3000.\nCreate Admin Account # However, the WUD installed using the above method does not have an admin account. It is fine for local access, but if you want to expose the service to the public network, you need to create an admin account. WUD provides two methods to create an admin account:\nPass the username and password to the WUD container using environment variables Use OpenidConnect (OIDC) to create an admin account For details, see the official documentation.\nThe first method is very simple and convenient, so I used it directly. We just need to add the following environment variables to the docker-compose.yml file:\n1 2 3 4 5 6 7 8 9 10 11 services: whatsupdocker: image: getwud/wud container_name: wud volumes: - /var/run/docker.sock:/var/run/docker.sock ports: - 3001:3000 environment: - WUD_AUTH_BASIC_ADMIN_USER=your_admin_name - WUD_AUTH_BASIC_ADMIN_HASH=your_admin_password_hash The password is stored in hash form, and we can use the htpasswd command to generate the password hash:\n1 htpasswd -nib admin doe If you do not have the htpasswd command installed, you can generate the password hash on this website.\nIt is important to note that the generated hash value starts with the username, and we only need to take the part after the colon. The part after the colon may contain the $ symbol, so we need to replace the $ symbol with $$. For example, if the original hash value is $apr1$2c4a3d5e$e0f8b7c6d7f8b7c6d7f8b7c6, we need to replace it with $$apr1$$2c4a3d5e$$e0f8b7c6d7f8b7c6d7f8b7c6.\nAfter restarting the WUD container and accessing it in the browser, you can see the following login interface: Enter your username and password to log in. Note that you need to enter the password, not the hash value.\nWUD Monitoring Container Updates # Through the web interface, you can monitor the update status of the containers. After logging in, you can see the following interface:\nIt can be seen that WUD has detected 37 containers, of which 10 have updated versions available. Click in to view details:\nYou can see the current version number of the container as well as the latest version number, and we can decide whether to update the container based on our needs.\nSet WUD Monitoring Rules # In WUD, the container status we see may look like this:\nThere are two issues here:\nThe tags used by our containers are in numeric form, but WUD monitors all tags, including some irregular tags. For example, the latest tag of authelia in the figure is feat-i18n-lang-attr. If we only want to monitor tags in numeric form, we need to set monitoring rules. When we deploy a service, we may use multiple containers to deploy this service. For example, the chevereto service requires two containers to deploy: one is chevereto-free, and the other is chevereto-free-database. We do not need to monitor all containers; we only need to monitor the chevereto-free container. Otherwise, if we detect that chevereto-free-database has an update available, while chevereto-free does not, and we only update chevereto-free-database, it may lead to compatibility issues. We also need to set monitoring rules to avoid this situation. Include Certain Tags # For the first case, we can add wud.tag.include to the docker-compose.yml file of the corresponding container to tell WUD which tags we only want to monitor. For example, if we only want to monitor tags in numeric form like xx.yy.zz, we can add the following configuration:\n1 2 3 4 5 services: some_service: image: some_image labels: - wud.tag.include=^\\d+\\.\\d+\\.\\d+$$ This method uses a regular expression to match the tags, where ^ indicates the start, \\d indicates a digit, + indicates one or more, \\. indicates a dot, and $$ indicates the end. This way, we can match tags in numeric form.\nIn tags in the form of xx.yy.zz, generally xx represents the major version number, yy represents the minor version number, and zz represents the patch number. We can set monitoring rules based on our needs. For example, if we only want to monitor updates to the major and minor version numbers and ignore the patch number, we can change the regular expression to:\n1 2 3 4 5 services: some_service: image: some_image labels: - wud.tag.include=^(\\d+\\.\\d+)\\.\\d+$$ =\u0026gt; $$1 In this case, we use parentheses to group xx.yy together and specify that we only want to monitor updates to xx.yy after the =\u0026gt;. This way, we can monitor updates to the major and minor version numbers only.\nMonitor Only Certain Containers # For the second case, we can add wud.watch to the docker-compose.yml file of the corresponding container to tell WUD whether we want to monitor this container. For example, if we do not want to monitor the chevereto-free-database container, we can add the following configuration:\n1 2 3 4 5 6 7 services: chevereto-free-app: image: chevereto-free chevereto-free-database: image: chevereto-free-database labels: - wud.watch=false This way, we can avoid monitoring the chevereto-free-database container.\n","date":"26 April 2025","externalUrl":null,"permalink":"/en/p/container-5-docker-best-practices-guide-container-update-monitoring-tool-wud-whats-up-docker/","section":"Posts","summary":" Background # This is the fifth article introducing Docker containers. Links to other articles in this series are as follows:\nContainer (1): Introduction to Container-related Knowledge - Containerization, Docker, Docker Compose, Kubernetes / K8s, etc. Container (2): Docker Best Practices Guide - Docker Compose and Portainer Container (3): Docker Best Practices Guide - Volume Management Container (4): Docker Best Practices Guide - Container Update, Upgrade, and Migration Container (6): Misconceptions, Bad Habits, and Issues When Using Docker The previous article mentioned how to update or upgrade containers. When I was updating Docker containers, I found that many of them had not been updated for two or three years, and the version gap was too large, which led to some containers being unable to update directly. Some containers, on the other hand, had not been updated much and could be updated directly. Since the update frequency of each container is different, and there are many containers created, I don’t want to manually check whether the container has updates available every few days. So I thought of a question: Is there a tool that can automatically monitor container updates?\n","title":"Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What's Up Docker)","type":"post"},{"content":"","date":"26 April 2025","externalUrl":null,"permalink":"/en/tags/container-upgrade/","section":"Tags","summary":"","title":"Container Upgrade","type":"tags"},{"content":"","date":"26 April 2025","externalUrl":null,"permalink":"/en/tags/docker-compose/","section":"Tags","summary":"","title":"Docker-Compose","type":"tags"},{"content":"","date":"26 April 2025","externalUrl":null,"permalink":"/en/tags/wud/","section":"Tags","summary":"","title":"WUD","type":"tags"},{"content":"","date":"2025年4月26日","externalUrl":null,"permalink":"/tags/%E5%AE%B9%E5%99%A8/","section":"Tags","summary":"","title":"容器","type":"tags"},{"content":"","date":"2025年4月26日","externalUrl":null,"permalink":"/tags/%E5%AE%B9%E5%99%A8%E5%8D%87%E7%BA%A7/","section":"Tags","summary":"","title":"容器升级","type":"tags"},{"content":" Background # This is the fourth article introducing Docker containers. Links to other articles in this series are as follows:\nContainer (1): Introduction to Container-related Knowledge - Containerization, Docker, Docker Compose, Kubernetes / K8s, etc. Container (2): Docker Best Practices Guide - Docker Compose and Portainer Container (3): Docker Best Practices Guide - Volume Management Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What\u0026rsquo;s Up Docker) Container (6): Misconceptions, Bad Habits, and Issues When Using Docker After creating a container, we may need to update or upgrade it to use new features or fix bugs. If we change servers or upgrade the hardware of our computer, we may need to migrate the container to the new server or to a new hard drive.\nDisclaimer: If the official documentation of the container has instructions on updating, upgrading, or migrating, please refer to the official documentation first. This article only introduces general container updates, upgrades, and migrations.\nPrerequisites # Docker and docker-compose must be installed Understand the basic concepts and usage of Docker Use docker-compose to manage containers (if you do not use docker-compose to manage containers, or do not want to use docker-compose to manage containers, then this article is for reference only) Changing Containers # Whether it is updating, upgrading, or migrating a container, the essence is to make some changes to the original container. Here we need to clarify which part of the container has been changed. Generally speaking, changes to the container involve the following parts:\nImage: Containers are created from images. If you want to use new features of the container, fix bugs, or revert to old features of the container, you need to update or replace the image. This is usually the most common and convenient change. Because ordinary users are usually users of the image, not creators of the image. We only need to pull the new image from the repository or delete the old image without worrying too much about other things.\nContainer Configuration: The configuration of the container refers to the runtime parameters of the container, such as environment variables, port mappings, data volume mounts, etc. Generally speaking:\nWhen updating or upgrading a container, we need to consult the official documentation or release notes to see if new configuration items have been added or old configuration items have been modified. We need to modify the container\u0026rsquo;s configuration according to the official documentation or release notes. Otherwise, we do not need to modify the container\u0026rsquo;s configuration when updating or upgrading the container. When migrating a container, we need to modify the container\u0026rsquo;s configuration according to the environment of the new server, such as modifying the mount path of the data volume, port mappings, etc. Data: The data here is divided into two categories\nSoftware Configuration Data: The software configuration here refers to the configuration of the software running in the container, such as database configuration, web service configuration, etc. Generally speaking, the software configuration is stored in the data volume specially created for the container.\nWhen updating or upgrading a container, we often do not need to do anything. The upgraded container will inherit the previous software configuration. When migrating a container, we need to migrate the data volume that stores the software configuration to the new server or new hard drive. User Data: User data refers to the data uploaded by users to the container, such as user-uploaded images, videos, etc., as well as account and password data of the software users themselves. Generally speaking, user data is stored in the data volume in the form of a database or folder. When updating or migrating a container, the most concerned are these data.\nBasic Process of Updating and Upgrading Containers # The basic process of updating and upgrading containers is as follows:\nDetermine Version # Before updating or upgrading a container, we need to determine which version to update or upgrade to. Generally speaking, the version number of the container is determined by the tag of the image. We can use the following command to view the current version number of the container:\n1 docker inspect \u0026lt;container_name\u0026gt; | grep Image or\n1 docker inspect \u0026lt;container_name\u0026gt; | grep RepoTags or view the container\u0026rsquo;s details in Portainer to find the image\u0026rsquo;s tag information.\nIt is important to note that many containers are often created with the latest tag, which refers to the latest version. Many tutorials online also use the latest tag for images. I used to create containers this way, but later found that it is not a good practice. If a container is not updated for a long time, there may be a significant gap between our container and the latest version in the repository. When updating or upgrading the container, we may encounter some issues. Moreover, we cannot intuitively see the gap between our container and the latest version. It is also difficult to mark the version when backing up and migrating containers.\nTherefore, I recommend using specific version numbers when creating containers, rather than the latest tag. This way, when we want to update the container, we can check the official repository for the latest version number. If the gap between the old and new version numbers is not large, we can update directly. If the gap is large, we may need to update one version at a time.\nUpdating and Upgrading Containers # When using docker-compose to manage containers, updating or upgrading a container generally involves three steps:\nStop the Container\n1 sudo docker-compose stop or\n1 sudo docker-compose down Modify the docker-compose.yml file\nChange the image tag to the new version number Modify other configuration items (if needed) Recreate the Container\n1 sudo docker-compose up -d This command will automatically pull the new image and recreate the container when it detects that we have updated the image version number.\nIf the current container version number and the latest version number differ significantly, we may need to update one version at a time. For example, after creating the GitLab container, I didn\u0026rsquo;t update it for nearly two years, resulting in a gap of two major versions and more than a dozen minor versions from the latest version. If I directly update to the latest version at this point, the container may fail to start due to compatibility issues. GitLab actually handles this quite well; if you check the logs, you\u0026rsquo;ll find it will prompt you:\n1 2 3 4 5 It seems you are upgrading from 16.0.4-ee to 16.7.0. It is required to upgrade to the latest 16.3.x version first before proceeding. Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/#upgrade-paths Thank you for using GitLab Docker Image! Current version: gitlab-ee=16.7.0-ee.0 This indicates that we cannot directly upgrade from 16.0.4 to 16.7.0, but need to first upgrade to 16.3.x. However, in my actual testing, I found that it seems we also cannot directly upgrade from 16.0.4 to 16.3.0, and can only upgrade to 16.1.0 first. In short, it took me nearly two hours to gradually update more than a dozen versions before I could update GitLab to the latest version.\nMigrating Containers # Migrating containers is a bit more complex, mainly because we need to migrate the data volumes to the new computer or new hard drive, and also ensure that the container can correctly read the data in the data volumes.\nDetermine Container Image Version and Data Volume Location # Before migrating a container, we need to determine the image version of the container to avoid compatibility issues. The issue of image version numbers has been mentioned earlier; when creating containers, we should try to use specific version numbers instead of the latest tag, so that we can clearly know the version number of the container during migration. If we don\u0026rsquo;t know, we can use the commands mentioned above to check.\nThe location of the data volumes can be found in the docker-compose.yml file, and there are several situations:\nNamed Volumes: The name of a named volume is unique and can be accessed by name. The data for named volumes is stored in the host\u0026rsquo;s /var/lib/docker/volumes directory.\nBind Mounts: The location of a bind mount data volume is the path defined before the colon in the definition. If it is a read-only bind mount, we generally do not need to worry about it during migration.\nTemporary Volumes: The data for temporary volumes is stored in the host\u0026rsquo;s /var/lib/docker/containers directory. The data for temporary volumes will be deleted when the container is stopped or removed, so we generally do not need to worry about it during migration.\nAnonymous Volumes: We should avoid using anonymous volumes. If we must use them, their data will be deleted when the container is stopped or removed, and we generally do not need to worry about it.\nMigrating Data Volumes # Migrating data volumes is different from regular file migration, mainly because we need to pay attention to the permission issues of the data volumes. If the permissions of certain files change after migration, it may cause the container to be unable to access the files in the data volumes. We can use the rsync command to migrate data volumes, and the specific command is as follows:\n1 rsync -aHAX --progress /path/to/source/ /path/to/destination/ The rsync command is a tool for file synchronization and backup, which can efficiently copy and synchronize files between local or remote computers. It supports incremental backups, compressed transfers, and preserving file permissions, making it very suitable for migrating data volumes. Among them, -a means to copy files in archive mode, preserving file permissions, timestamps, and other information; -H means to preserve hard links; -A means to preserve ACL permissions; -X means to preserve extended attributes; --progress means to display the copy progress.\nAdditionally:\nIf you encounter issues with certain files not being copied when using the above command, you can use sudo to elevate permissions, for example:\n1 sudo rsync -aHAX --progress /path/to/source/ /path/to/destination/ If the destination is another local directory that already has some files, rsync will automatically skip those files and only copy the files that do not exist in the source directory. If there are some files in the destination directory that do not exist in the source directory and you want to delete them, you can add the --delete option.\n1 rsync -aHAX --progress --delete /path/to/source/ /path/to/destination/ If the source and destination files are not on the same machine, you can use the address of the remote computer to copy, for example:\n1 rsync -aHAX --progress /path/to/source/ user@remote:/path/to/destination/ If the remote computer is not using port 22, you can use the -e option to specify the ssh port, for example:\n1 rsync -aHAX --progress -e \u0026#34;ssh -p 2222\u0026#34; /path/to/source/ user@remote:/path/to/destination/ If the remote computer uses ssh key login, you can use the -e option to specify the ssh key, for example:\n1 rsync -aHAX --progress -e \u0026#34;ssh -i /path/to/key\u0026#34; /path/to/source/ user@remote:/path/to/destination/ Migrating docker-compose.yml File # The reason for migrating the docker-compose.yml file after migrating the data is that the data volume paths on the new computer or new hard drive may change. We need to modify the data volume paths in the docker-compose.yml file after migrating the data.\nThis step is actually very simple, just don\u0026rsquo;t forget to modify the data volume paths.\nRecreating Containers # After migrating the data and the docker-compose.yml file, we can use the following command to recreate the containers:\n1 sudo docker-compose up -d If this is the first time starting the container, or if you need to rebuild the container, you can use:\n1 sudo docker-compose up -d --build If all goes well, the container migration will be successful!\n","date":"24 April 2025","externalUrl":null,"permalink":"/en/p/container-4-docker-best-practices-guide-container-update-upgrade-and-migration/","section":"Posts","summary":" Background # This is the fourth article introducing Docker containers. Links to other articles in this series are as follows:\nContainer (1): Introduction to Container-related Knowledge - Containerization, Docker, Docker Compose, Kubernetes / K8s, etc. Container (2): Docker Best Practices Guide - Docker Compose and Portainer Container (3): Docker Best Practices Guide - Volume Management Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What’s Up Docker) Container (6): Misconceptions, Bad Habits, and Issues When Using Docker After creating a container, we may need to update or upgrade it to use new features or fix bugs. If we change servers or upgrade the hardware of our computer, we may need to migrate the container to the new server or to a new hard drive.\n","title":"Container (4): Docker Best Practices Guide - Container Update, Upgrade, and Migration","type":"post"},{"content":"","date":"24 April 2025","externalUrl":null,"permalink":"/en/tags/container-migration/","section":"Tags","summary":"","title":"Container Migration","type":"tags"},{"content":"","date":"2025年4月24日","externalUrl":null,"permalink":"/tags/%E5%AE%B9%E5%99%A8%E8%BF%81%E7%A7%BB/","section":"Tags","summary":"","title":"容器迁移","type":"tags"},{"content":" Motivation # This is the third article in the Docker series. The links to the other articles in this series are as follows:\nContainer (1): Introduction to Container-related Knowledge——Containerization, Docker, Docker-compose, Kubernetes / K8s, etc. Container (2): The Best Practice Guide for Docker——docker-compose and Portainer Container (4): The Best Practice Guide for Docker——Container Update, Upgrade, and Migration Container (5): The Best Practice Guide for Docker——Container Update Monitoring Tool WUD (What\u0026rsquo;s Up Docker) Container (6): Misconceptions, Bad Habits, and Issues When Using Docker In the previous two articles, we introduced the basic concepts of containers and how to use them, as well as the best practices for using containers.\nIn the previous articles, we did not focus on the management of data volumes, but managing data volumes is actually a very important part of using and managing containers. Generally speaking, the data in a container is temporary; when the container stops or is deleted, the data in the container will also be deleted. This mechanism makes containers very lightweight, and users do not have to worry about the data in the container occupying too much storage space.\nHowever, when using containers to deploy certain services, we may need to persist some data in the container so that it does not get lost even if the container stops or is deleted. For example, in the article \u0026ldquo;Setting Up a Private Image Hosting Service with Chevereto-free\u0026rdquo;, we introduced how to set up a private image hosting service, where we need to persist the image data in Chevereto to the host machine so that these images can still be accessed after the container stops or is deleted. Or when we migrate Docker to another machine in the future, we can also migrate this data.\nData volumes are the mechanism in Docker used for data persistence, allowing data in containers to be saved to the host machine or shared between multiple containers.\nPrerequisites # Docker and docker-compose are installed Understand the basic concepts and basic usage of Docker Use docker-compose to manage containers (if you are not using docker-compose to manage containers, or do not want to use docker-compose, this article is for reference only) Introduction to Data Volumes # Data volumes are the mechanism in Docker used for data persistence, allowing data in containers to be saved to the host machine or shared between multiple containers. Data volumes can be shared between containers or between the host machine and containers.\nUsing data volumes can avoid the problem of data in containers being deleted when the container stops or is deleted. Using data volumes can also improve the performance of containers, as data volumes are directly mounted to the host machine rather than accessed over the network.\nWhen using docker-compose to manage containers, we can define data volumes and their usage in the docker-compose.yml file using the volumes field. Generally, there are several types of data volumes in Docker:\nNamed Volume: Named volumes are the most commonly used type of data volume in Docker and can be shared between multiple containers. The name of a named volume is unique, and it can be accessed by its name. The data of named volumes is stored in the /var/lib/docker/volumes directory on the host machine.\nWhen using named volumes in docker-compose, you can define them in the docker-compose.yml file using the volumes field, for example:\n1 2 3 4 5 6 7 8 version: \u0026#39;3\u0026#39; services: app: image: nginx volumes: - my_volume:/usr/share/nginx/html volumes: my_volume: Note that after defining a named volume in services, it must also be defined in the volumes field.\nBind Mount: A bind mount mounts a directory from the host machine into the container, allowing data to be shared between the host and the container. The data of a bind mount is stored in a specified directory on the host machine.\nWhen using bind mounts in docker-compose, you can define them in the docker-compose.yml file using the volumes field, for example:\n1 2 3 4 5 6 version: \u0026#39;3\u0026#39; services: app: image: nginx volumes: - ./data:/usr/share/nginx/html The above configuration mounts the ./data directory on the host machine to the /usr/share/nginx/html directory in the container. Bind mounts also have a special usage called Read-Only Mount, which mounts a directory from the host machine into the container in read-only mode, allowing data to be read in the container but not modified. Read-only mounts are typically used for reading specific data from the host machine, such as system timezone files or SSL certificates.\nWhen using read-only mounts in docker-compose, you can define them in the docker-compose.yml file using the volumes field, for example:\n1 2 3 4 5 6 version: \u0026#39;3\u0026#39; services: app: image: nginx volumes: - /etc/localtime:/etc/localtime:ro Here the configuration mounts the /etc/localtime file from the host machine to the /etc/localtime file in the container in read-only mode. This ensures that the timezone in the container is consistent with that of the host machine.\ntmpfs Mount: A tmpfs mount mounts the host machine\u0026rsquo;s memory into the container, allowing the container to use memory as data storage. The data stored in a tmpfs mount is kept in the host machine\u0026rsquo;s memory, and it will be lost when the container stops or is deleted.\nWhen using tmpfs mounts in docker-compose, you can define them in the docker-compose.yml file using the volumes field, for example:\n1 2 3 4 5 6 version: \u0026#39;3\u0026#39; services: app: image: nginx volumes: - /tmp:/usr/share/nginx/html:tmpfs Note that the syntax for mounting tmpfs is similar to that of bind mounts, but with :tmpfs added after the path to indicate that it is a tmpfs mount.\nAnonymous Volume: Anonymous volumes are a less commonly used type of data volume in Docker, which do not have a name and can be shared between multiple containers. The data of anonymous volumes is stored in the /var/lib/docker/volumes directory on the host machine.\nWhen using anonymous volumes in docker-compose, you can define them in the docker-compose.yml file using the volumes field, for example:\n1 2 3 4 5 6 version: \u0026#39;3\u0026#39; services: app: image: nginx volumes: - /usr/share/nginx/html The above configuration mounts the /usr/share/nginx/html directory in the container to an anonymous volume on the host machine.\nBest Practices for Using Data Volumes in Docker # Using data volumes in Docker can be very flexible, but it can also lead to some issues, such as permission problems with data volumes, backup and recovery of data volumes, etc. Therefore, when using data volumes, it is best to follow some principles:\nUnderstand the Purpose of Data Volumes # When creating a data volume, we should understand its purpose. We can categorize data volumes into the following types:\nApplication Data Volume: Stores application data, such as databases, caches, etc. Configuration Data Volume: Stores application configuration files, such as nginx, apache, etc. Log Data Volume: Stores application log files, such as nginx, apache, etc. Temporary Data Volume: Stores temporary files, such as uploaded files, cache files, etc. Shared Data Volume: Stores data shared between multiple containers, such as data shared between nginx and php-fpm. Classify Data Volumes # Based on the purpose of the data volume, determine the type of mount. Generally, we should classify them as follows:\nApplication Data Volume: Use named volumes or bind mounts. Configuration Data Volume: Use bind mounts. Log Data Volume: Use bind mounts. Temporary Data Volume: Use tmpfs mounts. Shared Data Volume: Use named volumes or bind mounts. Unified Management of Data Volumes # Unified management of data volumes includes several different aspects:\nFor the data volumes of the same application that need to use bind mounts, they should be uniformly mounted to the same directory or kept consistent with the path of the data volume in Docker. For example:\n1 2 3 4 5 6 7 8 9 version: \u0026#39;3\u0026#39; services: app: image: nextcloud volumes: - /media/user/docker_data/nextcloud/app/config:/var/www/html/config - /media/user/docker_data/nextcloud/app/custom_apps:/var/www/html/custom_apps - /media/user/docker_data/nextcloud/app/data:/var/www/html/data - /media/user/docker_data/nextcloud/app/themes:/var/www/html/themes We can mount all Nextcloud data volumes to the /media/user/docker_data/nextcloud/app directory, making it easy to manage these data volumes.\nFor different applications, we should also try to place their data volumes in the same directory, such as /media/user/docker_data, so that we can easily manage these data volumes.\nFor certain important data volumes, we can separate them from ordinary data volumes and store them on more stable storage media, such as RAID arrays. For example, in the above Nextcloud data volumes, the most important data volume is /var/www/html/data, which can be placed on a RAID array, such as mounting the RAID array at /media/user/raid, we can define the data volume as follows:\n1 2 3 4 5 6 version: \u0026#39;3\u0026#39; services: app: image: nextcloud volumes: - /media/user/raid/nextcloud/app/data:/var/www/html/data This way, we can store the data on the RAID array.\nBackup Important Data Volumes # For important data volumes, we should regularly back up the data in the volume to prevent data loss. We can use the docker cp command to copy data from the volume to the host machine, or use the docker volume export command to export the volume as a tar file. Before updating, migrating, or upgrading containers, we should also back up the data in the volume to prevent data loss.\n1 docker cp \u0026lt;container_id\u0026gt;:/path/to/data /path/to/backup Or\n1 docker volume export \u0026lt;volume_name\u0026gt; \u0026gt; backup.tar Regularly Clean Up Unused Data Volumes # When using Docker, we may create some temporary data volumes that are no longer needed after use. We should regularly clean up these unused data volumes to free up storage space. We can use the docker volume prune command to delete all unused data volumes.\n1 docker volume prune ","date":"23 April 2025","externalUrl":null,"permalink":"/en/p/container-3-the-best-practice-guide-for-dockermanaging-data-volumes/","section":"Posts","summary":" Motivation # This is the third article in the Docker series. The links to the other articles in this series are as follows:\nContainer (1): Introduction to Container-related Knowledge——Containerization, Docker, Docker-compose, Kubernetes / K8s, etc. Container (2): The Best Practice Guide for Docker——docker-compose and Portainer Container (4): The Best Practice Guide for Docker——Container Update, Upgrade, and Migration Container (5): The Best Practice Guide for Docker——Container Update Monitoring Tool WUD (What’s Up Docker) Container (6): Misconceptions, Bad Habits, and Issues When Using Docker In the previous two articles, we introduced the basic concepts of containers and how to use them, as well as the best practices for using containers.\n","title":"Container (3): The Best Practice Guide for Docker——Managing Data Volumes","type":"post"},{"content":"","date":"23 April 2025","externalUrl":null,"permalink":"/en/tags/data-volume/","section":"Tags","summary":"","title":"Data Volume","type":"tags"},{"content":"","date":"2025年4月23日","externalUrl":null,"permalink":"/tags/%E6%95%B0%E6%8D%AE%E5%8D%B7/","section":"Tags","summary":"","title":"数据卷","type":"tags"},{"content":"","date":"10 April 2025","externalUrl":null,"permalink":"/en/categories/computer/","section":"Categories","summary":"","title":"Computer","type":"categories"},{"content":"","date":"10 April 2025","externalUrl":null,"permalink":"/en/tags/ddns/","section":"Tags","summary":"","title":"DDNS","type":"tags"},{"content":"","date":"10 April 2025","externalUrl":null,"permalink":"/en/tags/dynamic-domain-name-system/","section":"Tags","summary":"","title":"Dynamic Domain Name System","type":"tags"},{"content":"Previously, I wrote an article about Dynamic Domain Name System using Alibaba Cloud\u0026rsquo;s API. Recently, I transferred the domain to Cloudflare and recorded how to use Cloudflare\u0026rsquo;s API for Dynamic Domain Name System.\nMotivation # In previous articles, I mentioned that due to work reasons, I often need to use ssh to log into personal computers at home and in the office. However, these computers generally do not have a fixed IP address, so when the IP address changes, it is often impossible to connect to the computer in time. Therefore, I considered using a domain name instead of an IP address to access these computers and purchased one or two inexpensive domains. For a long time, I used Alibaba Cloud\u0026rsquo;s domain resolution service. You can refer to the article “Establishing a Personal Website (Part 1): Purchasing a Personal Domain and Configuring Dynamic Domain Name System” for more information on dynamic domain name resolution on Alibaba Cloud.\nLater, I transferred the domain to Cloudflare, mainly because Cloudflare\u0026rsquo;s CDN service and DNS resolution speed are faster than Alibaba Cloud. Now, I will record how to use Cloudflare\u0026rsquo;s API for dynamic domain name resolution.\nPrerequisites # Own a Domain Name\nThe purchased domain name can be resolved on Cloudflare. If you do not have your own domain name, you can purchase an inexpensive domain from Namecheap. Have a Cloudflare Account\nIf you do not have a Cloudflare account, you can register for one at Cloudflare. After registration, log in to your Cloudflare account, add your domain, and point the domain\u0026rsquo;s DNS servers to the ones provided by Cloudflare. Domain Resolution to Server IP Address # The purchased domain name needs to be resolved to the server\u0026rsquo;s IP address in order to access the server directly via the domain name. If the server has a fixed IP address, you only need to bind the domain to the server on Cloudflare. If the server does not have a fixed IP address, you need to dynamically resolve the domain to the server\u0026rsquo;s IP address. Fixed IP - Direct Binding # Obtain IP Address\nYou can visit the IPv6 Test Website to check if your network supports IPv6: If you are using a Unix-like system, you can obtain the IPv4 address with the following command:\n1 curl ipinfo.io/ip You can obtain the IPv6 address with the following command:\n1 curl ipv6.icanhazip.com Bind Domain Name to IP Address\nLog in to your Cloudflare account, click on the menu in the upper left corner, and find the DNS option in the menu: Click the Add Record button to add an A record (IPv4) or AAAA record (IPv6) to resolve the domain name to the server\u0026rsquo;s IP address: Fill in the domain name to be resolved and the corresponding IP address (whether the IP address is correct or not is not important; if it is incorrect, running this script later will correct it; but the domain name must be the one you created in step 2 of \u0026ldquo;Fixed IP - Direct Binding\u0026rdquo;). If you have multiple domain names to resolve, you can add them yourself by referring to the above example, and both IPv4 and IPv6 are supported. Again, it is emphasized that the domain names to be resolved in this document must be the ones that have been created on Cloudflare in step 2 of \u0026ldquo;Fixed IP - Direct Binding\u0026rdquo;. Test if Binding is Successful After binding, it may take up to 10 minutes for the Cloudflare DNS servers to complete the update (usually it updates immediately after submission). You can then check if the binding was successful.\nIt is recommended to test using the \u0026ldquo;ping\u0026rdquo; command from another network environment 1 ping surface.jinli.cyou If you can ping it from the public network, the binding is successful. Done! After the binding is successful, you can access the server via the domain name. For example, if you need to use ssh to remotely log in to the server, you can directly use the domain name:\n1 ssh lijin@surface.jinli.cyou -X You can also set up NAS (Network Attached Storage) services on the server and access them via the domain name.\nDynamic Resolution After Binding for Non-Fixed IPs # If the computer or server does not have a fixed IP address, you need to first bind the current IP address on Cloudflare according to the above steps. Then monitor the local IP address, and once it changes, use the API provided by Cloudflare to upload it to the Cloudflare DNS servers and modify the originally bound IP address.\nDDNS Script\nThe DDNS (Dynamic Domain Name System) script I use is a bash script generated by ChatGPT: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 #!/bin/bash # Config API_TOKEN=\u0026#34;your_cloudflare_api_token\u0026#34; ZONE_ID=\u0026#34;your_cloudflare_zone_id\u0026#34; RECORDS=(\u0026#34;host1.example.com:A\u0026#34; \u0026#34;host1.example.com:AAAA\u0026#34; \u0026#34;host2.example.com:A\u0026#34; \u0026#34;host2.example.com:AAAA\u0026#34;) # Add as needed # Get current IPv4 and IPv6 addresses IPV4=$(curl -s https://ipv4.icanhazip.com) IPV6=$(ip -6 addr show scope global | grep inet6 | awk \u0026#39;{print $2}\u0026#39; | cut -d/ -f1 | head -n 1) if [ -z \u0026#34;$IPV4\u0026#34; ]; then echo \u0026#34;No IPv4 address found.\u0026#34; else echo \u0026#34;Detected IPv4: $IPV4\u0026#34; fi if [ -z \u0026#34;$IPV6\u0026#34; ]; then echo \u0026#34;No IPv6 address found.\u0026#34; else echo \u0026#34;Detected IPv6: $IPV6\u0026#34; fi # Function to update DNS record update_record() { local name=\u0026#34;$1\u0026#34; local type=\u0026#34;$2\u0026#34; local ip=\u0026#34;$3\u0026#34; echo \u0026#34;Processing $type record for $name\u0026#34; # Get record info RECORD=$(curl -s -X GET \u0026#34;https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=$type\u0026amp;name=$name\u0026#34; \\ -H \u0026#34;Authorization: Bearer $API_TOKEN\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34;) RECORD_ID=$(echo \u0026#34;$RECORD\u0026#34; | jq -r \u0026#39;.result[0].id\u0026#39;) CURRENT_IP=$(echo \u0026#34;$RECORD\u0026#34; | jq -r \u0026#39;.result[0].content\u0026#39;) if [ \u0026#34;$CURRENT_IP\u0026#34; == \u0026#34;$ip\u0026#34; ]; then echo \u0026#34; - $type is already up to date: $ip\u0026#34; return fi # Update the DNS record UPDATE=$(curl -s -X PUT \u0026#34;https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID\u0026#34; \\ -H \u0026#34;Authorization: Bearer $API_TOKEN\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ --data \u0026#34;{\\\u0026#34;type\\\u0026#34;:\\\u0026#34;$type\\\u0026#34;,\\\u0026#34;name\\\u0026#34;:\\\u0026#34;$name\\\u0026#34;,\\\u0026#34;content\\\u0026#34;:\\\u0026#34;$ip\\\u0026#34;,\\\u0026#34;ttl\\\u0026#34;:120,\\\u0026#34;proxied\\\u0026#34;:false}\u0026#34;) SUCCESS=$(echo \u0026#34;$UPDATE\u0026#34; | jq -r \u0026#39;.success\u0026#39;) if [ \u0026#34;$SUCCESS\u0026#34; == \u0026#34;true\u0026#34; ]; then echo \u0026#34; - Updated $type record to $ip\u0026#34; else echo \u0026#34; - Failed to update $type record: $UPDATE\u0026#34; fi } # Loop through all record names and update the specified type (A or AAAA) for record in \u0026#34;${RECORDS[@]}\u0026#34;; do # Split the record into name and type IFS=\u0026#39;:\u0026#39; read -r name type \u0026lt;\u0026lt;\u0026lt; \u0026#34;$record\u0026#34; if [ \u0026#34;$type\u0026#34; == \u0026#34;A\u0026#34; ] \u0026amp;\u0026amp; [ -n \u0026#34;$IPV4\u0026#34; ]; then update_record \u0026#34;$name\u0026#34; \u0026#34;A\u0026#34; \u0026#34;$IPV4\u0026#34; elif [ \u0026#34;$type\u0026#34; == \u0026#34;AAAA\u0026#34; ] \u0026amp;\u0026amp; [ -n \u0026#34;$IPV6\u0026#34; ]; then update_record \u0026#34;$name\u0026#34; \u0026#34;AAAA\u0026#34; \u0026#34;$IPV6\u0026#34; else echo \u0026#34; - Skipping $type record for $name due to missing IP address.\u0026#34; fi done This script requires the user to specify the domain name and record type (A or AAAA) they want to resolve, in the format domain_name:record_type, for example host1.example.com:A. If there are multiple domains to resolve, you can add multiple records in the script, in the format host1.example.com:A, host2.example.com:AAAA, etc. The script will automatically detect the current IPv4 and IPv6 addresses and update them on the Cloudflare DNS servers. The script uses the jq command to parse JSON-formatted data, so you need to install the jq command: 1 sudo apt install jq The script uses the Cloudflare API for dynamic domain resolution, so you need to generate an API Token and Zone ID on Cloudflare. Below is a brief introduction on how to obtain the API Token and Zone ID from Cloudflare. Fill in the Zone_ID into the ZONE_ID variable in the script above. Obtain API Token and Zone ID\nLog in to your Cloudflare account, click on your domain, and there is a menu on the right side. The lower part of the menu has the Zone ID and Account ID. Below these two IDs is the link \u0026ldquo;Get your API token\u0026rdquo;, click it. On the User API Token page, select \u0026ldquo;Create Token\u0026rdquo;. Here you need to choose the type of API Token to create, select \u0026ldquo;Edit Zone DNS\u0026rdquo; type, which means this Token can only modify DNS records. Then you will be prompted to set the Token\u0026rsquo;s permissions, set it according to your needs: After setting, click \u0026ldquo;Continue to summary\u0026rdquo;, and then click the \u0026ldquo;Create Token\u0026rdquo; button. Once created, the value of the Token will be displayed, and the page will also prompt you to run a curl command on the target computer to test whether the Token is valid. After testing is successful, copy the API Token and fill it into the API_TOKEN variable in the script above. Note: This API Token and the above Zone_ID are very important, and you must not expose them. If the API Token is leaked, you should immediately log in to your Cloudflare account and delete the Token. The Zone_ID and API Token are your credentials for operations on Cloudflare, and leaking them may lead to malicious modifications of your domain.\nTest Run DDNS Script\nAfter modifying the above script, save it as ddns.sh. Then\nEnsure that curl and jq commands are installed:\n1 sudo apt install curl jq Grant executable permissions to the script:\n1 chmod +x ddns.sh Run the script:\n1 ./ddns.sh If the script runs successfully, it will output results similar to the following:\n1 2 3 4 5 6 7 8 9 10 Detected IPv4: 111.111.111.111 Detected IPv6: 12aa3:4567:89ab:cdef Processing A record for host1.example.com - Updated A record to 111.111.111.111 Processing AAAA record for host1.example.com - Updated AAAA record to 12aa3:4567:89ab:cdef Processing A record for host2.example.com - Updated A record to 111.111.111.111 Processing AAAA record for host2.example.com - Updated AAAA record to 12aa3:4567:89ab:cdef Set Up Scheduled Tasks\nIf the computer/server does not have a fixed IP, the IP will change after a period of time, and we need to update the IP records on the Cloudflare DNS server as soon as the IP changes. Manually monitoring IP changes is not very convenient, so we can have the system automatically run the DDNS script every once in a while (for example, every hour), so that we can update it within an hour after the IP changes. Scheduled tasks in Unix-like systems can be implemented using the system program crontab, and you can see some methods for setting crontab here. Linux In Linux systems, setting up crontab scheduled tasks is very simple. We can enter the following command in the command line: shell crontab -e This command will open a configuration file containing crontab scheduled tasks. Add a line at the end of the file: shell 42 * * * * cd ~/.config/ddns \u0026amp;\u0026amp; ./ddns.sh This means that the ddns.sh script will run at the 42nd minute of every hour. You can modify the time as needed.\nAfter that, you can use the following command to check if the crontab scheduled task is running: 1 crontab -l macOS On macOS, due to the system's security considerations, some special settings are required before using the `crontab` command. 1. Grant `cron` command access to full disk: - Open \u0026quot;System Preferences\u0026quot; -\u0026gt; \u0026quot;Security \u0026amp; Privacy\u0026quot; -\u0026gt; \u0026quot;Privacy\u0026quot; tab, and click the lock icon in the lower-left corner to unlock the settings. - In the left list, select \u0026quot;Full Disk Access\u0026quot;, then click the plus button on the right to add the `/usr/sbin/cron` program. Note that after clicking the plus button, a Finder window will pop up for you to select the program to add. However, the `cron` program is hidden by default and cannot be found directly in the Finder window. You can press `Command + Shift + G` in Finder, enter `/usr/sbin/`, and then select the `cron` program to add it. - Close the \u0026quot;System Preferences\u0026quot; window, ensuring that the permissions have been granted. 2. The script requires the use of commands such as `curl`, `jq`, `yq`, and `ifconfig`, but the `cron` environment does not load user environment variables when running scripts. Therefore, you need to specify the full paths to these commands in the script. You can find the full paths by running the following command: ```shell which curl jq yq ifconfig ``` Then fill in the full paths of these commands in the script, for example: ```bash export PATH=\u0026quot;$PATH:/usr/local/bin:/usr/bin:/opt/local/bin:/sbin\u0026quot; ``` 3. To achieve this, I specifically wrote a script as a mediator script to call `ddns.sh`, named `cron_ddns.sh`, with the following content: ```bash #!/bin/bash echo \u0026quot;[$(date '+%Y-%m-%d %H:%M:%S')] Starting cron job for Cloudflare DDNS update...\u0026quot; export PATH=\u0026quot;$PATH:/usr/local/bin:/usr/bin:/opt/local/bin:/sbin\u0026quot; echo \u0026quot;PATH is $PATH\u0026quot; cd /Users/lijin/.config/ddns echo \u0026quot;Current directory is $(pwd)\u0026quot; echo \u0026quot;Running script cloudflare.sh\u0026quot; /bin/bash ddns.sh ``` 4. Grant execute permission to the script: ```shell chmod +x cron_ddns.sh ``` 5. Then add a scheduled task in `crontab`: ```shell crontab -e ``` Add a line at the end of the file: ```shell 42 * * * * /Users/lijin/.config/ddns/cron_ddns.sh \u0026gt;\u0026gt; /tmp/ddns.log 2\u0026gt;\u0026amp;1 ``` ","date":"10 April 2025","externalUrl":null,"permalink":"/en/p/dynamic-domain-name-system--cloudflare/","section":"Posts","summary":"Previously, I wrote an article about Dynamic Domain Name System using Alibaba Cloud’s API. Recently, I transferred the domain to Cloudflare and recorded how to use Cloudflare’s API for Dynamic Domain Name System.\nMotivation # In previous articles, I mentioned that due to work reasons, I often need to use ssh to log into personal computers at home and in the office. However, these computers generally do not have a fixed IP address, so when the IP address changes, it is often impossible to connect to the computer in time. Therefore, I considered using a domain name instead of an IP address to access these computers and purchased one or two inexpensive domains. For a long time, I used Alibaba Cloud’s domain resolution service. You can refer to the article “Establishing a Personal Website (Part 1): Purchasing a Personal Domain and Configuring Dynamic Domain Name System” for more information on dynamic domain name resolution on Alibaba Cloud.\n","title":"Dynamic Domain Name System -- Cloudflare","type":"post"},{"content":"","date":"2025年4月10日","externalUrl":null,"permalink":"/tags/%E5%8A%A8%E6%80%81%E5%9F%9F%E5%90%8D%E8%A7%A3%E6%9E%90/","section":"Tags","summary":"","title":"动态域名解析","type":"tags"},{"content":"","date":"2025年4月10日","externalUrl":null,"permalink":"/categories/%E8%AE%A1%E7%AE%97%E6%9C%BA/","section":"Categories","summary":"","title":"计算机","type":"categories"},{"content":"","date":"8 April 2025","externalUrl":null,"permalink":"/en/tags/remote-desktop/","section":"Tags","summary":"","title":"Remote Desktop","type":"tags"},{"content":" Motivation # Previously, I was using AnyDesk as my remote desktop software, but recently the IT department at my school mentioned that AnyDesk has security vulnerabilities and can no longer be used. On Linux, the only allowed remote desktop software are VNC and XRDP. Both VNC and RDP are remote desktop protocols, but VNC is an open-source protocol, while RDP is a proprietary protocol by Microsoft. The advantage of VNC is its cross-platform support, including Linux, Windows, macOS, etc.; although RDP is proprietary to Microsoft, there are open-source implementations available on Linux, such as XRDP.\nVNC\u0026rsquo;s drawback is its slower speed and higher latency. However, its advantage is that when using VNC on a Linux system, the remote desktop and local desktop are completely separate and do not interfere with each other. Additionally, many Linux systems and macOS come with built-in support for VNC services, eliminating the need for additional software installation.\nmacOS Connecting to Ubuntu # Ubuntu comes with a built-in VNC service, and macOS has a built-in VNC client, making it very easy to connect to Ubuntu from macOS. Before connecting, we need to do some preparation work.\nEnable VNC Service on Ubuntu. Although Ubuntu has a built-in VNC service, it is disabled by default, so we need to enable it first. If you are using the Gnome desktop environment, go to the \u0026ldquo;Settings\u0026rdquo; interface, click on \u0026ldquo;Sharing,\u0026rdquo; and then enable the \u0026ldquo;Remote Desktop\u0026rdquo; option:\nSet VNC Password. After enabling the \u0026ldquo;Remote Desktop\u0026rdquo; option, you will see the VNC service settings interface. Follow the prompts to set it up:\nEnable Password Authentication. After checking the \u0026ldquo;Enable Legacy VNC Protocol\u0026rdquo; option, click on the three dots on the right and check \u0026ldquo;Require a password\u0026rdquo;:\nIf you do not check this option, a confirmation window will pop up each time you connect, asking whether to allow the connection. If there is no one on the remote computer to confirm, you will not be able to connect:\nConnect to Ubuntu from macOS. You can use the built-in file browser Finder on macOS to connect to the remote computer via VNC. However, before doing so, we need to first connect to the remote computer using SSH and perform port forwarding:\n1 ssh -L 22590:localhost:5900 username@remote_ip The above command connects to the remote computer while mapping the remote computer\u0026rsquo;s port 5900 to the local port 22590. The reason for this is that the VNC service uses port 5900 by default, but for security reasons, many systems may have port 5900 closed, so we need to use SSH for port forwarding. Once the SSH connection is successful, you can connect to the remote computer in Finder:\nIn the pop-up window, enter the VNC password: If everything goes well, you should see the remote computer\u0026rsquo;s desktop.\n","date":"8 April 2025","externalUrl":null,"permalink":"/en/p/remote-desktop-ii-vnc-configuration/","section":"Posts","summary":" Motivation # Previously, I was using AnyDesk as my remote desktop software, but recently the IT department at my school mentioned that AnyDesk has security vulnerabilities and can no longer be used. On Linux, the only allowed remote desktop software are VNC and XRDP. Both VNC and RDP are remote desktop protocols, but VNC is an open-source protocol, while RDP is a proprietary protocol by Microsoft. The advantage of VNC is its cross-platform support, including Linux, Windows, macOS, etc.; although RDP is proprietary to Microsoft, there are open-source implementations available on Linux, such as XRDP.\n","title":"Remote Desktop (II): VNC Configuration","type":"post"},{"content":"","date":"8 April 2025","externalUrl":null,"permalink":"/en/series/remote-desktop-series/","section":"Series","summary":"","title":"Remote Desktop Series","type":"series"},{"content":"","date":"8 April 2025","externalUrl":null,"permalink":"/en/tags/vnc/","section":"Tags","summary":"","title":"VNC","type":"tags"},{"content":"","date":"2025年4月8日","externalUrl":null,"permalink":"/tags/%E8%BF%9C%E7%A8%8B%E6%A1%8C%E9%9D%A2/","section":"Tags","summary":"","title":"远程桌面","type":"tags"},{"content":"","date":"2025年4月8日","externalUrl":null,"permalink":"/series/%E8%BF%9C%E7%A8%8B%E6%A1%8C%E9%9D%A2%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"远程桌面系列","type":"series"},{"content":"","date":"5 December 2024","externalUrl":null,"permalink":"/en/tags/ffmpeg/","section":"Tags","summary":"","title":"Ffmpeg","type":"tags"},{"content":" Motivation # FFmpeg is a very powerful audio and video processing tool that can be used for editing, transcoding, merging, splitting, extracting video and audio, etc. FFmpeg is a free and open-source software, initially initiated by French programmer Fabrice Bellard and now maintained by Michael Niedermayer. Many players, video editing software, transcoding software on the market, such as Blender, Kodi, Plex, Shotcut, VLC media player, YouTube, etc., are based on FFmpeg. Of course, there are also many software that use FFmpeg\u0026rsquo;s code but do not comply with FFmpeg\u0026rsquo;s open-source agreement, and are nailed to the \u0026ldquo;FFmpeg Pillory\u0026rdquo;.\nFFmpeg is a command-line tool that is a bit complicated to use, but very powerful. I have only used a few simple functions, such as video clipping, merging, adjusting resolution, transcoding, video acceleration, slow motion, audio and video separation. But I have always used CPU for processing, now I have a graphics card on my computer, I want to use GPU for acceleration. It did take me a little effort, so here is a summary.\nThis is the second article in the \u0026ldquo;Nvidia GPU\u0026rdquo; series, which mainly records how to use Nvidia GPU, especially on Linux systems. Other articles in this series can be found at:\nNvidia GPU (1): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Ubuntu Nvidia GPU (3): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Fedora Nvidia GPU (4): Containerized Configuration of CUDA Programming and Deep Learning Environment for Nvidia GPU Prerequisites # NVIDIA graphics card on the computer Linux system Compile and Install FFmpeg from Source Code # I use Ubuntu 22.04 system, and my FFmpeg was installed through apt before, but this version does not support GPU acceleration. So I need to compile and install from source code. If you have installed FFmpeg through apt before, you need to uninstall it first:\n```bash sudo apt-get remove ffmpeg ``` The compilation and installation process I referred to the official documentation of NVIDIA. Unfortunately, this document seems a bit outdated. If you compile and install directly according to the steps in the document, you will encounter some problems. Here are the steps I successfully compiled and installed.\nInstall Dependencies # There are three main dependencies:\nNVIDIA graphics card driver. Please refer to my previous article Install NVIDIA Graphics Card Driver on Ubuntu 22.04.\nNVIDIA encoding interface library. Use the following command to compile and install from source code.\n1 2 git clone https://github.com/FFmpeg/nv-codec-headers.git cd nv-codec-headers \u0026amp;\u0026amp; sudo make install \u0026amp;\u0026amp; cd – FFmpeg dependency library. Use the following command to install.\n1 sudo apt-get install build-essential yasm cmake libtool libc6 libc6-dev unzip wget libnuma1 libnuma-devsudo apt-get install build-essential yasm cmake libtool libc6 libc6-dev unzip wget libnuma1 libnuma-dev Compile and Install # Download FFmpeg source code.\n1 git clone https://git.ffmpeg.org/ffmpeg.git Configure compilation parameters.\n1 ./configure --enable-nonfree --enable-cuda-nvcc --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --disable-static --enable-shared --disable-x86asm /usr/local/cuda here is the installation path of NVIDIA graphics card driver. If your installation path is different, please modify it.\n--enable-nonfree is to support non-free encoders.\n--enable-cuda-nvcc is to support CUDA acceleration.\n--enable-libnpp is to support NPP acceleration. NPP is the abbreviation of NVIDIA Performance Primitives, a high-performance image and signal processing function library provided by NVIDIA, which is not supported by FFmpeg by default.\n--disable-x86asm is to avoid the following error during compilation:\n1 nasm not found or too old. Please install/update nasm or use --disable-x86asm for a build without hand-optimized assembly. If the above command is executed without any problems, you can continue to compile.\nCompile.\n1 make -j8 -j8 here means using 8 threads for parallel compilation, you can adjust according to the number of CPU cores.\nInstall.\n1 sudo make install The above command will install FFmpeg in the /usr/local/bin directory.\nProblem Solving # After compiling and installing according to the above steps, when I run ffmpeg on the command line, I encountered the following error:\n1 ffmpeg: error while loading shared libraries: libavdevice.so.61: cannot open shared object file: No such file or directory This is because the FFmpeg library file is not correctly linked, and manual linking is required:\n1 sudo ldconfig If the above command does not solve the problem, it is because FFmpeg installed the linked library files in the /usr/local/lib directory during compilation and installation, while the system\u0026rsquo;s default linked library path is /usr/lib. In this case, you need to add /usr/local/lib to the linked library path:\n1 2 sudo echo \u0026#34;/usr/local/lib\u0026#34; \u0026gt; /etc/ld.so.conf.d/ffmpeg.conf sudo ldconfig Basic Usage of FFmpeg # Here are a few commands I have used:\nVideo clipping.\n1 ffmpeg -i input.mp4 -ss 00:00:00 -t 00:00:10 -c copy output.mp4 -i input.mp4: input file. -ss 00:00:00: start time. -t 00:00:10: duration. -c copy: copy encoding. Video transcoding.\nFor example, convert mkv format to mp4 format.\n1 ffmpeg -i input.mkv -codec copy output.mp4 -codec copy: copy encoding. Video acceleration or slow motion.\n1 ffmpeg -i input.mp4 -vf \u0026#34;setpts=0.5*PTS\u0026#34; output.mp4 -vf \u0026quot;setpts=0.5*PTS\u0026quot;: acceleration factor. Less than 1 means acceleration, greater than 1 means deceleration. Here 0.5 means 2 times acceleration. Video resolution adjustment.\n1 ffmpeg -i input.mp4 -vf scale=1920:1080 output.mp4 -vf scale=1920:1080: target resolution. Audio and video separation.\n1 2 ffmpeg -i input.mp4 -vn -acodec copy output.aac ffmpeg -i input.mp4 -an -vcodec copy output.mp4 -vn: no video. -acodec copy: copy audio encoding. -an: no audio. -vcodec copy: copy video encoding. Video merging.\nIf the resolution, frame rate, encoding, etc. of the videos to be merged are the same, and there are only two videos, you can use the following command:\n1 ffmpeg -i input1.mp4 -i input2.mp4 -filter_complex \u0026#34;[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1[outv][outa]\u0026#34; -map \u0026#34;[outv]\u0026#34; -map \u0026#34;[outa]\u0026#34; output.mp4 -filter_complex \u0026quot;[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1[outv][outa]\u0026quot;: merge video and audio. If there are more videos to be merged, it is recommended to write the video list into a text file, and then use the concat protocol to merge.\n1 ffmpeg -f concat -safe 0 -i list.txt -c copy output.mp4 -f concat: specify the protocol.\n-safe 0: allow reading any file.\n-i list.txt: video list file. The content is as follows:\n1 2 3 file \u0026#39;input1.mp4\u0026#39; file \u0026#39;input2.mp4\u0026#39; file \u0026#39;input3.mp4\u0026#39; -c copy: copy encoding.\nUsing GPU Acceleration # To use GPU acceleration, you need to add --enable-cuda-nvcc and --enable-libnpp parameters when compiling. The above compilation and installation steps have added these two parameters.\nTo use GPU acceleration, you need to specify the -hwaccel cuda parameter. For example:\n1 ffmpeg -y -hwaccel cuda -hwaccel_output_format cuda -i input.mkv -c:v h264_nvenc -ss 0:00:00 -to 0:01:23 output.mp4 -hwaccel cuda: specify to use CUDA acceleration. -hwaccel_output_format cuda: specify the output format as CUDA. -c:v h264_nvenc: specify to use NVIDIA\u0026rsquo;s h264 encoder. Of course, you can also use other encoders, such as hevc_nvenc. If you want to see the supported encoders, you can use the ffmpeg -h encoder=nvenc command. Of course, the commands introduced above can also use GPU acceleration by adding the -hwaccel cuda parameter.\nMy CPU is i5-9600K, and the graphics card is NVIDIA\u0026rsquo;s RTX 4060 Ti. When using only the CPU, processing a video with a resolution of 1920x1080, the CPU usage is about 100%, and the processing speed is about 30 frames per second. After using GPU acceleration, the GPU usage is about 33%, and the processing speed is about 500 frames per second. It can be seen that the processing speed is increased by about 16 times after GPU acceleration.\nClarity Issue # After using GPU acceleration, the video clarity may decrease. This is because when using GPU acceleration, FFmpeg will use the NPP library to process images, and the processing accuracy of the NPP library may not be as good as that of the CPU.\nIf you process the same video separately with CPU and GPU, and then compare the sizes of the two videos, you will find that the video processed by GPU is much smaller. For example, the size of a video clipped by CPU is 300MB, and the size of the same video clipped by GPU is only 50MB.\nIf you check the bitrates of the two videos, you will find that the bitrate of the video clipped by GPU is much smaller than that of the source video. This is because when processing with GPU, the video will be compressed, resulting in a decrease in video clarity. If you want to maintain the clarity of the video, you can specify the bitrate.\n1 ffmpeg -y -hwaccel cuda -hwaccel_output_format cuda -i input.mkv -c:v h264_nvenc -b:v 20M -ss 0:00:00 -to 0:01:23 output.mp4 -b:v 20M: specify the bitrate as 20M. Here 20M means 20Mbps, you can adjust according to your needs. FFmpeg prioritizes speed rather than clarity when using GPU acceleration. If you want to maintain the clarity of the video, you can also use the -preset slow parameter to specify the encoding speed.\n1 ffmpeg -y -hwaccel cuda -hwaccel_output_format cuda -i input.mkv -c:v h264_nvenc -preset slow -ss 0:00:00 -to 0:01:23 output.mp4 -preset slow: specify the encoding speed as slow. Here slow means slow, the processing speed is slower, but the clarity is higher. You can also combine the above two parameters.\n1 ffmpeg -y -hwaccel cuda -hwaccel_output_format cuda -i input.mkv -c:v h264_nvenc -b:v 20M -preset slow -ss 0:00:00 -to 0:01:23 output.mp4 ","date":"5 December 2024","externalUrl":null,"permalink":"/en/p/nvidia-gpu-2-accelerate-video-editing-and-transcoding-for-ffmpeg-using-gpu/","section":"Posts","summary":" Motivation # FFmpeg is a very powerful audio and video processing tool that can be used for editing, transcoding, merging, splitting, extracting video and audio, etc. FFmpeg is a free and open-source software, initially initiated by French programmer Fabrice Bellard and now maintained by Michael Niedermayer. Many players, video editing software, transcoding software on the market, such as Blender, Kodi, Plex, Shotcut, VLC media player, YouTube, etc., are based on FFmpeg. Of course, there are also many software that use FFmpeg’s code but do not comply with FFmpeg’s open-source agreement, and are nailed to the “FFmpeg Pillory”.\n","title":"Nvidia GPU (2): Accelerate Video Editing and Transcoding for FFmpeg Using GPU","type":"post"},{"content":"","date":"23 September 2024","externalUrl":null,"permalink":"/en/tags/attention-mechanism/","section":"Tags","summary":"","title":"Attention Mechanism","type":"tags"},{"content":"","date":"23 September 2024","externalUrl":null,"permalink":"/en/tags/language-model/","section":"Tags","summary":"","title":"Language Model","type":"tags"},{"content":" Background # This is the 7th article in the \u0026ldquo;Learn PyTorch by Examples\u0026rdquo; series. In the 6th article \u0026ldquo;Learn PyTorch by Examples (6): Language Model (I) \u0026ndash; Implementing a Word-Level Language Model with LSTM\u0026rdquo;, we briefly introduced how to implement a word-level language model using LSTM.\nLSTM and other models based on Recurrent Neural Networks (RNN) have been widely used in natural language processing, but these models have some problems when dealing with long-distance dependency problems, such as vanishing gradients and exploding gradients. To solve these problems, researchers proposed the Transformer model, which uses the attention mechanism to better handle long-distance dependency problems. In this article, we will briefly introduce the Transformer model and use the Transformer to implement a simple word-level language model. This article refers to the word_language_model example in the official PyTorch examples.\nThe code of this article can be found in the T06_word_lstm folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial.\nTransformer Model and Attention Mechanism # In 2017, Google researchers published a paper \u0026ldquo;Attention is All You Need\u0026rdquo;, which proposed the Transformer model. Due to its excellent performance in language modeling, it quickly replaced LSTM and GRU models and became the mainstream model in the field of natural language processing.\nThe structure of the Transformer model is as follows:\nAs we can see, the Transformer model consists of an encoder and a decoder. The encoder and decoder are both stacked with multiple identical layers, each layer containing a multi-head self-attention mechanism and a feed-forward neural network.\nSelf-Attention Mechanism # The so-called self-attention mechanism means that the model can simultaneously focus on different positions in the input sequence to better capture the information in the input sequence. Multi-head self-attention mechanism means that the model can simultaneously focus on different positions in the input sequence and learn different attention weights through multiple heads to better capture the information in the input sequence.\nFor example, suppose we have an input sequence [I, love, you], and we hope the model can predict love based on the relationship between I and you. In the LSTM model, the model will process each word in the input sequence one by one, but in the Transformer model, the model can simultaneously focus on I and you to better capture the relationship between them. Since the Transformer model can simultaneously focus on different positions in the input sequence, it can better handle long-distance dependency problems. However, since the Transformer model does not have a recurrent structure, it cannot handle the order information in the sequence like LSTM, so we need to add positional encoding to the input sequence to represent the position information of the words.\nThe calculation process of self-attention is as follows:\nFirst, we need to calculate the query, key, and value vectors. Here we use the word vectors of the input sequence as the query, key, and value vectors. These three vector representations can be obtained through a linear transformation, i.e., $Q = XW^Q$, $K = XW^K$, and $V = XW^V$, where $X$ is the word vector of the input sequence, and $W^Q$, $W^K$, and $W^V$ are the weights of the linear transformation.\nThen, we calculate the attention scores $A$, which are the dot product of the query vector $Q$ and the key vector $K$, divided by $\\sqrt{d_k}$, where $d_k$ is the dimension of the query vector $Q$. That is, $A = \\frac{QK^T}{\\sqrt{d_k}}$.\nNext, we calculate the attention weights $W$, which are obtained by applying the Softmax function to the attention scores $A$. That is, $W = \\text{Softmax}(A)$.\nFinally, we calculate the self-attention output $O$, which is the weighted sum of the attention weights $W$ and the value vector $V$. That is, $O = W \\cdot V$.\nIn practical applications, we usually use the multi-head self-attention mechanism, which means that the word vectors of the input sequence are transformed into multiple sets of query, key, and value vector representations through multiple linear transformations, and then multiple sets of attention scores, attention weights, and self-attention outputs are calculated separately, and finally the multiple sets of self-attention outputs are concatenated and passed through a linear transformation to obtain the final output.\nFeed-Forward Neural Network # The feed-forward neural network is another important component in the Transformer model. It consists of two fully connected layers and an activation function. The calculation process of the feed-forward neural network is as follows:\nFirst, we use a fully connected layer to obtain the intermediate representation $M$ of the self-attention output $O$, i.e., $M = O \\cdot W_1 + b_1$, where $W_1$ and $b_1$ are the weights and biases of the fully connected layer.\nThen, we use an activation function (usually ReLU) to obtain the output $F$ of the feed-forward neural network, i.e., $F = \\text{ReLU}(M)$.\nFinally, we use another fully connected layer to obtain the final output $O\u0026rsquo;$ of the feed-forward neural network, i.e., $O\u0026rsquo; = F \\cdot W_2 + b_2$, where $W_2$ and $b_2$ are the weights and biases of the fully connected layer.\nThe feed-forward neural network is used to perform a non-linear transformation on the self-attention output $O$ to better capture the information in the input sequence.\nEncoder and Decoder # Self-attention plus feed-forward neural network form a layer in the Transformer model, and the entire Transformer model is composed of multiple such layers stacked together, as shown in the figure below:\nThe Transformer model is generally divided into an encoder and a decoder. The encoder is used to encode the input sequence into a context vector, and the decoder is used to generate the output sequence based on the context vector. Both the encoder and decoder are stacked with multiple identical layers, each layer containing a multi-head self-attention mechanism and a feed-forward neural network.\nThe input of the encoder is a word sequence, and the output is a context vector. The input of the decoder is a context vector and a word sequence, and the output is a word sequence. In tasks such as machine translation, we can use the word sequence of the source language as the input of the encoder and the word sequence of the target language as the input of the decoder to achieve translation from the source language to the target language.\nThe difference between the encoder and decoder is that when calculating self-attention, the decoder also calculates the attention of the encoder\u0026rsquo;s output, which is to better capture the relationship between the input sequence and the output sequence. In addition, the self-attention mechanism used by the decoder is the masked self-attention mechanism, which means that when calculating the attention weights, the decoder can only focus on the positions before the current position, not the positions after the current position. This is because our goal is to predict the current and subsequent words, so naturally we cannot use the information of the subsequent words, otherwise it would be cheating.\nClassification of Transformer Models # Although the standard Transformer model consists of an encoder and a decoder, in practical applications, we can also use only the encoder or decoder, or use both the encoder and decoder at the same time. Moreover, the model that uses both the encoder and decoder is not necessarily better than the model that uses only the encoder or decoder, which depends on the specific task and dataset.\nEncoder Model # The encoder model only contains the encoder, which is used to encode the input sequence into a context vector. The encoder model is commonly used in tasks such as text classification and sentiment analysis. Commonly used encoder models include BERT and RoBERTa.\nDecoder Model # The decoder model only contains the decoder, which is used to generate the output sequence based on the context vector. The decoder model is commonly used in machine translation, text generation, and other tasks. Commonly used decoder models include T5 and GPT.\nEncoder-Decoder Model # The encoder-decoder model contains both the encoder and decoder, which is used to encode the input sequence into a context vector and generate the output sequence based on the context vector. The encoder-decoder model is also commonly used in machine translation, text generation, and other tasks. Commonly used encoder-decoder models include Transformer and BART.\nImplementing Word-Level Language Model with Transformer in PyTorch # PyTorch provides the torch.nn.Transformer module, which can be used to easily implement the Transformer model. In this article, we will replace the LSTM model in the previous article with the Transformer model to implement a simple word-level language model.\nPrepare Data # As in the previous article, we will use the WikiText-2 dataset. The method of downloading and processing the dataset can be found in the previous article, so I won\u0026rsquo;t repeat it here.\nDefine Model # PyTorch has a built-in torch.nn.Transformer module, which we can use to implement the Transformer model. However, before using the torch.nn.Transformer module, we need to define an embedding layer and a positional encoding layer.\nEmbedding Layer # The knowledge of word embedding has been introduced in the previous article. Here we can directly use the built-in torch.nn.Embedding module to define an embedding layer.\n1 input_embedding = nn.Embedding(vocab_size, embed_size) where vocab_size is the size of the dictionary in the dataset, and embed_size is the dimension of the word embedding.\nPositional Encoding Layer # The so-called positional encoding is to add position information to each word in the input sequence so that the model can better capture the information in the input sequence. We can use the following formula to calculate the positional encoding:\n$p_{(pos, 2i)} = \\sin(pos / 10000^{2i / d_{model}})$\n$p_{(pos, 2i+1)} = \\cos(pos / 10000^{2i / d_{model}})$\nwhere $d_{model}$ is the dimension of the word embedding, $pos$ is the position of the word, and $i$ is the index of the dimension of the word embedding. In simple terms, the effect of the above positional encoding formula is to add a sine or cosine function position information to each dimension of each word. Using sine and cosine functions for positional encoding ensures that the distance between positional encodings of different positions is equal, so that the model can better capture the information in the input sequence.\nWe can use the following code to implement the positional encoding. Here we refer to the PositionalEncoding class in the official PyTorch example code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 class PositionalEncoding(nn.Module): r\u0026#34;\u0026#34;\u0026#34;Inject some information about the relative or absolute position of the tokens in the sequence. The positional encodings have the same dimension as the embeddings, so that the two can be summed. Here, we use sine and cosine functions of different frequencies. .. math: \\text{PosEncoder}(pos, 2i) = sin(pos/10000^(2i/d_model)) \\text{PosEncoder}(pos, 2i+1) = cos(pos/10000^(2i/d_model)) \\text{where pos is the word position and i is the embed idx) Args: d_model: the embed dim (required). dropout: the dropout value (default=0.1). max_len: the max. length of the incoming sequence (default=5000). Examples: \u0026gt;\u0026gt;\u0026gt; pos_encoder = PositionalEncoding(d_model) \u0026#34;\u0026#34;\u0026#34; def __init__(self, d_model, dropout=0.1, max_len=5000): super(PositionalEncoding, self).__init__() self.dropout = nn.Dropout(p=dropout) pe = torch.zeros(max_len, d_model) position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1) div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model)) pe[:, 0::2] = torch.sin(position * div_term) pe[:, 1::2] = torch.cos(position * div_term) pe = pe.unsqueeze(0).transpose(0, 1) self.register_buffer(\u0026#39;pe\u0026#39;, pe) def forward(self, x): r\u0026#34;\u0026#34;\u0026#34;Inputs of forward function Args: x: the sequence fed to the positional encoder model (required). Shape: x: [sequence length, batch size, embed dim] output: [sequence length, batch size, embed dim] Examples: \u0026gt;\u0026gt;\u0026gt; output = pos_encoder(x) \u0026#34;\u0026#34;\u0026#34; x = x + self.pe[:x.size(0), :] return self.dropout(x) In the above code, we define a PositionalEncoding class, which inherits from the nn.Module class and is used to implement the positional encoding. In the __init__ method, we first define a positional encoding matrix pe, then calculate the positional encoding of the sine and cosine functions, and finally add the positional encoding matrix pe to the model\u0026rsquo;s buffer. In the forward method, we add the input sequence x and the positional encoding matrix pe together, and then get the final output through the Dropout layer.\nTransformer Model # With the embedding layer and positional encoding layer defined above, we can use the built-in torch.nn.Transformer module to define a Transformer model. In addition to inheriting from the nn.Module class and implementing the __init__ and forward methods, we also need to define a generate_square_subsequent_mask method to generate a mask matrix, which will be used when calculating the attention weights. In addition, we define an init_weights method to initialize the model\u0026rsquo;s weights.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 class TransformerModel(nn.Transformer): \u0026#34;\u0026#34;\u0026#34;Container module with an encoder, a recurrent or transformer module, and a decoder.\u0026#34;\u0026#34;\u0026#34; def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5): super(TransformerModel, self).__init__(d_model=ninp, nhead=nhead, dim_feedforward=nhid, num_encoder_layers=nlayers) self.model_type = \u0026#39;Transformer\u0026#39; self.src_mask = None self.pos_encoder = PositionalEncoding(ninp, dropout) self.input_emb = nn.Embedding(ntoken, ninp) self.ninp = ninp self.decoder = nn.Linear(ninp, ntoken) self.init_weights() def _generate_square_subsequent_mask(self, sz): return torch.log(torch.tril(torch.ones(sz,sz))) def init_weights(self): initrange = 0.1 nn.init.uniform_(self.input_emb.weight, -initrange, initrange) nn.init.zeros_(self.decoder.bias) nn.init.uniform_(self.decoder.weight, -initrange, initrange) def forward(self, src, has_mask=True): if has_mask: device = src.device if self.src_mask is None or self.src_mask.size(0) != len(src): mask = self._generate_square_subsequent_mask(len(src)).to(device) self.src_mask = mask else: self.src_mask = None src = self.input_emb(src) * math.sqrt(self.ninp) src = self.pos_encoder(src) output = self.encoder(src, mask=self.src_mask) output = self.decoder(output) return F.log_softmax(output, dim=-1) In the above code, we define a TransformerModel class, which inherits from the nn.Transformer class and is used to implement the Transformer model. In the __init__ method, we first call the super function to initialize the nn.Transformer class, then define the embedding layer, positional encoding layer, and linear layer, and finally call the init_weights method to initialize the model\u0026rsquo;s weights. In the forward method, we first generate a mask matrix, then pass the input sequence through the embedding layer and positional encoding layer, and finally pass the output through the encoder and decoder to get the final output.\nRun the Model # With the model defined above, we can define a training function and a testing function, and then encapsulate a main function to load data, train, and test the model. Note that compared to the LSTM model in the previous article, the Transformer model requires an additional parameter nhead to specify the number of attention heads. We can run our model by calling the following command:\n1 python language_transformer.py --plot Each time we run the training code, once the training exceeds one epoch, the original model file model.pt will be overwritten, so if you want to save the previous model, you need to manually rename the model.pt file, or specify the saved model file name (including the path) through the --save parameter when training again. After one epoch, the model will calculate the loss value on the validation set. If the loss value on the validation set is smaller than the previous minimum loss value, the model will be saved in the model.pt file.\nOn my personal computer, if training with GPU (Nvidia GeForce RTX 4060 Ti), each epoch takes about 28 seconds, and the memory usage is about 548MB; if training with CPU (Intel i5 9600K), each epoch takes about 516 seconds. Here I trained a total of 50 epochs, and the loss values of the training set and validation set are shown in the figure below:\nAs we can see, the result is similar to the LSTM model in the previous article. The Transformer model converges slightly faster than the LSTM model, but the loss value on the validation set is slightly larger.\nGenerate Text # After training the model, the code will save the model in the model.pt file by default. We can load this model and use it to generate some text. The method of generating text is similar to the previous article, and the code is in the generate_text.py file. We can call the following command to generate some text:\n1 python generate_text.py An odd situation is that generating text can only be done with GPU on my computer. If the --no-cuda parameter is specified to use CPU to generate text, my computer will crash and restart directly, and even the log is not output (at least I didn\u0026rsquo;t find it). One possible reason is that the CUDA version of my computer is 12.5, while the CUDA version of PyTorch is 12.1, and the crash and restart may be related to this (but I\u0026rsquo;m not sure, because training the model can be done without CUDA, only with CPU, theoretically generating text with CPU should not use CUDA, so the crash should not be related to CUDA). In addition, my computer has 64GB of memory, much larger than the GPU memory, so memory shortage should not be a problem. If someone has encountered similar problems, please leave a comment to let me know, and we can discuss it, thank you!\nThe speed of generating text with the Transformer model is slightly slower than the LSTM model. On my computer, generating 1000 words with GPU takes about 6 seconds. Since generating text with CPU will cause the computer to crash and restart, I did not test the speed of generating text with CPU.\nSummary # In this article, we briefly introduced the Transformer model and the attention mechanism, and then referred to the word_language_model example in the official PyTorch examples to implement a simple word-level language model using the torch.nn.Transformer module in PyTorch. We also introduced the classification of Transformer models, including the encoder model, decoder model, and encoder-decoder model. The model\u0026rsquo;s training results are similar to the LSTM model in the previous article. Finally, we can use the trained Transformer model to generate some text.\n","date":"23 September 2024","externalUrl":null,"permalink":"/en/p/learn-pytorch-by-examples-7-language-model-ii-implement-word-level-language-model-with-transformer/","section":"Posts","summary":" Background # This is the 7th article in the “Learn PyTorch by Examples” series. In the 6th article “Learn PyTorch by Examples (6): Language Model (I) – Implementing a Word-Level Language Model with LSTM”, we briefly introduced how to implement a word-level language model using LSTM.\nLSTM and other models based on Recurrent Neural Networks (RNN) have been widely used in natural language processing, but these models have some problems when dealing with long-distance dependency problems, such as vanishing gradients and exploding gradients. To solve these problems, researchers proposed the Transformer model, which uses the attention mechanism to better handle long-distance dependency problems. In this article, we will briefly introduce the Transformer model and use the Transformer to implement a simple word-level language model. This article refers to the word_language_model example in the official PyTorch examples.\n","title":"Learn PyTorch by Examples (7): Language Model (II) - Implement Word-Level Language Model with Transformer","type":"post"},{"content":"","date":"23 September 2024","externalUrl":null,"permalink":"/en/categories/machine-learning/","section":"Categories","summary":"","title":"Machine Learning","type":"categories"},{"content":"","date":"23 September 2024","externalUrl":null,"permalink":"/en/tags/machine-learning/","section":"Tags","summary":"","title":"Machine Learning","type":"tags"},{"content":"","date":"23 September 2024","externalUrl":null,"permalink":"/en/tags/neural-network/","section":"Tags","summary":"","title":"Neural Network","type":"tags"},{"content":"","date":"23 September 2024","externalUrl":null,"permalink":"/en/tags/pytorch/","section":"Tags","summary":"","title":"PyTorch","type":"tags"},{"content":"","date":"23 September 2024","externalUrl":null,"permalink":"/en/tags/transformer/","section":"Tags","summary":"","title":"Transformer","type":"tags"},{"content":"","date":"2024年9月23日","externalUrl":null,"permalink":"/series/%E5%AE%9E%E4%BE%8B%E5%AD%A6pytorch/","section":"Series","summary":"","title":"实例学PyTorch","type":"series"},{"content":"","date":"2024年9月23日","externalUrl":null,"permalink":"/categories/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/","section":"Categories","summary":"","title":"机器学习","type":"categories"},{"content":"","date":"2024年9月23日","externalUrl":null,"permalink":"/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/","section":"Tags","summary":"","title":"机器学习","type":"tags"},{"content":"","date":"2024年9月23日","externalUrl":null,"permalink":"/tags/%E6%B3%A8%E6%84%8F%E5%8A%9B%E6%9C%BA%E5%88%B6/","section":"Tags","summary":"","title":"注意力机制","type":"tags"},{"content":"","date":"2024年9月23日","externalUrl":null,"permalink":"/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/","section":"Tags","summary":"","title":"深度学习","type":"tags"},{"content":"","date":"2024年9月23日","externalUrl":null,"permalink":"/tags/%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C/","section":"Tags","summary":"","title":"神经网络","type":"tags"},{"content":"","date":"2024年9月23日","externalUrl":null,"permalink":"/tags/%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B/","section":"Tags","summary":"","title":"语言模型","type":"tags"},{"content":" Motivation # I installed the Steam game platform and Proton compatibility layer on my Ubuntu computer before, which allows running Windows games on Linux. However, I didn\u0026rsquo;t play games much, so I didn\u0026rsquo;t study it carefully. I just knew that the Proton compatibility layer is quite powerful, and most games run without any problems, even if the \u0026ldquo;system requirements\u0026rdquo; of the game only show support for Windows systems.\nLater, I installed an Nvidia RTX 4060 Ti graphics card, mainly to learn CUDA programming and deep learning. Recently, \u0026ldquo;Black Myth: Wukong\u0026rdquo; became very popular, and my roommate wanted to play on my computer, so he logged into his Steam account on my computer and downloaded \u0026ldquo;Black Myth: Wukong\u0026rdquo;. He found that it ran without any problems.\nAfter watching my roommate play for a while, I also wanted to try it, but I didn\u0026rsquo;t want to switch Steam accounts back and forth. So I created another system account on Ubuntu, started and logged into Steam, but I found that the games downloaded by my roommate on another account did not appear on my account. Since these games are very large, I don\u0026rsquo;t want to download them again, so I tried to run the games downloaded by my roommate on my account. In the process, I encountered one problem after another, and finally solved it. Here I record it for reference for those who encounter the same problem.\nPrerequisites # If you only want to play Steam games on Linux by yourself, you only need to install Steam and Proton. You can follow the steps in my previous article \u0026ldquo;Nvidia GPU Settings on Ubuntu: Games, CUDA, Deep Learning, Docker, etc.\u0026rdquo;. If you don\u0026rsquo;t want to click in and read, I\u0026rsquo;ll list the steps here briefly:\nInstall Steam # Download the Steam installation package: 1 wget https://cdn.cloudflare.steamstatic.com/client/installer/steam.deb Install Steam: 1 sudo dpkg -i steam.deb Install missing dependencies: 1 sudo apt-get install -f Run Steam: 1 steam Log in to your Steam account Install Proton # In Steam, select games that can run on Linux, then click Settings, in the Steam Play tab, check Enable Steam Play for supported titles and Enable Steam Play for all other titles, then select a Proton version from the Steam Play drop-down menu, and click OK.\nAfter Proton is installed, you can run Windows games on Linux.\nInstall Games # If you are using Steam for the first time and have not purchased any games, you can choose some free games for testing, such as \u0026ldquo;Dota 2\u0026rdquo;, \u0026ldquo;Counter-Strike: Global Offensive\u0026rdquo;, etc.\nShare Games Among Multiple Users # If you want multiple users to play the same game, and don\u0026rsquo;t want to download it again (after all, a large game may have tens of GBs or even hundreds of GBs), you can continue to see how I operate.\nSteam Runtime Environment # Before creating a second user and sharing the same game files, we need to understand the runtime environment of Steam on Linux and the location of the main files.\nAfter installing Steam and Proton according to the steps above, Steam is installed in the system directory, so all users can use the Steam client.\nWhen a user user1 starts the Steam client, Steam will create a .steam directory in the home directory of the user1 user, which contains the user\u0026rsquo;s Steam configuration files, game files, etc. In the .steam directory, there is a steam directory, which contains the user\u0026rsquo;s Steam client program, and a steamapps directory, which contains the user\u0026rsquo;s downloaded game files.\nTherefore, if we want to reuse a game, the essence is to let other users also access the game files in the /home/user1/.steam/steam/steamapps directory.\nCreate a Second User # We create a new system account for the second user, add it to the sudo group, and set a password:\n1 2 3 sudo adduser user2 sudo usermod -aG sudo user2 sudo passwd user2 Switch to the user2 user:\nClick the user icon in the upper right corner of the screen, select Switch Account, and then select the user2 user to log in.\nOpen and log in to Steam:\nAfter opening the Steam client, Steam will create a .steam directory in the home directory of the user2 user, but the .steam/steam/steamapps directory of the user2 user does not contain game files. Therefore, when you click the Library page, you will find that the library does not show the games downloaded by the user1 user.\nNext, we will mainly solve this problem, allowing the user2 user to access the game files downloaded by the user1 user and run the games normally.\nShare Game Files # The game files downloaded by the user1 user are located in the /home/user1/.steam/steam/steamapps directory. However, we want multiple users to share these game files, so it is best to move these game files to a public directory, such as /opt/games/steam.\n1 2 sudo mkdir -p /opt/games/steam sudo mv /home/user1/.steam/steam/steamapps /opt/games/steam To allow the user2 user to access the game files in the /opt/games/steam directory, we need to set the permissions of the /opt/games/steam directory to 755:\n1 sudo chmod 775 /opt/games/steam Next we create a new user group steam and add the user1 and user2 users to this user group:\n1 2 3 sudo groupadd steam sudo usermod -aG steam user1 sudo usermod -aG steam user2 Then, we set the user group of the /opt/games/steam directory to steam:\n1 sudo chown -R user1:steam /opt/games/steam Finally, we need to set the setgid permission for the shared folder, so that any files and directories created in the shared folder will have the same user group as the user group of the shared folder. That is, the user group of /opt/games/steam is steam, and the user group of files and directories created in /opt/games/steam will also be steam:\n1 sudo chmod g+s /opt/games/steam To allow the Steam client of the user1 user to access the game files in the /opt/games/steam directory, we need to create a symbolic link in the home directory of the user1 user:\n1 ln -s /opt/games/steam/steamapps /home/user1/.steam/steam/steamapps Then we open the Steam client under user1, click the Library page, and you will find that the previously downloaded game files are there. If you don\u0026rsquo;t see it, try restarting the Steam client or restarting the computer.\nSimilarly for the user2 user, we need to create a symbolic link in the home directory of the user2 user:\n1 ln -s /opt/games/steam/steamapps /home/user2/.steam/steam/steamapps Then we open the Steam client under user2 (you may need to restart the Steam client or restart the computer first), click the Library page, and you should see the previously downloaded game files.\nHowever, if you click the Play button, you will find that the game does not start, and there may be no prompts. If you want to find the problem through error logs, you can exit the Steam client, and then run the Steam client in the terminal, so you can see the output information of Steam in the terminal, something like this:\n1 pressure-vessel-wrap[44758]: E: openat(/opt/games/steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20240806.97927/files/.ref): Permission denied Or like this:\n1 2 3 4 5 pressure-vessel-wrap[4109614]: W: For best results, \u0026#34;/opt/games/steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20240820.99315/files\u0026#34; and \u0026#34;/opt/games/steam/steamapps/common/SteamLinuxRuntime_sniper/var/tmp-7P6EU2/usr\u0026#34; should both be on the same fully-featured Linux filesystem. Adding process 4109770 for gameID 1264970 wineserver: /opt/games/steam/steamapps/compatdata/1264970/pfx is not owned by you wine: using kernel write watches, use_kernel_writewatch 1. wine: \u0026#39;/opt/games/steam/steamapps/compatdata/1264970/pfx\u0026#39; is not owned by you You can also right-click on the game in the Library page, then select Properties, click Verify Integrity in the Local Files tab, and you will find that the integrity check fails.\nThe above problem is quite strange, because we have set the permissions of the /opt/games/steam directory to 755, and added the user1 and user2 users to the steam user group, but the error message above indicates that the Steam client under the user2 user still cannot access the /opt/games/steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20240806.97927/files/.ref file. Even more strange is that the size of this file is 0! I don\u0026rsquo;t know what this file is for, but after some exploration, I found some related discussions online:\nhttps://ubuntuforums.org/showthread.php?t=2494677 https://github.com/ValveSoftware/Proton/issues/4820 https://github.com/ValveSoftware/steam-for-linux/issues/3942 These discussions mainly propose two solutions:\nAfter switching system accounts, change the ownership of the /opt/games/steam directory to the current user:\n1 sudo chown -R user2:steam /opt/games/steam This way, the user2 user can access all files in the /opt/games/steam directory.\nHowever, this solution is a temporary solution, because every time you switch system accounts, you need to change the ownership of the /opt/games/steam directory, which is obviously inconvenient.\nThe essence of the above problem is that the Wine used by the Proton compatibility layer does not allow users to access prefix directories owned by other users for security reasons. Therefore, the solution is to create different prefix directories for different users when starting Steam. The prefix directory of Wine is an important concept of the Proton compatibility layer, which stores the game\u0026rsquo;s configuration files, cache files, etc. By default, Proton stores the prefix directory in the ~/.steam/steam/steamapps/compatdata/ directory, but we can modify the Proton code to create different prefix directories according to the current user.\nTo implement this solution, we need to modify the Proton code and then recompile Proton. In fact, there are only two lines of code to modify, see Proton pull request #4861. This pull request was created on GitHub in May 2021, but it has not been merged so far. I don\u0026rsquo;t know why the Proton developers have not merged this simple but useful pull request. Since the official has not added this feature, we will implement it ourselves next.\nModify and Compile Proton # First, we need to download the source code of Proton:\n1 2 git clone --recurse-submodules https://github.com/ValveSoftware/Proton.git proton cd proton Then, we need to switch to a stable branch, such as proton_9.0. We also need to update the submodules:\n1 2 git checkout proton_9.0 git submodule update --init --recursive Next, we need to modify the code of Proton. To do this, we create a new branch, for example, called myfeature-pr4861:\n1 git checkout -b myfeature-pr4861 Open the proton file in the root directory of the Proton project (this is a Python file), find the initialization function __init__ of the CompatData class. We need to modify the line self.base_dir and add a line. The modified code is as follows:\n1 2 3 4 class CompatData: def __init__(self, compatdata): self.base_dir = compatdata + \u0026#34;/\u0026#34; + str(os.getuid()) + \u0026#34;/\u0026#34; os.makedirs(self.base_dir, exist_ok=True) We only need to modify these two lines of code, see Proton pull request #4861.\nThis idea is actually very simple. The underlying layer of Proton uses Wine to run games, and the prefix directory of Wine is stored in the ~/.steam/steam/steamapps/compatdata/ directory. Before making the above modifications, when we run a game as user1, Proton will store the prefix directory in the ~/.steam/steam/steamapps/compatdata/ directory, and its ownership is user1. When we run the same game as user2, Wine will try to access the prefix directory in the ~/.steam/steam/steamapps/compatdata/ directory, but because its ownership is user1, Wine will refuse to access it.\nThe above modification is to let Proton create different prefix directories according to the current user. For example, if the uid of user1 is 1000 and the uid of user2 is 1001, when user1 runs a game, Proton will store the prefix directory in the ~/.steam/steam/steamapps/compatdata/1000/ directory, and its ownership is user1; when user2 runs the game, Proton will store the prefix directory in the ~/.steam/steam/steamapps/compatdata/1001/ directory, and its ownership is user2. This solves the ownership conflict problem mentioned above.\nNext, we compile Proton. Proton uses container technology, so we need a container tool, such as Docker or Podman. Docker is already installed on my computer, but Docker generally requires sudo permission to run, which will cause a series of permission issues. Therefore, I recommend using Podman, which does not require sudo permission.\nFirst, we need to install Podman:\n1 sudo apt install podman Then, we create a build folder, which can be placed at the same level as the Proton project:\n1 2 3 cd .. mkdir build cd build Next, we configure the build environment using the configuration tool provided by the Proton project:\n1 ../proton/configure.sh --enable-ccache --build-name=myfeature-9.0-pr4861 Finally, we start compiling Proton:\n1 make After the compilation is completed, we create the installation files of Proton:\n1 make redist This way, a redist directory will be generated in the build directory, which contains the installation files of Proton.\nInstall Proton. For user-defined compatibility layers, we can place them in the ~/.steam/root/compatibilitytools.d/ directory. We copy the redist directory to the ~/.steam/root/compatibilitytools.d/ directory:\n1 cp -r redist ~/.steam/root/compatibilitytools.d/myfeature-9.0-pr4861 Note that no matter which account uses Steam, we need to copy the redist directory to the ~/.steam/root/compatibilitytools.d/ directory and change the ownership of the directory to the current user. For example, if it is the user_x user, we can do this:\n1 sudo chown -R user_x:user_x ~/.steam/root/compatibilitytools.d/myfeature-9.0-pr4861 Use Customized Proton # After completing the above steps, Steam will be able to detect our customized Proton version.\nNo matter which account uses Steam, we only need to select our customized Proton version in Steam. After opening Steam, click Steam in the upper left corner, then select Settings, select Compatibility in the sidebar, select our customized Proton version myfeature-9.0-pr4861 from the drop-down menu under Run other products with, and then click OK.\nFinally, select a game and click Play, a dialog box will pop up asking you whether to run the game in compatibility mode. I found that it is actually okay not to select compatibility mode:\n![Run in compatibility mode](https://img.jinli.io/images/2024/09/19/steam_compatible_mode.md.png) Steam will spend some time compiling shaders, and then you can enter the game interface!\n![Black Myth: Wukong](https://img.jinli.io/images/2024/09/19/black_myth_wukong.md.png) A Small Issue # When playing games, I found that when the game plays a long cutscene, the monitor will display no input signal after about 30 seconds due to no operation. I suspect that it is caused by the power management mechanism of the graphics card. Running the xset -dpms command can disable power management, which ensures that the display signal will not be interrupted.\nWith xset q command, you can view the current power management settings, and with the xset command, you can view the help information of the xset command.\n","date":"19 September 2024","externalUrl":null,"permalink":"/en/p/linux-system-share-steam-games-among-multiple-users-ubuntu-as-an-example/","section":"Posts","summary":" Motivation # I installed the Steam game platform and Proton compatibility layer on my Ubuntu computer before, which allows running Windows games on Linux. However, I didn’t play games much, so I didn’t study it carefully. I just knew that the Proton compatibility layer is quite powerful, and most games run without any problems, even if the “system requirements” of the game only show support for Windows systems.\n","title":"Linux System: Share Steam Games Among Multiple Users (Ubuntu as an Example)","type":"post"},{"content":"","date":"19 September 2024","externalUrl":null,"permalink":"/en/tags/proton/","section":"Tags","summary":"","title":"Proton","type":"tags"},{"content":"","date":"19 September 2024","externalUrl":null,"permalink":"/en/tags/steam/","section":"Tags","summary":"","title":"Steam","type":"tags"},{"content":"","date":"9 September 2024","externalUrl":null,"permalink":"/en/series/learn-pytorch-by-examples/","section":"Series","summary":"","title":"Learn PyTorch by Examples","type":"series"},{"content":" Background # This is the sixth article in the \u0026ldquo;Learning PyTorch by Examples\u0026rdquo; series. In the fourth and fifth articles, we introduced the sequence prediction problem and implemented the prediction of the sine function with RNN, GRU, and LSTM.\n\u0026ldquo;Learn PyTorch by Examples (4): Sequence Prediction (I) \u0026ndash; Recurrent Neural Networks (RNN)\u0026rdquo; \u0026ldquo;Learn PyTorch by Examples (5): Sequence Prediction (II) \u0026ndash; Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM)\u0026rdquo; In the fourth article, we mentioned that in addition to time series like the sine function, sequence data can also be word sequences in language models. In this article, we will introduce how to implement a word-level language model with LSTM.\nThe code for this article can be found in the T06_word_lstm folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial.\nLanguage Model # A language model is an important problem in natural language processing. It is a model used to evaluate the probability of a sentence. Language models can be used to predict the next word, or to generate a sentence. Language models have a wide range of applications in machine translation, speech recognition, text generation, etc.\nA language model is also a sequence prediction problem, that is, when predicting the next word, we need to consider not only the current word but also the previous words. For example, if the current word is \u0026ldquo;apple\u0026rdquo;, when predicting the next word, we need to consider whether this \u0026ldquo;apple\u0026rdquo; refers to a fruit or a company, which requires information provided by the previous words. For example, if the previous words are \u0026ldquo;eat\u0026rdquo;, \u0026ldquo;banana\u0026rdquo;, \u0026ldquo;pear\u0026rdquo;, etc., then \u0026ldquo;apple\u0026rdquo; is likely to refer to a fruit; if the previous words are \u0026ldquo;phone\u0026rdquo;, \u0026ldquo;computer\u0026rdquo;, \u0026ldquo;Jobs\u0026rdquo;, etc., then \u0026ldquo;apple\u0026rdquo; is likely to refer to a company.\nIn the previous articles, we briefly introduced that the sequence prediction problem can be solved using recurrent neural networks (RNN) and its variants, such as long short-term memory networks (LSTM) and gated recurrent units (GRU). In the official PyTorch example code, there is an example of implementing a language model using RNN and Transformer. Here we will use LSTM to implement a simple word-level language model based on this example code. In addition, some of the images and code in this article are referenced and quoted from the YouTube blogger Donato Capitella\u0026rsquo;s video \u0026ldquo;LLM Chronicles #4.4: Building a Word-Level Language Model in PyTorch using RNNs\u0026rdquo;.\nOf course, in recent years, with the advent of Transformer, the performance of language models has been greatly improved. However, LSTM, as a classic recurrent neural network, still has a wide range of applications. So here we will first introduce how to implement a simple language model using LSTM. In the following articles, we will introduce Transformer and its variants, and how to use Transformer to implement a language model.\nLanguage models are divided into character-level, word-level, subword-level, etc., depending on how the sequence is segmented. Character-level means that single characters are used as units, word-level means that words are used as units, and subword-level means that words are segmented into morphemes. For example, the word \u0026ldquo;joyfulness\u0026rdquo; can be segmented into \u0026ldquo;joy\u0026rdquo;, \u0026ldquo;ful\u0026rdquo;, and \u0026ldquo;ness\u0026rdquo;.\nWord-Level Language Model # A word-level language model is a model used to predict the next word. The input of a word-level language model is a word sequence, and the output is also a word sequence. During training, we use the previous words in a sentence as input and the following words as output, and train the model by minimizing the difference between the predicted word and the true word. During testing, we can use the model to predict the next word.\nThe process of implementing a word-level language model with LSTM is similar to that of implementing a sine function prediction with LSTM or RNN:\nData Preparation: We need to convert the word sequence into an integer sequence so that it can be input into the model. Model Construction: We need to build an LSTM model to predict the next word. Model Training: We need to train the model using the dataset so that the model can predict the next word. Model Testing: We need to use the model to predict the next word. Data Preparation # Here we use the WikiText-2 dataset, which is a common language model dataset containing some Wikipedia entries. The content of the dataset can be found here: WikiText-2.\nThe difference between a language model and a sine function sequence model is that the input of the sine function sequence is a number, while the input of the language model is a word. PyTorch can handle mathematical tensors, so we need to convert words into numbers.\nData Preprocessing # The WikiText-2 dataset is a text file, and we need to convert the text file into a word sequence. PyTorch has a torchtext library that can be used to process text data, but after April 2024, this library was no longer maintained. So here we need to write some code to preprocess the training data ourselves.\nThe basic idea of preprocessing is:\nRead all the text and use a dictionary to save all the unique words. The keys of the dictionary are words, and the values are the numbers corresponding to the words. Replace all the words in the original text with numbers. The code we use here is from the official PyTorch example code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 import os from io import open import torch class Dictionary(object): def __init__(self): self.word2idx = {} self.idx2word = [] def add_word(self, word): if word not in self.word2idx: self.idx2word.append(word) self.word2idx[word] = len(self.idx2word) - 1 return self.word2idx[word] def __len__(self): return len(self.idx2word) class Corpus(object): def __init__(self, path): self.dictionary = Dictionary() self.train = self.tokenize(os.path.join(path, \u0026#39;train.txt\u0026#39;)) self.valid = self.tokenize(os.path.join(path, \u0026#39;valid.txt\u0026#39;)) self.test = self.tokenize(os.path.join(path, \u0026#39;test.txt\u0026#39;)) def tokenize(self, path): \u0026#34;\u0026#34;\u0026#34;Tokenizes a text file.\u0026#34;\u0026#34;\u0026#34; assert os.path.exists(path) # Add words to the dictionary with open(path, \u0026#39;r\u0026#39;, encoding=\u0026#34;utf8\u0026#34;) as f: for line in f: words = line.split() + [\u0026#39;\u0026lt;eos\u0026gt;\u0026#39;] for word in words: self.dictionary.add_word(word) # Tokenize file content with open(path, \u0026#39;r\u0026#39;, encoding=\u0026#34;utf8\u0026#34;) as f: idss = [] for line in f: words = line.split() + [\u0026#39;\u0026lt;eos\u0026gt;\u0026#39;] ids = [] for word in words: ids.append(self.dictionary.word2idx[word]) idss.append(torch.tensor(ids).type(torch.int64)) ids = torch.cat(idss) return ids Here we define a Dictionary class and a Corpus class. The Dictionary class is used to save the correspondence between words and numbers, and the Corpus class is used to read text files and convert them into integer sequences. Here we convert the training, validation, and test sets separately.\nAfter preprocessing, we get three integer sequences for the training, validation, and test sets, with lengths of 2088628, 217646, and 245569, respectively. The size of the dictionary is 33278.\nData Batching # After preprocessing, we get a very long integer sequence. To facilitate training, we divide this sequence into several small sequences. For example, if the length of the sequence is 10000, and we want to divide it into 20 batches, then the length of each batch is 500. In this way, we get a matrix of shape $500 \\times 20$.\nNote:\nThe converted matrix is column-major, i.e., the first column is the data of the first batch, the second column is the data of the second batch, and so on. If the length of the data cannot be divided by the batch length, we discard the extra data. The batches are independent of each other, i.e., there is no relationship between the last data of the first batch and the first data of the second batch. This also means that some context information will be lost after batching. The code for batching is very simple:\n1 2 3 4 5 6 7 8 def batchify(data, bsz): # Work out how cleanly we can divide the dataset into bsz parts. nbatch = data.size(0) // bsz # Trim off any extra elements that wouldn\u0026#39;t cleanly fit (remainders). data = data.narrow(0, 0, nbatch * bsz) # Evenly divide the data across the bsz batches. data = data.view(bsz, -1).t().contiguous() return data Word Embedding # In the previous articles, we used one-hot encoding to encode labels. But for words, one-hot encoding is not suitable because there are too many words (e.g., there are 33278 words in the dictionary of WikiText-2), and one-hot encoding will result in high dimensions and high computational complexity. Therefore, we need to use another method to represent words, mapping words to a low-dimensional space. This step is called word embedding.\nIn simple terms, the idea of word embedding is to use several features to represent a word, such as the part of speech, sentiment, semantics, etc. For example, we select 7 features: \u0026ldquo;living being\u0026rdquo;, \u0026ldquo;feline\u0026rdquo;, \u0026ldquo;human\u0026rdquo;, \u0026ldquo;gender\u0026rdquo;, \u0026ldquo;royalty\u0026rdquo;, \u0026ldquo;verb\u0026rdquo;, \u0026ldquo;plural\u0026rdquo;, and use a number between -1 and 1 to describe the degree of these features for a word, combining the values representing all features to get a 7-dimensional vector, which is the word embedding of this word. For example, for the word \u0026ldquo;man\u0026rdquo;, we can use the vector $[0.6, -0.2, 0.8, 0.9, -0.1, -0.9, -0.7]$ to represent its word embedding.\nSimilarly, we can represent all the words in the dictionary as word embeddings. The following figure shows some examples:\nWord embedding is a very important concept in language models because it can help our language model better represent the semantic information of words and clearly see the relationship between words. For example, if we subtract \u0026ldquo;man\u0026rdquo; from \u0026ldquo;king\u0026rdquo; and then add to \u0026ldquo;woman\u0026rdquo;, the result should be very close to \u0026ldquo;queen\u0026rdquo;.\nOf course, in practice, we will not use only 7 features, but generally use hundreds of features, so that we can fully represent the semantic information of words. For this problem, we choose to use 200 features, so we need to map the 33278 words in the dictionary to a 200-dimensional space. If we use a matrix to represent this, the word embedding matrix is a $33278 \\times 200$ matrix.\nThis idea seems reasonable, but how do we get this word embedding matrix? We can also train a neural network to get this word embedding matrix. The input of this neural network is the word index, and the output is the word embedding of this word. The training objective of this neural network is to minimize the difference between the predicted word embedding and the true word embedding. The training process of this neural network is similar to the training process of the language model, except that the input and output are different.\nOf course, here we don\u0026rsquo;t need to train this word embedding matrix ourselves, because researchers have trained many such word embedding matrices, and we can directly use these word embedding matrices. These word embedding matrices can be general or trained for a specific task. In PyTorch, we can use torch.nn.Embedding to load these word embedding matrices.\n1 torch.nn.Embedding(ntoken, emsize) The first parameter is the size of the dictionary, and the second parameter is the dimension of the word embedding. This function will return an Embedding object, which we will use as a layer in the LSTM model.\nLSTM Model # With the word embedding, we can build an LSTM model. Here we use torch.nn.LSTM in PyTorch as the basis to build a simple LSTM model. The structure of this model is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 import torch.nn as nn import torch.nn.functional as F class LanguageLSTM(nn.Module): \u0026#34;\u0026#34;\u0026#34;Container module with an encoder, a recurrent module, and a decoder.\u0026#34;\u0026#34;\u0026#34; def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0.5, tie_weights=False): super(LanguageLSTM, self).__init__() self.ntoken = ntoken self.drop = nn.Dropout(dropout) self.encoder = nn.Embedding(ntoken, ninp) self.rnn = getattr(nn, rnn_type)(ninp, nhid, nlayers, dropout=dropout) self.decoder = nn.Linear(nhid, ntoken) self.init_weights() self.rnn_type = rnn_type self.nhid = nhid self.nlayers = nlayers def init_weights(self): initrange = 0.1 nn.init.uniform_(self.encoder.weight, -initrange, initrange) nn.init.zeros_(self.decoder.bias) nn.init.uniform_(self.decoder.weight, -initrange, initrange) def forward(self, input, hidden): emb = self.drop(self.encoder(input)) output, hidden = self.rnn(emb, hidden) output = self.drop(output) decoded = self.decoder(output) decoded = decoded.view(-1, self.ntoken) return F.log_softmax(decoded, dim=1), hidden def init_hidden(self, bsz): weight = next(self.parameters()) return (weight.new_zeros(self.nlayers, bsz, self.nhid), weight.new_zeros(self.nlayers, bsz, self.nhid)) This model is similar to the previous LSTM model, but here we use a word embedding layer. The input of this model is an integer sequence, and the output is a probability distribution, indicating the probability that the next word is which word. The training objective of this model is to minimize the difference between the predicted probability distribution and the true probability distribution.\nTraining and Testing the Model # With the data and model, we can start training the model. The training code is similar to the previous LSTM model, but here we generally use CrossEntropyLoss as the loss function.\nIn addition to outputting the loss value on the command line during training, we also return the weighted average of the loss values of all batches in an epoch at the end, which is convenient for plotting later.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 def train(device, model, epoch, train_data, batch_size, criterion, lr, log_interval, seq_len): model.train() total_loss = 0. loss_all = [] data_cnt = [] start_time = time.time() hidden = model.init_hidden(batch_size) for batch, i in enumerate(range(0, train_data.size(0) - 1, seq_len)): data, targets = get_batch(train_data, i) data, targets = data.to(device), targets.to(device) model.zero_grad() hidden = repackage_hidden(hidden) output, hidden = model(data, hidden) loss = criterion(output, targets) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 0.25) for p in model.parameters(): p.data.add_(p.grad, alpha=-lr) total_loss += loss.item() loss_all.append(loss.item()) data_cnt.append(len(data)) if batch % log_interval == 0 and batch \u0026gt; 0: cur_loss = total_loss / log_interval elapsed = time.time() - start_time print(\u0026#39;| epoch {:3d} | {:5d}/{:5d} batches | lr {:02.2f} | ms/batch {:5.2f} | \u0026#39; \u0026#39;loss {:5.2f} | ppl {:8.2f}\u0026#39;.format( epoch, batch, len(train_data) // seq_len, lr, elapsed * 1000 / log_interval, cur_loss, math.exp(cur_loss))) total_loss = 0 start_time = time.time() return np.average(loss_all, weights=data_cnt) The testing code is similar to the previous LSTM model, and we don\u0026rsquo;t need to change it much.\nModel Performance # All the code can be found in the T06_word_lstm folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial. After setting up the environment, we can run language_lstm.py to train the model.\n1 python language_lstm.py On my personal computer, if training with GPU (Nvidia GeForce RTX 4060 Ti), each epoch takes about 26 seconds, and the memory usage is about 540MB; if training with CPU (Intel i5 9600K), each epoch takes about 506 seconds. Here I trained a total of 50 epochs, and the loss values of the training and validation sets are shown in the following figure:\nIt can be seen that after training for 20 epochs, the loss value of the model basically stabilized at around 4.1, and the loss value of the validation set also basically stabilized at around 4.7. This indicates that the model\u0026rsquo;s generalization ability is generally good. Moreover, our training data is not large, so training for 20 epochs is basically enough.\nGenerating Text with the Model # When the model training is completed, the trained model will be saved as the model.pt file in the current directory. We can use this model to generate text. The code for generating text is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 def generate_text(device, checkpoint, data_source, words, temperature, log_interval): with open(checkpoint, \u0026#39;rb\u0026#39;) as f: model = torch.load(f, map_location=device) model.eval() corpus = data.Corpus(data_source) ntokens = len(corpus.dictionary) hidden = model.init_hidden(1) input = torch.randint(ntokens, (1, 1), dtype=torch.long).to(device) generated_text = [] with torch.no_grad(): # no tracking history for i in range(words): output, hidden = model(input, hidden) word_weights = output.squeeze().div(temperature).exp().cpu() word_idx = torch.multinomial(word_weights, 1)[0] input.fill_(word_idx) word = corpus.dictionary.idx2word[word_idx] generated_text.append(word) if i % log_interval == 0: print(\u0026#39;| Generated {}/{} words\u0026#39;.format(i, words)) return generated_text In this function, checkpoint is the path of the model file, data_source is the path of the dataset, words is the number of words to generate, temperature is a parameter that controls the diversity of the generated text, and log_interval is the number of words to output every time. The complete code for generating text and saving it can be found in generate_text.py. Run this code:\n1 python generate_text.py We can get the generated text. Here I generated 1000 words, and part of the generated text is as follows:\n1 – \u0026lt;unk\u0026gt; , a year then with the software . It usually was sold for nearly half the day time . For this reason , the Nevermind run surpassed and a new group of canned \u0026lt;unk\u0026gt; . It had benefited from the unhealthy content , which have been leveled on the \u0026lt;unk\u0026gt; \u0026#39;s gates through the design the effects products associated with other birds and tested stewardship of those articles , ranging from an upright system with \u0026lt;unk\u0026gt; \u0026lt;unk\u0026gt; . Because the WikiText-2 dataset contains many non-ASCII characters, there may be some \u0026lt;unk\u0026gt; characters in the generated text, which is because these characters are not in our dictionary. In addition, the generated text is randomly generated, so it may not be coherent. We can adjust the temperature parameter to control the diversity of the generated text. The larger the temperature, the more diverse the generated text, and the smaller the temperature, the more conservative the generated text.\nSummary # This article introduces how to implement a simple word-level language model using LSTM. Language models are an important problem in natural language processing, which can be used to predict the next word or generate a sentence. Language models are a sequence prediction problem, and here we use LSTM to solve this problem, but other RNN models can also be used.\nAlthough using RNN and its variants to implement language models can achieve good results, their performance on complex tasks is still limited. In recent years, with the advent of Transformer, the performance of language models has been greatly improved. In the following articles, we will introduce Transformer and its variants, and how to use Transformer to implement a language model.\n","date":"9 September 2024","externalUrl":null,"permalink":"/en/p/learn-pytorch-by-examples-6-language-model-i--implementing-a-word-level-language-model-with-lstm/","section":"Posts","summary":" Background # This is the sixth article in the “Learning PyTorch by Examples” series. In the fourth and fifth articles, we introduced the sequence prediction problem and implemented the prediction of the sine function with RNN, GRU, and LSTM.\n“Learn PyTorch by Examples (4): Sequence Prediction (I) – Recurrent Neural Networks (RNN)” “Learn PyTorch by Examples (5): Sequence Prediction (II) – Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM)” In the fourth article, we mentioned that in addition to time series like the sine function, sequence data can also be word sequences in language models. In this article, we will introduce how to implement a word-level language model with LSTM.\n","title":"Learn PyTorch by Examples (6): Language Model (I) -- Implementing a Word-Level Language Model with LSTM","type":"post"},{"content":"","date":"9 September 2024","externalUrl":null,"permalink":"/en/tags/lstm/","section":"Tags","summary":"","title":"LSTM","type":"tags"},{"content":"","date":"9 September 2024","externalUrl":null,"permalink":"/en/tags/recurrent-neural-network/","section":"Tags","summary":"","title":"Recurrent Neural Network","type":"tags"},{"content":"","date":"2024年9月9日","externalUrl":null,"permalink":"/tags/%E5%BA%8F%E5%88%97%E9%A2%84%E6%B5%8B/","section":"Tags","summary":"","title":"序列预测","type":"tags"},{"content":"","date":"13 August 2024","externalUrl":null,"permalink":"/en/tags/chatgpt/","section":"Tags","summary":"","title":"ChatGPT","type":"tags"},{"content":"","date":"13 August 2024","externalUrl":null,"permalink":"/en/tags/lobechat/","section":"Tags","summary":"","title":"LobeChat","type":"tags"},{"content":" Motivation # After installing a graphics card on my server, I want to make full use of this graphics card, so I thought of deploying a large language model. In this way, I don\u0026rsquo;t need to subscribe to ChatGPT every month. After all, I don\u0026rsquo;t use ChatGPT very often, and the 20 USD monthly subscription fee is a bit expensive. If I can deploy a private large language model, I can use various large language models, not limited to ChatGPT. Even if the open-source large language model is not as good as the paid version of ChatGTP, I can use OpenAI\u0026rsquo;s API to call ChatGPT\u0026rsquo;s interface, which can achieve almost the same effect as using the paid version of ChatGPT, but the monthly fee should be less than 20 USD.\nPrerequisites # Already have a server or local computer Docker and docker-compose are installed If you have a graphics card, the relevant driver is installed Background Knowledge # A large language model (LLM) is a deep learning-based natural language processing model that can generate natural language text. Training a large language model requires a lot of computing resources, so it is usually trained on a graphics card. There are many open-source large language models, such as GPT-2, GPT-3, T5, etc. These models are based on the Transformer architecture and have achieved good results in natural language processing tasks.\nTo deploy a large language model by yourself, you generally need three components:\nTrained model: Large language models trained by companies or research institutions are usually saved in the form of PyTorch or TensorFlow model files. Model files are usually large and require several GB to tens of GB of storage space. Generally, choose according to the size of your graphics card memory, for example, many models are 7B, which means that the model has 700 million parameters and can generally run on an 8GB graphics card. Framework for running the model: A program that can load model files and run models on a graphics card. Here we use ollama, which can load various model files and allow users to train, fine-tune, and deploy large models. User interaction front-end interface: An interface that can interact with users, users can enter text, and the model will generate a reply. Here we use LobeChat, which is a web-based user interaction interface that can be integrated with ollama. ollama # ollama is an open-source large language model framework that can load various model files and allow users to train, fine-tune, and deploy large models. ollama supports PyTorch and TensorFlow model files and can run on CPU and GPU. ollama provides a RESTful API that users can call the model through HTTP requests.\nollama supports Windows, Linux, and macOS systems and can run on local computers or servers, as well as in Docker containers. Here we use Docker containers to run ollama.\nInstallation # We use Docker containers to deploy ollama, using the method we introduced in the article \u0026ldquo;Docker Best Practices Guide - docker-compose and Portainer\u0026rdquo;, to deploy ollama in a Docker container.\nCreate a subdirectory ollama under ~/docker to store ollama-related files.\nCreate a docker-compose.yml file in the ~/docker/ollama directory to define the ollama configuration.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 services: ollama: container_name: ollama image: ollama/ollama environment: - OLLAMA_ORIGINS=* - OLLAMA_HOST=0.0.0.0 - OLLAMA_MODELS=/root/.ollama/models ports: - \u0026#34;11434:11434\u0026#34; deploy: resources: reservations: devices: - driver: nvidia capabilities: [\u0026#34;gpu\u0026#34;] count: all volumes: - ollama:/root/.ollama restart: always volumes: ollama: Run the docker-compose up -d command to start the ollama container.\nUsage # Common commands in ollama include:\nollama train: Used to train the model. ollama pull: Used to download pre-trained models. ollama serve: Used to start the ollama service. We can enter the ollama container by executing the docker exec -it ollama bash command and then execute the command.\nFirst, we need to download a pre-trained model, for example, we can download a llama3.1 model:\n1 ollama pull llama3.1 Of course, there are many other models, you can check them in the ollama model library.\nThen we test whether we can use this model:\n1 2 3 4 5 6 7 curl http://127.0.0.1:11434/api/generate -d \u0026#39;{ \u0026#34;model\u0026#34;: \u0026#34;llama3.1\u0026#34;, \u0026#34;prompt\u0026#34;: \u0026#34;Why is the sky blue?\u0026#34;, \u0026#34;options\u0026#34;: { \u0026#34;num_ctx\u0026#34;: 4096 } }\u0026#39; We send an HTTP POST request to ollama\u0026rsquo;s RESTful API to ask llama3.1 why the sky is blue. If our deployed ollama runs well, llama3.1 will return a JSON response containing the generated text.\nLobeChat # LobeChat is a web-based user interaction interface that can be integrated with ollama. Users can enter text in LobeChat, and ollama will generate a reply. LobeChat provides a simple interface that users can use in a browser.\nLobeChat also supports voice synthesis, image recognition, multimodal, plugins, and other functions, allowing users to interact with ollama in multiple ways.\nInstallation # LobeChat is a Node.js-based application that can run on various operating systems. We can use Docker containers to deploy LobeChat.\nCreate a subdirectory lobe-chat under ~/docker to store LobeChat-related files.\nCreate a docker-compose.yml file in the ~/docker/lobe-chat directory to define the LobeChat configuration.\n1 2 3 4 5 6 7 8 9 services: lobe-chat: container_name: lobe-chat image: lobe-chat/lobe-chat environment: - OLLAMA_URL=http://127.0.0.1:11434 ports: - \u0026#34;3000:3000\u0026#34; restart: always Run the docker-compose up -d command to start the LobeChat container.\nUsage # After completing the above steps, we can access the LobeChat interface in the browser by visiting http://localhost:3000 (the first time you open this page may take a few seconds to complete initialization):\nThen, we can use LobeChat to interact with ollama, just like using ChatGPT. It should be noted that we need to select the model that has been downloaded in ollama. For example, we downloaded the Llama3.1 8B model, so we can choose the Llama3.1 model in LobeChat.\nConfigure Other Models # LobeChat also supports many other models, such as Open AI, Google\u0026rsquo;s Gemini, Tongyi Qianwen, etc. You can click on the avatar in the upper left corner, select \u0026ldquo;Settings\u0026rdquo; in the menu, open the settings page, and then select the \u0026ldquo;Language Model\u0026rdquo; tab to see all supported models. You can choose to enable or disable a model, or enter the API Key required by the model to use these models.\nRemote Access # If you want to use LobeChat remotely, you can set up a reverse proxy with Nginx according to the method introduced in the article \u0026ldquo;Access Personal Website from the Public Network - Nginx Reverse Proxy Configuration\u0026rdquo;, bind it with your domain name, and then access LobeChat through the domain name.\n","date":"13 August 2024","externalUrl":null,"permalink":"/en/p/local-large-language-model-1-deploy-on-local-or-server-with-ollama-and-lobechat/","section":"Posts","summary":" Motivation # After installing a graphics card on my server, I want to make full use of this graphics card, so I thought of deploying a large language model. In this way, I don’t need to subscribe to ChatGPT every month. After all, I don’t use ChatGPT very often, and the 20 USD monthly subscription fee is a bit expensive. If I can deploy a private large language model, I can use various large language models, not limited to ChatGPT. Even if the open-source large language model is not as good as the paid version of ChatGTP, I can use OpenAI’s API to call ChatGPT’s interface, which can achieve almost the same effect as using the paid version of ChatGPT, but the monthly fee should be less than 20 USD.\n","title":"Local Large Language Model (1): Deploy on Local or Server with ollama and LobeChat","type":"post"},{"content":" Motivation # This is the second article in the series introducing Docker containers. The links to other articles in this series are as follows:\nContainer(1): Introduction to Container-related Knowledge——Containerization, Docker, Docker-compose, Kubernetes / K8s, etc. Container(3): Docker Best Practices Guide——Volume Management Container(4): Docker Best Practices Guide——Container Update, Upgrade, and Migration Container(5): Docker Best Practices Guide——Container Update Monitoring Tool WUD (What\u0026rsquo;s Up Docker) Container(6): Misconceptions, Bad Habits, and Issues When Using Docker In \u0026ldquo;Container(1): Introduction to Container-related Knowledge——Containerization, Docker, Docker-compose, Kubernetes / K8s, etc.\u0026rdquo;, we introduced the concept of containerization and how to use Docker for containerization. When using Docker in practice, we may encounter some problems, such as:\nAfter running multiple containers, how can we clearly know where these containers are, and how is their running status? When we need to update the container, how can we ensure that the updated container is consistent with the original container? When we need to run multiple containers, how can we ensure the dependency between these containers? At the beginning, when I used Docker, some containers were started with the docker run command, and some containers were started with docker-compose, and the location of the docker-compose.yml file was not unified, which made me very confused when managing these containers. Later, I reorganized them, started all containers with docker-compose, and placed the docker-compose.yml file in a unified directory, which made it much easier to manage.\nOf course, calling it the \u0026ldquo;best practice guide\u0026rdquo; may be a bit exaggerated, but this is indeed the best way I think to use Docker at the current stage.\nPrerequisites # Docker and docker-compose are installed Understand the basic concepts and basic usage of Docker If you have not installed Docker and docker-compose, or do not understand the basic concepts and basic usage of Docker and containerization, you can refer to \u0026ldquo;Container(1): Introduction to Container-related Knowledge——Containerization, Docker, Docker-compose, Kubernetes / K8s, etc.\u0026rdquo;, which provides a more detailed introduction.\nDocker Best Practices # Manage containers with docker-compose # Create a directory to store the docker-compose.yml file and related files, such as ~/docker. Create a subdirectory in this directory for each container that needs to be run, such as ~/docker/nginx. Create a docker-compose.yml file in the subdirectory to define the configuration of the container, such as ~/docker/nginx/docker-compose.yml. We can define some templates for creating docker-compose.yml files, so that each time a new container is created, you only need to copy the template and modify the configuration. For example\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 version: \u0026#39;3\u0026#39; services: service-name: container_name: container-name image: image-source/image-name:tag environment: - SOME_ENV_VAR=some_value ports: - \u0026#34;host_port:container_port\u0026#34; deploy: resources: reservations: devices: - driver: nvidia capabilities: [\u0026#34;gpu\u0026#34;] count: all volumes: - /path/on/host:/path/in/container restart: always This template defines a service, the service name is service-name, the container name is container-name, uses the image-source/image-name:tag image, sets the environment variable SOME_ENV_VAR, maps the port host_port:container_port, sets resource limits, mounts volumes, and sets the container restart policy. And you can use Nvidia\u0026rsquo;s GPU.\nStart containers with docker-compose # After defining the docker-compose.yml file in the subdirectory, you can start the container using the docker-compose command. Execute in the subdirectory:\n1 docker-compose up -d If it is the first time to start the container, or you need to rebuild the container, you can use:\n1 docker-compose up -d --build If you need to stop the container, you can use:\n1 docker-compose down If you want to delete the container\u0026rsquo;s data volume, you can use:\n1 docker-compose down -v Manage containers with Portainer # Portainer is a lightweight container management tool that can be used to manage Docker containers, images, networks, etc. It includes viewing the running status, logs, resource usage, etc. of all containers, and can start, stop, and delete containers through the Web interface.\nPortainer has a paid commercial version (Portainer BE) and a free community version (Portainer CE), and we can use the free community version.\nWe can start Portainer with docker-compose and manage containers through the Web interface. The specific steps are as follows:\nCreate a subdirectory portainer in the ~/docker directory, and create a docker-compose.yml file in the portainer directory with the following content:\n1 2 3 4 5 6 7 8 9 10 11 version: \u0026#39;3\u0026#39; services: portainer: image: portainer/portainer-ce:latest ports: - \u0026#34;8001:8001\u0026#34; - \u0026#34;9443:9443\u0026#34; restart: always volumes: - /var/run/docker.sock:/var/run/docker.sock - ./portainer_data:/data Note that when I used Portainer, the latest version of the official image portainer/portainer-ce:latest is 2.19.4, but the 2.19 version of Portainer does not fully support Docker 26 and above. And the Docker version I use is 27.0.3, so I use the 2.20.1 version of Portainer, that is, portainer/portainer-ce:2.20.1.\nExecute in the portainer directory:\n1 docker-compose up -d Open a browser, visit http://localhost:8001, enter the username and password (you need to set the username and password for the first login), and you can enter the Portainer management interface.\nClick Local, you can view all containers, images, networks, etc.:\nIf you want to remotely manage Docker on a machine through Portainer, you can set up a reverse proxy with Nginx according to the method introduced in the previous article \u0026ldquo;Access Personal Website from Public Network——Nginx Reverse Proxy Configuration\u0026rdquo;, bind it with your domain name, and then access Portainer through the domain name.\n","date":"12 August 2024","externalUrl":null,"permalink":"/en/p/container-2-the-best-practice-guide-for-dockerdocker-compose-and-portainer/","section":"Posts","summary":" Motivation # This is the second article in the series introducing Docker containers. The links to other articles in this series are as follows:\nContainer(1): Introduction to Container-related Knowledge——Containerization, Docker, Docker-compose, Kubernetes / K8s, etc. Container(3): Docker Best Practices Guide——Volume Management Container(4): Docker Best Practices Guide——Container Update, Upgrade, and Migration Container(5): Docker Best Practices Guide——Container Update Monitoring Tool WUD (What’s Up Docker) Container(6): Misconceptions, Bad Habits, and Issues When Using Docker In “Container(1): Introduction to Container-related Knowledge——Containerization, Docker, Docker-compose, Kubernetes / K8s, etc.”, we introduced the concept of containerization and how to use Docker for containerization. When using Docker in practice, we may encounter some problems, such as:\n","title":"Container (2): The Best Practice Guide for Docker——docker-compose and Portainer","type":"post"},{"content":" Motivation # Recently, I bought an Nvidia RTX 4060 Ti (8GB) graphics card. To make full use of it, I want to use this graphics card on Ubuntu for games, CUDA programming, deep learning, etc. However, using Nvidia graphics cards on Ubuntu is not an easy task and requires some settings. Here I record the settings I use Nvidia graphics cards on Ubuntu.\nUpdate (2024-09-19): I have migrated from Ubuntu to Fedora, and the installation of Nvidia graphics card drivers in Fedora is different from that in Ubuntu. For details, please refer to the third and fourth articles in this series. Other articles in this series can be found at:\nNvidia GPU (2): Accelerate Video Editing and Transcoding for FFmpeg Using GPU Nvidia GPU (3): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Fedora Nvidia GPU (4): Containerized Configuration of CUDA Programming and Deep Learning Environment for Nvidia GPU Installing Nvidia Graphics Card Driver on Ubuntu # Check Graphics Card Information # First, we need to check our graphics card information. Open the terminal and enter the following command:\n1 lspci | grep VGA If you have an Nvidia graphics card on your computer, you will see output similar to the following:\n1 01:00.0 VGA compatible controller: NVIDIA Corporation Device 2803 (rev a1) For some reason, my computer displays NVIDIA Corporation Device 2803 instead of RTX 4060 Ti, but it doesn\u0026rsquo;t matter. We just need to know that this is an Nvidia graphics card.\nInstall Nvidia Graphics Card Driver # First, remove any Nvidia graphics card drivers that may have been installed: 1 2 3 sudo apt-get remove --purge \u0026#39;^nvidia-.*\u0026#39; sudo apt-get remove --purge \u0026#39;^libnvidia-.*\u0026#39; sudo apt-get remove --purge \u0026#39;^cuda-.*\u0026#39; Install any missing dependencies: 1 sudo apt-get install linux-headers-$(uname -r) Add the Nvidia graphics card driver PPA repository and update: 1 2 sudo add-apt-repository ppa:graphics-drivers sudo apt-get update Install the Nvidia graphics card driver: 1 sudo ubuntu-drivers autoinstall Restart the computer\nConfirm that the graphics card driver is installed successfully:\n1 nvidia-smi If you see output similar to the following, congratulations, your Nvidia graphics card driver is installed successfully:\n1 2 3 4 5 6 7 8 9 10 11 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 12.5 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA RTX 4060 Ti Off | 00000000:01:00.0 Off | N/A | | N/A 41C P8 10W / N/A | 0MiB / 7611MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ Disable Nouveau Graphics Card Driver # Nouveau is an open-source Nvidia graphics card driver, but its performance is not as good as the official Nvidia closed-source driver, so we need to disable the Nouveau graphics card driver.\nCheck if the Nouveau graphics card driver is loaded: 1 lsmod | grep nouveau If you see output similar to the following, it means that the Nouveau graphics card driver is loaded:\n1 2 3 4 5 6 7 nouveau 2457600 1 mxm_wmi 16384 1 nouveau ttm 106496 1 nouveau drm_kms_helper 217088 1 nouveau drm 552960 3 drm_kms_helper,nouveau,ttm wmi 36864 2 mxm_wmi,nouveau video 49152 1 nouveau Disable the Nouveau graphics card driver: 1 cat \u0026lt;\u0026lt;EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf\\nblacklist nouveau\\noptions nouveau modeset=0\\nEOF Update initramfs: Then we need to update the kernel modules loaded at boot time:\n1 sudo update-initramfs -u Restart the computer\nConfirm that the Nouveau graphics card driver is disabled:\n1 lsmod | grep nouveau If there is no output, it means that the Nouveau graphics card driver has been disabled.\nTest Graphics Card # We can use glmark2 to test the graphics card performance.\nInstall glmark2: 1 sudo apt-get install glmark2 Run glmark2: 1 glmark2 If you see output similar to the following in the terminal, it means that the graphics card performance test is successful:\n1 2 3 4 5 6 7 8 ======================================================= glmark2 2021.02 ======================================================= OpenGL Information GL_VENDOR: NVIDIA Corporation GL_RENDERER: NVIDIA GeForce RTX 4060 Ti/PCIe/SSE2 GL_VERSION: 4.6.0 NVIDIA 555.58.02 ======================================================= And a window will pop up showing the content being tested. After the test is completed, the terminal will display the test score.\nGames # Linux systems are not very suitable for playing games, but with the promotion of Steam, more and more games can be run on Linux using Proton. Proton is a tool developed by Valve based on Wine, which can run Windows games on Linux.\nInstall Steam # Download the Steam installation package: 1 wget https://cdn.cloudflare.steamstatic.com/client/installer/steam.deb Install Steam: 1 sudo dpkg -i steam.deb Install any missing dependencies: 1 sudo apt-get install -f Run Steam: 1 steam Log in to your Steam account Install Proton # In Steam, select games that can run on Linux, then click Settings, in the Steam Play tab, check Enable Steam Play for supported titles and Enable Steam Play for all other titles, then select a Proton version from the Steam Play drop-down menu, and click OK.\nAfter Proton is installed, you can run Windows games on Linux.\nInstall Games # If you are using Steam for the first time and have not purchased any games, you can choose some free games to test, such as \u0026ldquo;Dota 2\u0026rdquo;, \u0026ldquo;Counter-Strike: Global Offensive\u0026rdquo;, etc.\nCUDA Programming # CUDA is a parallel computing platform and programming model developed by Nvidia, which can use the parallel computing power of the GPU to accelerate compute-intensive applications. CUDA programming requires the installation of the Nvidia graphics card driver and the CUDA toolkit. The CUDA version and the Nvidia graphics card driver version have a certain correspondence, and you need to choose the appropriate CUDA version according to your graphics card driver version.\nInstall CUDA # Check the CUDA version required by the Nvidia graphics card driver: 1 nvidia-smi In the CUDA Version line, you can see the CUDA version required by the Nvidia graphics card driver, for example, CUDA Version: 12.5. That means we need to install CUDA 12.5.\nDownload the CUDA installation package: Download the appropriate CUDA installation package from the Nvidia website, select the appropriate operating system, architecture, distribution, version, etc.\nNote that we need to install CUDA 12.5, but the default display on the official website is 12.6. However, this is not a problem. We choose to install via the network, and you will see the following installation guide:\nWe only need to change the last line of the command to the CUDA version we need.\nInstall CUDA: 1 2 3 4 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get install cuda-toolkit-12-5 Configure environment variables: If the installation is successful, you will see the CUDA installation files in the /usr/local/cuda-12.5 directory. At the same time, there is a symbolic link /usr/local/cuda pointing to the CUDA installation directory. So that when you update the CUDA version, the symbolic link will point to the new version, and you don\u0026rsquo;t need to change the environment variables.\nWe need to configure environment variables so that CUDA can be found. Here we use /usr/local/cuda instead of /usr/local/cuda-12.5, so that when we update the CUDA version, we don\u0026rsquo;t need to change the environment variables. Similarly, also add the library path to the environment variables.\nIf you are using bash, you can add the following content to the ~/.bashrc file:\n1 2 3 echo \u0026#39;export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}\u0026#39; \u0026gt;\u0026gt; ~/.bashrc echo \u0026#39;export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}\u0026#39; \u0026gt;\u0026gt; ~/.bashrc source ~/.bashrc If you use another shell, you can add the above content to the corresponding configuration file.\nTest CUDA: 1 nvcc --version If you see output similar to the following, it means that the CUDA programming environment is set up successfully:\n1 2 3 4 5 nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Jun__6_02:18:23_PDT_2024 Cuda compilation tools, release 12.5, V12.5.82 Build cuda_12.5.r12.5/compiler.34385749_0 Compile CUDA Program # You can use the CUDA sample program provided by Nvidia to test the CUDA programming environment.\nDownload the CUDA sample program: 1 git clone https://github.com/NVIDIA/cuda-samples.git Compile the CUDA sample program: 1 cd cuda-samples This sample project uses CMake to compile, so we need to install CMake if it is not installed:\n1 sudo apt-get install cmake Then we can compile the CUDA sample program:\n1 2 3 mkdir build \u0026amp;\u0026amp; cd build cmake .. make Run the CUDA sample program: 1 2 cd Samples/1_Utilities/deviceQuery ./deviceQuery If you see output similar to the following, it means that the CUDA programming environment is set up successfully:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 ./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: \u0026#34;NVIDIA GeForce RTX 4060 Ti\u0026#34; CUDA Driver Version / Runtime Version 12.5 / 12.5 CUDA Capability Major/Minor version number: 8.9 Total amount of global memory: 7810 MBytes (8188919808 bytes) (034) Multiprocessors, (128) CUDA Cores/MP: 4352 CUDA Cores GPU Max Clock rate: 2565 MHz (2.57 GHz) Memory Clock rate: 9001 Mhz Memory Bus Width: 128-bit L2 Cache Size: 33554432 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total shared memory per multiprocessor: 102400 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 1536 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 2 copy engine(s) Run time limit on kernels: Yes Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): Yes Device supports Managed Memory: Yes Device supports Compute Preemption: Yes Supports Cooperative Kernel Launch: Yes Supports MultiDevice Co-op Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0 Compute Mode: \u0026lt; Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) \u0026gt; deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.5, CUDA Runtime Version = 12.5, NumDevs = 1 Result = PASS Here you can see that the graphics card on my computer is NVIDIA GeForce RTX 4060 Ti, the CUDA version is 12.5, and other information such as memory size, CUDA core count, GPU clock frequency, etc. If you see similar output, it means that the CUDA programming environment is set up successfully.\nNow you can start writing your own CUDA program.\nDeep Learning # Deep learning is one of the most commonly used machine learning methods, which can be used to solve problems such as image recognition, natural language processing, recommendation systems, etc. Deep learning usually requires a large amount of data and computing resources, so using GPUs to accelerate deep learning training is very common.\nInstall Deep Learning Framework # Currently, popular deep learning frameworks include TensorFlow, PyTorch, Keras, etc., all of which support GPU acceleration. Before installing a deep learning framework, we need to install CUDA and cuDNN.\nInstall cuDNN: cuDNN is a deep learning library provided by Nvidia, which can accelerate the operation of deep learning frameworks. We need to download the corresponding cuDNN installation package from the Nvidia website, and then install it.\nInstall the deep learning framework: For example, PyTorch:\n1 pip install torch torchvision torchaudio Test the deep learning framework: 1 2 import torch print(torch.cuda.is_available()) If you see True, it means that PyTorch is installed successfully and can be used to accelerate with GPU.\nTrain Model # Now you can use the GPU to accelerate the training of deep learning models. For example, we can download the example provided by the PyTorch official repository to test.\n1 2 3 4 git clone https://github.com/pytorch/examples.git cd examples/time_sequence_prediction python generate_sine_wave.py python train.py This is an example of using a Long Short-Term Memory (LSTM) network to predict a sine wave. You can modify the model and data according to your needs.\nDocker Containers # I have run many applications using Docker before, but they were all running on the CPU. Now that I have an Nvidia graphics card, I want to use GPU acceleration in Docker containers, such as migrating the large language model I previously ran to the GPU.\nInstall Nvidia Container Toolkit # Nvidia Container Toolkit is a tool provided by Nvidia that allows Docker containers to access Nvidia graphics cards.\nAdd the Nvidia Container Toolkit PPA repository: 1 2 3 4 5 curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \\ sed \u0026#39;s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g\u0026#39; | \\ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update Install Nvidia Container Toolkit: 1 2 sudo apt-get install nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker Restart the Docker service: 1 sudo systemctl restart docker Use GPU Acceleration in Docker Containers # Now we can use GPU acceleration in Docker containers, for example:\n1 docker run --gpus all nvidia/cuda:12.5-base nvidia-smi I usually use docker-compose to manage Docker containers, and you can add GPU configuration in the docker-compose.yml file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 version: \u0026#39;3\u0026#39; services: service-name: container_name: container-name image: image-source/image-name:tag environment: - SOME_ENV_VAR=some_value ports: - \u0026#34;host_port:container_port\u0026#34; deploy: resources: reservations: devices: - driver: nvidia capabilities: [\u0026#34;gpu\u0026#34;] count: all volumes: - /path/on/host:/path/in/container restart: always After defining the docker-compose.yml file, you can start the container using the docker-compose command:\n1 docker-compose up -d Troubleshooting # The first time I installed the Nvidia driver and restarted the computer, the computer could not enter the desktop and was stuck in the command line interface, and I could not enter any commands. I suspect that the Nouveau graphics card driver was not disabled, causing the Nvidia graphics card driver to not work properly.\nI seem to have not set up a recovery mode, so I can\u0026rsquo;t enter recovery mode to fix the problem. I don\u0026rsquo;t want to reinstall the system because there are many important things in the original system.\nFinally, I had to start a Ubuntu system with a USB flash drive, mount the original system partition, enter the original system through the chroot command, uninstall the Nvidia driver, and then restart the computer.\nBut there were some small twists and turns in this process. Because the Ubuntu system in the USB flash drive is 24.04, and the system in the original computer is 22.04, so after entering the original system through chroot, when I ran apt-get update, it seemed to install some wrong drivers, which caused the computer to lose the network after restarting, and neither wireless nor wired connections could be made. And the kernel version number was updated to a new version, which seemed to cause some compatibility issues.\nSo I had to create a new Ubuntu system with 22.04 in the USB flash drive, then chroot into the original system again, update with apt-get update, and add some additional Linux modules dpkg -s linux-modules-extra-$(uname -r) | grep status, and finally update the initramfs update-initramfs -u, update the grub update-grub. After restarting the computer, everything was back to normal, except that the system\u0026rsquo;s kernel was updated to a new version, and everything else should have been restored to its original state.\nFinally, I reinstalled the Nvidia graphics card driver, this time without restarting the computer, but first disabled the Nouveau graphics card driver, then updated the initramfs and grub, and finally restarted the computer, and finally successfully installed the Nvidia graphics card driver.\n","date":"12 August 2024","externalUrl":null,"permalink":"/en/p/nvidia-gpu-1-games-cuda-programming-deep-learning-docker-containers-etc.-on-ubuntu/","section":"Posts","summary":" Motivation # Recently, I bought an Nvidia RTX 4060 Ti (8GB) graphics card. To make full use of it, I want to use this graphics card on Ubuntu for games, CUDA programming, deep learning, etc. However, using Nvidia graphics cards on Ubuntu is not an easy task and requires some settings. Here I record the settings I use Nvidia graphics cards on Ubuntu.\n","title":"Nvidia GPU (1): Games, CUDA Programming, Deep Learning, Docker Containers, etc. on Ubuntu","type":"post"},{"content":"","date":"4 August 2024","externalUrl":null,"permalink":"/en/tags/gru/","section":"Tags","summary":"","title":"GRU","type":"tags"},{"content":" Background # This is the fifth article in the \u0026ldquo;Learn PyTorch by Examples\u0026rdquo; series. In the fourth article \u0026ldquo;Learn PyTorch by Examples (4): Sequence Prediction (I) \u0026ndash; Recurrent Neural Networks (RNN)\u0026rdquo;, we introduced the sequence prediction problem and how to use a simple Recurrent Neural Network (RNN) to predict the sine function. In this article, we will go further and introduce two other commonly used neural networks for sequence prediction: Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM).\nThe code for this article can be found in the T05_series_rnn folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial.\nProblems with RNN and Solutions # In the fourth article, we introduced the basic concepts and working principles of Recurrent Neural Networks (RNNs). RNN is a type of neural network that can handle sequence data, and it saves the previous data information at each time step, so it can handle sequence data. However, RNN also has some problems, such as the vanishing gradient and exploding gradient problems. The vanishing gradient and exploding gradient are common problems in deep learning, and they can cause the model to fail to converge or converge very slowly.\nThe vanishing gradient and exploding gradient are common problems in deep learning. In the backpropagation algorithm, the gradient is calculated using the chain rule:\n$D_n = \\sigma^{\u0026rsquo;}(z_1) w_1 \\cdot \\sigma^{\u0026rsquo;}(z_2) w_2 \\cdot \\ldots \\cdot \\sigma^{\u0026rsquo;}(z_{n-1}) w_{n-1} \\cdot \\sigma^{\u0026rsquo;}(z_n) w_n$\nwhere $D_n$ is the gradient of the $n$-th layer, $\\sigma^{\u0026rsquo;}(z_i)$ is the derivative of the activation function of the $i$-th layer, and $w_i$ is the weight of the $i$-th layer. It can be seen that the gradient is calculated by multiplying the derivative of the activation function and the weight of each layer. If the derivative of the activation function is less than 1, the gradient will decrease exponentially with the increase of the number of layers, leading to the vanishing gradient; if the derivative of the activation function is greater than 1, the gradient will increase exponentially with the increase of the number of layers, leading to the exploding gradient.\nLong Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) are proposed to solve the vanishing gradient and exploding gradient problems in RNN. They control the flow of information by introducing gate mechanisms to solve the long-term dependency problem in RNN.\nIntroduction to LSTM and GRU # LSTM and GRU both introduce gate mechanisms to control the flow of information. The so-called gate mechanism multiplies the data by a coefficient between 0 and 1 to control whether to pass the data and how much data to pass, and this coefficient is calculated by a sigmoid activation function.\nThe difference between LSTM and GRU is that LSTM has three gates: Forget Gate, Input Gate, and Output Gate, while GRU has only two gates: Reset Gate and Update Gate. GRU has fewer parameters and less computation than LSTM, but LSTM generally performs better than GRU.\nLong Short-Term Memory (LSTM) # Long Short-Term Memory (LSTM) is a type of gated recurrent neural network proposed by Hochreiter and Schmidhuber in 1997. LSTM introduces three gates: Forget Gate, Input Gate, and Output Gate to control the flow of information, thus solving the long-term dependency problem in RNN.\nThe structure of LSTM is shown in the following figure:\nThe derivation of the specific calculation formulas for the Forget Gate, Input Gate, and Output Gate is not given here. Interested readers can refer to this article Understanding LSTM Networks. Here we simply introduce the working principle of LSTM:\nWe assume that the memory data in LSTM is $C_t$, the hidden state is $h_t$, the input data is $x_t$, the Forget Gate is $f_t$, the Input Gate is $i_t$, and the Output Gate is $o_t$. The working principle of LSTM is as follows:\nForget Gate: The input of the Forget Gate is the input data $x_t$ at the current time step and the hidden state $h_{t-1}$ at the previous time step. The output of these two inputs after passing through the sigmoid function is a coefficient $f_t$ between 0 and 1. $f_t$ determines how much data from the previous time step needs to be retained. If $f_t$ is close to 0, the data from the previous time step will be forgotten; if $f_t$ is close to 1, the data from the previous time step will be retained.\nInput Gate: The input of the Input Gate is the input data $x_t$ at the current time step and the hidden state $h_{t-1}$ at the previous time step. The output of these two inputs after passing through the sigmoid function is a coefficient $i_t$ between 0 and 1. $i_t$ determines how much data from the current time step needs to be retained. If $i_t$ is close to 0, the data from the current time step will be ignored; if $i_t$ is close to 1, the data from the current time step will be retained.\nUpdate Memory: The formula for updating the memory is $C_t = f_t \\cdot C_{t-1} + i_t \\cdot \\tilde{C}t$, where $\\tilde{C}t$ is the output of the input data $x_t$ at the current time step and the hidden state $h{t-1}$ at the previous time step after passing through the tanh function. $C_t$ is the memory data at the current time step, $f_t \\cdot C{t-1}$ is the memory data at the previous time step, and $i_t\\cdot\\tilde{C}_t$ is the input data at the current time step.\nOutput Gate: The input of the Output Gate is the input data $x_t$ at the current time step and the hidden state $h_{t-1}$ at the previous time step. The output of these two inputs after passing through the sigmoid function is a coefficient $o_t$ between 0 and 1. $o_t$ determines how much output data $h_t$ at the current time step needs to be retained. If $o_t$ is close to 0, the output data at the current time step will be ignored; if $o_t$ is close to 1, the output data at the current time step will be retained.\nGated Recurrent Unit (GRU) # The Gated Recurrent Unit (GRU) is a simplified version of LSTM proposed by Cho et al. in 2014. GRU has only two gates: Reset Gate and Update Gate. Compared to LSTM, GRU has fewer parameters and less computation. Although LSTM generally performs better, GRU is also popular due to its simplicity.\nThe Reset Gate and Update Gate in GRU are actually simplified versions of the three gates in LSTM. The derivation of the specific calculation formulas is not given here. Interested readers can refer to this article Understanding LSTM Networks. Here we simply introduce the working principle of GRU:\nWe assume that the memory data in GRU is $h_t$, the input data is $x_t$, the Reset Gate is $r_t$, and the Update Gate is $z_t$. The working principle of GRU is as follows:\nReset Gate: The input of the Reset Gate is the input data $x_t$ at the current time step and the hidden state $h_{t-1}$ at the previous time step. The output of these two inputs after passing through the sigmoid function is a coefficient $r_t$ between 0 and 1. $r_t$ determines how much data from the previous time step needs to be retained. If $r_t$ is close to 0, the data from the previous time step will be ignored; if $r_t$ is close to 1, the data from the previous time step will be retained.\nUpdate Memory: The formula for updating the memory is $\\tilde{h}t = \\tanh(W [r_t h{t-1}, x_t]) = \\tanh(W_{xh} x_t + r_t \\odot W_{hh} h_{t-1})$, where $\\odot$ is element-wise multiplication. $\\tilde{h}t$ is the memory data at the current time step, $W{xh} x_t$ is the input data at the current time step, and $r_t \\odot W_{hh} h_{t-1}$ is the hidden state at the previous time step.\nUpdate Gate: The input of the Update Gate is the input data $x_t$ at the current time step and the hidden state $h_{t-1}$ at the previous time step. The output of these two inputs after passing through the sigmoid function is a coefficient $z_t$ between 0 and 1. $z_t$ determines how much memory data at the current time step needs to be retained. If $z_t$ is close to 0, the memory data at the current time step will be ignored; if $z_t$ is close to 1, the memory data at the current time step will be retained.\nCode Implementation # Data Preparation # We continue to use the sine wave data generated in the previous article. For details, please refer to \u0026ldquo;Learn PyTorch by Example (4): Sequence Prediction with Recurrent Neural Networks (I)\u0026rdquo;.\nModel Definition # LSTM Model # PyTorch has implemented the LSTM model, and we encapsulate it here for this problem. The code is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 import torch import torch.nn as nn class SimpleLSTM(nn.Module): def __init__(self, input_size=1, hidden_size=10, output_size=1, num_layers=1): super(SimpleLSTM, self).__init__() self.hidden_size = hidden_size self.num_layers = num_layers self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, dropout=(0 if num_layers == 1 else 0.05), num_layers=num_layers, batch_first=True) self.fc = nn.Linear(hidden_size, output_size) # Linear layer is output of model def forward(self, x): out, _ = self.lstm(x) out = self.fc(out[:, -1, :]) # Use the last output of the LSTM return out GRU Model # PyTorch has also implemented the GRU model, and we encapsulate it here for this problem. The code is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 import torch import torch.nn as nn class SimpleGRU(nn.Module): def __init__(self, input_size=1, hidden_size=10, output_size=1, num_layers=1): super(SimpleGRU, self).__init__() self.gru = nn.GRU(input_size=input_size, hidden_size=hidden_size, dropout=(0 if num_layers == 1 else 0.05), num_layers=num_layers, batch_first=True) self.fc = nn.Linear(hidden_size, output_size) def forward(self, x): out, _ = self.gru(x) out = self.fc(out[:, -1, :]) # Use the last output of the GRU return out Train the Model # On the basis of the code used in the previous article, we only need to make some simple modifications.\nFirst, we put the LSTM and GRU models defined above into the main file, and then modify the part of the model call. Here we add a command-line parameter model_type to the main() function to specify whether to use the RNN, LSTM, or GRU model:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 if model_type == \u0026#39;RNN\u0026#39;: model = SimpleRNN(hidden_size=hidden_size).float().to(device) elif model_type == \u0026#39;LSTM\u0026#39;: model = SimpleLSTM(hidden_size=hidden_size).float().to(device) elif model_type == \u0026#39;GRU\u0026#39;: model = SimpleGRU(hidden_size=hidden_size).float().to(device) else: exit(\u0026#39;Invalid model type. Please choose one of the following: RNN, LSTM, GRU\u0026#39;) loss_function = nn.MSELoss() optimizer = optim.Adam(model.parameters(), lr=lr) if os.path.exists(\u0026#39;train_data.pt\u0026#39;) and os.path.exists(\u0026#39;test_data.pt\u0026#39;): print(\u0026#39;Loading train data \\\u0026#34;train_data.pt\\\u0026#34; and test data \\\u0026#34;test_data.pt\\\u0026#34;\u0026#39;) train_data = torch.load(\u0026#39;train_data.pt\u0026#39;, weights_only=False) test_data = torch.load(\u0026#39;test_data.pt\u0026#39;, weights_only=False) else: exit(\u0026#39;No train data or test data found. Please generate data first by running the script with the --generate-data flag.\u0026#39;) dataloader = DataLoader(train_data, batch_size=batch_size, shuffle=False) test_dataloader = DataLoader(test_data, batch_size=test_batch_size, shuffle=True) train_loss = [] test_loss = [] for epoch in range(1, epochs+1): train_loss_tmp = train(model, device, dataloader, loss_function, optimizer, epoch, 10) train_loss.append(train_loss_tmp) test_loss_tmp = test(model, device, test_dataloader, loss_function) test_loss.append(test_loss_tmp) To compare the performance of the three models, we write another Python script to call these three models. The code for comparing the performance is in the T05_series_gru_lstm folder in the compare_results.py file.\nRun the Code # We run the script to compare the performance, and the results are as follows:\nLoss curves of the three models:\nPredictions of the three models:\nThe compare_results.py script runs the RNN, LSTM, and GRU models in turn. If I run it on my computer with a GPU (NVIDIA GeForce GTX 4060 Ti), it takes about 30 seconds, with a maximum memory usage of about 354 MB. If I run it on a CPU (Intel i5-9600K), it takes about 3 minutes and 33 seconds.\nI have run this comparison performance script multiple times, and the results vary each time. In most cases, all three models can fit the sine function sequence data well, but overall, LSTM and GRU perform slightly better than RNN.\nSummary # In this article, we introduced the working principles of Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) and implemented these two models using PyTorch. We used these two models to predict the sine function sequence data and compared them with a simple Recurrent Neural Network (RNN). We found that LSTM and GRU perform better than RNN in capturing long-term dependencies in sequence data, thus improving the model\u0026rsquo;s performance.\n","date":"4 August 2024","externalUrl":null,"permalink":"/en/p/learn-pytorch-by-examples-5-sequence-prediction-ii--gated-recurrent-unit-gru-and-long-short-term-memory-lstm/","section":"Posts","summary":" Background # This is the fifth article in the “Learn PyTorch by Examples” series. In the fourth article “Learn PyTorch by Examples (4): Sequence Prediction (I) – Recurrent Neural Networks (RNN)”, we introduced the sequence prediction problem and how to use a simple Recurrent Neural Network (RNN) to predict the sine function. In this article, we will go further and introduce two other commonly used neural networks for sequence prediction: Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM).\n","title":"Learn PyTorch by Examples (5): Sequence Prediction (II) -- Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM)","type":"post"},{"content":"","date":"4 August 2024","externalUrl":null,"permalink":"/en/tags/sequence-prediction/","section":"Tags","summary":"","title":"Sequence Prediction","type":"tags"},{"content":"","date":"4 August 2024","externalUrl":null,"permalink":"/en/tags/time-series/","section":"Tags","summary":"","title":"Time Series","type":"tags"},{"content":"","date":"2024年8月4日","externalUrl":null,"permalink":"/tags/%E5%BE%AA%E7%8E%AF%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C/","section":"Tags","summary":"","title":"循环神经网络","type":"tags"},{"content":"","date":"2024年8月4日","externalUrl":null,"permalink":"/tags/%E6%97%B6%E9%97%B4%E5%BA%8F%E5%88%97/","section":"Tags","summary":"","title":"时间序列","type":"tags"},{"content":"","date":"21 July 2024","externalUrl":null,"permalink":"/en/tags/conda/","section":"Tags","summary":"","title":"Conda","type":"tags"},{"content":"","date":"21 July 2024","externalUrl":null,"permalink":"/en/tags/env/","section":"Tags","summary":"","title":"Env","type":"tags"},{"content":"","date":"21 July 2024","externalUrl":null,"permalink":"/en/tags/environment/","section":"Tags","summary":"","title":"Environment","type":"tags"},{"content":"","date":"21 July 2024","externalUrl":null,"permalink":"/en/tags/mamba/","section":"Tags","summary":"","title":"Mamba","type":"tags"},{"content":"","date":"21 July 2024","externalUrl":null,"permalink":"/en/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":" Motivation # When developing with Python, I encountered many situations where I needed to configure the environment. Sometimes I need to install different versions of Python, sometimes I need to install different Python packages, and sometimes I need to install different versions of the same Python package.\nAt the beginning, I basically followed the tutorials on the Internet. Different Python projects may use different environment management tools, and I have used pip, conda, mamba, virtualenv, venv, etc. These tools have their own advantages and disadvantages. Sometimes I also mix them, which leads to confusion of environment variables and conflicts.\nSince there has been no consistency, the Python environment on my computer is quite messy, and I feel confused myself. So I decided to summarize the usage of these tools and their advantages and disadvantages. Finally, I will give some recommendations.\nClassification of Python Management Tools # Python management tools can be divided into two categories:\nPackage management tools: used to install, uninstall, and update Python packages, such as pip, conda, mamba, etc.\nEnvironment management tools: used to manage Python environments, such as venv, virtualenv, conda, mamba, etc.\nHere we mainly discuss environment management tools, because the usage of package management tools is relatively simple, and in most cases we will use pip, so we will not go into details here.\nThere is also another post about using module to manage software packages and environment variables on Linux, which can be found at \u0026ldquo;Use Environment Module to Manage Software Packages and Environment Variables in Linux\u0026rdquo;\nEnvironment Management Tools # venv # venv is a built-in environment management tool in Python after version 3.3, which is the simplest and least troublesome to use.\nUsage # Create a new environment:\n1 python -m venv .venv This will create a .venv folder in the current directory, which contains a new Python environment. Of course, .venv can be replaced with any name you want. Using .venv will create an automatically hidden folder, which is easy to ignore in git.\nNote that python here is the Python version you want to create the environment. If you have multiple Python versions, you can use python3 or python3.10, etc.\nActivate this environment:\nIf you are using a Linux or MacOS system, you can use the following command:\n1 source .venv/bin/activate If you are using a Windows system, you can use the following command:\n1 .venv\\Scripts\\activate Install Python packages:\nAfter activating the environment, using pip to install Python packages will install the packages into this environment, not the global environment. The installed packages will be placed in the .venv/lib/python3.10/site-packages directory.\n1 pip install numpy Exit the environment:\nYou can exit the environment by using the deactivate command.\n1 deactivate Advantages and Disadvantages # Advantages:\nNative tool, simple and easy to use, no need to install additional software. Fast, because no additional software packages need to be downloaded. Environment isolation, will not affect the global environment. Can be created anywhere, no need for administrator privileges. Each environment in each project is in the project directory, which is physically isolated from other projects and not easy to confuse. Disadvantages:\nEach project needs to create a new environment, which can take up a lot of space if there are many projects. Cannot share environments, need to be recreated if multiple projects use the same environment. Not good at switching Python versions, each environment is a Python version and cannot be switched. Only applicable to Python 3.3 and above. In short, if you don\u0026rsquo;t care about space usage, don\u0026rsquo;t need to switch Python versions, and just want the simplest environment management tool, then venv is a good choice.\nvirtualenv # Since venv has a large space usage problem, someone developed virtualenv, which is a third-party environment management tool that can solve some of the problems of venv.\nThe usage of virtualenv is basically the same as venv, but you need to install the virtualenv package.\nUsage # Install virtualenv:\n1 pip install virtualenv Create a new environment:\n1 virtualenv .venv If you want to specify the Python version, you can use the following command:\n1 virtualenv -p python3.10 .venv Activate this environment:\n1 .venv\\Scripts\\activate Install Python packages:\n1 pip install numpy Exit the environment:\n1 deactivate But the difference between virtualenv and venv is that virtualenv will reuse the system\u0026rsquo;s Python library and will not reinstall it, so it will take up much less space. If you don\u0026rsquo;t want to reuse the system\u0026rsquo;s Python library, you can use the --no-site-packages option to install a completely new Python library in this project.\nAdvantages and Disadvantages # Advantages:\nCan specify Python versions. Can share environments, no need to recreate. Takes up less space, does not reinstall Python libraries. Applicable to Python 2.7 and above. Disadvantages:\nNon-native tool, need to install additional software. In short, if you care about space usage and want to share environments, and are using Python 2.7 and above, then virtualenv is a good choice.\nconda # conda is a very powerful environment management tool that can manage Python environments as well as environments for other languages such as R, Julia, etc.\nRelationship between conda, Anaconda, miniconda, mamba, and micromamba # conda is part of Anaconda. Anaconda is a tool package commonly used by data scientists, which contains many software packages commonly used by data scientists, such as numpy, pandas, scipy, etc. However, some users have complained that the software packages installed by Anaconda are too many and take up too much space, so someone developed miniconda, which is a slim version of Anaconda that only contains conda and some basic software packages.\nWhether it is Anaconda or miniconda, they use the conda package management tool to manage environments. However, conda is single-threaded when installing software packages, so it is slow. Therefore, someone developed mamba, which is an accelerated version of conda and is much faster than conda.\nmamba and conda have almost the same usage, except that mamba is used instead of conda when installing software packages. Users can basically think that the mamba command is an alias of the conda command.\nmicromamba is a slim version of mamba. The base environment of micromamba is empty and does not contain any software packages.\nUsage (using mamba as an example) # Install conda or mamba:\nYou can find the minimal conda or mamba installation package on miniforge, download and install it.\nCreate a new environment:\n1 mamba create -n myenv python=3.10 Here, myenv is the name of the environment, and python=3.10 specifies the Python version.\nActivate this environment:\n1 conda activate myenv Install Python packages:\n1 mamba install numpy Exit the environment:\n1 conda deactivate Advantages and Disadvantages # Advantages:\nCan specify Python versions. Can share environments, no need to recreate. Can install environments for other languages. Fast, because there is mamba to accelerate. Can activate the environment anywhere, no need to create the environment in the project directory. Can share software packages, no need to download repeatedly. Disadvantages:\nNon-native tool, need to install additional software. The environment is separate from the specific project, and for environments that have not been used for a long time, you may forget what the environment is for. For projects that have not been used for a long time, you may forget which environment the project uses. In short, if you want a powerful environment management tool and want to install packages quickly, then conda or mamba is a good choice. But when developing a project, it is best to mark which environment the project uses to avoid forgetting.\nFreeze Environment # Regardless of which environment management tool you use, you can use the pip freeze command to freeze the environment, that is, save the list of software packages in the current environment to a file.\n1 pip freeze \u0026gt; requirements.txt This will create a requirements.txt file in the current directory, which contains the list of software packages in the current environment. When you need to install these software packages in another environment, you can use the following command:\n1 pip install -r requirements.txt This will install all the software packages listed in the requirements.txt file.\nSummary # If you don\u0026rsquo;t care about space usage, use Python versions higher than 3.3, don\u0026rsquo;t need to switch Python versions, and just want the simplest environment management tool, then venv is a good choice.\nIf you care about space usage, only want to install packages when needed, you can install micromamba on your main development computer.\nIf you don\u0026rsquo;t care about space usage, you can use Anaconda on your main development computer.\nRecommended Usage # If you don\u0026rsquo;t care about space usage, and may use many scientific computing packages, you can install Anaconda on your main development computer.\nIf you care about space usage, only want to install packages when needed, you can install micromamba on your main development computer.\nIf you don\u0026rsquo;t care about space usage, you can use venv on your test computer or server.\n","date":"21 July 2024","externalUrl":null,"permalink":"/en/p/python-environment-management-with-venv/conda/mamba/","section":"Posts","summary":" Motivation # When developing with Python, I encountered many situations where I needed to configure the environment. Sometimes I need to install different versions of Python, sometimes I need to install different Python packages, and sometimes I need to install different versions of the same Python package.\nAt the beginning, I basically followed the tutorials on the Internet. Different Python projects may use different environment management tools, and I have used pip, conda, mamba, virtualenv, venv, etc. These tools have their own advantages and disadvantages. Sometimes I also mix them, which leads to confusion of environment variables and conflicts.\n","title":"Python Environment Management with venv/conda/mamba","type":"post"},{"content":"","date":"2024年7月21日","externalUrl":null,"permalink":"/tags/%E7%8E%AF%E5%A2%83/","section":"Tags","summary":"","title":"环境","type":"tags"},{"content":"","date":"14 July 2024","externalUrl":null,"permalink":"/en/tags/anydesk/","section":"Tags","summary":"","title":"AnyDesk","type":"tags"},{"content":" Motivation # I have several computers at home and in the office, running operating systems including Windows, macOS, and Linux.\nIntroduction # There are many remote desktop software on the market, such as Teamviewer, Chrome Remote Desktop, ToDesk, RustDesk, etc. I used Teamviewer before, but the free version of Teamviewer has a time limit. I used it for a while and then I don\u0026rsquo;t know why my free use qualification was canceled, so I started using AnyDesk.\nAnyDesk is a cross-platform remote desktop software that supports Windows, macOS, Linux, and Android systems. The features of AnyDesk include:\nFree for personal use, paid for commercial use. Support for multiple operating systems. Fast connection speed and clear picture quality. Installation # Windows and macOS # Download the installation package for the corresponding system from the AnyDesk official website, and then install it.\nLinux # AnyDesk provides .deb and .rpm installation packages for Linux systems, you can download the corresponding installation package from the AnyDesk official website, Debian/Ubuntu systems use .deb package:\n```bash sudo dpkg -i anydesk_x.x.x-x_amd64.deb ``` RedHat/Fedora/OpenSUSE systems use .rpm package:\n```bash sudo rpm -i anydesk-x.x.x-x.x86_64.rpm ``` Android # Anydesk also has an Android mobile version, which can be downloaded and installed from Google Play.\nThis is also a great advantage of Anydesk, because many other remote desktop software do not have an Android version. With the Android version, you can use your phone to do some simple control of the remote computer (complex operations are still too cramped on the phone), which is still very useful in some special cases.\nConfiguration and Usage # Basic usage is very simple, so I won\u0026rsquo;t go into details here. Some useful features include:\nIf you are using AnyDesk on macOS, you can enable screen sharing permissions in system settings. If you want to remotely control the computer without anyone present, you need to enable the Unattended Access feature. The Unattended Access feature requires setting a password, and then entering the address and password of the remote computer to connect. In the User Interface setting, you can set an alias for your computer, so you don\u0026rsquo;t have to remember the long string of numbers of the computer address when connecting. Troubleshooting # When using AnyDesk on a Linux system, I encountered an error: The session has ended. Status: display_server_not_supported.\nThis problem is because AnyDesk does not support the Wayland display server, only the X11 display server. The solution is to select the X11 display server at the login screen, and then log in.\nAnother error occurs when trying to connect to Anydesk remotely after a system restart: The session has ended. Status: desk_rt_ipc_error.\nThis is because AnyDesk cannot work properly after a restart without logging into the system. The solution is to log into the system after a system restart, and then connect to AnyDesk. But if you are not next to the remote computer and cannot log into the system, you can set up automatic login, so that AnyDesk can work properly after a restart.\n1 /etc/gdm3/custom.conf enable automatic login Or you can also edit the /etc/gdm3/custom.conf file and add the following content:\n1 2 3 4 5 [daemon] AutomaticLoginEnable=True AutomaticLogin=username WaylandEnable=false Where username is your username.\nWaylandEnable=false is to disable Wayland, which is not supported by AnyDesk.\n","date":"14 July 2024","externalUrl":null,"permalink":"/en/p/remote-desktop-1-install-and-use-anydesk-on-all-platforms/","section":"Posts","summary":" Motivation # I have several computers at home and in the office, running operating systems including Windows, macOS, and Linux.\nIntroduction # There are many remote desktop software on the market, such as Teamviewer, Chrome Remote Desktop, ToDesk, RustDesk, etc. I used Teamviewer before, but the free version of Teamviewer has a time limit. I used it for a while and then I don’t know why my free use qualification was canceled, so I started using AnyDesk.\n","title":"Remote Desktop (1): Install and Use AnyDesk on All Platforms","type":"post"},{"content":"","date":"30 June 2024","externalUrl":null,"permalink":"/en/tags/arch-linux/","section":"Tags","summary":"","title":"Arch Linux","type":"tags"},{"content":" Motivation # I have an old Surface Pro 6 from 2018, which is now a bit outdated in performance, but still usable for playing around. So I decided to install one or two lightweight Linux systems to familiarize myself with different Linux distributions and prepare for updating the Linux system on my main machine.\nI have always heard that Arch Linux is the \u0026ldquo;cult\u0026rdquo; of Linux distributions, known for its minimalist, highly customizable, and rolling update features, so I decided to give it a try and see how \u0026ldquo;cult\u0026rdquo; Arch Linux really is.\nIntroduction to Arch Linux # Arch Linux is an extremely lightweight Linux distribution with a design philosophy of simplicity, lightness, and flexibility. The main features of Arch Linux are:\nRolling Update: Arch Linux is a rolling update distribution, unlike Ubuntu, Debian, and other distributions with fixed version numbers, it continuously updates software packages, allowing users to use the latest system. However, this also means that users need to update the system frequently, otherwise there may be software package incompatibility issues, and they need to bear the risk of new system bugs or instability, commonly known as \u0026ldquo;rolling hang\u0026rdquo; when updating the system (although I think the frequency of \u0026ldquo;rolling hang\u0026rdquo; in Arch Linux systems is already very low now).\nSimple: The design philosophy of Arch Linux is simplicity, it provides a clean system, and users can customize the system according to their needs. Arch Linux installation packages do not include graphical interfaces or pre-installed software, users need to choose to install them themselves.\nRich Software: Arch Linux provides a powerful software package management tool pacman, users can install, uninstall, and update software packages through pacman, and can also install third-party software packages through AUR (Arch User Repository). pacman+AUR may be the largest software repository in Linux distributions, users can find almost all software packages in AUR, but the packaging quality of some software may not be high, users need to pay attention to it themselves.\nThese features make Arch Linux have advantages and disadvantages. In general, Arch Linux is suitable for users with some experience, who like to tinker, have cleanliness, and pursue new features.\nArch Linux Installation # In the early days, the installation of Arch Linux relied entirely on users typing commands, but now Arch Linux provides an installation wizard archinstall, which greatly simplifies the installation process. Nevertheless, installing Arch Linux still requires users to know some basic knowledge, such as disk partitioning, file systems, boot methods, network configuration, etc.\nPrepare Installation USB Drive # First, we need to make a boot disk, it is recommended to use the Ventoy tool. Ventoy is a very easy-to-use tool that can write multiple ISO files to a USB drive, making it easy for us to choose different system installations without affecting the normal use of the USB drive. For specific usage, please refer to the official documentation.\nDownload Arch Linux ISO # Download the latest Arch Linux ISO file from the Arch Linux official website, and then copy the ISO file to the USB drive. Arch Linux system images are generally updated on the 1st of each month.\nInstallation # Insert the USB drive, start the Surface Pro 6, press and hold the Volume Up key to enter the BIOS settings, and set the USB drive as the first boot item. For other devices, you may need to press F2, F12, ESC, etc. to enter the BIOS settings.\nSelect Boot Arch Linux (x86_64) to enter the Arch Linux installation interface. Then you will see a bunch of scrolling commands, wait a while and you will enter the command line interface of the Root user.\nBefore starting the installation, we need to connect to the network. Arch Linux generally automatically recognizes wired networks, if it is a wireless network, you need to use the command line tool iwctl to connect. Common commands include\ndevice list station \u0026lt;device_name\u0026gt; connect \u0026lt;wifi_name\u0026gt; and then enter the password station \u0026lt;device_name\u0026gt; show to view the connection status Run archinstall to start the installation\nIt is recommended to partition the disk in advance Select manual partition during installation, select the disk and partition in turn, and assign a mount point to the partition. As for how to partition, you can refer to Multi-System Boot Loader Configuration As for network options, I didn\u0026rsquo;t quite understand, but the basic principle is: If you choose not to install the graphical interface, choose to copy the network settings in the ISO, that is, the network settings before executing archinstall If you choose to install the graphical interface, choose NetworkManager. But Network Manager seems to need to disable iwd, and you need to install the Surface-specific driver. After the installation is complete, it will automatically chroot into the installed system, where you can execute post-installation commands.\nInstall the driver needed for Surface networking: pacman -S linux-firmware-marvell. If you don\u0026rsquo;t install the graphical interface, to connect to the network smoothly after restarting, you need to install iwd and dhcpcd. Restart\nI use rEFInd as the boot loader, rEFInd will scan two Arch boot items, but one of them cannot boot Connect to the network after restarting\nIf you use the graphical interface, if there is no network icon, either NetworkManager is not installed or the driver is not installed If you don\u0026rsquo;t install the graphical interface, you need to use iwd to connect to the network, start iwd first: 1 sudo systemctl start iwd Then start dhcpcd.service 1 sudo systemctl start dhcpcd.service You can also use NetworkManager to connect to the network in the command line interface: nmcli device show nmcli device wifi connect \u0026lt;wifi_name\u0026gt; password \u0026lt;password\u0026gt; ","date":"30 June 2024","externalUrl":null,"permalink":"/en/p/installing-and-configuring-arch-linux/","section":"Posts","summary":" Motivation # I have an old Surface Pro 6 from 2018, which is now a bit outdated in performance, but still usable for playing around. So I decided to install one or two lightweight Linux systems to familiarize myself with different Linux distributions and prepare for updating the Linux system on my main machine.\nI have always heard that Arch Linux is the “cult” of Linux distributions, known for its minimalist, highly customizable, and rolling update features, so I decided to give it a try and see how “cult” Arch Linux really is.\n","title":"Installing and Configuring Arch Linux","type":"post"},{"content":"","date":"30 June 2024","externalUrl":null,"permalink":"/en/categories/os/","section":"Categories","summary":"","title":"OS","type":"categories"},{"content":"","date":"24 May 2024","externalUrl":null,"permalink":"/en/tags/efi/","section":"Tags","summary":"","title":"EFI","type":"tags"},{"content":" Motivation # I have installed dual or multiple systems on my computer before, but I have always been confused about the boot entries of multiple systems, often encountering problems such as missing boot entries and duplicate boot entries. This time I plan to study the configuration of multi-system boot entries in depth, including the settings of boot loaders, EFI/Grub/rEFInd, and how to customize the theme of multi-system boot loaders.\nOperating System Boot Process # The computer boot process is mainly divided into 4 steps: 1. Power on; 2. BIOS/UEFI boot; 3. Load boot entries; 4. Operating system boot. Steps 1 and 4 generally do not require user intervention, while steps 2 and 3 require user configuration. So here we mainly introduce the content of steps 2 and 3.\nBIOS/UEFI Boot # BIOS (Basic Input/Output System) and UEFI (Unified Extensible Firmware Interface) are the firmware of the computer, responsible for booting the computer and loading the operating system. BIOS is the old boot method, and UEFI is the new boot method, UEFI has more functions and better performance than BIOS. Whether UEFI or BIOS, they are firmware on the motherboard.\nBIOS Boot Process # BIOS is generally used on older computers, and its boot process is relatively fixed, with low user customization. It is mainly divided into the following steps:\nBIOS POST (Power-On Self-Test) self-test\nBIOS reads the boot program in the MBR (Master Boot Record)\nThe boot program reads the partition table and finds the active partition\nThe boot program reads the boot sector of the active partition and loads the operating system\nThe location of the MBR is fixed, generally in the first sector of the hard disk, with a size of 512 bytes. The MBR contains the partition table and the boot program, the partition table records the partition information of the hard disk, and the boot program is responsible for loading the operating system.\nUEFI Boot Process # UEFI is also a kind of BIOS, generally used on newer computers. Its boot process is more flexible, with more functions and higher user customization. Although UEFI has more functions, its process is more unified, mainly divided into the following steps:\nUEFI POST (Power-On Self-Test) self-test\nUEFI reads the boot program in the EFI partition\nThe boot program reads the boot file in the EFI partition and loads the operating system\nCompared with BIOS, UEFI uses the EFI partition, and the location of the EFI partition is not fixed, and there can even be multiple EFI partitions on one hard disk.\nThe boot program and boot file of UEFI are stored in the EFI partition, including the boot program, boot file, driver program, etc. As long as the boot file meets the UEFI specification, whether it is the boot program of Windows, Linux, or macOS, and no matter where the boot file is placed in the EFI partition, UEFI can find and load it.\nSince UEFI is more flexible and updated, the following multi-system boot entry configuration is based on UEFI.\nConcept Distinction: Boot Loader, Boot Entry, Boot File # Before discussing the configuration of multi-system boot entries, let\u0026rsquo;s first sort out the concepts of boot loader, boot entry, and boot file.\nBoot Loader: Responsible for loading the operating system\u0026rsquo;s boot program, each boot program is a boot entry. Common boot loaders include Grub (Grub2), rEFInd, Clover, etc.\nBoot Entry: Each operating system loaded by the boot loader is a boot entry, and each boot entry corresponds to an operating system. The boot entry contains the operating system\u0026rsquo;s boot file, kernel, driver program, etc.\nBoot File: The boot file in the boot entry is the boot program of the operating system, responsible for loading the operating system. The boot file of Windows is bootmgfw.efi, and the boot file of Linux is vmlinuz.\nOn this basis, let\u0026rsquo;s take a look at how UEFI starts the operating system.\nEach boot loader corresponds to a .efi file, and this .efi file is the boot file of the boot loader, responsible for loading the boot loader. After UEFI completes the self-test, it reads all .efi files in the EFI partition. If you enter the BIOS settings at this time, you will see the boot entries in the EFI partition. These boot entries have a priority sequence, and their order can be adjusted in the BIOS settings.\nFor example, the highest priority boot entry in the figure below is rEFInd, followed by Windows Boot Manager, and finally ubuntu. When the computer starts, it will first try to load rEFInd, if rEFInd does not exist, it will try to load Windows Boot Manager, and so on.\nIf rEFInd is successfully loaded, rEFInd will read the refind.conf configuration file in the EFI partition, generate a boot menu according to the configuration file, and display it on the screen. Each menu item corresponds to a boot entry, and boot entries are generally divided into two types: operating system kernel images and boot loaders.\nOperating System Kernel Image: Corresponds to the operating system\u0026rsquo;s kernel file, such as Linux\u0026rsquo;s vmlinuz file, Windows\u0026rsquo;s bootmgfw.efi file. Boot Loader: Corresponds to the .efi file of the boot loader, such as Grub2\u0026rsquo;s grubx64.efi file. After the user selects a menu item, rEFInd will load the corresponding boot entry. If the boot entry is an operating system kernel image, rEFInd will directly load the operating system; if the boot entry is a boot loader, rEFInd will load the boot loader, and the boot loader will load the operating system.\nIf the boot loader wants to start a Linux system, it will first load the Linux kernel file vmlinuz-xxx, then load the Linux initial file system file initrd-xxx.img or initramfs-xxx.img, and finally load the Linux system. These two files are in the /boot directory of the Linux system.\nBoot Entry Configuration During Multi-System Installation # As mentioned above, the boot files of multiple systems are stored in the same EFI partition, so when installing multiple systems, each system will create a boot file in the EFI partition, so there will be multiple boot files, each boot file corresponds to a system. In this way, you can start different systems by selecting different boot files.\nWhen installing multiple operating systems, we can put the boot files of each system in the same EFI partition, or put the boot files of each system in different EFI partitions. Both methods have their advantages and disadvantages, and the specific choice depends on personal preferences.\nSame EFI Partition\nIn this case, the hard disk partition structure is as follows:\nAdvantages: Easy to manage, only one EFI partition is needed, no need for multiple EFI partitions. Easier to configure boot loaders.\nDisadvantages: All boot files are in one EFI partition. If the EFI partition is damaged, all systems will not be able to start.\nThe specific operation of installing multiple systems is as follows:\nHard Disk Partition: Divide an EFI partition on the hard disk, with a size of at least 1GB, and format it as FAT32. Other partitions can be determined according to the requirements of each operating system, for example, Windows generally requires at least 52GB of space, and Linux generally requires at least 20GB of space. Install Windows: If you are installing multiple operating systems with both Windows and Linux, it is recommended to install Windows first. Because the boot program of Windows will overwrite the boot program of other previously installed operating systems, causing the previously installed Linux to fail to start. When installing Windows, select the partition for Windows, and Windows will automatically create a boot file in the EFI partition. Install Each Linux: For each Linux system, when installing, select custom partitioning, manually mount the EFI partition to /boot/efi, and mount the partition prepared for Linux to /, and create a swap partition and /home partition if necessary. After installation, Linux will create a boot file in the EFI partition. Configure Boot Loader: After installing multiple operating systems, we need to configure the boot loader so that we can select the operating system to start. These contents will be detailed later. Different EFI Partitions\nIn this case, the hard disk partition structure is as follows:\nAdvantages: Each system has its own EFI partition, which does not interfere with each other. If the EFI partition of one system is damaged, other systems are not affected.\nDisadvantages: Multiple EFI partitions are needed, which are more cumbersome to manage. The boot files of each system are in different EFI partitions, making it difficult to configure boot loaders.\nThe specific operation of installing multiple systems is as follows:\nHard Disk Partition: Divide a space for each system on the hard disk. Do not create an EFI partition first, the EFI partition will be created in each system\u0026rsquo;s partition when installing the system. Install Windows: When installing Windows, select the partition for Windows, and Windows will automatically create an EFI partition in that partition. Install Linux: For each Linux system, when installing, select the partition for this system, create an EFI partition of about 200MB in this partition, mount point is /boot/efi; mount the remaining space as a partition prepared for Linux, mount point is /; create a swap partition and /home partition if necessary. Configure Boot Loader: After installing multiple operating systems, we need to configure the boot loader so that we can select the operating system to start. These contents will be detailed later. Boot Directory Structure # Let\u0026rsquo;s take a look at the structure of the boot directory and what files are created when installing an operating system. Here, it is assumed that the boot files of multiple operating systems are placed in the same EFI partition. We have installed 3 operating systems: Windows, Ubuntu, and Arch Linux. Ubuntu uses Grub2 as the boot loader, and Arch Linux uses systemd-boot as the boot loader. We also installed rEFInd as the boot loader in the Ubuntu system.\nEFI Partition\nThe EFI partition is a FAT32 formatted independent partition, generally with a size of about 1GB. The EFI partition mainly contains the boot files of various boot loaders.\nThe directory structure of the EFI partition is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 EFI ├── Boot │ └── bootx64.efi ├── Microsoft │ └── Boot │ ├── BCD │ ├── BCD.LOG │ ├── BCD.LOG1 │ ├── BCD.LOG2 │ ├── bootmgfw.efi │ ├── bootmgr.efi │ └── boot.stl ├── systemd │ └── systemd-bootx64.efi └── ubuntu ├── grub.cfg ├── grubx64.efi ├── mmx64.efi └── shimx64.efi EFI/Boot/bootx64.efi: The default boot file of UEFI, generally points to the boot file of Windows EFI/Microsoft/Boot/bootmgfw.efi. EFI/Microsoft/Boot/bootmgfw.efi: The boot file of Windows, responsible for loading the Windows operating system. EFI/systemd/systemd-bootx64.efi: The boot file of systemd-boot, responsible for loading the Linux operating system. EFI/ubuntu/grubx64.efi: The boot file of Grub2, responsible for loading the Linux operating system. EFI/ubuntu/shimx64.efi: The boot file of Secure Boot, used to load Grub2. EFI/ubuntu/mmx64.efi: The boot file of Grub2, used to load Grub2. EFI/ubuntu/grub.cfg: The configuration file of Grub2, containing the configuration of each boot entry. Install Operating System\nWhen installing Ubuntu, select custom partitioning, manually mount the EFI partition to /boot/efi, and mount the partition prepared for Linux to /. After installation, Ubuntu will create the EFI/ubuntu directory in the EFI partition, which contains the boot files and configuration files of Grub2. Ubuntu will create two files in the /boot directory: vmlinuz and initrd.img, these two files are the kernel file and initial file system file of Linux. The boot file of Grub2 will recognize these two files and create boot entries, select the corresponding boot entry at startup, and load the Ubuntu system.\nIn the Ubuntu system, you can enter /boot/efi/EFI to view the directory structure of the EFI partition.\n1 2 3 4 5 6 7 8 9 /boot ├── efi │ └── EFI │ ├── Boot │ ├── Microsoft │ ├── systemd │ └── ubuntu ├── vmlinuz └── initrd.img Note that although /boot/efi/EFI is under /boot, the files in /boot/efi/EFI are in the EFI partition, while the other files in /boot are in the partition of the Ubuntu system. The files vmlinuz and initrd.img in /boot cannot be accessed in other systems, but if other systems also mount the EFI partition, then other systems can also access the files in /boot/efi/EFI.\nIf you continue to install more operating systems, the operating system will generally ask you to choose a boot loader, you can choose to use Grub2, systemd-boot, or other boot loaders. The configuration of the boot loader is introduced below.\nBoot Loader # The boot loader (Boot Loader) is responsible for loading the operating system\u0026rsquo;s boot program, and each boot program is a boot entry (Boot Entry). Common boot loaders include Grub (Grub2), rEFInd, Clover, etc. Here we mainly introduce Grub2 and rEFInd.\nGrub2 # Grub2 (GNU GRand Unified Bootloader 2) is a commonly used boot loader on Linux systems, powerful, and supports multi-system boot. Grub2\u0026rsquo;s configuration file is /boot/grub/grub.cfg, generally generated by the grub-mkconfig command.\nGrub2\u0026rsquo;s configuration file is relatively complex but powerful, and can customize various boot entries. Grub2\u0026rsquo;s theme can also be customized, but it is relatively complex.\nGrub2 Configuration # Configuration File\nGrub2\u0026rsquo;s main configuration file is /boot/grub/grub.cfg, which can be edited to configure Grub2\u0026rsquo;s boot entries.\n1 sudo vim /boot/grub/grub.cfg However, it is generally not recommended to edit the grub.cfg file directly because this file is generated by the grub-mkconfig command. If you edit the grub.cfg file directly, it will be overwritten when updating Grub2 next time. It is recommended to configure Grub2 through the /etc/default/grub file and the configuration files in the /etc/grub.d/ directory.\nBoot Entry Configuration\nGrub2\u0026rsquo;s boot entry configuration is relatively complex, for example:\n1 2 3 4 5 GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2\u0026gt; /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT=\u0026#34;quiet splash\u0026#34; GRUB_CMDLINE_LINUX=\u0026#34;\u0026#34; GRUB_DEFAULT: The number of the default boot entry, starting from 0. GRUB_TIMEOUT: The time the boot menu is displayed, in seconds. GRUB_DISTRIBUTOR: The system\u0026rsquo;s distributor. GRUB_CMDLINE_LINUX_DEFAULT: The default boot parameters of the Linux kernel. GRUB_CMDLINE_LINUX: The boot parameters of the Linux kernel. Customize Theme\nGrub2\u0026rsquo;s theme can be customized, you can put the desired theme files in the /boot/grub/themes directory. The theme directory contains theme files, font files, background images, etc., and a configuration file named theme.txt that can configure various properties of the theme.\nYou can specify the theme file in the configuration file /etc/default/grub:\n1 GRUB_THEME=\u0026#34;/boot/grub/themes/theme-name/theme.txt\u0026#34; More Grub2 configurations can be found at Grub2 Configuration.\nrEFInd # rEFInd is a UEFI-based boot loader, powerful, and supports multi-system boot. rEFInd\u0026rsquo;s configuration file is /boot/efi/EFI/refind/refind.conf, generally generated by the refind-install command.\nrEFInd\u0026rsquo;s configuration file is relatively simple but powerful, and can customize various boot entries. rEFInd\u0026rsquo;s theme can also be customized, relatively simple.\nrEFInd Installation # rEFInd installation is relatively simple, you can directly download the rEFInd installation package, unzip it to the EFI partition. You can also install rEFInd using the refind-install command.\n1 refind-install After installation, rEFInd will create the EFI/refind directory in the EFI partition, which contains rEFInd\u0026rsquo;s boot files and configuration files.\nrEFInd Configuration # rEFInd\u0026rsquo;s configuration file is /boot/efi/EFI/refind/refind.conf, you can edit this file to configure rEFInd\u0026rsquo;s boot entries.\n1 sudo vim /boot/efi/EFI/refind/refind.conf Boot Entry Configuration\nrEFInd\u0026rsquo;s boot entry configuration is relatively simple, for example:\n1 2 3 timeout 5 hideui singleuser scanfor manual,external timeout: The time the boot menu is displayed, in seconds. hideui: Hide the boot menu and only display single-user mode. scanfor: The way to scan boot entries, you can scan manually configured boot entries, or scan boot entries on external devices. Customize Theme\nrEFInd\u0026rsquo;s theme can be customized, you can put the desired theme files in the /boot/efi/EFI/refind/theme directory. The theme file is a CSS file, you can modify the color, font, background, etc.\nYou can specify the theme file in the configuration file /boot/efi/EFI/refind/refind.conf:\n1 include themes/theme-name/theme.conf Hide Boot Entries\nSometimes rEFInd will scan many unnecessary boot entries, for example, if you choose to use Grub2 as the boot loader when installing the system (Deepin, Fedora, etc. systems will install Grub2 by default, and it seems that you can\u0026rsquo;t cancel it during installation), then rEFInd will scan Grub2\u0026rsquo;s boot entries. You can hide these boot entries by configuring the file /boot/efi/EFI/refind/refind.conf. rEFInd can choose to ignore certain files (dont_scan_files), tools (dont_scan_tools), directories (dont_scan_dirs), and volumes (dont_scan_volumes).\n1 2 dont_scan_files shim.efi,mmx64.efi dont_scan_dirs ESP:/EFI/ubuntu,EFI/deepin,EFI/fedora Troubleshooting # Install System on SD Card or USB Drive\nIf your computer has an SD card slot, you can choose to install the system on the SD card. Similar to the method of installing multiple operating systems mentioned above, there are two EFI partition methods for installing the system on the SD card, one is to create an EFI partition on the SD card, and the other is to mount the EFI partition on the hard disk to /boot/efi when installing the system. For the /boot partition, you can choose to create it directly on the SD card, or you can create a 500M ext4 partition on the hard disk first, and then mount it to /boot when installing the system. The /boot partition mainly stores kernel files and boot files, generally about 200M of space is enough, but if you choose to back up multiple kernels, you will need more space. Fedora system will limit the /boot partition to no less than 512M. If you choose to put the /boot partition on the SD card, you may encounter some security restrictions, for example, Secure Boot may prevent booting from the SD card. You need to register the registration key or disable Secure Boot. I generally choose to put the EFI partition and /boot partition on the hard disk. I haven\u0026rsquo;t tried installing the system on a USB drive yet, but the principle should be the same as installing it on an SD card.\nFedora Fails to Boot\nAfter installing Fedora on the SD card, I encountered a problem that the system could not boot. rEFInd can scan Fedora\u0026rsquo;s kernel files vmlinuz-xxx and initramfs-xxx.img, but it will hang halfway through the boot process, reporting:\n1 2 3 4 5 6 7 Failed to switch root: Specified switch root path /sysroot does not seem to be an OS tree. os-release file is missing. initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE initrd-switch-root.service: Failed with result \u0026#39;exit-code\u0026#39;. Failed to start initrd-switch-root.service: Switch Root. Startup finished in 1.073s (kernel) + 1.000s (initrd) + 1.000s (userspace) = 3.073s. initrd-switch-root.service: Triggering OnFailure= dependencies. Started emergency.service - Emergency Shell. After checking in the Emergency Shell, I found that the /sysroot directory was empty, with no files. Normally, the /sysroot directory should contain the files of the operating system after loading the kernel and initializing the file system.\nSo this problem is because the SD card is not correctly mounted when starting the kernel and initializing the file system. You can manually mount the SD card in the Emergency Shell, then exit to exit the Emergency Shell, and the system can start normally.\n1 2 mount /dev/sda1 /sysroot exit However, this is obviously not a good solution because you need to manually mount the SD card every time you start it. So the correct solution is to let rEFInd correctly mount the SD card when starting the kernel, which requires adding a root parameter to the boot options. You can add a menu item with an options parameter in the rEFInd configuration file /boot/efi/EFI/refind/refind.conf.\n1 2 3 4 5 6 menuentry \u0026#34;Fedora\u0026#34; { volume \u0026#34;SD-boot\u0026#34; loader /vmlinuz-xxx initrd /initramfs-xxx.img options \u0026#34;root=/dev/sda1\u0026#34; } However, this solution is not very good because\nAfter each kernel update, you need to manually modify the rEFInd configuration file. If you have multiple kernels, this menu item cannot generate submenus automatically, you need to manually add menu items for each kernel. So a better solution is to let the entire boot partition use the root option. The correct way is to create a refind_linux.conf file in the root directory of the /boot partition, and then add a root parameter in the rEFInd configuration file /boot/efi/EFI/refind/refind.conf.\n1 \u0026#34;Boot with standard options\u0026#34; \u0026#34;root=/dev/sda1\u0026#34; The problem is finally solved!\n","date":"24 May 2024","externalUrl":null,"permalink":"/en/p/multi-system-boot-loader-configuration/","section":"Posts","summary":" Motivation # I have installed dual or multiple systems on my computer before, but I have always been confused about the boot entries of multiple systems, often encountering problems such as missing boot entries and duplicate boot entries. This time I plan to study the configuration of multi-system boot entries in depth, including the settings of boot loaders, EFI/Grub/rEFInd, and how to customize the theme of multi-system boot loaders.\n","title":"Multi-System Boot Loader Configuration","type":"post"},{"content":"","date":"24 May 2024","externalUrl":null,"permalink":"/en/tags/refind/","section":"Tags","summary":"","title":"REFInd","type":"tags"},{"content":" Motivation # After deciding to build a personal server on my own computer, I bought a domain name and configured dynamic domain name resolution, so I could access my server through the domain name. Later, I built a lot of websites on this server and used Nginx to configure reverse proxy to access these websites from the public network. Here I record how I configured Nginx reverse proxy.\nPrerequisites # A computer running Linux (as a private server) nginx (as an HTTP server) Private domain name (for detailed operations, please refer to the previous post \u0026ldquo;How This Website Was Built 1 \u0026ndash; Purchase and Configure a Personal Domain Name\u0026rdquo;) Install and Configure nginx # Install nginx # Install nginx 1 sudo apt install nginx Start nginx service 1 sudo systemctl enable nginx Nginx Configuration File # Configuration File Structure # The nginx configuration file is generally located in the /etc/nginx directory, and the directory structure is generally as follows:\n1 2 3 4 5 6 7 8 /etc/nginx ├── ... ├── nginx.conf ├── sites-available │ ├── default │ └── ... └── sites-enabled └── ... where nginx.conf is the main configuration file of nginx, and the sites-available directory contains all the website configuration files. Each time a new website is added, we can put the configuration file in the sites-available directory.\nIn the nginx.conf file, there is a default configuration as follows:\n1 2 3 4 http { ... include /etc/nginx/sites-enabled/*; } This means that nginx will automatically read all the configuration files in the sites-enabled directory. Therefore, we can create a new configuration file in the sites-available directory, and then create a symbolic link in the sites-enabled directory pointing to this configuration file, so that nginx will read this configuration file.\nConfiguration File Example # General Static Website\nA general static website refers to a website with only static files, such as HTML, CSS, JS, etc., or the front-end part of a website that is separated from the back-end. The configuration file for this type of website is generally as follows:\n1 2 3 4 5 6 7 server { listen 80; server_name www.your.domain.name; root /home/YourUserName/Documents/www/website1/public; location / { } } where:\nlisten 80; means listening on port 80, the port of the HTTP protocol. server_name www.your.domain.name; means that the domain name www.your.domain.name points to this website. root /home/YourUserName/Documents/www/website1/public; means the root directory of the website. location / {} means that all requests will be forwarded to the root directory of the website. Note that here we point the domain name to the ~/Documents/www/website1 directory\u0026rsquo;s public folder, rather than the entire website1 directory. Because we don\u0026rsquo;t want to expose some private files.\nIf you need to configure HTTPS, you can use the following configuration:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 server{ listen 80; server_name www.your.domain.name; return 301 https://www.your.domain.name$request_uri; } server{ listen 443 ssl; ssl_certificate /home/YourUserName/Documents/www/website1/private/cert.pem; ssl_certificate_key /home/YourUserName/Documents/www/website1/private/key.pem; root /home/YourUserName/Documents/www/website1/public; server_name www.your.domain.name; location /{ } } where ssl_certificate and ssl_certificate_key are the paths to the SSL certificate and private key.\nGeneral Dynamic Website\nA general dynamic website refers to a website with backend code, such as PHP, Python, Node.js, etc., or a website hosted in a Docker container. This type of website is generally accessed through a specific port, such as localhost:3000. The configuration file for this type of website is generally as follows:\n1 2 3 4 5 6 7 server { listen 80; server_name www.your.domain.name; location / { proxy_pass http://localhost:3000; } } where:\nproxy_pass http://localhost:3000; means that all requests are forwarded to localhost:3000, the address of the backend service. If you need to configure HTTPS, you can use the following configuration:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 server{ listen 80; server_name www.your.domain.name; return 301 https://www.your.domain.name$request_uri; } server{ listen 443 ssl http2; listen [::]:443 ssl http2; server_name www.your.domain.name; ssl_certificate /home/YourUserName/Documents/www/website1/private/cert.pem; ssl_certificate_key /home/YourUserName/Documents/www/website1/private/key.pem; location /{ proxy_pass http://localhost:3000; } } Enable Configuration File # Create a new configuration file in the sites-available directory according to the configuration file example above, such as website1.conf.\nCheck if there are any syntax errors in the configuration file\n1 sudo nginx -t If there are no errors, it will display nginx: configuration file /etc/nginx/nginx.conf test is successful.\nCreate a symbolic link in the sites-enabled directory pointing to this configuration file.\n1 sudo ln -s /etc/nginx/sites-available/website1.conf /etc/nginx/sites-enabled/website1.conf Restart the nginx service\n1 sudo service nginx reload You can check the status of nginx with the following command:\n1 sudo nginx status Access www.your.domain.name through a browser, and you will see the content of the website.\nLoad Balancing # If we have multiple backend services, we can use Nginx\u0026rsquo;s load balancing feature. Load balancing can distribute requests to multiple backend services to improve performance and reliability. Since I only have one server at the moment, I will not go into detail here. I will update this part when I have enough money to buy multiple servers.\n","date":"8 May 2024","externalUrl":null,"permalink":"/en/p/access-personal-website-from-public-networknginx-reverse-proxy-configuration/","section":"Posts","summary":" Motivation # After deciding to build a personal server on my own computer, I bought a domain name and configured dynamic domain name resolution, so I could access my server through the domain name. Later, I built a lot of websites on this server and used Nginx to configure reverse proxy to access these websites from the public network. Here I record how I configured Nginx reverse proxy.\n","title":"Access Personal Website from Public Network——Nginx Reverse Proxy Configuration","type":"post"},{"content":"","date":"8 May 2024","externalUrl":null,"permalink":"/en/tags/domain-name/","section":"Tags","summary":"","title":"Domain Name","type":"tags"},{"content":"","date":"8 May 2024","externalUrl":null,"permalink":"/en/tags/nginx/","section":"Tags","summary":"","title":"Nginx","type":"tags"},{"content":"","date":"24 April 2024","externalUrl":null,"permalink":"/en/tags/latex/","section":"Tags","summary":"","title":"LaTeX","type":"tags"},{"content":"","date":"24 April 2024","externalUrl":null,"permalink":"/en/tags/overleaf/","section":"Tags","summary":"","title":"Overleaf","type":"tags"},{"content":"","date":"24 April 2024","externalUrl":null,"permalink":"/en/tags/self-host/","section":"Tags","summary":"","title":"Self-Host","type":"tags"},{"content":" Motivation # I have been using Overleaf to collaborate on LaTeX documents with others when working on assignments or papers. Although it is not as customizable as local editors (such as VS Code), it is still very convenient to collaborate with others. When writing my doctoral dissertation, I purchased a one-year student subscription for Git synchronization. Due to the reduced frequency of use after graduation, I did not renew it. Recently, I needed to recompile my doctoral dissertation, only to find that the compilation time for free users of Overleaf is limited. Due to the large size of the doctoral dissertation, the compilation time exceeded the limit, which prevented me from compiling successfully.\nI have also tried to use the open-source community version of Overleaf to build an online LaTeX service, but after setting it up, I still used the official Overleaf service more, so I did not maintain my own service. This time, because I need to recompile my doctoral dissertation, I tried to compile it on my own Overleaf, but found that many packages were missing, which prevented me from compiling. So I decided to rebuild an Overleaf open-source community version service and install the full version of TeX Live to be able to compile all LaTeX documents.\nIntroduction to Overleaf # Overleaf is an online LaTeX editor that supports collaborative editing, provides rich templates and packages, and can compile LaTeX documents online. Overleaf has a free version and a paid version. The free version has some limitations, such as compilation time limits and project number limits. The paid version of Overleaf provides more features, such as unlimited compilation time, unlimited number of projects, Git synchronization, etc.\nThe open-source community version of Overleaf is the open-source version of Overleaf, which allows you to build a LaTeX online editor similar to Overleaf on your own server. The Overleaf open-source community version provides similar features to the official Overleaf service, such as online compilation of LaTeX documents, collaborative editing, template and package support, etc. Unfortunately, the Overleaf community version does not support integrated Git synchronization. If you want to use Git synchronization and some other features, you need to purchase Overleaf Server Pro. Here I only need the Overleaf community version.\nPrerequisites # Docker and Docker Compose Domain name (second-level domain name is sufficient) Installation of Overleaf Community Edition # We can use Docker to easily and quickly build the Overleaf community edition service. There are two installation methods: one is to use the docker-compose.yml file provided by the Overleaf community edition repository, and the other is to use the toolbox provided by Overleaf. Overleaf officially recommends the second method, but here are both methods.\nMethod 1: Install using the docker-compose.yml file # You can find the docker-compose.yml file in the Overleaf community edition repository and download it to your local machine, then use Docker Compose to start the service.\nYou may need to modify the docker-compose.yml file according to your actual situation, such as changing the port, data volume path, etc. After modifying it, use the following command to start the service:\n1 docker-compose up -d Reference: Overleaf Community Edition Quick Start Guide.\nMethod 2: Install using the Overleaf toolbox # Overleaf provides a toolbox, which wraps some common docker commands and can be used to initialize, start, stop, diagnose, and upgrade the Overleaf service. Although I think this toolbox makes the process of deploying docker more complicated compared to using docker-compose.yml directly, this toolbox does provide more flexible customization options.\nTo deploy the Overleaf community edition service using this toolbox, you can refer to the Overleaf Community Edition Toolbox Documentation. In short, there are several steps:\nDownload the toolbox:\n1 git clone https://github.com/overleaf/toolkit.git Enter the toolbox directory:\n1 cd toolkit Initialize the installation configuration:\n1 bin/init Running this command will generate a config folder in the current directory, which contains three configuration files:\noverleaf.rc: Overleaf configuration file. Users can configure some parameters of Overleaf in this file, such as port, data volume path, etc. variables.env: Environment variable configuration file version: Select the Overleaf version. Note that after version 5.0.0, Overleaf replaced the original ShareLaTeX trademark with the Overleaf trademark, so if the environment variable configuration file uses variables with the OVERLEAF prefix, you need to select a version after 5.0.0. After modifying the configuration files, run the following command to start the service (may need to elevate permissions with sudo):\n1 bin/up This command will start the service in command line debug mode, and you can see the output log of the service after running it. If you want to stop the service, you can press Ctrl+C.\nIf you want to start the service in the background mode, you can use the following command (may need to elevate permissions with sudo):\n1 bin/up -d If you want to stop the service, you can use the following command (may need to elevate permissions with sudo):\n1 bin/stop Configure Nginx Reverse Proxy # The Overleaf community edition docker-compose.yml file contains an nginx container, and if you use the Overleaf toolbox, you can also configure Nginx in the overleaf.rc configuration file. If you need TSL/SSL encryption, you can use the bin/init --tls command when initializing. This will generate an nginx folder in the config directory, which contains Nginx configuration files and an example SSL certificate.\nHowever, since I had already deployed a separate Nginx service on the server to manage all websites, I chose to use a separate Nginx service to reverse proxy the Overleaf service. The specific process can be found in Building a Personal Website (II): Building a Personal Website Using the Hugo Framework.\nUpgrade TeX Live # The Overleaf docker image comes with a basic version of TeX Live, but this version may not include all packages. If you need to compile some special LaTeX documents, you may need to install the full version of TeX Live. You can refer to the Overleaf toolbox documentation on upgrading TeX Live. The main steps are as follows:\nInstall the full version of TeX Live # Use the following command to enter the Overleaf container:\n1 docker exec -it sharelatex bash Check the current TeX Live version:\n1 tlmgr --version Update TeX Live:\n1 tlmgr install scheme-full When I ran this command, I encountered the following problem:\n1 2 3 4 tlmgr: Local TeX Live (2023) is older than remote repository (2024). Cross release updates are only supported with update-tlmgr-latest(.sh/.exe) --update See https://tug.org/texlive/upgrade.html for details. This is because the versions of TeX Live are different. You can use the following command to specify the appropriate version of TeX Live:\n1 tlmgr option repository https://ftp.math.utah.edu/pub/tex/historic/systems/texlive/2023/tlnet-final After specifying the appropriate version, run the update command again:\n1 tlmgr install scheme-full If everything goes well, this command will download and install the full version of TeX Live. This will install more than 4000 packages and may take some time.\nAdd the path:\nAfter the installation is complete, you need to use the following command to add the path:\n1 tlmgr path add If you do not run this command, you may encounter problems such as being unable to compile EPS images when compiling LaTeX documents in Overleaf.\nSave changes to the image # After making the above changes, the upgraded TeX Live will only be saved in the current container. If the container is deleted, these changes will also be lost. If you want to save these changes to the image and use them when creating containers later, you can perform the following steps:\nUse the following command to update the image (may need to elevate permissions with sudo):\n1 docker commit sharelatex sharelatex/sharelatex:with-texlive-full Then you can add a docker-compose.override.yml file in the config directory to specify the new image when creating containers later:\n1 2 3 4 5 --- version: \u0026#39;2.2\u0026#39; services: sharelatex: image: sharelatex/sharelatex:with-texlive-full Stop and delete the previous container, then recreate the container (may need to elevate permissions with sudo):\n1 2 3 bin/stop bin/docker-compose rm -f sharelatex bin/up -d ","date":"24 April 2024","externalUrl":null,"permalink":"/en/p/self-hosted-online-latex-compilation-and-preview-service-overleaf-community-edition/","section":"Posts","summary":" Motivation # I have been using Overleaf to collaborate on LaTeX documents with others when working on assignments or papers. Although it is not as customizable as local editors (such as VS Code), it is still very convenient to collaborate with others. When writing my doctoral dissertation, I purchased a one-year student subscription for Git synchronization. Due to the reduced frequency of use after graduation, I did not renew it. Recently, I needed to recompile my doctoral dissertation, only to find that the compilation time for free users of Overleaf is limited. Due to the large size of the doctoral dissertation, the compilation time exceeded the limit, which prevented me from compiling successfully.\n","title":"Self-hosted Online LaTeX Compilation and Preview Service: Overleaf Community Edition","type":"post"},{"content":"","date":"12 February 2024","externalUrl":null,"permalink":"/en/tags/development/","section":"Tags","summary":"","title":"Development","type":"tags"},{"content":"","date":"12 February 2024","externalUrl":null,"permalink":"/en/tags/git/","section":"Tags","summary":"","title":"Git","type":"tags"},{"content":"","date":"12 February 2024","externalUrl":null,"permalink":"/en/tags/github/","section":"Tags","summary":"","title":"GitHub","type":"tags"},{"content":"","date":"12 February 2024","externalUrl":null,"permalink":"/en/tags/gitlab/","section":"Tags","summary":"","title":"GitLab","type":"tags"},{"content":" Background # If you see an interesting project on GitHub and want to participate in its development, you generally need to follow a certain process. Here\u0026rsquo;s a general project collaboration development process.\nOf course, this process is not only applicable to GitHub, but also to other code hosting platforms such as GitLab.\nPrerequisites # Git is installed GitHub account is registered Git settings # Set username and email\nThere are two modes for setting the username and email.\nIf you only use one GitHub account, you can set the global username and email.\n1 2 git config --global user.name \u0026#34;Your Name\u0026#34; git config --global user.email \u0026#34;Your Email\u0026#34; If you have multiple GitHub accounts, you can set the username and email for each Git repository separately.\n1 2 3 cd /path/to/your/repo git config user.name \u0026#34;Your Name\u0026#34; git config user.email \u0026#34;Your Email\u0026#34; Login credential management\nAfter setting the username and email in the previous step, when cloning a repository and every time you git push, you need to enter your GitHub username and password, which is very cumbersome. You can use a credential manager to save your username and password.\nWindows\nIf the version number of Git you installed is higher than 2.29, Git has integrated support for GitHub OAuth. When you first clone a GitHub private repository via HTTPS, Git will prompt you to log in to GitHub using a browser and authorize Git to access your GitHub account. After that, Git will automatically save your GitHub credentials, so you don\u0026rsquo;t need to enter your username and password again.\nmacOS\nMethod 1 (Old method):\nYou need to install Git Credential Manager. You can run the following command in the terminal:\n1 brew install --cask git-credential-manager After installation, when you first clone a GitHub private repository via HTTPS, Git will prompt you to log in to GitHub using a browser and authorize Git to access your GitHub account. After that, Git will automatically save your GitHub credentials, so you don\u0026rsquo;t need to enter your username and password again.\nGit Credential Manager supports various Git hosting platforms, including GitHub, GitLab, Bitbucket, etc., and you can use this tool for credential management.\nMethod 2 (Recommended method):\nFor GitHub accounts, you can use the GitHub CLI tool for credential management. You can run the following command in the terminal to install the GitHub CLI tool:\n1 brew install gh After installation, run the following command to log in to your GitHub account:\n1 gh auth login Follow the prompts to log in to your GitHub account via a browser. After a successful login, the GitHub CLI tool will automatically configure Git\u0026rsquo;s credential management, so you don\u0026rsquo;t need to enter your username and password again.\nLinux\nLinux systems are similar to macOS and can choose either Git Credential Manager or GitHub CLI tool for credential management. Just use the package manager of your Linux distribution to install it.\nGitHub collaboration development process # Overview # To participate in the development of others\u0026rsquo; projects, the general idea is:\nFork the project to your own GitHub account on the GitHub website. Clone the forked project to your local machine and add the original repository as a remote repository. This way: Your local repository corresponds to two remote repositories: one is the Fork repository (origin), and the other is the original repository (upstream). Do not modify the main branch of the origin remote repository on GitHub, and do not develop on the main branch locally. Always keep the local main branch, the main branch of the origin remote repository, and the main branch of the upstream remote repository in sync. When you want to add a new feature or fix a bug, first create a new branch from the main branch and develop on this new branch. After completing development locally, push the new local branch to the origin remote repository. GitHub will remind you x commits ahead of upstream/main, y commits behind upstream/main, where x represents the number of commits on your new branch, and y represents the number of new commits on the main branch of the upstream remote repository while you were developing the new feature/fixing the bug. If y is not 0, it means the main branch of the upstream remote repository has new commits, and you need to synchronize your new branch with the main branch of the upstream remote repository. First, pull the latest code of the main branch of the upstream remote repository to the local main branch. Then merge the local main branch into your new branch. This process has two cases: If there are no conflicts, you can directly merge or rebase. If there are conflicts, you need to resolve these conflicts and then commit. If you want to merge your new feature/bug fix into the original repository, you need to create a Pull Request on GitHub, wait for the author of the original repository to review your Pull Request, and if necessary, you need to make changes according to the feedback from the author of the original repository. Below we will introduce this process in detail.\nOperations on the GitHub website —— Fork # Open the GitHub page of the project you want to participate in, click the Fork button in the upper right corner, and fork the project to your own GitHub account. We call the original project the original repository and the project forked to your own account the Fork repository.\nIn your GitHub account, find the forked project (i.e., Fork repository), click the Clone or download button, and copy the URL of the project.\nLocal operations —— Clone # In the terminal on your local machine, switch to the directory where you want to store the project, and run the following command to clone the project to your local machine.\n1 git clone https://github.com/your-username/project-name.git Here, your-username is your GitHub username, and project-name is the name of the project.\nEnter the project directory, run the following command to add the original repository as a remote repository.\n1 git remote add upstream https://github.com/authors-usename/project-name.git Here, authors-username is the GitHub username of the author of the original repository, and project-name is the name of the project.\nThen run the following command to check the status of the remote repository.\n1 git remote -v If everything is normal, you will see output similar to the following:\n1 2 3 4 origin https://github.com/your-username/project-name.git (fetch) origin https://github.com/your-username/project-name.git (push) upstream https://github.com/authors-username/project-name.git (fetch) upstream https://github.com/authors-username/project-name.git (push) Here, origin is your Fork repository, and upstream is the original repository.\nLocal development # In the terminal on your local machine, switch to the project directory, run the following command to create a new branch.\n1 git checkout -b new-branch-name Here, new-branch-name is the name of your new branch, which can be the name of the new feature you want to add, or the name of the bug you want to fix.\nIn the terminal on your local machine, develop, modify code/add new features/fix bugs. Then run the following command to add the modified files to the staging area and commit them to the local repository.\n1 2 git add . git commit -m \u0026#34;Your commit message\u0026#34; You can also push the changes in the local repository to the Fork repository.\n1 git push origin new-branch-name Repeat the operations in step 2 until your new feature/bug fix is complete.\nOn GitHub, open your Fork repository. GitHub will remind you x commits ahead of upstream/main, y commits behind upstream/main, where x represents the number of commits on your new branch, and y represents the number of new commits on the main branch of the upstream remote repository while you were developing the new feature/fixing the bug. If y is not 0, it means the main branch of the upstream remote repository has new commits, and you need to synchronize your new branch with the main branch of the upstream remote repository.\n1 2 git checkout main git pull upstream main Then you can merge the latest code of the main branch into your new-branch-name branch locally. There are two ways to do this \u0026ldquo;merge\u0026rdquo;: merge and rebase:\nmerge: Merge the latest code of the main branch into the new-branch-name branch, the commit history of the new-branch-name branch remains unchanged, forming a new commit history.\n1 2 git checkout new-branch-name git merge main rebase: Will package the commit history of the new-branch-name branch and place it after the latest commit history of the main branch, forming a new commit history.\n1 2 git checkout new-branch-name git rebase main Whether merge or rebase, if the main branch of the original repository has modified the files you modified on the new-branch-name branch, conflicts will occur. You need to resolve these conflicts. After resolving the conflicts, run the following command to add the modified files to the staging area and commit them to the local repository.\n1 2 git add . git commit -m \u0026#34;Your commit message\u0026#34; Finally, run the following command again to push the modified files to the Fork repository.\n1 git push origin new-branch-name This way, your new-branch-name branch will not have conflicts when merged into the main branch of the original repository.\nOperations on the GitHub website —— Pull Request # Before creating a Pull Request, first switch to your new-branch-name branch on GitHub and confirm that the y in This branch is x commits ahead of upstream/main, y commits behind upstream/main displayed on GitHub is 0 (i.e., only shows This branch is x commits ahead of upstream/main, not showing behind), indicating that your new-branch-name branch is already synchronized with the main branch of the original repository. If y is not 0, return to the previous step, step 4, and first synchronize your new-branch-name branch with the main branch of the original repository.\nOpen the GitHub page of your Fork repository, click the New pull request button, and create a new Pull Request. Note that if the original repository provides a Pull Request template, you need to fill in the title and content of the Pull Request according to the Pull Request template of the original repository.\nWait for the author of the original repository to review your Pull Request. If necessary, you need to make changes according to the feedback from the author of the original repository.\nIf the author of the original repository accepts your Pull Request, congratulations, your code will be merged into the original repository.\nLocal operations —— Update local repository # In the terminal on your local machine, switch to the main branch, run the following command to pull the latest code of the main branch of the original repository to your local machine.\n1 2 git checkout main git pull upstream main You can also use the git fetch upstream main command to check the latest code of the main branch of the original repository, and then use the git merge upstream/main command to merge the latest code of the main branch of the original repository into the main branch of your local machine.\n1 2 3 git fetch upstream main git checkout main git merge upstream/main This way, your local main branch will be synchronized with the main branch of the original repository.\nThen you can push the main branch of your local machine to the Fork repository.\n1 git push origin main (Optional) If your new-branch-name branch has been merged into the main branch of the original repository, you can delete the new-branch-name branch.\n1 git branch -d new-branch-name You can also delete the remote branch of the new-branch-name branch.\n1 git push origin --delete new-branch-name This way, you have completed the work of collaborating with others to add a new feature/fix a bug on GitHub.\nContinue to Develop New Features/Fix New Bugs # If you need to continue adding new features/fixing new bugs, you can first synchronize your main branch of the Fork repository with the main branch of the original repository to ensure that you start working from the latest version of the project:\n```bash git checkout main git pull upstream main git push origin main ``` Then, you can create a new branch on the main branch to continue developing new features/fixing new bugs.\n```bash git checkout -b another-new-branch-name ``` The subsequent operations are the same as after local development.\n","date":"12 February 2024","externalUrl":null,"permalink":"/en/p/project-collaboration-development-process-on-github/gitlab/","section":"Posts","summary":" Background # If you see an interesting project on GitHub and want to participate in its development, you generally need to follow a certain process. Here’s a general project collaboration development process.\nOf course, this process is not only applicable to GitHub, but also to other code hosting platforms such as GitLab.\nPrerequisites # Git is installed GitHub account is registered Git settings # Set username and email\n","title":"Project Collaboration Development Process on GitHub/GitLab","type":"post"},{"content":" Background # This is the fourth article in the \u0026ldquo;Learn PyTorch by Examples\u0026rdquo; series. In the previous three articles:\n\u0026ldquo;Learn PyTorch by Examples (1): MNIST Handwritten Digit Recognition (I) \u0026ndash; PyTorch Basics and Neural Network Basics\u0026rdquo; \u0026ldquo;Learn PyTorch by Examples (2): MNIST Handwritten Digit Recognition (II) \u0026ndash; Parameter Selection in Neural Network\u0026rdquo; \u0026ldquo;Learn PyTorch by Examples (3): MNIST Handwritten Digit Recognition (III) \u0026ndash; Convolutional Neural Networks\u0026rdquo; In the articles, we introduced how to solve image classification problems using PyTorch. Another important problem in machine learning is sequence prediction. Unlike image classification, sequence prediction requires considering the correlation between data. Recurrent Neural Network (RNN) is a neural network that can handle sequence data, as it saves the previous data information at each time step. In this article, we will use a simple RNN to predict the sine function.\nThere is an example in the PyTorch official repository https://github.com/pytorch/examples/, which uses Long Short-Term Memory (LSTM) to predict the sine function. We will not consider LSTM for now, but use a simple RNN to predict the sine function.\nThe code for this article can be found in the T04_series_rnn folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial.\nRecurrent Neural Network (RNN) Introduction # In the previous articles, we introduced feedforward neural networks, including fully connected neural networks and convolutional neural networks. Both of them are feedforward neural networks. A disadvantage of feedforward neural networks is that they cannot handle sequence data because they do not store previous data information. Recurrent Neural Network (RNN) is a neural network that can handle sequence data. It saves the previous data information at each time step, so it can handle sequence data. It is worth noting that the sequence data here is not necessarily time series data, but can also be spatial sequence data. For example, sentences in natural language processing are spatial sequence data, and audio data in speech recognition is sequence data. For convenience, we call each element in the sequence a time step.\nThe structure of RNN is not complicated. Its core has two points:\nDuring training, the training data needs to be unfolded according to time steps, and then the loss function is calculated by traversing each time step. Finally, the weights are updated through the backpropagation algorithm. For each time step, there is not only an output but also a hidden state. This hidden state is passed to the next time step, thus retaining the previous data information. For more information about RNN, you can refer to the CS230 course slides from Stanford University. Here we use an animated image to briefly illustrate how RNN works:\nHere, $x$ is the input, $h$ is the hidden state, $y$ is the output, and $t$ is the time step. It can be seen that the hidden state $h$ is passed to the next time step at each time step, thus retaining the previous data information. $x$, $h$, and $y$ are each a layer of the neural network. $h$ is the hidden layer, $x$ is the input layer, and $y$ is the output layer. The size of each layer can be determined according to the specific problem.\nSine Function Prediction # The sine function can be regarded as a time series. At some moments, the value of the sine function may be the same, but the values after that may be different. For example, for $y = \\sin(x)$, when $x = 0$ and $x = \\pi$, $y$ is both $0$, but at the next time step of these two moments (assuming the time step size is 0.01, then the next two time steps are $x = 0.01$ and $x = \\pi + 0.01$), $y$ is different. To predict the value of the next time step, we need to know not only the value of the current time step but also the values of the previous several time steps. This is exactly the purpose of the Recurrent Neural Network.\nRNN Model Design # This problem is relatively simple, and we only need to use a Recurrent Neural Network. The input of our Recurrent Neural Network is a sequence, and the output is the next value of this sequence. The input sequence is the value of the sine function, and the output is the next value of the sine function. The structure of our Recurrent Neural Network is as follows:\nInput layer: The size of the input layer is 1, that is, there is only one input at each time step. Hidden layer: The size of the hidden layer can be chosen arbitrarily. Considering that this problem is relatively simple, we choose 10 neurons as the hidden layer. Output layer: The size of the output layer is 1, that is, there is only one output at each time step. In this way, the structure of our Recurrent Neural Network is determined. We can use the nn.RNN class in PyTorch to implement this Recurrent Neural Network:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 class SimpleRNN(nn.Module): def __init__(self, rnn_type, input_size, hidden_size, num_layers): super(SimpleRNN, self).__init__() self.hidden_size = hidden_size self.num_layers = num_layers self.rnn = nn.RNN(input_size=input_size, hidden_size=hidden_size, dropout=(0 if num_layers == 1 else 0.05), num_layers=num_layers, batch_first=True) self.out = nn.Linear(hidden_size, 1) # Linear layer is output of model def forward(self, x, h_state): # Define our forward pass, we take some input sequence and an initial hidden state. r_out, h_state = self.rnn(x, h_state) final_y = self.out(r_out[:, -1, :]) # Return only the last output of RNN. return final_y, h_state Here we define a SimpleRNN class, which inherits from PyTorch\u0026rsquo;s nn.Module class. In the __init__ function, we define an instance of the nn.RNN class, which is our Recurrent Neural Network. In the forward function, we define the forward propagation process of the Recurrent Neural Network, that is, how we calculate the output. Here we only need to return the output of the last time step.\nData Preparation # In the previous three articles, we used datasets that others had prepared when training neural networks. But here, we need to prepare the data ourselves. For this problem, data preparation is very simple. We only need to generate some sine function data. After generating the data, we need to encapsulate it into a PyTorch Dataset class so that we can easily load the data using PyTorch\u0026rsquo;s DataLoader class.\nThe complete code for generating the dataset can be found in the SineWaveDataset.py file in the T04_series_rnn folder in the GitHub repository corresponding to this article https://github.com/jin-li/pytorch-tutorial. The specific code is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 import numpy as np import torch from torch.utils.data import Dataset class RNNDataset(Dataset): def __init__(self, x, y=None): self.data = x self.labels = y def __len__(self): return self.data.shape[0] def __getitem__(self, idx): if self.labels is not None: return self.data[idx], self.labels[idx] else: return self.data[idx] def create_dataset(sequence_length, train_percent=0.8): # Create sin wave at discrete time steps. num_time_steps = 2000 time_steps = np.linspace(start=0, stop=1000, num=num_time_steps, dtype=np.float32) discrete_sin_wave = (np.sin(time_steps * 2 * np.pi / 20)).reshape(-1, 1) # Take (sequence_length + 1) elements \u0026amp; put as a row in sequence_data, extra element is value we want to predict. # Move one time step and keep grabbing till we reach the end of our sampled sin wave. sequence_data = [] for i in range(num_time_steps - sequence_length): sequence_data.append(discrete_sin_wave[i: i + sequence_length + 1, 0]) sequence_data = np.array(sequence_data) # Split for train/val. num_total_samples = sequence_data.shape[0] num_train_samples = int(train_percent * num_total_samples) train_set = sequence_data[:num_train_samples, :] test_set = sequence_data[num_train_samples:, :] print(\u0026#39;{} total sequence samples, {} used for training\u0026#39;.format(num_total_samples, num_train_samples)) # Take off the last element of each row and this will be our target value to predict. x_train = train_set[:, :-1][:, :, np.newaxis] y_train = train_set[:, -1][:, np.newaxis] x_test = test_set[:, :-1][:, :, np.newaxis] y_test = test_set[:, -1][:, np.newaxis] train_data = RNNDataset(x_train, y_train) test_data = RNNDataset(x_test, y_test) torch.save(train_data, \u0026#39;train_data.pt\u0026#39;) torch.save(test_data, \u0026#39;test_data.pt\u0026#39;) if __name__ == \u0026#39;__main__\u0026#39;: create_dataset(sequence_length=80) First, we define an RNNDataset class to store the training data. It inherits from PyTorch\u0026rsquo;s Dataset class so that we can use PyTorch\u0026rsquo;s DataLoader class to load the data. Then we define a create_dataset function to generate the sine function data. This function has two input parameters. One is sequence_length, which indicates how many time steps of data we use to predict the next time step of data. The other is train_percent, which indicates how much of the data we use for training, and the rest is used for testing. The main work of this function is to generate the sine function data and encapsulate it into an instance of the RNNDataset class. Finally, we save the training data and test data to the files train_data.pt and test_data.pt. In the create_dataset function, we first generate 2000 time steps of sine function data. Then we generate some sequence data for training from this sine data. The method of generating training data is: Starting from the first time step, take 50 consecutive time steps of data as a sequence, i.e., $x_1, x_2, \\cdots, x_{80}$. The next time step after these 50 time steps is the value to be predicted, $y = x_{81}$. Repeat the above process until all time steps are taken. Here we have 2000 time steps, so we can generate a total of $2000 - 80 = 1920$ sequences. We divide these 1950 sequence data into two parts, 80% for training ($1920 \\times 0.8 = 1536$) and 20% for testing. Finally, we encapsulate the training data and test data into instances of the RNNDataset class and save them to the files train_data.pt and test_data.pt. x_train is a 3-D tensor with a size of $1536 \\times 80 \\times 1$, where $1536$ is the number of training samples, $80$ is the sequence length, and $1$ is the input size. y_train is a 2-D tensor with a size of $1536 \\times 1$, where $1536$ is the number of training samples, and $1$ is the output size. Model Training # As in the previous articles, we need to define a training function to train our model. The code for this training function is also simple:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 def train(model, device, train_dataloader, loss_function, optimizer, epoch_idx, log_interval): model.train() trained_cnt = 0 for batch_idx, (x_batch, y_batch) in enumerate(train_dataloader): x_batch, y_batch = x_batch.to(device), y_batch.to(device) h_state = torch.zeros([model.num_layers, x_batch.size()[0], model.hidden_size]).to(device) optimizer.zero_grad() output, _ = model(x_batch, h_state) loss = loss_function(output, y_batch) loss.backward() optimizer.step() trained_cnt += len(x_batch) if batch_idx % log_interval == 0: print(\u0026#39;Train Epoch: {:5d} [{:5d} / {:5d} ({:3.0f}%)]\\tLoss: {:.3e}\u0026#39;.format( epoch_idx, trained_cnt, len(train_dataloader.dataset), 100. * (batch_idx + 1) / len(train_dataloader), loss.item())) The input parameters of this training function are similar to the training function in the previous articles, namely the model, training data loader, loss function, and optimizer, which are not repeated here.\nModel Testing # In addition to the training function, we define a testing function. After training an epoch, we need to test the performance of our model on the test data. The code for the testing function is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 def test(model, device, test_dataloader, loss_function): model.eval() loss_all = [] for x_batch, y_batch in test_dataloader: x_batch, y_batch = x_batch.to(device), y_batch.to(device) h_state = torch.zeros([model.num_layers, x_batch.size()[0], model.hidden_size]).to(device) output, _ = model(x_batch, h_state) loss = loss_function(output, y_batch) loss_all.append(loss.cpu().data.numpy()) print(\u0026#39;Test loss: \u0026#39;, np.mean(loss_all)) return np.mean(loss_all) Model Prediction # Finally, we define a prediction function to test whether our model can predict the values of a sine function sequence based on a given sequence of data. The code for the prediction function is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 def predict(model, device, dataloader, prediction_steps): model.eval() h_state = torch.zeros([model.num_layers, 1, model.hidden_size]).to(device) # Adjusted to 3-D with batch size 1 initial_input = next(iter(dataloader))[1].to(device) # Grab one initial sequence of data for use in prediction. if initial_input.dim() == 2: initial_input = initial_input.unsqueeze(0) initial = initial_input.squeeze().cpu().numpy().tolist() predictions = [] for _ in range(prediction_steps): # Predict prediction_steps steps ahead pred, h_state = model(initial_input, h_state) predictions.append(pred.item()) initial_input = pred.unsqueeze(0) # Ensure pred has the same dimensions as test_input[:, 1:, :] return initial, predictions Run the Model # We integrate the above code into one file and call the above functions in the main() function. As before, we add some command-line parameters to control the training and testing of the model. The complete code can be found in the T04_series_rnn folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial, in the time_series_rnn.py file.\nFirst, we generate the sine function dataset:\n1 python SineWaveDataset.py Then we train the model, use the model to predict a sine function sequence, and plot the prediction results with the true results:\n1 python time_series_rnn.py --plot This code runs for about 20 seconds on the GPU, with a memory usage of about 208M; if using the CPU, the running time increases to about 1 minute and 30 seconds. After running this command, we can see the model prediction results as shown in the figure below:\nThe first 80 data points are an initial sequence, and the next 150 data points are the model\u0026rsquo;s prediction results. It can be seen that the model\u0026rsquo;s prediction results are very close to the true results.\nSummary # In this article, we introduced how to use PyTorch to implement a simple Recurrent Neural Network (RNN) to predict a sine function sequence.\nIn this article, I ran the example with different random seeds multiple times. The results shown above are from one of the runs with a random seed of 18 (the default value in the code on GitHub). Readers can adjust the model parameters according to the parameter selection method we introduced in the second article of this series \u0026ldquo;Learn PyTorch by Examples (2): MNIST Handwritten Digit Recognition (II) \u0026ndash; Parameter Selection in Neural Network\u0026rdquo; to see if they can get better results.\nWith this simple example as a foundation, we will introduce how to use other neural networks, such as Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), etc., to predict the sine function sequence in the next article.\n","date":"27 January 2024","externalUrl":null,"permalink":"/en/p/learn-pytorch-by-examples-4-sequence-prediction-i--recurrent-neural-networks-rnn/","section":"Posts","summary":" Background # This is the fourth article in the “Learn PyTorch by Examples” series. In the previous three articles:\n“Learn PyTorch by Examples (1): MNIST Handwritten Digit Recognition (I) – PyTorch Basics and Neural Network Basics” “Learn PyTorch by Examples (2): MNIST Handwritten Digit Recognition (II) – Parameter Selection in Neural Network” “Learn PyTorch by Examples (3): MNIST Handwritten Digit Recognition (III) – Convolutional Neural Networks” In the articles, we introduced how to solve image classification problems using PyTorch. Another important problem in machine learning is sequence prediction. Unlike image classification, sequence prediction requires considering the correlation between data. Recurrent Neural Network (RNN) is a neural network that can handle sequence data, as it saves the previous data information at each time step. In this article, we will use a simple RNN to predict the sine function.\n","title":"Learn PyTorch by Examples (4): Sequence Prediction (I) -- Recurrent Neural Networks (RNN)","type":"post"},{"content":"","date":"27 January 2024","externalUrl":null,"permalink":"/en/tags/rnn/","section":"Tags","summary":"","title":"RNN","type":"tags"},{"content":"","date":"23 January 2024","externalUrl":null,"permalink":"/en/tags/cnn/","section":"Tags","summary":"","title":"CNN","type":"tags"},{"content":"","date":"23 January 2024","externalUrl":null,"permalink":"/en/tags/convolutional-neural-networks/","section":"Tags","summary":"","title":"Convolutional Neural Networks","type":"tags"},{"content":"","date":"23 January 2024","externalUrl":null,"permalink":"/en/tags/handwritten-digit-recognition/","section":"Tags","summary":"","title":"Handwritten Digit Recognition","type":"tags"},{"content":" Background # This is the third article in the \u0026ldquo;Learn PyTorch by Examples\u0026rdquo; series. In the previous two articles \u0026ldquo;Learn PyTorch by Examples (1): MNIST Handwritten Digit Recognition (I) \u0026ndash; PyTorch Basics and Neural Network Basics\u0026rdquo; and \u0026ldquo;Learn PyTorch by Examples (2): MNIST Handwritten Digit Recognition (II) \u0026ndash; Parameter Selection in Neural Network\u0026rdquo;, we introduced the basic concepts and usage of PyTorch, and implemented MNIST handwritten digit recognition using a simple three-layer fully connected neural network. In this article, we will use a convolutional neural network to implement MNIST handwritten digit recognition, which is an extension of the previous two articles.\nThe code for this article can be found in the T03_mnist_cnn folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial. The code is adapted from PyTorch\u0026rsquo;s official example code https://github.com/pytorch/examples.\nConvolutional Neural Networks (CNNs) Introduction # In the previous two articles, we used a simple fully connected neural network to solve the MNIST handwritten digit recognition problem. The fully connected neural network performed well, but it did not consider the local features of the image. In image recognition, local features of the image are very important, such as edges, textures, etc. Convolutional Neural Networks (CNNs) are neural networks specifically designed to process images. They can effectively extract local features of the image, thereby improving the accuracy of image recognition.\nThe key points of convolutional neural networks are the convolutional layer and the pooling layer. The convolutional layer convolves the input image with a convolutional kernel to extract local features of the image. The pooling layer pools the output of the convolutional layer using a pooling kernel to reduce the size of the feature map and improve computational efficiency. The convolutional layer and the pooling layer appear alternately, and the output is obtained through the fully connected layer. Next, we will design a simple convolutional neural network to implement MNIST handwritten digit recognition.\nThe convolutional kernel is the core of the convolutional neural network. It is a small matrix used to extract local features of the image. The size, stride, padding, and other parameters of the convolutional kernel are hyperparameters that need to be adjusted. The size of the convolutional kernel determines the size of the area the convolutional kernel can sense, i.e., the local receptive field. The stride is the distance the convolutional kernel moves each time, and padding is to pad a circle of 0 around the image to keep the size of the image unchanged. The size and stride of the pooling kernel are also hyperparameters that need to be adjusted. For more information about the convolutional kernel, you can refer to this article. Here we use a few figures from this article to illustrate the working principle of the convolutional kernel:\nConvolution Calculation Normal Convolution Dilated Convolution Deconvolution MNIST Handwritten Digit Recognition # CNN Model Design # We can first determine the input and output. Obviously, the input of this neural network is a 28x28 grayscale image, and the output is a number between 0 and 9.\nWe need to choose a neural network type, such as a fully connected neural network, a convolutional neural network, a recurrent neural network, etc. Here we choose to use a simple convolutional neural network.\nWe need to determine the structure of the network, including the number of layers, the number of neurons in each layer, the activation function, etc. Here we choose a simple convolutional neural network, including two convolutional layers and two fully connected layers.\nThe code is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 import torch import torch.nn as nn import torch.nn.functional as F class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(1, 32, 3, 1) self.conv2 = nn.Conv2d(32, 64, 3, 1) self.dropout1 = nn.Dropout(0.25) self.dropout2 = nn.Dropout(0.5) self.fc1 = nn.Linear(9216, 128) self.fc2 = nn.Linear(128, 10) def forward(self, x): x = self.conv1(x) x = F.relu(x) x = self.conv2(x) x = F.relu(x) x = F.max_pool2d(x, 2) x = self.dropout1(x) x = torch.flatten(x, 1) x = self.fc1(x) x = F.relu(x) x = self.dropout2(x) x = self.fc2(x) output = F.log_softmax(x, dim=1) return output Here we define a class named Net that inherits from nn.Module. In the __init__ method, we define two convolutional layers conv1 and conv2, and two fully connected layers fc1 and fc2. The structure of the network is as follows:\nSince the MNIST dataset is a grayscale image, the input channel is 1. If it is an RGB color image, the input channel is 3. The output channel represents the number of convolution kernels, i.e., the number of features that each convolutional layer wants to extract. Here we assume that the first convolutional layer extracts 32 features, and the second convolutional layer extracts 64 features. The size of the convolution kernel is the size of the area that each convolution kernel can sense, i.e., the local receptive field. Here we assume that the size of the convolution kernel is 3x3. The stride is the distance the convolution kernel moves each time, here we assume the stride is 1. The 28x28 input passes through a 3x3 convolution kernel with a stride of 1, the output is 26x26; then through the second convolutional layer, the output is 24x24; then through the max pooling layer, the output is 12x12. Two dropout layers are used to prevent overfitting, without affecting the shape of the output parameters. Therefore, the number of input neurons in the first fully connected layer is 12x12x64=9216. The middle Dropout layer is used to prevent overfitting. Dropout is a regularization method that randomly sets some neuron outputs to 0, reducing the dependence between neurons. After two fully connected layers, the last output is 10 neurons, representing the numbers 0 to 9. Finally, the F.log_softmax() function is used to convert the output to probabilities. In the forward method, we define the forward propagation process of the network, i.e., the input data passes through each layer for calculation, and finally outputs the prediction result. Where: Both convolutional layers use the ReLU activation function, and the second convolutional layer is followed by a max pooling layer. Then perform a Dropout operation, flatten the output into a one-dimensional vector, and input it into two fully connected layers. After the first fully connected layer, another ReLU activation function is performed, and then another Dropout operation is performed. Finally, output 10 neurons, representing the numbers 0 to 9. Data Loading, Preprocessing, Training, and Testing # This part is actually the same as the simple fully connected neural network introduced in the previous article, except that the SimpleNet class defined in the previous article needs to be replaced with the Net class defined here.\nThe code here is actually the example code provided by PyTorch, which can be found here, or in the T03_mnist_cnn folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial.\nHere we create a Python script to run this CNN model and plot the performance curve of the model.\nThe Python environment required for the code in this article is the same as the previous two articles. You can activate the environment with conda activate pytorch-mnist and then run the code with the following command:\n1 python plot_performance.py Running Results # I ran this CNN model using both CPU (Intel i5-9600K) and GPU (NVIDIA GeForce RTX 4060 Ti), and the running time was 12 minutes and 56 seconds and 2 minutes and 25 seconds, respectively. It can be seen that the running speed is much faster using the GPU. The GPU resources required for running are actually not high, with a GPU utilization rate of about 12% and a memory utilization rate of about 740MB.\nThe performance of the model is as follows:\nIt can be seen that the accuracy of this CNN model on the MNIST dataset is about 99.2%, which is higher than the simple fully connected neural network.\nUsing the Model for Digit Recognition # This article is already the third article on MNIST handwritten digit recognition, but we have only been training the model and have not actually used the trained model. Now we will use the trained CNN model to recognize some handwritten digits.\nFirst, when training the model, we need to save the model parameters so that we can load these parameters directly when using the model, without having to retrain the model. The code to save the model parameters is as follows:\n1 torch.save(model.state_dict(), \u0026#34;mnist_cnn.pt\u0026#34;) The code in mnist_cnn.py already has this functionality. We need to specify the --save-model parameter when running mnist_cnn.py, so that the model parameters will be saved to the mnist_cnn.pt file:\n1 python mnist_cnn.py --save-model Then we can use the saved model parameters to recognize handwritten digits. We can use the PIL library to read images, use the torchvision library to preprocess images, and use our trained CNN model to recognize the digits in the images.\nIt should be noted that:\nThe images in the MNIST training set are black background white text 28x28 pixel images, so we need to keep the images we create black background white text. The digits we write ourselves are not necessarily 28x28 pixels, so we need to scale the images to 28x28 pixels. The images in the MNIST training set satisfy a normal distribution with a mean of 0.1307 and a standard deviation of 0.3081, so we need to normalize the images during preprocessing. The code for these operations is as follows:\n1 2 3 4 5 6 7 8 from torchvision import transforms transform = transforms.Compose([ transforms.Grayscale(num_output_channels=1), transforms.Resize((28, 28)), transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)) ]) The complete code is in the classify_image file.\nHere I used the mouse to write a few digits on the computer, and saved each digit separately in the T03_mnist_cnn folder in the numbers directory:\nFinally, we can use classify_image.py to recognize a handwritten digit:\n1 python classify_image.py numbers/number1.png I recognized these 10 handwritten digits and found that the model correctly recognized 8 digits. The two digits that were not recognized were the digits 2 and 9, which the model recognized as 4 and 8, respectively. Although some of the results were puzzling, the model\u0026rsquo;s performance was still acceptable.\nSummary # In this article, we used a convolutional neural network to implement MNIST handwritten digit recognition. Compared to the simple fully connected neural network, the CNN model achieved a significant improvement in accuracy. We also used the trained CNN model to recognize some handwritten digits, and the model\u0026rsquo;s performance was quite good.\nMNIST is a classic dataset in the field of computer vision and a good introductory dataset. We have written three articles to discuss it, so let\u0026rsquo;s take a break here and study some other machine learning or deep learning problems first. If we have the opportunity in the future, we will use the MNIST dataset to study some other deep learning algorithms.\n","date":"23 January 2024","externalUrl":null,"permalink":"/en/p/learn-pytorch-by-examples-3-mnist-handwritten-digit-recognition-iii--convolutional-neural-networks/","section":"Posts","summary":" Background # This is the third article in the “Learn PyTorch by Examples” series. In the previous two articles “Learn PyTorch by Examples (1): MNIST Handwritten Digit Recognition (I) – PyTorch Basics and Neural Network Basics” and “Learn PyTorch by Examples (2): MNIST Handwritten Digit Recognition (II) – Parameter Selection in Neural Network”, we introduced the basic concepts and usage of PyTorch, and implemented MNIST handwritten digit recognition using a simple three-layer fully connected neural network. In this article, we will use a convolutional neural network to implement MNIST handwritten digit recognition, which is an extension of the previous two articles.\n","title":"Learn PyTorch by Examples (3): MNIST Handwritten Digit Recognition (III) -- Convolutional Neural Networks","type":"post"},{"content":"","date":"23 January 2024","externalUrl":null,"permalink":"/en/tags/mnist/","section":"Tags","summary":"","title":"MNIST","type":"tags"},{"content":"","date":"23 January 2024","externalUrl":null,"permalink":"/en/tags/neural-networks/","section":"Tags","summary":"","title":"Neural Networks","type":"tags"},{"content":"","date":"2024年1月23日","externalUrl":null,"permalink":"/tags/%E5%8D%B7%E7%A7%AF%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C/","section":"Tags","summary":"","title":"卷积神经网络","type":"tags"},{"content":"","date":"2024年1月23日","externalUrl":null,"permalink":"/tags/%E6%89%8B%E5%86%99%E6%95%B0%E5%AD%97%E8%AF%86%E5%88%AB/","section":"Tags","summary":"","title":"手写数字识别","type":"tags"},{"content":"","date":"20 January 2024","externalUrl":null,"permalink":"/en/tags/hyperparameter/","section":"Tags","summary":"","title":"Hyperparameter\"","type":"tags"},{"content":" Background # This is the second article in the \u0026ldquo;Learn PyTorch by Examples\u0026rdquo; series. In the previous article \u0026ldquo;Learn PyTorch by Examples (1): PyTorch Basics and MNIST Handwritten Digit Recognition (1)\u0026rdquo;, we introduced the basic concepts and usage of PyTorch, and implemented MNIST handwritten digit recognition using a simple three-layer fully connected neural network, which is the \u0026ldquo;Hello World\u0026rdquo; in the field of deep learning. In this article, we will discuss the selection of parameters in this simple three-layer neural network and compare the impact of different parameter choices on the model performance.\nThe code for this article can be found in the T02_mnist_cnn folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial. The code is based on the official PyTorch example code https://github.com/pytorch/examples.\nParameter Selection # Based on the three-layer fully connected neural network in the previous article, we can adjust the parameters to observe the change in model performance.\nThe parameters in machine learning models can be divided into two categories: hyperparameters and model structure. Hyperparameters are parameters set before training the model, such as learning rate, number of iterations, batch size, etc. Model structure refers to the network structure of the model, such as the number of layers in the network, the number of neurons in each layer, activation functions, loss functions, regularization methods, etc.\nModel Structure # In the previous article, we used a simple three-layer fully connected neural network. We can try to increase or decrease the number of layers in the network, the number of neurons in each layer, activation functions, etc. to observe the change in model performance. We can also try different loss functions and regularization methods.\nNumber of Layers and Number of Neurons in Each Layer\nIn machine learning, the number of layers and the number of neurons in each layer of a neural network are very important hyperparameters. Increasing the number of layers and the number of neurons in each layer can increase the expressive power of the model, but it will also increase the complexity of the model, which may lead to overfitting. Conversely, reducing the number of layers and the number of neurons in each layer can reduce the complexity of the model, but it may also lead to a decrease in the accuracy of the model. Therefore, we need to strike a balance between the two.\nHere we try several different network structures, such as using 1 layer, 3 layers, 5 layers of hidden layers, and different values for the number of neurons in each layer, such as 64, 128, 256, etc.\nActivation Functions\nThe activation function is a very important concept in neural networks, and it is also the key to the ability of neural networks to fit various models. We can understand the activation function in this way: no matter what model, its essence is to make some judgments, to judge what the output should be based on different inputs, this judgment may be a single judgment, or a comprehensive judgment of many judgments. The activation function introduces a non-linear factor into the neuron, providing the neuron with a judgment ability.\nCommon activation functions include ReLU, Sigmoid, Tanh, etc. It can be seen that different activation functions are actually very different, but they all introduce a non-linear factor. Here we also try some different activation functions.\nLoss Functions\nThe loss function is used to measure the difference between the predicted value of the model and the true value, that is, the evaluation index of the model. The loss function is very important because it determines the optimization direction of the model.\nCommon loss functions include cross-entropy loss function, mean square error loss function, etc. Here we also try some different loss functions. It should be noted that the mean square error loss function is generally used for regression problems, and the cross-entropy loss function is generally used for classification problems. But we can still try to use the mean square error loss function. Since the target value in the mean square error loss function parameters (target) needs to input a one-hot encoded vector, and the labels in the original MNIST dataset are integers, we need to encode the labels into one-hot encoding. The code corresponding to this article makes a judgment on the loss function. If the loss function is the mean square error loss function, the labels are encoded into one-hot encoding.\n1 2 if loss_function == F.mse_loss: target = F.one_hot(target, num_classes=10).float() The so-called one-hot encoding is to convert an integer into a vector, the length of the vector is equal to the number of categories, with only one element being 1 and the rest being 0. For example, for the MNIST dataset, there are 10 categories in total, we can convert label 0 to [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], label 1 to [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], and so on.\nRegularization Methods\nRegularization is mainly used to prevent overfitting. Why can regularization prevent overfitting?\nFirst, let\u0026rsquo;s analyze the reason for overfitting. Overfitting means that the model performs well on the training set, but poorly on the test set. The reason for overfitting is that the model learns the noise of the training set on the training set, which leads to poor generalization ability of the model on the test set. Regularization is to add the model parameters to the loss function, so that the model parameters are as small as possible, thereby reducing the complexity of the model and preventing the model from learning noise on the training set.\nCommon regularization methods include L1 regularization, L2 regularization, Dropout, etc. Here we also try some different regularization methods.\nHyperparameters # Hyperparameters are parameters set before training the model, such as learning rate, number of iterations, batch size, etc. These parameters have a great impact on the performance of the model, so we need to carefully select these parameters.\nLearning Rate\nThe learning rate is the step size at which the model updates the parameters, which determines the speed at which the model parameters are updated. A learning rate that is too small may slow down the convergence of the model, while a learning rate that is too large may prevent the model from converging.\nEpochs\nThe number of epochs refers to the number of iterations the model iterates on the training set. Too few epochs may lead to underfitting, while too many epochs may lead to overfitting.\nBatch Size\nWhen training the model, we usually divide the training set into several batches, each batch containing several samples. This can reduce memory usage and speed up model training. A batch size that is too small may slow down the convergence of the model, while a batch size that is too large may lead to overfitting.\nModify the Code to Make Parameters Customizable # In the previous article, the model structure was fixed, and hyperparameters could be specified through command-line parameters. Now we modify the code, extract the main function, and pass all these parameters as parameters to the main function. In this way, we can write another script to call the main function, pass in different parameters, and achieve different model structures and hyperparameter choices.\nHere we won\u0026rsquo;t post the complete code, just give the prototype of the modified main function:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 def main(batch_size = 64, epochs = 14, lr = 1.0, test_batch_size = 1000, no_cuda = False, no_mps = False, dry_run = False, seed = 1, log_interval = 10, save_model = False, hidden_layers = [128], activation_functions = [F.relu], loss_function = F.nll_loss, regularizations = [None] ): The first three parameters are hyperparameters, the next four parameters are model structure parameters, and the middle few parameters are some auxiliary parameters.\nFinally, because we want to compare the model performance under different parameters, we let the main function return the loss and accuracy during training, so that we can draw a comparison of the model performance under different parameters in the script calling the main function. For this purpose, we define a class to store the training performance:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 def main(batch_size = 64, epochs = 14, lr = 1.0, test_batch_size = 1000, no_cuda = False, no_mps = False, dry_run = False, seed = 1, log_interval = 10, save_model = False, hidden_layers = [128], activation_functions = [F.relu], loss_function = F.nll_loss, regularizations = [None] ): In the training process, we store the loss and accuracy of the training and test sets in this class, and finally return them from the main function. The specific code can be found in the code corresponding to this article in the GitHub repository.\nPerformance Comparison # We focus on three hyperparameters and four model parameters here, a total of seven parameters, and each parameter has multiple choices. Assuming we only choose three values for each parameter, then there are a total of $3^7=2187$ combinations, which is not a small number!\nDue to the variety of parameter combinations, finding an optimal parameter combination is like finding a recipe through trial and error, so many people metaphorically call machine learning \u0026ldquo;alchemy\u0026rdquo;. Indeed, this is very similar to the work of ancient alchemists.\nIn practical applications, we can use some heuristic methods to select parameters, such as grid search, random search, Bayesian optimization, etc. These methods can help us find an optimal parameter combination more quickly.\nHere we use the method of controlling variables, taking the model structure and hyperparameters in the previous article as the baseline, and adjusting only one parameter at a time to observe the change in model performance. This can help us better understand the impact of each parameter on the model performance. The specific code can be found in the T02_mnist_parameters folder in my GitHub repository https://github.com/jin-li/pytorch-tutorial, in the parametric_study.py file.\nAfter each set of parameters is trained, we draw the training and test loss and accuracy:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 fig, axs = plt.subplots(2, 1, figsize=(10, 10)) fig.suptitle(f\u0026#39;Parameter Study: {param_names[idx]}\u0026#39;) axs[0].plot(metrics_ref.test_count, metrics_ref.test_loss, label=ref_labels[idx]) for i, metric in enumerate(metrics): axs[0].plot(metric.test_count, metric.test_loss, label=param_labels[idx][i]) axs[0].legend() axs[0].set_title(\u0026#39;Training Loss\u0026#39;) axs[0].set_xlabel(\u0026#39;data count\u0026#39;) axs[0].set_ylabel(\u0026#39;loss\u0026#39;) axs[1].plot(metrics_ref.test_count, metrics_ref.test_accuracy, label=ref_labels[idx]) for i, metric in enumerate(metrics): axs[1].plot(metric.test_count, metric.test_accuracy, label=param_labels[idx][i]) axs[1].legend() axs[1].set_title(\u0026#39;Test Accuracy\u0026#39;) axs[1].set_xlabel(\u0026#39;data count\u0026#39;) axs[1].set_ylabel(\u0026#39;accuracy\u0026#39;) plt.savefig(f\u0026#39;test{idx}_{param_names[idx]}.png\u0026#39;) Here you need the same Python virtual environment as in the previous article, which can be activated by conda activate pytorch-mnist. Then you can use the following command to run all the tests:\n1 python parametric_study.py Model Structure Parameters # Number of Hidden Layers\nHere I tried 1, 3, and 5 hidden layers. The number of neurons in the middle layers varies between 64, 128, and 256. The following figure shows the test results:\nIt can be seen that the model with 1 hidden layer has performed very well, and increasing the number of hidden layers does not significantly improve the model performance. Of course, this may also be related to other parameters in the model. Perhaps increasing the number of hidden layers also requires adjusting other parameters of the neural network to make the hidden layers work. But from this experimental result, 1 hidden layer is enough for this problem.\nActivation Functions\nHere I tried ReLU, Sigmoid, Tanh, and other activation functions. The following figure shows the test results:\nIt can be seen that the performance of these three activation functions is similar, but ReLU performs relatively better, and the model performance of Sigmoid and Tanh is relatively worse. This is also in line with our expectations, because the ReLU activation function is the most commonly used activation function at present. Its advantages are simple calculation, fast convergence, and not easy to encounter the problem of gradient disappearance.\nLoss Functions\nIn addition to the negative log-likelihood loss function nll_loss() used in the model, I also tried the cross-entropy loss function cross_entropy() and the mean square error loss function mse_loss(). The following figure shows the test results:\nIt can be seen that the cross-entropy loss function and the negative log-likelihood loss function perform similarly, with good performance, and the mean square error loss function has the worst model performance. This is also in line with our expectations, because the mean square error loss function is generally used for regression problems, the cross-entropy loss function is generally used for classification problems.\nRegularization Methods\nRegularization methods are special. In the code corresponding to this article, using regularization requires too many modifications to the code, so I did not test the effect of regularization here, and I will leave it for future research.\nHyperparameters # Batch Size\nHere I tried three batch sizes: 16, 64, and 256. The following figure shows the test results:\nIt can be seen that the model performs best with batch sizes of 64 and 256, and worst with a batch size of 16. I also recorded the training time, the training time is longest with a batch size of 16, and relatively shorter with batch sizes of 64 and 256:\n1 2 3 batch size 16: 164.6s batch size 64: 133.9s batch size 256: 125.0s In the case of sufficient memory and computing resources, we can choose a larger batch size, which can speed up the training of the model.\nNumber of Epochs\nThe number of epochs refers to the number of iterations the model iterates on the training set. I tried up to 25 epochs. The following figure shows the test results:\nIt can be seen that as the number of epochs increases, the loss value gradually decreases, and the accuracy gradually increases. However, when the number of epochs exceeds a certain value, the model performance no longer improves, and may even overfit. Therefore, we need to choose the appropriate number of epochs based on the performance of the model. For this problem, 14 epochs are enough.\nLearning Rate\nThe learning rate is a very important hyperparameter. I tried three learning rates: 0.1, 1.0, and 10.0. The following figure shows the test results:\nIt can be seen that the model performs best with a learning rate of 1.0, and worst with a learning rate of 10.0. This is also in line with our expectations, because a learning rate that is too small may slow down the convergence of the model, while a learning rate that is too large may prevent the model from converging.\nComputing Resources # Above experiments were run on my personal computer, using an NVIDIA GeForce GTX 4060 Ti graphics card. Each example uses almost the same amount of computing resources, basically not consuming much graphics card resources, the graphics card utilization rate is about 5%, the graphics memory occupies about 180 MB, and the running time is about 2 minutes and 15 seconds. If using the CPU, the running time is about 3 minutes.\nSummary # In this article, we discussed the parameter selection problem in neural networks, including model structure and hyperparameters. We modified the code so that the model structure and hyperparameters can be specified through command-line parameters, and then adjusted each parameter one by one through the method of controlling variables to observe the change in model performance.\nDue to limited computing resources, we only tested a part of the parameter combinations, but this is enough to illustrate the impact of parameter selection on model performance. In practical applications, we can use some heuristic methods to select parameters, such as grid search, random search, Bayesian optimization, etc. These methods can help us find an optimal parameter combination more quickly. We will continue to discuss these methods in subsequent articles.\nInterested readers can use the code corresponding to this article in the GitHub repository https://github.com/jin-li/pytorch-tutorial (in the T02_mnist_cnn folder) to try different parameter combinations and observe the change in model performance.\n","date":"20 January 2024","externalUrl":null,"permalink":"/en/p/learn-pytorch-by-examples-2-mnist-handwritten-digit-recognition-ii--parameter-selection-in-neural-network/","section":"Posts","summary":" Background # This is the second article in the “Learn PyTorch by Examples” series. In the previous article “Learn PyTorch by Examples (1): PyTorch Basics and MNIST Handwritten Digit Recognition (1)”, we introduced the basic concepts and usage of PyTorch, and implemented MNIST handwritten digit recognition using a simple three-layer fully connected neural network, which is the “Hello World” in the field of deep learning. In this article, we will discuss the selection of parameters in this simple three-layer neural network and compare the impact of different parameter choices on the model performance.\n","title":"Learn PyTorch by Examples (2): MNIST Handwritten Digit Recognition (II) -- Parameter Selection in Neural Network","type":"post"},{"content":"","date":"20 January 2024","externalUrl":null,"permalink":"/en/tags/parameter-selection/","section":"Tags","summary":"","title":"Parameter Selection","type":"tags"},{"content":"","date":"2024年1月20日","externalUrl":null,"permalink":"/tags/%E5%8F%82%E6%95%B0%E9%80%89%E6%8B%A9/","section":"Tags","summary":"","title":"参数选择","type":"tags"},{"content":"","date":"2024年1月20日","externalUrl":null,"permalink":"/tags/%E8%B6%85%E5%8F%82%E6%95%B0/","section":"Tags","summary":"","title":"超参数","type":"tags"},{"content":" Background # I have almost all my websites deployed using Docker. When I talked to my friends about how I built these websites, I found that many people said they had heard of Docker or containers, but didn\u0026rsquo;t know what they were.\nIn fact, when I first started using Docker, I came into contact with many container-related concepts, but because I had not practiced them, I was also at a loss about these concepts. Now that I have been using container technology for a long time, I want to summarize the related knowledge and hope to help others.\nThis is the first article about Docker containers. The links to other articles in this series are as follows:\nContainer (2): The Best Practice Guide for Docker —— docker-compose and Portainer Container (3): Docker Best Practices Guide - Volume Management Container (4): Docker Best Practices Guide - Container Update, Upgrade, and Migration Container (5): Docker Best Practices Guide - Container Update Monitoring Tool WUD (What\u0026rsquo;s Up Docker) Container (6): Misconceptions, Bad Habits, and Issues When Using Docker Containerization # What is containerization # Containerization is a virtualization technology that can package applications and their dependencies together so that they can run in different environments. Containerized applications can run anywhere, including developers\u0026rsquo; laptops, physical servers, virtual machines, container clusters, and public clouds.\nWhy containerization is needed # Looking directly at the concept of containerization, it may still be difficult to understand. We use an example to illustrate why containerization is needed, so that the concept of containerization will be very clear.\nFirst, if you are just a user of a program or service and don\u0026rsquo;t care how the program or service runs on the machine, you don\u0026rsquo;t need to understand the concept of containerization at all.\nSecond, if the program or service you are running is just a simple one, and you just need it to run on your computer, you don\u0026rsquo;t need to understand the concept of containerization either.\nSo, here we assume that you are a developer (of course, you don\u0026rsquo;t really have to develop a program, because for most beginners, they don\u0026rsquo;t develop any programs, they just use programs they get from open source websites or elsewhere), you have a program you want to run, this program can provide some services, and you want to be able to access these services from different places. For example, you have built a website using a popular website framework, and you want to be able to access this website from different devices in different places, and you also want others to be able to access your website. At this point, you deploy the website to your computer, assuming your network operator assigns you a static IP and opens a port, you can open your website on the public network, so you can access the website from anywhere that can connect to the public network.\nAfter a while, you bought a domain name and bound it to your website using a web server (such as Nginx or Apache), so you can access your website through the domain name.\nAfter a while, you found that you had deployed more and more services on your website, and your website was becoming more and more popular, and the number of visitors was increasing. Your computer\u0026rsquo;s weak performance was no longer able to handle such a large number of visitors, so you decided to buy a more powerful computer or server and move your website there.\nSo you assembled a server, installed a Linux system, and started migrating the website. You copied the website files from the original computer to the new computer, and then started running them, only to find that the website couldn\u0026rsquo;t run at all. Because your website has many dependencies, such as Python libraries and some miscellaneous libraries, you can only install these libraries one by one. Then you will find that many of the default versions of the libraries are newer than the versions you installed on the original computer, and the new libraries are simply not compatible with your old code, so you can only find old versions of the libraries on the Internet and install them one by one. If you\u0026rsquo;re lucky, you can install the old versions of the libraries smoothly; if you\u0026rsquo;re unlucky, you may find that your operating system also depends on some libraries, and your operating system crashes after installing the old versions.\nThis is not the end. Even if you successfully solve the dependency problem, you need to reconfigure the database, web server, etc. that you configured for your website on the original computer.\nAt this point, you may think, if you could just pack up the website, database, web server, and their dependencies and runtime environment on the original computer, and then run them directly on the new computer, that would be great. If you\u0026rsquo;ve used virtual machines before, you might think, isn\u0026rsquo;t this just creating a virtual machine that\u0026rsquo;s the same as the original computer and then migrating the virtual machine to the new computer?\nBut virtual machines are too heavy, because virtual machines contain a complete operating system, which generally takes up a lot of resources and starts very slowly. So we don\u0026rsquo;t want to create the entire operating system, we just want to create a runtime environment that is the same as the original program. This is actually containerization.\nHow to containerize # Now that our goal is clear, we want to package a program and its dependencies and runtime environment, and then run this packaged program in different places. So, simply put, containerization consists of two steps: 1. Packaging the program and its dependencies and runtime environment; 2. Isolating an independent runtime environment on the machine and running the packaged program.\nPackaging # This step is relatively easy, just package the program and its dependencies and runtime environment. The packaged program and its dependencies and runtime environment are called images.\nIsolating the environment # The technology for isolating the environment depends on the characteristics of the operating system. In the Linux system, the isolation of the environment depends on three technologies: namespace, control group, and chroot:\nNamespace: Namespaces can isolate the view of processes, so that processes can only see their own view and not the view of other processes. Control Group: Control groups can limit the resource usage of processes, such as CPU, memory, disk IO, etc. chroot: chroot can change the root directory of a process. A detailed introduction to these three technologies is beyond the scope of this article, and interested readers can search for them on their own.\nWindows systems also have container technology, but it is different from Linux systems. In Docker Desktop for Windows, containerization technology depends on the Hyper-V virtual machine.\nContainers and virtual machines # From the above description, it can be seen that containers and virtual machines both provide the function of isolating the environment, and both can be called sandbox technology. But there are also significant differences between the two:\nContainers are implemented at the operating system level and provide process-level isolation, and must share the operating system kernel with the host machine. Virtual machines are implemented at the hardware level and provide operating system-level isolation, and can have their own operating system kernel. Because containers share the operating system kernel with the host machine, the startup speed of containers (seconds) is much faster than that of virtual machines (minutes), and the resource usage of containers is also much lower than that of virtual machines.\nDocker # The three technologies required for containerization appeared in 1979 (chroot), 2002 (namespace), and 2007 (cgroup), but the original intention of these three technologies was not containerization. It wasn\u0026rsquo;t until 2013 that Docker emerged and combined these three technologies, and containerization began to become popular.\nDocker is simple in composition and use, as shown in the figure below:\nDocker\u0026rsquo;s composition # The figure contains the three core concepts of Docker: image, container, and repository.\nImage: An image is a read-only template that contains everything needed to run a program, including code, runtime, libraries, environment variables, and configuration files. The image is the basis of the container. Repository: A repository is a place to store images, and can be understood as a collection of images. Repositories are divided into public and private repositories, public repositories are open, and anyone can upload and download images; private repositories are private, and only the owner can upload and download images. Container: A container is a running instance of an image, and it contains the image and everything it needs at runtime, including the file system, system environment, network configuration, etc. The container is the runtime state of the image. Docker usage (single container) # There are two ways to use Docker: using someone else\u0026rsquo;s image and making your own image.\nUsing someone else\u0026rsquo;s image # Using someone else\u0026rsquo;s image is very simple, just two steps:\nDownload the image: Use the docker pull command to download the image, for example:\n1 docker pull ubuntu:latest This command will download an image named ubuntu from Docker Hub, and the tag latest indicates that the downloaded image is the latest version.\nRun the container: Use the docker run command to run the container, for example:\n1 docker run -it --rm ubuntu:latest /bin/bash The above command will run a container named ubuntu, with the tag latest, and enter the bash terminal of the container.\nUsing your own image # Using your own image requires three steps:\nWrite a Dockerfile: The Dockerfile is a text file that contains a series of commands used to build the image. For example:\n1 2 3 4 5 FROM python:3.12 WORKDIR /app COPY . /app RUN pip install -r requirements.txt CMD [\u0026#34;python\u0026#34;, \u0026#34;app.py\u0026#34;] In the Dockerfile above, the FROM command indicates that the base image is python:3.12, the WORKDIR command indicates that the working directory is /app, the COPY command indicates that all files in the current directory are copied to the /app directory, the RUN command indicates that the dependencies in requirements.txt are installed, and the CMD command indicates the command to run when the container starts.\nBuild the image: Use the docker build command to build the image, for example:\n1 docker build -t myapp . The above command will build an image named myapp in the current directory.\nRun the container: Use the docker run command to run the container, for example:\n1 docker run -d -p 5000:5000 myapp The above command will run a container named myapp and map the container\u0026rsquo;s 5000 port to the host\u0026rsquo;s 5000 port.\nDocker usage (multiple containers) # In most cases, we need more than one container to deploy a service, for example, we need a container to run the cloud disk service, and another container to run the database service. And some complex services may require more containers.\nAt the same time, the technology for deploying and managing multiple containers is called container orchestration, and Docker provides a tool called docker-compose to achieve container orchestration.\ndocker-compose installation # When Docker client was first released, it did not have the ability to orchestrate containers, so developers developed the docker-compose tool to achieve container orchestration. This version of docker-compose is called the first version (v1) of docker-compose.\nLater, Docker released a new version of the Docker client, which has the ability to orchestrate containers, called the second version (v2) of docker-compose.\nIn Ubuntu 22, docker-compose can be installed using the apt package manager, and the apt search docker-compose command can be used to see three versions of docker-compose in the apt source:\n1 2 3 4 5 6 7 8 9 10 11 $ apt search docker-compose Sorting... Done Full Text Search... Done docker-compose/jammy,jammy,now 1.29.2-1 all [installed] define and run multi-container Docker applications with YAML docker-compose-plugin/jammy 2.24.5-1~ubuntu.22.04~jammy amd64 [upgradable from: 2.24.1-1~ubuntu.22.04~jammy] Docker Compose (V2) plugin for the Docker CLI. docker-compose-v2/jammy-updates 2.20.2+ds1-0ubuntu1~22.04.1 amd64 tool for running multi-container applications on Docker The first one is the first version, which we won\u0026rsquo;t consider. The second and third ones are both the second version.\nThe first one is the v1, we won\u0026rsquo;t consider it The second one is a plugin for docker-cli, and the command to use is docker compose The third one is a standalone tool, and the command to use is docker-compose We generally choose to install the third one, because many tutorials on the Internet use the docker-compose command, not the docker compose command.\ndocker-compose usage # docker-compose uses a file named docker-compose.yml to define the configuration of multiple containers. For example, when we deploy a personal cloud disk service using Nextcloud, the docker-compose.yml file we use is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 version: ‘3’ services: db: image: mariadb container_name: nextcloud-mariadb networks: - nextcloud_network volumes: - ./db:/var/lib/mysql - /etc/localtime:/etc/localtime:ro environment: - MYSQL_ROOT_PASSWORD=PASSWORD1 - MYSQL_PASSWORD=PASSSWORD2 - MYSQL_DATABASE=nextcloud - MYSQL_USER=nextcloud restart: unless-stopped app: image: nextcloud:latest container_name: nextcloud-app networks: - nextcloud_network ports: - 7080:80 depends_on: - db volumes: - ./nextcloud:/var/www/html - ./app/config:/var/www/html/config - ./app/custom_apps:/var/www/html/custom_apps - ./app/data:/var/www/html/data - ./app/themes:/var/www/html/themes - /etc/localtime:/etc/localtime:ro environment: - VIRTUAL_HOST=your.cloud.domain.name restart: unless-stopped networks: nextcloud_network: This file defines two services: db and app. The db service uses the mariadb image, and the app service uses the nextcloud image. Both the db service and the app service use the nextcloud_network network.\nWe can use the docker-compose command to start these two services:\n1 docker-compose up -d If you want to stop these two services, you can use the docker-compose command to stop them:\n1 docker-compose down Kubernetes / K8s # As mentioned earlier, the docker-compose tool was developed by Docker, and its functionality is relatively simple, suitable for small projects that are only used on a single computer or server. For large projects that include several computer clusters, docker-compose\u0026rsquo;s functionality is somewhat lacking, and this is where Kubernetes (K8s) comes in.\nKubernetes (K8s, because there are 8 letters between K and s) is an open-source container orchestration engine that can automate the deployment, scaling, and management of containerized applications, and its functionality is very powerful.\nIt should be noted that the containerization standard is defined by the Open Container Initiative (OCI), and Docker is just one implementation of OCI. Kubernetes is also an implementation of OCI, and it used to use Docker as the container runtime, but now it no longer depends on Docker, but uses containerd as the container runtime.\nI haven\u0026rsquo;t used Kubernetes yet, so I won\u0026rsquo;t go into detail here.\n","date":"18 January 2024","externalUrl":null,"permalink":"/en/p/container1-about-containerization--containerization-docker-docker-compose-kubernetes-/-k8s-etc./","section":"Posts","summary":" Background # I have almost all my websites deployed using Docker. When I talked to my friends about how I built these websites, I found that many people said they had heard of Docker or containers, but didn’t know what they were.\nIn fact, when I first started using Docker, I came into contact with many container-related concepts, but because I had not practiced them, I was also at a loss about these concepts. Now that I have been using container technology for a long time, I want to summarize the related knowledge and hope to help others.\n","title":"Container(1): About Containerization -- Containerization, Docker, Docker-Compose, Kubernetes / K8s, etc.","type":"post"},{"content":"","date":"17 January 2024","externalUrl":null,"permalink":"/en/tags/aws/","section":"Tags","summary":"","title":"AWS","type":"tags"},{"content":"","date":"17 January 2024","externalUrl":null,"permalink":"/en/tags/flask/","section":"Tags","summary":"","title":"Flask","type":"tags"},{"content":" Introduction # In the previous four articles of this series, we developed a simple weather forecast website using the Flask framework of Python, and deployed this website to a private server using Docker, so that our website can be accessed on the public network.\nHowever, not everyone has their own server. To make the deployment of the website easier, we can use cloud servers provided by cloud service providers such as AWS or Google Cloud to deploy our website.\nPrerequisites # Know how to develop a simple web application using the Flask framework (see \u0026ldquo;Python Web Development Learning (1): Using Flask Framework\u0026rdquo;) Have installed PostgreSQL database and know how to use SQLAlchemy to operate the database (see \u0026ldquo;Python Web Development Learning (2): Using PostgreSQL and SQLAlchemy\u0026rdquo; Have an AWS account and know how to use their cloud servers Have a domain name Extensions # If you want to know how to get user input through input boxes and how to use APIs to get information on the Internet, please refer to \u0026ldquo;Python Web Development Learning (3): Using Input Boxes and APIs\u0026rdquo; If you want to know how to deploy a website using Docker, please refer to \u0026ldquo;Python Web Development Learning (4): Deploying Website Using Docker, Gunicorn, and Nginx\u0026rdquo; Deploying to AWS # In general, deploying to AWS involves the following steps:\nCreate an AWS account and get the Access Key and Secret Key Configure the AWS command line tool Create a security group Create a database Create a container image for the app Create an App Runner Create an AWS account and get the Access Key and Secret Key # First, we need to register an account on the AWS website and then create a new IAM user in the console. You can search for \u0026ldquo;IAM\u0026rdquo; in the search box at the top to find the IAM service.\nIn the IAM service, select \u0026ldquo;Users\u0026rdquo; and then click \u0026ldquo;Add user\u0026rdquo;.\nThe username can be any, no need to check \u0026ldquo;Enable console access\u0026rdquo; Check the \u0026ldquo;AdministratorAccess\u0026rdquo; permission After checking, select create After creating, go to the user details page, select the \u0026ldquo;Security credentials\u0026rdquo; tab, and click \u0026ldquo;Create access key\u0026rdquo;.\nCheck the \u0026ldquo;Command Line Interface\u0026rdquo; option when creating Click \u0026ldquo;Next\u0026rdquo; to create Record the Access Key and Secret Key in a secure place. These two keys are displayed only once. If you forget the Secret Key, you can only create a new Access Key. Configure the AWS command line tool # To facilitate the use of AWS services and avoid the tedious operations on the web, we can use the AWS command line tool provided by AWS to manage our cloud servers.\nInstall the AWS command line tool\nYou can refer to the AWS official documentation to install the AWS command line tool. Be sure to choose the installation method that suits your operating system.\nAfter installation, you can check if it is installed successfully by entering aws --version in the command line.\nConfigure the AWS command line tool\nEnter aws configure in the command line, and then enter the Access Key and Secret Key you just created, as well as the default region and output format.\n1 2 3 4 5 $ aws configure AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: us-west-2 Default output format [None]: json After configuration, you can enter aws configure list in the command line to view the configuration information.\n1 aws configure list Create a security group # AWS has high requirements for the security of cloud services. We need to create a security group to control the rules for accessing our cloud server. Here, the main purpose is to allow access to the MySQL database we are going to create through port 3306 from the outside.\nSearch for \u0026ldquo;security group\u0026rdquo; in the AWS console to enter the security group page.\nClick \u0026ldquo;Create security group\u0026rdquo;.\nSecurity group name: can be any, such as \u0026ldquo;flask-weather\u0026rdquo; Description: can be left blank VPC: select the default VPC Add rule: add a rule, allow TCP protocol, port 3306, source \u0026ldquo;Anywhere\u0026rdquo; Finally, click \u0026ldquo;Create security group\u0026rdquo; Create a database # Next, we need to create a database on AWS to store the data of our website. The database hosting service provided by AWS is called RDS.\nSearch for \u0026ldquo;RDS\u0026rdquo; in the AWS console to enter the RDS service page.\nClick \u0026ldquo;Create database\u0026rdquo;.\nSelect the database creation method: select \u0026ldquo;Standard create\u0026rdquo; Select the database engine: select MySQL Select the database instance size: select \u0026ldquo;Free tier\u0026rdquo; Set the database instance identifier, master username, and password Set the database instance category, storage, VPC, subnet group, security group, etc., and select the security group created earlier Finally, click \u0026ldquo;Create database\u0026rdquo; It takes some time to create the database. After the database is created, you can find the database endpoint on the database details page, and then you can use some database connection tools to connect to the database.\nThe database connection tool I often use is the VS Code plugin MySQL, which can connect to the database directly in VS Code, which is very convenient.\nImage source: database-client.com\nIf you use JetBrains IDE (such as IntelliJ IDEA), you can also use the database tool built into the IDE to connect to the database.\nCreate a container image for the app # Modern web applications are generally deployed using container technology. We can use Docker to create a container image and then deploy this image to AWS\u0026rsquo;s App Runner.\nFirst, we need to create an ECR (Elastic Container Registry) on AWS to store our container image.\nSearch for \u0026ldquo;ECR\u0026rdquo; in the AWS console to enter the ECR service page.\nClick \u0026ldquo;Create repository\u0026rdquo;.\nRepository name: can be any, such as \u0026ldquo;flask-weather\u0026rdquo; Tags: can be left blank Encryption: can be left blank Finally, click \u0026ldquo;Create repository\u0026rdquo; After creating, record the URI of the repository, which will be used later.\nCreate a Dockerfile in the project directory to build the container image.\n1 2 3 4 5 6 7 8 9 10 FROM python:3.9-slim WORKDIR /app COPY requirements.txt requirements.txt RUN pip install -r requirements.txt COPY . . CMD [\u0026#34;gunicorn\u0026#34;, \u0026#34;-b\u0026#34;, \u0026#34;0.0.0.0:5001\u0026#34;, \u0026#34;app:app\u0026#34;] The content of this Dockerfile is basically the same as the Dockerfile used to deploy the website using Docker, except that we use the slim version of Python 3.9 as the base image here.\nCreate a requirements.txt file in the project directory to record the project\u0026rsquo;s dependencies.\n1 2 3 4 5 Flask==2.0.1 Flask-SQLAlchemy==2.5.1 gunicorn==20.1.0 psycopg2-binary==2.9.1 requests==2.26.0 The content of this file is basically the same as the requirements.txt file used to deploy the website using Docker, except that we use Flask 2.0.1 here.\nCreate a .dockerignore file in the project directory to ignore some unnecessary files.\n1 2 3 4 5 6 7 8 9 10 11 __pycache__ *.pyc *.pyo *.pyd .DS_Store .env .venv .git .gitignore .dockerignore .vscode Build the container image\nBuild the container image in the project directory with the following command.\n1 docker build -t flask-weather . This command will build a container image named flask-weather locally.\nPush the container image to ECR\nFirst, we need to log in to ECR.\n1 aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-west-2.amazonaws.com This command will get the login password of ECR and then log in to ECR using Docker.\nNext, tag the container image.\n1 docker tag flask-weather:latest 123456789012.dkr.ecr.us-west-2.amazonaws.com/flask-weather:latest This command will tag the local flask-weather container image and then push it to ECR.\n1 docker push 123456789012.dkr.ecr.us-west-2.amazonaws.com/flask-weather:latest This command will push the local flask-weather container image to ECR.\nAfter the push is complete, you can see the pushed container image in the repository of ECR.\nCreate an App Runner # We use the App Runner provided by AWS to deploy our website.\nSearch for \u0026ldquo;App Runner\u0026rdquo; in the AWS console to enter the App Runner service page.\nClick \u0026ldquo;Create service\u0026rdquo;.\nSelect the deployment method: select \u0026ldquo;Container\u0026rdquo; Select the container image: select the container image pushed to ECR just now Select the port: select 5001 Click \u0026ldquo;Next\u0026rdquo; Configure the service\nService name: can be any, such as \u0026ldquo;flask-weather\u0026rdquo; Environment variables: add the database connection information, such as DATABASE_URL, DATABASE_USER, DATABASE_PASSWORD, etc. Click \u0026ldquo;Next\u0026rdquo; Configure the network\nVPC: select the default VPC Subnet: select the default subnet Security group: select the security group created earlier Click \u0026ldquo;Next\u0026rdquo; Configure the domain name\nDomain name: can be any, such as \u0026ldquo;flask-weather\u0026rdquo; Click \u0026ldquo;Next\u0026rdquo; Deploy the service\nClick \u0026ldquo;Deploy service\u0026rdquo; After deployment, you can see the service just created in the App Runner service page.\nClick the service name to see the details of the service, including the domain name of the service.\nClick the domain name of the service to access our website in the browser.\nCustomizing the domain name # After completing the deployment above, we can access our website through the domain name provided by AWS App Runner. However, the domain name provided by AWS is generally long and contains a random string, which is not easy to remember. We can bind the website deployed on App Runner to our own domain name.\nDirectly binding our own domain name to App Runner will result in a \u0026ldquo;Create Failed\u0026rdquo; error. The reason is that App Runner needs to issue a certificate for our domain name to use HTTPS, but our domain name does not list Amazon as a trusted certificate authority. We need to add a CAA record to the domain\u0026rsquo;s DNS server so that App Runner can issue a certificate for our domain name.\nCertificate Authority Authorization (CAA) record\nAdd a CAA record to the domain\u0026rsquo;s DNS server, with the name @ and the value amazon.com. This means that Amazon can issue certificates for any subdomain under our domain name.\nBind the domain name on App Runner\nIn the App Runner service page, click the service name, then click the \u0026ldquo;Domain\u0026rdquo; tab, and click \u0026ldquo;Bind domain\u0026rdquo;.\nDomain name: enter our own domain name Click \u0026ldquo;Bind domain\u0026rdquo; After binding, you can access our website in the browser.\n","date":"17 January 2024","externalUrl":null,"permalink":"/en/p/python-web-development-learning-5-deploying-website-on-aws/","section":"Posts","summary":" Introduction # In the previous four articles of this series, we developed a simple weather forecast website using the Flask framework of Python, and deployed this website to a private server using Docker, so that our website can be accessed on the public network.\nHowever, not everyone has their own server. To make the deployment of the website easier, we can use cloud servers provided by cloud service providers such as AWS or Google Cloud to deploy our website.\n","title":"Python Web Development Learning (5): Deploying Website on AWS","type":"post"},{"content":"","date":"17 January 2024","externalUrl":null,"permalink":"/en/series/python-web-development-series/","section":"Series","summary":"","title":"Python Web Development Series","type":"series"},{"content":"","date":"2024年1月17日","externalUrl":null,"permalink":"/series/python-web%E5%BC%80%E5%8F%91%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"Python Web开发系列","type":"series"},{"content":" Motivation # I used TensorFlow when I took a machine learning course many years ago, but my understanding of deep learning was not deep enough at that time, so I only used it briefly. Later, I did not take any machine learning courses, but I still used some machine learning knowledge in my research work, so I learned some machine learning knowledge intermittently.\nNow deep learning has become the mainstream method of machine learning, and PyTorch is a very popular deep learning framework. I have also learned about PyTorch before, but I have not learned it systematically, so I recently decided to learn PyTorch systematically.\nAlthough there are already many tutorials on PyTorch online, I still want to start a new series \u0026ldquo;Learn PyTorch by Examples\u0026rdquo;, mainly to deepen my understanding. The goal of this series is to start with the basics of PyTorch and use the examples provided by the PyTorch official to learn how to implement some classic machine learning models using PyTorch.\nThe PyTorch official provides many examples, including MNIST handwritten digit recognition, CIFAR-10 image classification, IMDB sentiment analysis, etc., and the GitHub repository address is https://github.com/pytorch/examples. These examples are the Hello World of deep learning and are very suitable for beginners to learn. However, the PyTorch official only provides code without explanation, which may not be friendly enough for beginners. So I plan to use this series to explain these examples to help beginners learn PyTorch better.\nHere I have created a new GitHub repository, the main body of which is the PyTorch official examples, but I will add some auxiliary code and documents to each example to facilitate beginners\u0026rsquo; learning. The address of this repository is https://github.com/jin-li/pytorch-tutorial. Welcome to Star and Fork.\nPyTorch Basics # Introduction to PyTorch # PyTorch is an open-source deep learning framework developed by Facebook\u0026rsquo;s artificial intelligence research team. PyTorch provides two main functions:\nA multi-dimensional tensor library, similar to NumPy, but can run on GPUs. An automatic differentiation engine for building and training neural networks. Advantages of PyTorch:\nPyTorch is a dynamic graph framework that allows for more flexible definition of neural networks. PyTorch\u0026rsquo;s API is more Pythonic and easier to learn and use. PyTorch\u0026rsquo;s community is more active, with more tutorials and examples. PyTorch Runtime Environment # PyTorch supports multiple operating systems, including Linux, Windows, and macOS. PyTorch supports multiple hardware devices, including CPUs, GPUs, and TPUs. PyTorch supports multiple programming languages, including Python, C++, and Java.\nHere my runtime environment is a desktop computer with Ubuntu 22.04, equipped with an Intel Core i5-9600K processor and an NVIDIA GeForce GTX 4060 Ti graphics card. The CPU memory is 64GB, and the GPU memory is 8GB. However, in this series of tutorials, I will try to use both the CPU and the GPU, both for performance comparison and to allow readers to run on different hardware devices.\nPyTorch Installation # PyTorch installation is very simple, just use the pip command. But if you install directly with pip, you may mess up your Python environment, so we will use conda to create a virtual environment each time, and then run the PyTorch instance in the virtual environment.\nFor Python environment management, I wrote an article \u0026ldquo;Python Environment Management Methods Summary\u0026rdquo;, interested readers can refer to it.\nMNIST Handwritten Digit Recognition # MNIST is a very classic handwritten digit recognition dataset, which contains 60,000 training images and 10,000 test images. Each image is a 28x28 pixel grayscale image, and the label is a number between 0 and 9. MNIST has become the Hello World of deep learning, especially in the field of computer vision (CV), almost all deep learning frameworks have MNIST examples. Here we start with MNIST to start our PyTorch learning journey.\nProcess Overview # The general process of implementing a deep learning model with PyTorch is as follows:\nPrepare the dataset: download the dataset, convert the dataset to a PyTorch dataset. Define the model: define the neural network model, including the network structure and parameters. Train the model: train the model using the training dataset, adjust the model parameters. Test the model: test the model using the test dataset, evaluate the model performance. Here we will implement MNIST handwritten digit recognition according to this process.\nPrepare the Dataset # For the MNIST handwritten digit recognition problem, the dataset preparation is very simple because PyTorch has built-in the MNIST dataset. We only need to use the torchvision.datasets.MNIST class, here we can specify train=True to indicate the training dataset, train=False to indicate the test dataset.\n1 2 3 4 5 6 7 from torchvision import datasets dataset1 = datasets.MNIST(root=\u0026#39;data\u0026#39;, train=True, download=True) # 训练数据集 dataset2 = datasets.MNIST(root=\u0026#39;data\u0026#39;, train=False, download=True) # 测试数据集 train_loader = torch.utils.data.DataLoader(dataset1, batch_size=64, shuffle=True) test_loader = torch.utils.data.DataLoader(dataset2, batch_size=64, shuffle=False) The MNIST dataset is a series of images and labels, each image is a 28x28 grayscale image, and each label is a number between 0 and 9.\nHere we use datasets.MNIST() to get the dataset, and then use torch.utils.data.DataLoader() to convert the dataset to a PyTorch dataset. DataLoader is an iterator that can easily batch process the dataset, here we specify batch_size=64 to indicate that 64 samples are taken each time, and shuffle=True to indicate that the order of samples is shuffled each time.\nDefine the Model # Here we choose to use a neural network to implement handwritten digit recognition. Once the type of model is determined, we need to consider the specific structure of the model, including the number of layers in the network, the number of neurons in each layer, the activation function, etc. The choice of these parameters depends on the specific problem and depends heavily on experience.\nWe can first determine the input and output. Obviously, the input of this neural network is a 28x28 grayscale image, and the output is a number between 0 and 9.\nWe need to choose a neural network type, such as a fully connected neural network, a convolutional neural network, a recurrent neural network, etc. Here we choose to use a fully connected neural network.\nWe need to determine the structure of the network, including the number of layers in the network, the number of neurons in each layer, the activation function, etc. Here we choose a simple neural network, including an input layer, a hidden layer, and an output layer.\nThe code to create this neural network is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 import torch import torch.nn as nn import torch.nn.functional as F class SimpleNN(nn.Module): def __init__(self): super(SimpleNN, self).__init__() self.fc1 = nn.Linear(28 * 28, 128) # Input layer to hidden layer self.fc2 = nn.Linear(128, 10) # Hidden layer to output layer def forward(self, x): x = x.view(-1, 28 * 28) # Flatten the input image x = F.relu(self.fc1(x)) # Apply ReLU activation x = self.fc2(x) # Output layer return F.log_softmax(x, dim=1) # Apply log-softmax for classification Here we define a class named Simple that inherits from nn.Module. In PyTorch, all neural network models need to inherit from the nn.Module class and implement the __init__ and forward methods.\nIn the __init__ method, we define two fully connected layers fc1 and fc2, representing the input layer to the hidden layer and the hidden layer to the output layer, respectively. nn.Linear represents a fully connected layer, the first parameter represents the number of input neurons, and the second parameter represents the number of output neurons. In the forward method, we define the forward propagation process of the network, that is, how we calculate the output given the input data. Where: x.view(-1, 28 * 28) means to flatten the input data into a one-dimensional vector, that is, to flatten a 28x28 image into a 784-dimensional vector. F.relu(self.fc1(x)) means to input the data into the first fully connected layer, and then apply the ReLU activation function. self.fc2(x) means to input the data after ReLU activation into the second fully connected layer to get the output result. F.log_softmax(x, dim=1) means to convert the output result to a probability, that is, take the logarithm of the probability of each class. Train the Model # With the dataset and model, we can start training the model. The training process is to use the gradient descent algorithm to continuously adjust the model parameters to minimize the error between the model\u0026rsquo;s prediction and the true result.\nThe general process of training the model is as follows:\nInitialize the model parameters. Take a batch of data from the dataset. Input the data into the model to get the model\u0026rsquo;s prediction. Calculate the error between the model\u0026rsquo;s prediction and the true result. Update the model parameters using the gradient descent algorithm. Repeat steps 2 to 5 until the model converges. The most critical steps are steps 4 and 5, that is, calculating the error and updating the parameters. PyTorch provides the torch.optim module to implement the gradient descent algorithm, and the torch.nn.functional module to implement the loss function. The process of updating the parameters is implemented by the backpropagation algorithm, PyTorch provides the loss.backward() method to calculate the gradient, and the optimizer.step() method to update the parameters.\nThe loss function and backpropagation algorithm are the core of deep learning because the loss function determines the optimization goal of the model, and the backpropagation algorithm determines how to adjust the model parameters.\nLoss Function # The loss function is used to measure the difference between the model\u0026rsquo;s prediction and the true result, that is, the model\u0026rsquo;s error. PyTorch provides many common loss functions:\nClassification problem: cross-entropy loss function torch.nn.CrossEntropyLoss(), negative log-likelihood loss function torch.nn.NLLLoss(), etc. Regression problem: mean square error loss function torch.nn.MSELoss(). Binary classification problem: binary cross-entropy loss function torch.nn.BCELoss(). Multi-label classification problem: multi-label cross-entropy loss function torch.nn.BCEWithLogitsLoss(). Backpropagation # The backpropagation algorithm is used to calculate the gradient of the model parameters, that is, the derivative of the model\u0026rsquo;s error with respect to the parameters. PyTorch provides the loss.backward() method to calculate the gradient, and then uses the optimizer.step() method to update the parameters.\nTraining Code # We encapsulate this process into a train function:\n1 2 3 4 5 6 7 8 9 10 import torch.nn.functional as F def train(args, model, device, train_loader, optimizer, epoch): model.train() for batch_idx, (data, target) in enumerate(train_loader): data, target = data.to(device), target.to(device) optimizer.zero_grad() output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step() model.train() sets the model to training mode, so that the Dropout layer and BatchNorm layer in the model will take effect. optimizer.zero_grad() sets the gradient of the optimizer to zero because PyTorch defaults to accumulate gradients. output = model(data) inputs the data into the model to get the model\u0026rsquo;s prediction. loss = F.nll_loss(output, target) calculates the error between the model\u0026rsquo;s prediction and the true result, using the negative log-likelihood loss function. loss.backward() calculates the gradient of the model parameters using the backpropagation algorithm. optimizer.step() updates the model parameters using the gradient descent algorithm. Note that since we have converted the dataset to a PyTorch dataset in advance, each time a batch of data is taken, that is, data is a tensor, for the MNIST dataset, the shape of data is (batch_size, 1, 28, 28), and the shape of target is (batch_size,).\nTraining all the data is called completing an epoch, and we can iterate over the training dataset multiple times. However, iterating over the training dataset multiple times does not necessarily improve the model\u0026rsquo;s performance, as it may lead to overfitting. Therefore, we need to monitor the model\u0026rsquo;s performance during training and stop training in time.\nWhy do we train in batches? Because training all the data at once may lead to insufficient memory, and training in batches can speed up the training process.\nHow to choose the batch size? The batch size is a hyperparameter that needs to be chosen based on the specific problem and hardware device. Generally speaking, the larger the batch size, the faster the training speed, but also the larger the memory consumption. The choice of batch size also affects the convergence speed and generalization ability of the model.\nTest the Model # After training the model, we need to test the model\u0026rsquo;s performance. The general process of testing the model is as follows:\nTake a batch of data from the test dataset. Input the data into the model to get the model\u0026rsquo;s prediction. Calculate the error between the model\u0026rsquo;s prediction and the true result. Repeat steps 1 to 3 until the test dataset is traversed. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 def test(model, device, test_loader): model.eval() test_loss = 0 correct = 0 with torch.no_grad(): for data, target in test_loader: data, target = data.to(device), target.to(device) output = model(data) test_loss += F.nll_loss(output, target, reduction=\u0026#39;sum\u0026#39;).item() # sum up batch loss pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability correct += pred.eq(target.view_as(pred)).sum().item() test_loss /= len(test_loader.dataset) print(\u0026#39;\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n\u0026#39;.format( test_loss, correct, len(test_loader.dataset), 100. * correct / len(test_loader.dataset))) model.eval() sets the model to evaluation mode, so that the Dropout layer and BatchNorm layer in the model will not take effect. with torch.no_grad(): means that we do not need to calculate the gradient because during testing we only need to calculate the model\u0026rsquo;s prediction, not update the model parameters. The error here is the average error of the entire test dataset. pred = output.argmax(dim=1, keepdim=True) means to take the class with the highest probability from the prediction result. correct += pred.eq(target.view_as(pred)).sum().item() means to calculate the number of samples predicted correctly. Main Program # With the above preparation, we can start training and testing the model. Here we define a main function to call the train and test functions.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 def main(): transform=transforms.Compose([ transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)) ]) dataset1 = datasets.MNIST(\u0026#39;../data\u0026#39;, train=True, download=True, transform=transform) dataset2 = datasets.MNIST(\u0026#39;../data\u0026#39;, train=False, transform=transform) train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs) test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs) model = SimpleNN().to(device) optimizer = optim.Adadelta(model.parameters(), lr=args.lr) for epoch in range(1, args.epochs + 1): train(args, model, device, train_loader, optimizer, epoch) test(model, device, test_loader) transforms.Compose() means to combine multiple data transformation operations.\ntransforms.ToTensor() means to convert the data to a tensor.\ntransforms.Normalize() means to standardize the data, that is, subtract the mean and divide by the standard deviation. Here the mean and standard deviation are the mean and standard deviation of the MNIST dataset, which can be calculated, the specific code is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 import torch from torchvision import datasets, transforms # Load the MNIST dataset without any transformations dataset = datasets.MNIST(\u0026#39;../data\u0026#39;, train=True, download=True, transform=transforms.ToTensor()) # Compute the mean and standard deviation loader = torch.utils.data.DataLoader(dataset, batch_size=60000, shuffle=False) data = next(iter(loader))[0] # Get all the images in a single batch mean = data.mean().item() std = data.std().item() print(f\u0026#39;Mean: {mean}, Std: {std}\u0026#39;) I put this code in the file get_mnist_statistics.py, which can be run directly.\nmodel = SimpleNN().to(device) means to move the model to the specified device, here we can specify CPU or GPU.\nComplete Code # The above is all the main code needed to implement MNIST handwritten digit recognition. I have integrated these codes into a file mnist_nn.py. All the above codes are in the T01_mnist_nn folder of my GitHub repository https://github.com/jin-li/pytorch-tutorial.\nIn addition to the above code, I also defined some code to parse command line parameters, so that we can specify some parameters through the command line, such as learning rate, batch size, number of iterations, etc. You can view all parameters by entering the following command:\n1 python mnist_nn.py --help Python Environment # Before running the code, we need to create a Python virtual environment and install PyTorch and other dependencies. There are many Python environment management tools, see my previous article \u0026ldquo;Python Environment Management with venv/conda/mamba\u0026rdquo;. Here I use conda to create a virtual environment for this series of tutorials and install PyTorch and other dependencies.\n1 2 3 conda create -n pytorch-mnist conda activate pytorch-mnist pip install -r requirements.txt Run the Code # We run this code to see the model\u0026rsquo;s performance on the test dataset:\n1 python mnist_nn.py The above code defaults to using the GPU, on my machine (NVIDIA GeForce GTX 4060 Ti) it takes about 2 minutes and 15 seconds. If you don\u0026rsquo;t have a GPU, you can specify --no-cuda to use only the CPU, on my machine (Intel Core i5-9600K) it takes about 3 minutes and 1 second, slightly slower than using the GPU, but not much slower, this is because our model is relatively simple, and the MNIST dataset is relatively small.\nAfter running 14 epochs, you will get the following output:\n1 Test set: Average loss: 0.1077, Accuracy: 9790/10000 (98%) That is, our simple three-layer neural network model has an accuracy of 98% on the MNIST dataset, which is already very good.\nSummary # In this article, we introduced the basic concepts and usage of PyTorch, and then implemented MNIST handwritten digit recognition using a simple three-layer fully connected neural network. This is the \u0026ldquo;Hello World\u0026rdquo; of deep learning. However, the parameters in this neural network are directly given, we did not discuss how these parameters are obtained, nor did we discuss how the choice of these parameters affects the model\u0026rsquo;s performance. In the next article, we will try different parameter choices to see how they affect the model\u0026rsquo;s performance and how to adjust these parameters to improve the model\u0026rsquo;s performance.\n","date":"15 January 2024","externalUrl":null,"permalink":"/en/p/learn-pytorch-by-examples-1-mnist-handwritten-digit-recognition-i--pytorch-basics-and-neural-network-basics/","section":"Posts","summary":" Motivation # I used TensorFlow when I took a machine learning course many years ago, but my understanding of deep learning was not deep enough at that time, so I only used it briefly. Later, I did not take any machine learning courses, but I still used some machine learning knowledge in my research work, so I learned some machine learning knowledge intermittently.\n","title":"Learn PyTorch by Examples (1): MNIST Handwritten Digit Recognition (I) -- PyTorch Basics and Neural Network Basics","type":"post"},{"content":" Motivation # Due to copyright issues, many video and music resources are unavailable abroad. Therefore, I wanted to set up a VPN to access these resources from abroad. I rented a server from a cloud service provider in China and set up a Shadowsocks service on it to achieve this goal. I encountered several problems during this process, and I am documenting how I solved them.\nThis article is the first in the \u0026ldquo;Private Virtual Network VPN\u0026rdquo; series, and subsequent articles will introduce the basic concepts of VPNs and how to use Tailscale to set up a private virtual network. The other articles in this series are as follows:\nPrivate Virtual Network VPN (2): Basic Concepts and Tailscale Setup Guide Prerequisites # A cloud server located in China Installing Shadowsocks # The process of installing Shadowsocks mainly refers to this article: Shadowsocks Tutorial for Beginners. However, following the steps in this article, I failed many times because there are many pitfalls in this article. Here, I mainly record how I solved these problems.\nCloud Server Network # The tutorial for installing Shadowsocks above requires downloading some resources located on GitHub. However, due to well-known reasons, GitHub is inaccessible in China. Therefore, we need to solve this problem first. That is, if we want to bypass the Great Firewall and access resources in China from abroad, we need to allow the cloud server in China to access the internet.\nIf it were an ordinary computer in China, we could just use some common VPN software. But our cloud server runs in command-line mode, so we need to bypass the firewall in the command line. There are two solutions here:\nSolution 1: Use SSH Proxy # First of all, since we are already abroad, we don\u0026rsquo;t need to buy a VPN specifically for our cloud server in China, we just need to use our computer abroad.\nSSH from the computer abroad (hereinafter referred to as \u0026ldquo;computer\u0026rdquo;) to the cloud server in China (hereinafter referred to as \u0026ldquo;cloud server\u0026rdquo;):\n1 ssh username@cloud_server_ip SSH from the cloud server to the computer, using a proxy port (the port number can be any number from 1 to 65535; here I chose 11111):\n1 ssh -D 11111 username@foreign_computer_ip Set the network proxy for the cloud server, with the proxy port set to 11111:\n1 export ALL_PROXY=socks5://127.0.0.1:11111 This way, all network requests from the cloud server will be forwarded to port 11111, and then tunneled through SSH to the computer. This allows the cloud server to access foreign networks through the computer\u0026rsquo;s network, effectively bypassing the firewall.\nCheck the IP address of the cloud server at this time:\n1 curl ipinfo.io If the displayed IP address is the computer\u0026rsquo;s IP address, then we have succeeded.\nThen follow the steps in the tutorial above to install Shadowsocks, but I found that there were still problems. The problem seems to be that the script for installing Shadowsocks in the tutorial uses wget to download some files, but wget does not seem to go through the proxy we set, so it still prompts a connection timeout. However, curl can go through our proxy network, so we can replace all wget requests in the script with curl requests, which can solve this problem.\nSolution 2: Download Locally and Upload to Cloud Server # The above solution is relatively complex to set up and may encounter various issues. In fact, I may not frequently use the cloud server to connect to foreign networks in the future. If it is just for this one-time need, I can completely download the required files in advance and then upload them to the cloud server.\nUsing this solution means we cannot use the single command from the tutorial above; we need to split this command:\nDownload the Shadowsocks installation script on the computer:\n1 wget --no-check-certificate -O shadowsocks-all.sh https://raw.githubusercontent.com/teddysun/shadowsocks_install/master/shadowsocks-all.sh Upload the script to the cloud server:\n1 scp shadowsocks-all.sh root@cloud_server_ip:/root Install Shadowsocks on the cloud server:\n1 bash shadowsocks-all.sh 2\u0026gt;\u0026amp;1 | tee shadowsocks-all.log It is important to note that running this installation script now will still fail for two reasons:\nUnable to connect to GitHub to download some files Unable to find the installation package for Python Python Version # The above installation script requires the use of python (python2), but the apt source in Ubuntu 22 does not have python, and it is necessary to explicitly specify the Python version, such as clearly indicating python2 or python3.\nTherefore, two things need to be done here:\nChange python in apt_depends in the script to python2, and python-dev to python2-dev Create a symbolic link in the /usr/bin directory to point python to python2 1 ln -s /usr/bin/python2 /usr/bin/python Then follow the installation steps in the tutorial above. The version of Shadowsocks I chose to install is Shadowsocks-Python.\nComplete Shadowsocks Installation # Confirm Shadowsocks is Running Normally # After the installation is complete, we can use the following command to check the running status of Shadowsocks:\n1 /etc/init.d/shadowsocks status If the displayed message is Shadowsocks (pid xxxxx) is running..., then it means Shadowsocks is running normally.\nOpen Firewall Ports on Cloud Server # It is necessary to open the Shadowsocks port (default is 8388) in the firewall of the cloud server, otherwise, you will not be able to connect to the Shadowsocks service. Run the following command:\n1 2 ufw allow 8388 ufw enable Then check the status of the firewall:\n1 ufw status If the displayed message is Status: active, and the port used by Shadowsocks is in the ALLOW state, then it means the firewall has been opened.\nEnable Port in Cloud Server Management Interface # For many cloud servers, it is not enough to just open the firewall in the command line; you also need to enable the port in the cloud server\u0026rsquo;s management interface. Find the Security Group in the management interface, and then add an inbound rule to set the port to the one used by Shadowsocks (default is 8388), set the protocol to TCP, and set the source address to 0.0.0.0/0.\nUse Shadowsocks to Bypass Restrictions # Download the Shadowsocks client, and then configure the Shadowsocks client. Set the server address to the IP address of the cloud server, set the port to the one used by Shadowsocks (default is 8388), set the password to the password of the Shadowsocks service, and ensure that the encryption method is consistent with that of the Shadowsocks server.\nIf the previous settings are all normal, you can now connect to the domestic network.\n","date":"9 December 2023","externalUrl":null,"permalink":"/en/p/private-virtual-network-vpn-1-setting-up-a-vpn-with-shadowsocks-on-a-cloud-server/","section":"Posts","summary":" Motivation # Due to copyright issues, many video and music resources are unavailable abroad. Therefore, I wanted to set up a VPN to access these resources from abroad. I rented a server from a cloud service provider in China and set up a Shadowsocks service on it to achieve this goal. I encountered several problems during this process, and I am documenting how I solved them.\n","title":"Private Virtual Network VPN (1): Setting Up a VPN with Shadowsocks on a Cloud Server","type":"post"},{"content":"","date":"9 December 2023","externalUrl":null,"permalink":"/en/tags/shadowsocks/","section":"Tags","summary":"","title":"Shadowsocks","type":"tags"},{"content":" Background # In the first article of this series \u0026ldquo;Building a Private Mail Server using mailcow and Docker\u0026rdquo;, we built a private email server with mailcow and docker on our own computer or server. Since many operators will block port 25, causing emails to not be sent from the server, we used email relay as an alternative.\nRecently, I found that my email could not be sent to the outside world again. After some testing, I found that AT\u0026amp;T\u0026rsquo;s email relay service was not working, and it seemed that their email system had a problem. I couldn\u0026rsquo;t even log into the att.net email I registered with them. Every time I logged in, it would always redirect to the AT\u0026amp;T page, and after half a day of loading, it would pop up an error page. I can\u0026rsquo;t help but spit out, as a well-known network operator, the website can be so bad, this is also a great wonder.\nAfter many attempts, I decided to give up AT\u0026amp;T\u0026rsquo;s email relay service and use SendGrid email relay service instead. Here are some of the problems and solutions I encountered.\nPrerequisites # A private email server has been built using mailcow and docker (if not, please refer to \u0026ldquo;Building a Private Mail Server using mailcow and Docker\u0026rdquo;) SendGrid Settings # Register a SendGrid Account # Register an account on the SendGrid website. In addition to the general information on the website, SendGrid also requires you to fill in company information, which can be filled in at will. Finally, you need to set up two-step verification using your mobile phone.\nI chose the free version, which allows you to send up to 100 emails per day, which is enough for me.\nGenerate an API Key # After registering, log in to the SendGrid management interface, click on the left menu bar Email API, select Integration Guide, then select SMTP Relay, and click Get Started,\nThen fill in the name of the API Key, click Create API Key, and finally check I've updated my settings., and click Next: Verify Integration.\nEmail Relay # The settings for email relay can be found in the previous article \u0026ldquo;Building a Private Mail Server using mailcow and Docker\u0026rdquo;, so I won\u0026rsquo;t go into them here.\nOne thing to note is that if you use SendGrid\u0026rsquo;s email relay service in other network applications, you need to set Relayhost to smtp.sendgrid.net:587, not smtp.mail.your_domain_name, the username is \u0026ldquo;apikey\u0026rdquo;, and the password is the API Key generated in the previous step.\nEmail Blocking Issues # Reasons for being blocked # When I used SendGrid\u0026rsquo;s email relay service in one of my network applications, I found that I didn\u0026rsquo;t receive the email after waiting for a long time. Then I checked the recent activity records in the SendGrid management interface:\nClick on the undelivered record, and you can see that the reason the email was blocked is 550 DMARC check failed:\nThe blocked email can be viewed in the QQ mailbox. Just click on your own avatar in the upper right corner, select Self-service query, and then click on the Receive mail query section, and you can see the blocked email:\nAfter searching online, I found that the reason for this problem is that I used SendGrid\u0026rsquo;s email relay service to send the email, but the actual sender of the email is my self-built email server, that is, the actual sender\u0026rsquo;s domain name and the nominal sender\u0026rsquo;s domain name are inconsistent. When the receiving mail server queries the DMARC record of the sender, it finds that it is inconsistent, and it is likely to refuse to receive the email.\nIn fact, this can also be seen when receiving emails. Click on the details of the sender, and you can see that the actual sender\u0026rsquo;s domain name is sendgrid.net, while the nominal sender\u0026rsquo;s domain name is jinli.io:\nSolution # The solution to the above problem is to authenticate your domain name in the SendGrid management system. Click on your username in the upper left corner, select Setup Guide, and then complete the second step Authenticate your sending domain. SendGrid will ask you to add 3 CNAME records to your domain\u0026rsquo;s DNS server. After adding them, click verify.\nAfter verification, you can send an email from your email to test it. This time, SendGrid will use your domain name to generate a sender address, and then use this user to send the email. In this way, the actual sender\u0026rsquo;s domain name and the nominal sender\u0026rsquo;s domain name are consistent, and can pass the DMARC check of the receiving server.\nSummary # In fact, the process of building a self-built email server is not complicated. There are generally no problems receiving emails, but the real problem is how to send emails. Here, the SendGrid email relay service is used to solve the problem of sending emails. Although it is a bit troublesome, it is still a free solution to the problem.\nBuilding a self-built email server is actually a very time-consuming task, especially the problem of email relay. There are actually many discussions online, and some netizens have also proposed some free or relatively cheap solutions (about $10 per year). You can refer to the discussion on Reddit. Currently, I am using mailcow and SendGrid without any problems. If there are problems in the future, I may also turn to the solutions mentioned in this post.\n","date":"3 December 2023","externalUrl":null,"permalink":"/en/p/build-a-private-email-server-with-mailcow-and-docker-2-use-email-relay-and-reduce-the-probability-of-being-blocked/","section":"Posts","summary":" Background # In the first article of this series “Building a Private Mail Server using mailcow and Docker”, we built a private email server with mailcow and docker on our own computer or server. Since many operators will block port 25, causing emails to not be sent from the server, we used email relay as an alternative.\n","title":"Build a Private Email Server with mailcow and Docker (2) - Use Email Relay and Reduce the Probability of being Blocked","type":"post"},{"content":"","date":"3 December 2023","externalUrl":null,"permalink":"/en/tags/dmarc/","section":"Tags","summary":"","title":"DMARC","type":"tags"},{"content":"","date":"3 December 2023","externalUrl":null,"permalink":"/en/tags/email-relay/","section":"Tags","summary":"","title":"Email Relay","type":"tags"},{"content":"","date":"3 December 2023","externalUrl":null,"permalink":"/en/tags/mail-server/","section":"Tags","summary":"","title":"Mail Server","type":"tags"},{"content":"","date":"3 December 2023","externalUrl":null,"permalink":"/en/series/mail-server-series/","section":"Series","summary":"","title":"Mail Server Series","type":"series"},{"content":"","date":"3 December 2023","externalUrl":null,"permalink":"/en/tags/mailcow/","section":"Tags","summary":"","title":"Mailcow","type":"tags"},{"content":"","date":"2023年12月3日","externalUrl":null,"permalink":"/series/%E9%82%AE%E4%BB%B6%E6%9C%8D%E5%8A%A1%E5%99%A8%E7%B3%BB%E5%88%97/","section":"Series","summary":"","title":"邮件服务器系列","type":"series"},{"content":"","date":"21 November 2023","externalUrl":null,"permalink":"/en/tags/api/","section":"Tags","summary":"","title":"API","type":"tags"},{"content":" Introduction # In the previous three articles of this series, we used the Flask framework of Python to develop a simple weather forecast website. So far, our website is still limited to access on our own computer. In this article, we will use Docker, Gunicorn and Nginx to deploy this website, so that our website can be accessed on the public network.\nPrerequisites # Already know how to develop a simple web application using the Flask framework (see \u0026ldquo;Learn Web Development with Python (1): Use Flask Framework\u0026rdquo;) Already installed PostgreSQL database and know how to use SQLAlchemy to operate the database (see \u0026ldquo;Learn Web Development with Python (2): Use PostgreSQL and SQLAlchemy\u0026rdquo;) Already installed Docker and docker-compose Already have a domain name Project Structure (Development Version) # Up to the previous article \u0026ldquo;Learn Web Development with Python (3): Use Input and API\u0026rdquo;, our project structure is as follows:\n1 2 3 4 5 6 learn_flask ├── app.py ├── static │ └── style.css └── templates └── weather.html In this article, we will use Docker and Gunicorn to deploy this website, so we need to add some files. For convenience, we will adjust the project structure to the following structure:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 learn_flask ├── docker-compose.yml ├── .env ├── .env.db └── learn_flask ├── app │ ├── config.py │ ├── __init__.py │ ├── static │ │ └── style.css │ └── templates │ └── weather.html ├── Dockerfile ├── manage.py └── requirements.txt Next we will create these files step by step and introduce the functions of these files.\nAdd Dependencies # Before developing this web application, we have defined a Python virtual environment, which contains the dependencies we need. We can use the following command to export the dependencies in the current environment to the requirements.txt file:\n1 pip freeze \u0026gt; requirements.txt For some reason, there are some unnecessary dependencies in it, such as click, itsdangerous, Jinja2, MarkupSafe, Werkzeug, we can manually delete these dependencies. Finally, our requirements.txt file is as follows:\n1 2 3 4 5 Flask==3.0.0 Flask-SQLAlchemy==3.1.1 Jinja2==3.1.2 gunicorn==20.1.0 psycopg2-binary==2.9.9 Package the Application as a Python Package # We rename the original app.py file to __init__.py and move it to the learn_flask/app directory.\nThen we can create a config.py file in the learn_flask/app directory to store our database configuration information:\n1 2 3 4 5 6 7 import os basedir = os.path.abspath(os.path.dirname(__file__)) class Config(object): SQLALCHEMY_DATABASE_URI = os.getenv(\u0026#34;DATABASE_URL\u0026#34;, \u0026#34;sqlite://\u0026#34;) SQLALCHEMY_TRACK_MODIFICATIONS = False This way we can delete the database configuration information in __init__.py:\n1 2 app.config[\u0026#39;SQLALCHEMY_DATABASE_URI\u0026#39;] = \u0026#39;postgresql://test:test_password@localhost:5432/weather_db\u0026#39; app.config[\u0026#39;SQLALCHEMY_TRACK_MODIFICATIONS\u0026#39;] = False We create a manage.py file outside the app directory to manage our application. We can add the following code to manage.py:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 from flask.cli import FlaskGroup from sqlalchemy.exc import ProgrammingError from app import app cli = FlaskGroup(app) @cli.command(\u0026#34;create_db\u0026#34;) def create_db(): from app import db from app import Weather with app.app_context(): try: db.create_all() db.session.commit() except ProgrammingError: pass @cli.command(\u0026#34;drop_db\u0026#34;) def drop_db(): from app import db from app import Weather with app.app_context(): try: db.drop_all() db.session.commit() except ProgrammingError: pass if __name__ == \u0026#34;__main__\u0026#34;: cli() Here we use FlaskGroup to manage our application, and the create_db and drop_db commands are used to create and delete the database. Here we use ProgrammingError to determine whether the database exists. If the database does not exist, the database will not be deleted.\nThen we can delete the code for creating and deleting the database in __init__.py:\n1 2 3 4 5 6 7 8 if __name__ == \u0026#39;__main__\u0026#39;: with app.app_context(): try: db.create_all() db.session.commit() except ProgrammingError: pass app.run() Deploy the Application with Docker # Create Dockerfile # We create a Dockerfile file under the learn_flask directory, which is used to build our application. We can add the following code to Dockerfile:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # pull official base image FROM python:3.11.4-slim-buster as builder # set work directory WORKDIR /app # Copy the current directory contents into the container at /app COPY . /app # Install any needed packages specified in requirements.txt COPY ./requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Run app.py when the container launches CMD [\u0026#34;gunicorn\u0026#34;, \u0026#34;-b\u0026#34;, \u0026#34;0.0.0.0:5001\u0026#34;, \u0026#34;app:app\u0026#34;] The function of each command is marked in the comments.\nCreate docker-compose.yml # We create a docker-compose.yml file under the learn_flask directory, which is used to manage our application. We can add the following code to docker-compose.yml:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 version: \u0026#34;3.8\u0026#34; services: learn_flask: build: ./learn_flask command: python manage.py run -h 0.0.0.0 volumes: - ./learn_flask:/app/ ports: - 5001:5000 env_file: - ./.env depends_on: - db db: image: postgres:13 volumes: - ./postgres_data_prod:/var/lib/postgresql/data/ env_file: - ./.env.db volumes: postgres_data: In this docker-compose.yml file, we define two services, one is the application learn_flask, and the other is the database db.\nCreate .env File # In the above docker-compose.yml file, we used the .env file and .env.db file, we can create these two files under the learn_flask directory to store our environment variables. We can add the following environment variables to the .env file:\n1 2 3 4 5 6 FLASK_APP=app/__init__.py FLASK_DEBUG=1 DATABASE_URL=postgresql://your_postgre_user_name:your_postgres_password@db:5432/weather_db SQL_HOST=db SQL_PORT=5432 DATABASE=postgres Add database environment variables to the .env.db file:\n1 2 3 POSTGRES_USER=your_postgre_user_name POSTGRES_PASSWORD=your_postgres_password POSTGRES_DB=weather_db Deploy the Application (Development Version) # Build the Image # When building the image for the first time, our application has not yet created the database, so we need to create the database first. We need to change the command in docker-compose.yml to the following code:\n1 2 3 4 command: \u0026gt; sh -c \u0026#34;python manage.py drop_db \u0026amp;\u0026amp; python manage.py create_db \u0026amp;\u0026amp; python manage.py run -h Then we can use the following command to build the image:\n1 docker-compose up -d --build If everything goes well, you can access our website in the browser by entering http://localhost:5001.\nDeploy the Application # After the first successful build, we can temporarily stop the application, and then change the command in docker-compose.yml back to the original command:\n1 command: python manage.py run -h 0.0.0.0 Then we can use the following command to deploy the application:\n1 docker-compose up -d Deploy the Application (Production Version) # After completing the deployment according to the above steps, our program is still running in development mode. We can see the following prompt information in the docker log:\n1 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. This prompts us that the above deployment is not enough for the production environment, so we need to make some adjustments to make our application run in the production environment.\ndocker-compose # First, we create a new docker-compose.yml file for the production environment deployment. We can create a docker-compose.prod.yml file under the learn_flask directory:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 version: \u0026#34;3.8\u0026#34; services: learn_flask: build: context: ./learn_flask dockerfile: Dockerfile.prod command: gunicorn --bind 0.0.0.0:5000 manage:app ports: - 5001:5000 env_file: - ./.env.prod depends_on: - db db: image: postgres:13 volumes: - postgres_data_prod:/var/lib/postgresql/data/ env_file: - ./.env.prod.db volumes: postgres_data_prod: Note that we no longer specify volumes for the application, because we no longer need to mount the code of the application to the container, but package the application into the image.\nDockerfile # In the docker-compose.prod.yml file above, we used a new Dockerfile.prod file, we can create a Dockerfile.prod file under the learn_flask/learn_flask directory to build our application image:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 ########### # BUILDER # ########### # pull official base image FROM python:3.11.3-slim-buster as builder # set work directory WORKDIR /usr/src/app # set environment variables ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 # install system dependencies RUN apt-get update \u0026amp;\u0026amp; \\ apt-get install -y --no-install-recommends gcc # lint RUN pip install --upgrade pip # RUN pip install flake8==6.0.0 COPY . /usr/src/app/ # RUN flake8 --ignore=E501,F401 . # install python dependencies COPY ./requirements.txt . RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt ######### # FINAL # ######### # pull official base image FROM python:3.11.3-slim-buster # create directory for the app user RUN mkdir -p /home/app # create the app user RUN addgroup --system app \u0026amp;\u0026amp; adduser --system --group app # create the appropriate directories ENV HOME=/home/app ENV APP_HOME=/home/app/learn_flask RUN mkdir $APP_HOME WORKDIR $APP_HOME # install dependencies RUN apt-get update \u0026amp;\u0026amp; apt-get install -y --no-install-recommends netcat COPY --from=builder /usr/src/app/wheels /wheels COPY --from=builder /usr/src/app/requirements.txt . RUN pip install --upgrade pip RUN pip install --no-cache /wheels/* # copy entrypoint-prod.sh COPY ./entrypoint.prod.sh $APP_HOME # copy project COPY . $APP_HOME # chown all the files to the app user RUN chown -R app:app $APP_HOME # change to the app user USER app # run entrypoint.prod.sh ENTRYPOINT [\u0026#34;/home/app/learn_flask/entrypoint.prod.sh\u0026#34;] entrypoint # In the Dockerfile.prod file above, we used an entrypoint.prod.sh file, we can create an entrypoint.prod.sh file under the learn_flask/learn_flask directory to run our application:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 #!/bin/sh if [ \u0026#34;$DATABASE\u0026#34; = \u0026#34;postgres\u0026#34; ] then echo \u0026#34;Waiting for postgres...\u0026#34; while ! nc -z $SQL_HOST $SQL_PORT; do sleep 0.1 done echo \u0026#34;PostgreSQL started\u0026#34; fi if [ \u0026#34;$FLASK_DEBUG\u0026#34; = \u0026#34;1\u0026#34; ] then echo \u0026#34;Creating the database tables...\u0026#34; python manage.py create_db echo \u0026#34;Tables created\u0026#34; fi exec \u0026#34;$@\u0026#34; Note: After creating the entrypoint.prod.sh file, we need to change its permissions to executable, otherwise an error will be reported when running docker later:\n1 chmod +x entrypoint.prod.sh Environment Variables # Similar to deploying in the development environment, we need to create .env.prod and .env.prod.db files to store the environment variables for the production environment. We can add the following environment variables to the .env.prod file:\n1 2 3 4 5 6 FLASK_APP=app/__init__.py FLASK_DEBUG=0 DATABASE_URL=postgresql://your_postgre_user_name:your_postgres_password@db:5432/weather_db_prod SQL_HOST=db SQL_PORT=5432 DATABASE=postgres Add database environment variables to the .env.prod.db file:\n1 2 3 POSTGRES_USER=your_postgre_user_name POSTGRES_PASSWORD=your_postgres_password POSTGRES_DB=weather_db_prod Build the Image # We can use the following command to build the image:\n1 sudo docker-compose -f docker-compose.prod.yml up -d --build At this time, we only built the image, and the database is still empty, so we need to create the database in the container. We can use the following command to create the database in the container:\n1 sudo docker-compose -f docker-compose.prod.yml exec learn_flask python manage.py create_db Then we can enter http://localhost:5001 in the browser to see our website.\nProject Structure (Production Version) # Finally, our directory structure for the production environment is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 learn_flask ├── docker-compose.prod.yml ├── env.prod ├── env.prod.db └── learn_flask ├── app │ ├── config.py │ ├── __init__.py │ ├── static │ │ └── style.css │ └── templates │ └── weather.html ├── Dockerfile.prod ├── entrypoint.prod.sh ├── manage.py └── requirements.txt Use Nginx Reverse Proxy # After completing the above deployment, we can still only access our website locally. If we want to access it on the public network, we need to use Nginx reverse proxy. Since Nginx has been installed on my server, I only need to add a configuration file on the server. We can create a learn_flask.conf file under the /etc/nginx/sites-available directory on the server to store our configuration information. We can add the following code to the learn_flask.conf file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 upstream learn_flask.jinli.io { server 127.0.0.1:5001; } server { listen 80; server_name learn_flask.jinli.io; return 301 https://learn_flask.jinli.io$request_uri; } server { listen 443 ssl; server_name learn_flask.jinli.io; ssl_certificate /media/lijin/learn_flask/cert/cert.pem; ssl_certificate_key /media/lijin/learn_flask/cert/key.pem; location / { proxy_redirect off; proxy_pass http://learn_flask.jinli.io; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Ssl on; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Frame-Options SAMEORIGIN; client_max_body_size 100m; } } Then we can use the following command to activate this configuration file:\n1 sudo service nginx reload ","date":"21 November 2023","externalUrl":null,"permalink":"/en/p/learn-web-development-with-python-4-deploy-website-with-docker-gunicorn-and-nginx/","section":"Posts","summary":" Introduction # In the previous three articles of this series, we used the Flask framework of Python to develop a simple weather forecast website. So far, our website is still limited to access on our own computer. In this article, we will use Docker, Gunicorn and Nginx to deploy this website, so that our website can be accessed on the public network.\n","title":"Learn Web Development with Python (4): Deploy Website with Docker, Gunicorn and Nginx","type":"post"},{"content":"","date":"21 November 2023","externalUrl":null,"permalink":"/en/categories/programming/","section":"Categories","summary":"","title":"Programming","type":"categories"},{"content":"","date":"21 November 2023","externalUrl":null,"permalink":"/en/categories/web/","section":"Categories","summary":"","title":"Web","type":"categories"},{"content":"","date":"2023年11月21日","externalUrl":null,"permalink":"/categories/%E7%BC%96%E7%A8%8B/","section":"Categories","summary":"","title":"编程","type":"categories"},{"content":" Introduction # This is the third article in this series, which mainly introduces how to use the input box to get user input, and how to use the API to get information on the Internet.\nPrerequisites # Already know how to develop a simple web application using the Flask framework (see \u0026ldquo;Learn Web Development with Python (1): Use Flask Framework\u0026rdquo;) Already installed PostgreSQL database and know how to use SQLAlchemy to operate the database (see \u0026ldquo;Learn Web Development with Python (2): Use PostgreSQL and SQLAlchemy\u0026rdquo;) Get User Input # Add Input Box # First, we can use the \u0026lt;input\u0026gt; tag in the HTML page to create an input box. For example, we can add a form in templates/index.html to let users enter a city:\n1 2 3 4 5 6 7 8 9 \u0026lt;section class=\u0026#34;top-banner\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;container\u0026#34;\u0026gt; \u0026lt;h1 class=\u0026#34;heading\u0026#34;\u0026gt;Weather App\u0026lt;/h1\u0026gt; \u0026lt;form method=\u0026#34;post\u0026#34;\u0026gt; \u0026lt;input type=\u0026#34;text\u0026#34; name=\u0026#34;city\u0026#34; placeholder=\u0026#34;Enter a city\u0026#34; autocomplete=\u0026#34;off\u0026#34; required\u0026gt; \u0026lt;button type=\u0026#34;submit\u0026#34;\u0026gt;Search\u0026lt;/button\u0026gt; \u0026lt;/form\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/section\u0026gt; The style of the input box comes from the style.css file. This style file comes from https://webdesign.tutsplus.com/build-a-simple-weather-app-with-vanilla-javascript\u0026ndash;cms-33893t, and the effect of the input box is as follows:\nGet User Input # When the user enters the city name in the input box and clicks the \u0026ldquo;Search\u0026rdquo; button or presses the Enter key, the user\u0026rsquo;s input will be passed to the name defined in the \u0026lt;input\u0026gt; tag. We can add a POST request route in app.py to process user input. Since we have already added a route for the / path before, we only need to add a POST request for the / path:\n1 2 3 4 5 6 7 @app.route(\u0026#39;/\u0026#39;, methods=[\u0026#39;POST\u0026#39;, \u0026#39;GET\u0026#39;]) def index(): if request.method == \u0026#39;POST\u0026#39;: city_name = request.form.get(\u0026#39;city\u0026#39;) else: #for default name city_name = \u0026#39;Beijing\u0026#39; Here we use request.form.get('city') to get the city name entered by the user and assign it to the variable city_name.\nGet Weather Information Using API # OpenWeather API # OpenWeather is a website that provides weather information. After the user registers an account, the API provided by OpenWeather can be used to obtain weather information. We can register an account at https://openweathermap.org/, and then get the API Key on the https://home.openweathermap.org/api_keys page. Here we use the free version of the API Key, which can send up to 60 requests per minute and up to 1000 requests per day. The free version is enough for me.\nGet Weather Information Using API # In the previous step, we used the OpenWeather API to get weather information, but this information is returned in JSON format. In Python, we can use a dictionary to access the data in JSON and store it in the database. As for the weather information in JSON, you can check https://openweathermap.org/current#current_JSON\n1 2 3 4 def get_weather(city_name): url = \u0026#39;http://api.openweathermap.org/data/2.5/weather?q={}\u0026amp;units=metric\u0026amp;appid={}\u0026#39;.format(city_name, API_KEY) response = urlopen(url).read() return json.loads(response) Note:\nThe API_KEY here is the API Key we obtained on the OpenWeather website, and it needs to be assigned to the variable API_KEY.\nThe city name obtained earlier may contain spaces, but the OpenWeather API does not support spaces. Spaces in the OpenWeather API are replaced by +. Therefore, we need to replace the spaces in the city name with + and convert the city name to lowercase. For example, New York will be converted to new+york. We can use the following regular expression to implement this function:\n1 re.sub(r\u0026#34;\\s+\u0026#34;, \u0026#39;+\u0026#39;, str(city_name).lower()) Process the Weather Information Obtained # In the previous step, we used the OpenWeather API to get weather information, but this information is returned in JSON format. In Python, we can use a dictionary to access the data in JSON and store it in the database. As for the weather information in JSON, you can check https://openweathermap.org/current#current_JSON\n1 2 3 4 5 6 7 8 9 10 11 data = get_weather(city_name) db.session.add(Weather( name = data[\u0026#39;name\u0026#39;], country = data[\u0026#39;sys\u0026#39;][\u0026#39;country\u0026#39;], temp = round(data[\u0026#39;main\u0026#39;][\u0026#39;temp\u0026#39;]), feels_like = round(data[\u0026#39;main\u0026#39;][\u0026#39;feels_like\u0026#39;]), icon = data[\u0026#39;weather\u0026#39;][0][\u0026#39;icon\u0026#39;], description = data[\u0026#39;weather\u0026#39;][0][\u0026#39;description\u0026#39;], )) db.session.commit() Other parts are the same as the previous code, get the weather information you want to display from the database, and then pass it to the template. The effect is as follows: Display the Latest Weather Information # Finally, we add a slightly more complex database operation, such as displaying the latest weather information. We can add a piece of code in app.py to select the latest 4 weather information in the database, and then display them on the page:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 subquery = ( db.session.query( Weather.name, func.max(Weather.time).label(\u0026#39;max_time\u0026#39;) ) .group_by(Weather.name) .order_by(func.max(Weather.time).desc()) .limit(4) .subquery() ) query = ( db.session.query(Weather) .join(subquery, and_( Weather.name == subquery.c.name, Weather.time == subquery.c.max_time )) .order_by(Weather.time.desc()) ) wd = query.all() Note that we used a subquery of SQLAlchemy here, so we need to import from sqlalchemy.sql import func, and_. Finally, when we run the entire application, we first create the database. If the database already exists, it will not be created:\n1 2 3 4 5 6 7 if __name__ == \u0026#39;__main__\u0026#39;: with app.app_context(): try: db.create_all() except ProgrammingError: pass app.run() Here we also need to import related modules from sqlalchemy.exc import ProgrammingError.\n","date":"19 November 2023","externalUrl":null,"permalink":"/en/p/learn-web-development-with-python-3-use-input-and-api/","section":"Posts","summary":" Introduction # This is the third article in this series, which mainly introduces how to use the input box to get user input, and how to use the API to get information on the Internet.\nPrerequisites # Already know how to develop a simple web application using the Flask framework (see “Learn Web Development with Python (1): Use Flask Framework”) Already installed PostgreSQL database and know how to use SQLAlchemy to operate the database (see “Learn Web Development with Python (2): Use PostgreSQL and SQLAlchemy”) Get User Input # Add Input Box # First, we can use the \u003cinput\u003e tag in the HTML page to create an input box. For example, we can add a form in templates/index.html to let users enter a city:\n","title":"Learn Web Development with Python (3): Use Input and API","type":"post"},{"content":"","date":"18 November 2023","externalUrl":null,"permalink":"/en/tags/database/","section":"Tags","summary":"","title":"Database","type":"tags"},{"content":" Background # In the previous article \u0026ldquo;Learn Web Development with Python (1): Use Flask Framework\u0026rdquo;, we used the Flask framework to develop a simple web application to display weather information for some cities. But these weather information are hard-coded in the code. If we have a lot of weather information, it is not convenient to hard-code them in the code. Therefore, we need a database to store these weather information, and then read these weather information from the database and display them on the web page.\nThis article will introduce how to use the PostgreSQL database and SQLAlchemy to operate the database to read weather information from the database.\nPrerequisites # Before starting this article, make sure you understand how to use the Flask framework to develop a simple web application. If you don\u0026rsquo;t know, you can refer to the previous article \u0026ldquo;Learn Web Development with Python (1): Use Flask Framework\u0026rdquo;.\nPostgreSQL # Introduction to PostgreSQL # PostgreSQL is an open source relational database management system. It is characterized by powerful functions, good scalability, support for SQL standards, support for transactions, support for complex queries, support for multiple programming languages, and support for multiple operating systems. PostgreSQL is one of the most popular relational database management systems, so we choose to use the PostgreSQL database as the database for the backend of the web application.\nInstall and configure PostgreSQL # Installing PostgreSQL on Ubuntu 22 is very simple, just run the following command in the terminal:\n1 sudo apt install postgresql You can use the following command to view the installed version of PostgreSQL:\n1 psql --version After the installation is complete, PostgreSQL will automatically create a user named postgres. This user is a super user and can be used to manage the database. We can use the following command to switch to the postgres user:\n1 sudo su postgres After logging in, we log in to the PostgreSQL database in the command line:\n1 psql The following are commonly used commands in PostgreSQL:\n\\l：List all databases \\?：List all commands \\c \u0026lt;database_name\u0026gt;：Connect to the specified database \\d：List all tables in the current database \\d \u0026lt;table_name\u0026gt;：List the structure of the specified table \\q：Exit PostgreSQL \\du：List all users \\du+：List all users and user details \\password \u0026lt;user_name\u0026gt;：Change the password of the specified user \\conninfo：Display the information of the current connected database Use \\du to list all users, you can see that there is currently only one user postgres. We need to create a new user to manage our database. Use the following command to create a new user:\n1 CREATE USER test WITH PASSWORD \u0026#39;test_password\u0026#39;; Here test is the user name, and test is the password. Then use the following command to grant superuser permissions to this user:\n1 ALTER USER test WITH SUPERUSER; Then use the following command to exit PostgreSQL:\n1 \\q Then use the following command to exit the postgres user:\n1 exit Then we log in to the PostgreSQL database again with the newly created user test:\n1 psql -U test You may encounter the following error message:\n1 FATAL: Peer authentication failed for user \u0026#34;test\u0026#34; To solve this problem, you need to modify the configuration file of PostgreSQL (refer to https://zhuanlan.zhihu.com/p/467644334). Use the following command to open the configuration file of PostgreSQL:\n1 sudo vim /etc/postgresql/14/main/pg_hba.conf Here 14 is the version number of PostgreSQL. If your version number is not 14, please modify it to your version number. At the end of the configuration file, you can see the following lines:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # Database administrative login by Unix domain socket local all postgres peer # TYPE DATABASE USER ADDRESS METHOD # \u0026#34;local\u0026#34; is for Unix domain socket connections only local all all peer # IPv4 local connections: host all all 127.0.0.1/32 scram-sha-256 # IPv6 local connections: host all all ::1/128 scram-sha-256 # Allow replication connections from localhost, by a user with the # replication privilege. local replication all peer host replication all 127.0.0.1/32 scram-sha-256 host replication all ::1/128 scram-sha-256 We need to change the peer in the line local all all peer to md5, and then save and exit. Finally, use the following command to restart PostgreSQL:\n1 sudo service postgresql restart Now try to log in to PostgreSQL again, and there will be no error:\n1 psql -U test Finally, create a new database for our web application:\n1 CREATE DATABASE weather_db; Here weather_db is the name of the database. Then use the following command to exit PostgreSQL:\n1 \\q Use PostgreSQL in Flask # Install psycopg2 and SQLAlchemy # psycopg2 is a Python library for connecting to the PostgreSQL database, and SQLAlchemy is a Python library for operating the database. We need to install these two libraries. First make sure that we have activated the Python virtual environment, and then use the following command:\n1 2 pip install psycopg2-binary pip install SQLAlchemy Create database model # Connect to the PostgreSQL database in the Flask application, use the following code:\n1 2 3 4 5 6 7 from flask import Flask from flask_sqlalchemy import SQLAlchemy app = Flask(__name__) app.config[\u0026#39;SQLALCHEMY_DATABASE_URI\u0026#39;] = \u0026#39;postgresql://test:test_password@localhost:5432/weather_db\u0026#39; app.config[\u0026#39;SQLALCHEMY_TRACK_MODIFICATIONS\u0026#39;] = False db = SQLAlchemy(app) In the above code, we first import the Flask class and the SQLAlchemy class, then create a Flask application, and then use app.config to configure the database connection information, where SQLALCHEMY_DATABASE_URI is the database connection address, SQLALCHEMY_TRACK_MODIFICATIONS is used to turn off the monitoring of model modifications, because this function will consume extra memory, but we don’t need this function. Finally, we create a db object to operate the database.\nNote: The database uses port number 5432 here, which is the default port number of PostgreSQL. If the port number used by your PostgreSQL is not 5432, please change it to your port number. You can view the port number used by PostgreSQL in the configuration file of PostgreSQL. The path of the configuration file is /etc/postgresql/14/main/postgresql.conf, where 14 is the version number of PostgreSQL. If your version number is not 14, please change it to your version number.\nThen we need to create a database model to use SQLAlchemy to operate the database in Flask. We first import the SQLAlchemy library, and then create a db object to operate the database. Then we need to define a Weather class to represent the weather information of the city. The Weather class inherits from the db.Model class, so the Weather class can use the db object to operate the database. The Weather class has several attributes, which are similar to the dictionary we defined before. Finally, we need to use the db.create_all() method to create a database table. This method will create a database table according to the database model we defined. The code is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 db = SQLAlchemy(app) class Weather(db.Model): id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String(50), nullable=False) country = db.Column(db.String(50), nullable=False) temp = db.Column(db.Integer, nullable=False) feels_like = db.Column(db.Integer, nullable=False) icon = db.Column(db.String(50), nullable=False) description = db.Column(db.String(50), nullable=False) db.create_all() Use the database in the application # Previously we used a dictionary to save weather information, and now we change to use the database to save. The code is as follows: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 db.session.add(Weather( name = \u0026#39;London\u0026#39;, country = \u0026#39;UK\u0026#39;, temp = \u0026#39;12\u0026#39;, feels_like = \u0026#39;11\u0026#39;, icon = \u0026#39;10d\u0026#39;, description = \u0026#39;Moderate rain\u0026#39; )) db.session.add(Weather( name = \u0026#39;New York\u0026#39;, country = \u0026#39;US\u0026#39;, temp = \u0026#39;20\u0026#39;, feels_like = \u0026#39;19\u0026#39;, icon = \u0026#39;01d\u0026#39;, description = \u0026#39;Sunny\u0026#39; )) db.session.commit() Then we need to read the weather information from the database and pass it to the template. Note that the HTML template previously received an array of dictionaries as a parameter, and now we need to pass an array of Weather objects, so we need to modify the template: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 \u0026lt;!DOCTYPE html\u0026gt; \u0026lt;html lang=\u0026#34;en\u0026#34;\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;utf-8\u0026#34;\u0026gt; \u0026lt;title\u0026gt;Weather App\u0026lt;/title\u0026gt; \u0026lt;link rel=\u0026#34;stylesheet\u0026#34; href=\u0026#34;{{ url_for(\u0026#39;static\u0026#39;, filename=\u0026#39;style.css\u0026#39;) }}\u0026#34; type=\u0026#34;text/css\u0026#34;\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;section class=\u0026#34;ajax-section\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;container\u0026#34;\u0026gt; \u0026lt;ul class=\u0026#34;cities\u0026#34;\u0026gt; {% for wd in weather_data %} \u0026lt;li class=\u0026#34;city\u0026#34;\u0026gt; \u0026lt;h2 class=\u0026#34;city-name\u0026#34;\u0026gt;{{ wd.name }}, \u0026lt;sup\u0026gt;{{ wd.country }}\u0026lt;/sup\u0026gt;\u0026lt;/h2\u0026gt; \u0026lt;div class=\u0026#34;city-temp\u0026#34;\u0026gt;{{ wd.temp }}\u0026lt;sup\u0026gt;°C\u0026lt;/sup\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;figcaption\u0026gt;{{ wd.description }}\u0026lt;/figcaption\u0026gt; \u0026lt;img class=\u0026#34;city-icon\u0026#34; src=\u0026#34;https://s3-us-west-2.amazonaws.com/s.cdpn.io/162656/{{ wd.icon }}.svg\u0026#34; alt=\u0026#34;{{ wd.description }}\u0026#34;\u0026gt; \u0026lt;/li\u0026gt; {% endfor %} \u0026lt;/ul\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/section\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; The code for getting database data in Python and passing it to the template is as follows: 1 2 cities_data = Weather.query.all() return render_template(\u0026#39;weather.html\u0026#39;, cities_data=cities_data) Here cities_data is a list, and the elements in the list are Weather objects. We can use the attributes of this object in the template to get weather information. Run the web application # Try to run our web application with the following command:\n1 python app.py You may encounter the following error message:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Traceback (most recent call last): File \u0026#34;/home/lijin/Documents/www/learn_flask/app.py\u0026#34;, line 21, in \u0026lt;module\u0026gt; db.create_all() File \u0026#34;/home/lijin/Documents/www/learn_flask/env/lib/python3.11/site-packages/flask_sqlalchemy/extension.py\u0026#34;, line 900, in create_all self._call_for_binds(bind_key, \u0026#34;create_all\u0026#34;) File \u0026#34;/home/lijin/Documents/www/learn_flask/env/lib/python3.11/site-packages/flask_sqlalchemy/extension.py\u0026#34;, line 871, in _call_for_binds engine = self.engines[key] ^^^^^^^^^^^^ File \u0026#34;/home/lijin/Documents/www/learn_flask/env/lib/python3.11/site-packages/flask_sqlalchemy/extension.py\u0026#34;, line 687, in engines app = current_app._get_current_object() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \u0026#34;/home/lijin/Documents/www/learn_flask/env/lib/python3.11/site-packages/werkzeug/local.py\u0026#34;, line 508, in _get_current_object raise RuntimeError(unbound_message) from None RuntimeError: Working outside of application context. This typically means that you attempted to use functionality that needed the current application. To solve this, set up an application context with app.app_context(). See the documentation for more information. This is because our application needs an application context. We need to add the following code to app.py:\n1 2 with app.app_context(): db.create_all() The final code of app.py is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 from flask import Flask, render_template from flask_sqlalchemy import SQLAlchemy from datetime import datetime app = Flask(__name__) app.config[\u0026#39;SQLALCHEMY_DATABASE_URI\u0026#39;] = \u0026#39;postgresql://test:test_password@localhost:5432/weather_db\u0026#39; app.config[\u0026#39;SQLALCHEMY_TRACK_MODIFICATIONS\u0026#39;] = False db = SQLAlchemy(app) class Weather(db.Model): id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String(50), nullable=False) country = db.Column(db.String(50), nullable=False) temp = db.Column(db.Integer, nullable=False) feels_like = db.Column(db.Integer, nullable=False) icon = db.Column(db.String(50), nullable=False) description = db.Column(db.String(50), nullable=False) @app.route(\u0026#39;/\u0026#39;) def index(): db.session.add(Weather( name = \u0026#39;London\u0026#39;, country = \u0026#39;UK\u0026#39;, temp = \u0026#39;12\u0026#39;, feels_like = \u0026#39;11\u0026#39;, icon = \u0026#39;10d\u0026#39;, description = \u0026#39;Moderate rain\u0026#39; )) db.session.add(Weather( name = \u0026#39;New York\u0026#39;, country = \u0026#39;US\u0026#39;, temp = \u0026#39;20\u0026#39;, feels_like = \u0026#39;19\u0026#39;, icon = \u0026#39;01d\u0026#39;, description = \u0026#39;Sunny\u0026#39; )) db.session.commit() wd = Weather.query.all() print(len(wd)) return render_template(\u0026#39;weather.html\u0026#39;, weather_data=wd) if __name__ == \u0026#34;__main__\u0026#34;: with app.app_context(): db.create_all() app.run() Then we run the application again:\n1 python app.py If everything goes well, we can see that our website displays the weather information of two cities in two cards: Note: Every time you run the application, two pieces of data will be added to the database, so if you run the application multiple times, it may cause multiple duplicate data in the database, and multiple duplicate cards will appear on the web page. At this time, we can use the following code to delete duplicate data:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 from sqlalchemy import text sql_text = text( \u0026#34;\u0026#34;\u0026#34;DELETE FROM weather WHERE id IN ( SELECT id FROM ( SELECT id, ROW_NUMBER() OVER (PARTITION BY name) AS row_num FROM weather ) AS numbered_rows WHERE row_num \u0026gt; 1 ); \u0026#34;\u0026#34;\u0026#34;) db.session.execute(sql_text) db.session.commit() Summary # So far, we have successfully created a PostgreSQL database, and then used SQLAlchemy to operate the database in the Flask application. But the data in the database is still manually added one by one, which is very inflexible and inconvenient. In the next article, we will introduce how to get user input, and then use the API to get weather information based on the city entered by the user, and then save the obtained weather information to the database.\n","date":"18 November 2023","externalUrl":null,"permalink":"/en/p/learn-web-development-with-python-2-use-postgresql-and-sqlalchemy/","section":"Posts","summary":" Background # In the previous article “Learn Web Development with Python (1): Use Flask Framework”, we used the Flask framework to develop a simple web application to display weather information for some cities. But these weather information are hard-coded in the code. If we have a lot of weather information, it is not convenient to hard-code them in the code. Therefore, we need a database to store these weather information, and then read these weather information from the database and display them on the web page.\n","title":"Learn Web Development with Python (2): Use PostgreSQL and SQLAlchemy","type":"post"},{"content":"","date":"18 November 2023","externalUrl":null,"permalink":"/en/tags/postgresql/","section":"Tags","summary":"","title":"PostgreSQL","type":"tags"},{"content":"","date":"18 November 2023","externalUrl":null,"permalink":"/en/tags/sqlalchemy/","section":"Tags","summary":"","title":"SQLAlchemy","type":"tags"},{"content":" Background # There are many popular web development languages, such as Java, PHP, Python, Ruby, etc. I know Python best, so I want to use Python to develop web applications. There are many web frameworks in Python, such as Django, Flask, Tornado, etc. These frameworks have their own characteristics. Among them, Flask is a lightweight web framework, so I choose to start learning web development with Flask framework.\nIn this series of articles, I will start from scratch and introduce how to use the Flask framework to develop web applications step by step. This article will introduce how to use the Flask framework to develop a simple web application, including URL parsing and template rendering. This series of articles includes:\nDevelop a simple web application using the Flask framework: URL routing, template rendering (this article) Use PostgreSQL database: create PostgreSQL database, use SQLAlchemy to operate database (see \u0026ldquo;Learn Web Development with Python (2): Use PostgreSQL and SQLAlchemy\u0026rdquo; Data receiving and data requesting: form, get data from third-party web API (see \u0026ldquo;Learn Web Development with Python (3): Use Input and API\u0026rdquo; Deploy web applications using Docker: docker-compose, Nginx, Gunicorn (see \u0026ldquo;Learn Web Development with Python (4): Deploy Website with Docker, Gunicorn and Nginx\u0026rdquo; Introduction to Flask # Flask is a lightweight web framework. Its core is the WSGI toolkit Werkzeug and the template engine Jinja. The core of the Flask framework is the WSGI toolkit Werkzeug, which is a WSGI toolkit that can be used to handle HTTP requests and responses, as well as other tasks related to web applications. The template engine Jinja is a modern and elegant template engine that can be used to generate HTML pages.\nThe first simple web application # We need to create a Flask environment and then write a simple web application.\nCreate a directory for learning Flask development, such as learn_flask, and then enter the directory:\n1 2 mkdir learn_flask cd learn_flask Create a Python virtual environment to install the Flask framework and other dependencies:\n1 2 3 python3 -m venv env # Create a Python virtual environment source env/bin/activate # Activate the Python virtual environment pip install flask # Install the Flask framework Create a Python file to write the Flask application, such as app.py:\n1 2 3 4 5 6 7 8 9 10 from flask import Flask app = Flask(__name__) @app.route(\u0026#39;/\u0026#39;) def index(): return \u0026#39;Hello, World!\u0026#39; if __name__ == \u0026#34;__main__\u0026#34;: app.run() The above code creates a Flask application. When the root URL is accessed, Hello, World! will be returned. Let\u0026rsquo;s analyze this code:\nfrom flask import Flask: Import the Flask framework app = Flask(__name__): Create a Flask application. __name__ is the name of the current module. If the current module is the main module, the value of __name__ is __main__, otherwise the value of __name__ is the name of the current module. @app.route('/'): Register a URL with the app.route() decorator. Here the root URL, /, is registered. route() is a routing function. When a user accesses a URL, the routing function will call the corresponding Python function according to the URL request. def index():: Define a Python function to handle the request of the root URL. return 'Hello, World!': When a user accesses the root URL, the index() function will be called, which will return Hello, World!. if __name__ == \u0026quot;__main__\u0026quot;:: If the current module is the main module, execute app.run() to start the Flask application. Run the Flask application:\n1 python app.py The Flask application runs on port 5000 by default, so you can access http://localhost:5000/ in your browser to see Hello, World!: So far, we have created a simple Flask application. But now our application can only return a string, which is obviously not enough. In actual applications, we often need to return an HTML page, which requires the use of the template engine Jinja.\nUse template # In the following example, we use this application to display some actual information, such as the weather in a city. We need to create an HTML page to display the weather information, then pass the weather information to the HTML page, and finally return the HTML page to the user.\nTemplate file # Create a templates directory under the learn_flask directory, and then create a weather.html file under the templates directory. The content of the weather.html file is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 \u0026lt;!DOCTYPE html\u0026gt; \u0026lt;html lang=\u0026#34;en\u0026#34;\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;utf-8\u0026#34;\u0026gt; \u0026lt;title\u0026gt;Weather App\u0026lt;/title\u0026gt; \u0026lt;link rel=\u0026#34;stylesheet\u0026#34; href=\u0026#34;{{ url_for(\u0026#39;static\u0026#39;, filename=\u0026#39;style.css\u0026#39;) }}\u0026#34; type=\u0026#34;text/css\u0026#34;\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;section class=\u0026#34;ajax-section\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;container\u0026#34;\u0026gt; \u0026lt;ul class=\u0026#34;cities\u0026#34;\u0026gt; {% for city in cities_data %} \u0026lt;li class=\u0026#34;city\u0026#34;\u0026gt; \u0026lt;h2 class=\u0026#34;city-name\u0026#34;\u0026gt;{{ city.name }}, \u0026lt;sup\u0026gt;{{ city.country }}\u0026lt;/sup\u0026gt;\u0026lt;/h2\u0026gt; \u0026lt;div class=\u0026#34;city-temp\u0026#34;\u0026gt;{{ city.temp }}\u0026lt;sup\u0026gt;°C\u0026lt;/sup\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;figcaption\u0026gt;{{ city.description }}\u0026lt;/figcaption\u0026gt; \u0026lt;img class=\u0026#34;city-icon\u0026#34; src=\u0026#34;{{ city.icon }}\u0026#34; alt=\u0026#34;{{ city.description }}\u0026#34;\u0026gt; \u0026lt;/li\u0026gt; {% endfor %} \u0026lt;/ul\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/section\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; Create a static directory under the learn_flask directory, and then create a style.css file under the static directory. The style of the card that displays the weather information comes from https://webdesign.tutsplus.com/build-a-simple-weather-app-with-vanilla-javascript\u0026ndash;cms-33893t, as shown below:\nPass parameters to the template # In the weather.html file, we are ready to receive a cities_data variable, and then display the weather information in the city. The cities_data variable is a dictionary. The key of the dictionary is the city name, and the value of the dictionary is a list. The elements in the list are dictionaries. The dictionary contains weather information.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 from flask import Flask, render_template app = Flask(__name__) class WeatherData: def __init__(self, data): self.name = data[\u0026#39;name\u0026#39;] self.country = data[\u0026#39;country\u0026#39;] self.temp = data[\u0026#39;temp\u0026#39;] self.feels_like = data[\u0026#39;feels_like\u0026#39;] self.icon = f\u0026#39;https://s3-us-west-2.amazonaws.com/s.cdpn.io/162656/{data[\u0026#34;icon\u0026#34;]}.svg\u0026#39; self.description = data[\u0026#39;description\u0026#39;] @app.route(\u0026#39;/\u0026#39;) def index(): cities_data = [] wd1 = WeatherData({ \u0026#39;name\u0026#39;: \u0026#39;London\u0026#39;, \u0026#39;country\u0026#39;: \u0026#39;UK\u0026#39;, \u0026#39;temp\u0026#39;: \u0026#39;12\u0026#39;, \u0026#39;feels_like\u0026#39;: \u0026#39;11\u0026#39;, \u0026#39;icon\u0026#39;: \u0026#39;10d\u0026#39;, \u0026#39;description\u0026#39;: \u0026#39;Moderate rain\u0026#39; }) cities_data.append(wd1) wd2 = WeatherData({ \u0026#39;name\u0026#39;: \u0026#39;New York\u0026#39;, \u0026#39;country\u0026#39;: \u0026#39;US\u0026#39;, \u0026#39;temp\u0026#39;: \u0026#39;20\u0026#39;, \u0026#39;feels_like\u0026#39;: \u0026#39;19\u0026#39;, \u0026#39;icon\u0026#39;: \u0026#39;01d\u0026#39;, \u0026#39;description\u0026#39;: \u0026#39;Sunny\u0026#39; }) cities_data.append(wd2) return render_template(\u0026#39;weather.html\u0026#39;, cities_data=cities_data) if __name__ == \u0026#34;__main__\u0026#34;: app.run() Run the application # Run the application again:\n1 python app.py Access http://localhost:5000/ in the browser, and you can see that our website displays the weather information of two cities in two cards: ","date":"17 November 2023","externalUrl":null,"permalink":"/en/p/learn-web-development-with-python-1use-flask-framework/","section":"Posts","summary":" Background # There are many popular web development languages, such as Java, PHP, Python, Ruby, etc. I know Python best, so I want to use Python to develop web applications. There are many web frameworks in Python, such as Django, Flask, Tornado, etc. These frameworks have their own characteristics. Among them, Flask is a lightweight web framework, so I choose to start learning web development with Flask framework.\n","title":"Learn Web Development with Python (1)：Use Flask Framework","type":"post"},{"content":" Background # Recently, I have been learning to use SQL language, mainly to do LeetCode problems. But since I am a beginner, I am not very familiar with SQL databases, so I often need to debug the SQL statements I write, which requires a MySQL database that I can connect to. Most of the tutorials online are about installing MySQL on Windows, but my usual development environment is to connect to a remote Ubuntu server using VSCode on my MacBook, so I want to install MySQL on Ubuntu and be able to connect to it using VSCode remote.\nI have also seen some tutorials on installing and using MySQL on Ubuntu, but most of them are quite outdated, and I have tried many of them and encountered various problems. After a lot of trial and error, I finally configured a MySQL development and testing environment that I can use more comfortably, and I will record it here.\nSQL Introduction # SQL (Structured Query Language) is an abbreviation for Structured Query Language, which is a language used for database operations, and is a standard database language used to access and operate database systems. SQL language can be used to access and operate data in a database, as well as to create and modify tables and views in a database. SQL language is a standard database language, and different database systems support SQL language, but different database systems have different levels of support for SQL language, so when using SQL language, you need to pay attention to the differences in database systems. Currently, commonly used database systems include MySQL, Oracle, SQL Server, PostgreSQL, SQLite, etc.\nMySQL # MySQL is an open-source relational database management system developed by Swedish company MySQL AB, and is currently owned by Oracle. MySQL is one of the most popular relational database management systems, known for its speed, ease of installation, small size, and low overall cost. MySQL has a paid commercial version, but here we only use the free version for learning database knowledge. If you need a commercial version and have a large amount of data, it is recommended to use Oracle database, which is much faster than MySQL.\nInstalling MySQL # Installing MySQL on Ubuntu 22 is very simple, just run the following command in the terminal:\n1 sudo apt install mysql-server Changing MySQL Account Settings # After installation, only the super user can access the default account, which is very inconvenient. Therefore, we need to modify the MySQL account settings, create a normal account, and then allow us to connect to this database remotely using VS Code.\nFirst log in to MySQL as a super user: 1 sudo mysql -u root Create a new MySQL account: 1 2 3 CREATE User \u0026#39;test\u0026#39;@\u0026#39;localhost\u0026#39; IDENTIFIED BY \u0026#39;\u0026#39;; GRANT ALL ON *.* TO \u0026#39;test\u0026#39;@\u0026#39;localhost\u0026#39;; FLUSH PRIVILEGES; Here, test is the account name, localhost is the host name allowed to access, *.* means allowing access to all databases and tables, and WITH GRANT OPTION means allowing this account to create new accounts. Then enter exit to exit MySQL. Note: The password here is empty. Because setting a password can also connect to MySQL, but sometimes when running SQL code in VS Code, an error \u0026ldquo;ACCESS DENIED\u0026rdquo; may occur, so no password is set here. Of course, sometimes it can run successfully, and I don\u0026rsquo;t understand why. This time log in to MySQL as a normal user, specifying the username as test: 1 mysql -h 127.0.0.1 -P 3306 -u test Then run the following SQL statement to view the current user information: 1 SELECT User, Host FROM mysql.user; If everything went well, you should see output similar to the following: 1 2 3 4 5 6 7 8 9 10 11 mysql\u0026gt; SELECT User,Host FROM mysql.user; +------------------+-----------+ | User | Host | +------------------+-----------+ | debian-sys-maint | localhost | | mysql.infoschema | localhost | | mysql.session | localhost | | mysql.sys | localhost | | test | localhost | +------------------+-----------+ 5 rows in set (0.01 sec) VS Code # Installing MySQL Plugin # To connect to the MySQL database using VSCode remote, you need to install a MySQL plugin MySQL:\nwhere Host is the hostname of the MySQL database, Port is the port number of the MySQL database 3306, User is the username of the MySQL database test, and Password is the password of the MySQL database. If everything goes well, you can connect to the MySQL database.\nDebugging SQL Code # After connecting to the database, create a new file with the suffix .sql, and then enter the SQL code in the file. The Active Connection will be displayed at the top of the file, indicating that the MySQL database has been successfully connected.\nThere is also an Execute button in front of each line of SQL code, click this button to execute the SQL code on this line. Click the plugin button in the sidebar to see the real-time updated content in the database:\n","date":"12 October 2023","externalUrl":null,"permalink":"/en/p/configuring-sql-development-and-testing-environment-mysql-vscode-on-ubuntu-22/","section":"Posts","summary":" Background # Recently, I have been learning to use SQL language, mainly to do LeetCode problems. But since I am a beginner, I am not very familiar with SQL databases, so I often need to debug the SQL statements I write, which requires a MySQL database that I can connect to. Most of the tutorials online are about installing MySQL on Windows, but my usual development environment is to connect to a remote Ubuntu server using VSCode on my MacBook, so I want to install MySQL on Ubuntu and be able to connect to it using VSCode remote.\n","title":"Configuring SQL Development and Testing Environment (MySQL+VSCode) on Ubuntu 22","type":"post"},{"content":"","date":"12 October 2023","externalUrl":null,"permalink":"/en/tags/sql/","section":"Tags","summary":"","title":"SQL","type":"tags"},{"content":"","date":"12 October 2023","externalUrl":null,"permalink":"/en/tags/ubuntu/","section":"Tags","summary":"","title":"Ubuntu","type":"tags"},{"content":"","date":"2023年10月12日","externalUrl":null,"permalink":"/tags/%E6%95%B0%E6%8D%AE%E5%BA%93/","section":"Tags","summary":"","title":"数据库","type":"tags"},{"content":"","date":"16 September 2023","externalUrl":null,"permalink":"/en/tags/code-repository/","section":"Tags","summary":"","title":"Code Repository","type":"tags"},{"content":"","date":"16 September 2023","externalUrl":null,"permalink":"/en/tags/hostinger/","section":"Tags","summary":"","title":"Hostinger","type":"tags"},{"content":" Motivation # Hostinger is a network custody service provider, providing services such as domain name registration, virtual hosting, VPS and other services. Recently, a domain name was started on Hostinger, and intended to migrate the previous website to the new domain name. I have been using acme.sh to generate an SSL certificate for my website. But hostinger does not provide APIs that can be retrieved by acme.sh, so it is impossible to use acme.sh to generate SSL certificates for domain names on hostinger. Cloudflare is a service provider with content distribution network based on reverse agents, also provides domain name analysis services, and provides API available in acme.sh. So I want to migrate the domain name service provider from Hostinger to Cloudflare.\nMigrate Domain Name # Create a Cloudflare Account # First of all, you need to create a Cloudflare account. Just create it directly. There are not many special requirements for the creation process.\nAdd Domain Name # After logging in to the cloudflare account, enter the homepage, click add a site on the homepage, enter the domain name, and click Add Site.\nThen you need to choose a package. The free package here is enough for me. Click Confirm Plan, and then click Confirm.\nNext, Cloudflare scan the DNS record of the domain name, position the domain name resolution server used before your domain name, and then Cloudflare will prompt you to change the domain name analysis server to the cloudflare domain analysis server.\nHere you need to log in to the account of the previous domain name service provider. For example, I need to log in to the account of the hostinger and change the domain name parsing server to the cloudflare domain name analysis server.Select the domain name you need to migrate, the status of the domain name, and the name of the domain name analysis server, click CHANGE, and change the domain name parsing server to the cloudflare domain name parsing server according to the prompt given by Cloudflare in the previous step.\nAfter changing the change, click Done, Check NameServers.Then you need to wait for a period of time to make Cloudflare confirm that the domain name parsing server has been changed to Cloudflare\u0026rsquo;s domain name analysis server. This process may take several minutes to a few hours. Cloudflare will send an email after confirmation.Essence\nAdd DNS Records # On the homepage of Cloudflare, click DNS, and then click add record to add a DNS record.\nAdd all the DNS records previously on the Hostinger to the Cloudflare.\nMigrate Websites # Generate SSL Certificate # Here we use acme.sh to generate SSL certificates on the website. For installation and use of acme.sh How This Website Was Built 2 \u0026ndash; Hugo Framework for Personal Blog Website。\nIt should be noted that the method of generating SSL certificates introduced in the link above is for Alibaba Cloud, and Cloudflare\u0026rsquo;s API and Alibaba Cloud\u0026rsquo;s APIs are not the same. They need to use different account passwords, which are account names (CF_ACCOUNT_ID)., Regional name (CF_ZONE_ID) and passwords (CF_TOKEN).These can be found in the cloudflare account.For the specific process, please refer to acme.sh\u0026rsquo;s Document.\nHere I re-generated the SSL certificates for all websites.\nChange the nginx Reverse Proxy Configuration # The previous reverse proxy configuration files were placed in the /etc/nginx/vhost directory. Here I replace all the old domain names in all configuration files into a new domain name. Note that if the SSL certificate is changed in the previous step, the storage of the SSL certificateThe path also needs to change the certificate path in the configuration file to a new path.\nChange the docker-compose Configuration # Finally, the old domain names in the docker-compose configuration file that previously built a website are replaced by the new domain name, and then the Docker-Compose service is restarted.\nSome issues # Cloudflare\u0026rsquo;s SSL/TLS Encryption Settings # After the above settings are done, the error of ERR_TOO_MANY_REDIRECTS appears when accessing the website through the new domain name: This is because SSL/TLS encryption is set to \u0026ldquo;Flexible\u0026rdquo; in CloudFlare. Here you need to set SSL/TLS encryption to \u0026ldquo;Full\u0026rdquo;: In this way, the previous website can be accessed through the new domain name!\n","date":"16 September 2023","externalUrl":null,"permalink":"/en/p/migrate-the-domain-name-service-provider-from-hostinger-to-cloudflare/","section":"Posts","summary":" Motivation # Hostinger is a network custody service provider, providing services such as domain name registration, virtual hosting, VPS and other services. Recently, a domain name was started on Hostinger, and intended to migrate the previous website to the new domain name. I have been using acme.sh to generate an SSL certificate for my website. But hostinger does not provide APIs that can be retrieved by acme.sh, so it is impossible to use acme.sh to generate SSL certificates for domain names on hostinger. Cloudflare is a service provider with content distribution network based on reverse agents, also provides domain name analysis services, and provides API available in acme.sh. So I want to migrate the domain name service provider from Hostinger to Cloudflare.\n","title":"Migrate the Domain Name Service Provider from Hostinger to Cloudflare","type":"post"},{"content":" Motivation # The previous code was basically placed on GitHub, but because some codes involve some scientific research projects, even if they are placed in the private repository, there may be potential security issues, so I want to build a private code repository and put them on my own computer or server.\nPrerequisites # docker和docker compose Domain name (second-level domain name is sufficient) acme.sh (Generate SSL certificate) nginx (Reverse proxy) Preparation # Domain Name and SSL Certificate # For the installation and preparation of the above software and tools, please refer to \u0026ldquo;Building a Personal Cloud Network Disk using Nextcloud and Docker\u0026rdquo;\nInstall GitLab # Use docker-compose to install Jellyfin. The specific operation refer to https://docs.gitlab.com/ee/install/docker.html.HTML. Create the following docker-compose.yml file under the installation directory/media/gitlab file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 version: \u0026#39;3.6\u0026#39; services: web: image: \u0026#39;gitlab/gitlab-ee:latest\u0026#39; restart: always hostname: \u0026#39;git.jinli.io\u0026#39; environment: GITLAB_OMNIBUS_CONFIG: | external_url \u0026#39;http://git.jinli.io\u0026#39; gitlab_rails[\u0026#39;gitlab_shell_ssh_port\u0026#39;] = 22 ports: - \u0026#39;1080:80\u0026#39; - \u0026#39;1443:443\u0026#39; - \u0026#39;1022:22\u0026#39; volumes: - \u0026#39;./config:/etc/gitlab\u0026#39; - \u0026#39;./logs:/var/log/gitlab\u0026#39; - \u0026#39;./data:/var/opt/gitlab\u0026#39; shm_size: \u0026#39;256m\u0026#39; Then execute the following command to start the GitLab service:\n1 sudo docker-compose up -d Website Initialization and Settings # If all the above configurations are successfully completed, you can access the website through the browser, you can enter http: //0.0.0.0: 8080 in the address bar, or use the domain name https: // git.jinli.cyouaccess. If the website has not yet been initialized, you will see the prompt 502 Bad Gateway when you visit the browser, you need to wait patiently for a few minutes until the website is initialized.\nWhen you visit the website for the first time, you need to set up an administrator account and password, and then you can create a common account for daily use.\nConfiguration System Notification Mailbox # After the above configuration, gitlab can be used normally.However, we also want to use the system notification function, such as sending the system to notify the email, the email verification when registering the account, etc., we need to configure the system to notify the mailbox. Here I use my own mailbox server. The specific configuration is as follows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 ### GitLab email server settings ###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html ###! **Use smtp instead of sendmail/postfix.** gitlab_rails[\u0026#39;smtp_enable\u0026#39;] = true gitlab_rails[\u0026#39;smtp_address\u0026#39;] = \u0026#34;mail.jinli.cyou\u0026#34; gitlab_rails[\u0026#39;smtp_port\u0026#39;] = 587 gitlab_rails[\u0026#39;smtp_user_name\u0026#39;] = \u0026#34;mailer@jinli.cyou\u0026#34; gitlab_rails[\u0026#39;smtp_password\u0026#39;] = \u0026#34;your_email_password\u0026#34; gitlab_rails[\u0026#39;smtp_domain\u0026#39;] = \u0026#34;mail.jinli.cyou\u0026#34; gitlab_rails[\u0026#39;smtp_authentication\u0026#39;] = \u0026#34;login\u0026#34; gitlab_rails[\u0026#39;smtp_enable_starttls_auto\u0026#39;] = true gitlab_rails[\u0026#39;smtp_tls\u0026#39;] = false # gitlab_rails[\u0026#39;smtp_pool\u0026#39;] = false ###! **Can be: \u0026#39;none\u0026#39;, \u0026#39;peer\u0026#39;, \u0026#39;client_once\u0026#39;, \u0026#39;fail_if_no_peer_cert\u0026#39;** ###! Docs: http://api.rubyonrails.org/classes/ActionMailer/Base.html gitlab_rails[\u0026#39;smtp_openssl_verify_mode\u0026#39;] = \u0026#39;none\u0026#39; # gitlab_rails[\u0026#39;smtp_ca_path\u0026#39;] = \u0026#34;/etc/ssl/certs\u0026#34; # gitlab_rails[\u0026#39;smtp_ca_file\u0026#39;] = \u0026#34;/etc/ssl/certs/ca-certificates.crt\u0026#34; ### Email Settings # gitlab_rails[\u0026#39;gitlab_email_enabled\u0026#39;] = true ##! If your SMTP server does not like the default \u0026#39;From: gitlab@gitlab.example.com\u0026#39; ##! can change the \u0026#39;From\u0026#39; with this setting. gitlab_rails[\u0026#39;gitlab_email_from\u0026#39;] = \u0026#39;mailer@jinli.cyou\u0026#39; gitlab_rails[\u0026#39;gitlab_email_display_name\u0026#39;] = \u0026#39;GitLab Mailer\u0026#39; gitlab_rails[\u0026#39;gitlab_email_reply_to\u0026#39;] = \u0026#39;mailer@jinli.cyou\u0026#39; # gitlab_rails[\u0026#39;gitlab_email_subject_suffix\u0026#39;] = \u0026#39;\u0026#39; # gitlab_rails[\u0026#39;gitlab_email_smime_enabled\u0026#39;] = false # gitlab_rails[\u0026#39;gitlab_email_smime_key_file\u0026#39;] = \u0026#39;/etc/gitlab/ssl/gitlab_smime.key\u0026#39; # gitlab_rails[\u0026#39;gitlab_email_smime_cert_file\u0026#39;] = \u0026#39;/etc/gitlab/ssl/gitlab_smime.crt\u0026#39; # gitlab_rails[\u0026#39;gitlab_email_smime_ca_certs_file\u0026#39;] = \u0026#39;/etc/gitlab/ssl/gitlab_smime_cas.crt\u0026#39; In this way, our own Gitlab server also has a systematic notification function.\n","date":"16 September 2023","externalUrl":null,"permalink":"/en/p/use-gitlab-and-docker-to-build-a-personal-code-repository/","section":"Posts","summary":" Motivation # The previous code was basically placed on GitHub, but because some codes involve some scientific research projects, even if they are placed in the private repository, there may be potential security issues, so I want to build a private code repository and put them on my own computer or server.\nPrerequisites # docker和docker compose Domain name (second-level domain name is sufficient) acme.sh (Generate SSL certificate) nginx (Reverse proxy) Preparation # Domain Name and SSL Certificate # For the installation and preparation of the above software and tools, please refer to “Building a Personal Cloud Network Disk using Nextcloud and Docker”\n","title":"Use Gitlab and docker to Build a Personal Code Repository","type":"post"},{"content":"","date":"2 September 2023","externalUrl":null,"permalink":"/en/tags/almeida-cv/","section":"Tags","summary":"","title":"Almeida-Cv","type":"tags"},{"content":"","date":"2 September 2023","externalUrl":null,"permalink":"/en/tags/cv-template/","section":"Tags","summary":"","title":"CV Template","type":"tags"},{"content":"","date":"2 September 2023","externalUrl":null,"permalink":"/en/tags/hugo-theme-stack/","section":"Tags","summary":"","title":"Hugo-Theme-Stack","type":"tags"},{"content":"","date":"2023年9月2日","externalUrl":null,"permalink":"/tags/hugo%E4%B8%BB%E9%A2%98/","section":"Tags","summary":"","title":"Hugo主题","type":"tags"},{"content":" Motivation # When setting up this website, I used the hugo theme [hugo-theme-stack] (https://github.com/caijimmy/hugo-theme-stack), you can add personal resumes on the `page.However, the style of this page is not the style of the traditional resume, but the style of a blog post, which is not in line with my requirements. My requirements for the resume template are:\nThe style of the traditional resume, don’t spend a lot of bind Can be exported to PDF format The size of the size is the size of A4 paper or the size of Letter paper, so that the size of the traditional resume is the same So I found a resume template that roughly met the requirements on the Internet.The themes are all used, just want to use its resume template. After checking on the Internet, I found [some information] (https://stackoverflow.com/questions/59208233/Hugo-theme-within-another-theme) showing that there is indeed a way to use two Hugo themes on the same website.\nImplementation # Download the Themes # We need to download two themes, one is [hugo-theme-stack] (https://github.com/caijimmy/hugo-thememe-stack)Ineesalmeida/Almeida-CV).Since I have made some modifications on the Hugo-theme-Stack, I fork Jimmy Cai\u0026rsquo;s warehouse on Github. My modified warehouse is [hugo-theme-stack] (https://github.com/Jin-Li/Hugo-Theme-Stack).\nEnter the root directory of the website, download these two themes, and put them in the Themes folder:\n1 2 git clone https://github.com/jin-li/hugo-theme-stack themes/hugo-theme-stack git clone https://github.com/ineesalmeida/almeida-cv themes/almeida-cv Some Modifications # Modify the Configuration File. First of all, we need to specify two themes in Config/_default/Config.yaml:\n1 2 3 theme: - hugo-theme-stack - almeida-cv Then merge the settings in the Almeida-CV theme configuration file to the config/_default/config.yaml:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 enableRobotsTXT: true enableEmoji: true params: enableMetaTags: true colorLight: \u0026#39;#fff\u0026#39; colorDark: \u0026#39;#666\u0026#39; colorPageBackground: \u0026#39;#ddd\u0026#39; colorPrimary: \u0026#39;#e3bfb8\u0026#39; colorSecondary: \u0026#39;#aaa\u0026#39; colorIconPrimary: \u0026#39;#fff\u0026#39; colorIconBackground: \u0026#39;#e3bfb8\u0026#39; colorRightColumnBackground: \u0026#39;#f5f5f5\u0026#39; colorRightColumnHeadingText: \u0026#39;#666\u0026#39; colorRightColumnBodyText: \u0026#39;#666\u0026#39; colorRightColumnIconPrimary: \u0026#39;#fff\u0026#39; colorRightColumnIconBackground: \u0026#39;#e3bfb8\u0026#39; pages: 1 swapColumns: false New CV Layout.In the theme of Almeida-CV, the style of the resume is defined in the layouts/index.html. We copy the index.html file to the layouts/_default folder, and rename it as cv.html:\n1 cp themes/almeida-cv/layouts/index.html themes/almeida-cv/layouts/_default/cv.html Then we delete the Asseests/SCSS/_custom.scss file in the Almeida-CV theme.This file was originally an empty file, but it will cover the asseests/scss/custom.scss file in the hugo-theme-Stack theme, which causes our configuration in Hugo-theme-Stack to be invalid. So we need to delete it.\nCopy Static Files. We need to copy the template static file provided in the Almeida-CV theme to copy the template static file provided in the website to the root directory of the website. There are two files that need to be followed: -exampleSite/data/content.yaml file, this file defines the content of the resume, we need to copy it to the data folder in the website root directory.After that, we can modify the content in this file to generate our own resume. -exampleSite/static/img/avatar.jpg file, this file is the avatar in the resume.Since we actually have a avatar in Hugo-Theme-Stack, we can just re-specify the path of the avatar without using this file.For example, the avatar I used in Hugo-Theme-Stack is stored in the static/favicon.png under the root directory of the website, so you only need to change the path of the avatar in the path of the avatar in the data/content.yaml.\nModify the About Page. I want to put the resume on the website \u0026ldquo;About\u0026rdquo; page, so I need to modify the \u0026ldquo;About\u0026rdquo; page defined by the hugo-theme-stack. The \u0026ldquo;About\u0026rdquo; page of the website is defined by the content/page/about/index.md file of the website root directory. We need to specify the cv.html template defined in the second step,Therefore, the content of the file in the index.md file is:\n1 2 3 4 5 6 7 8 9 10 11 --- title: \u0026#34;关于\u0026#34; date: 2023-08-22 layout: \u0026#34;cv\u0026#34; slug: \u0026#34;about\u0026#34; menu: main: weight: -70 params: icon: archives --- In the Markdown file, you only need to tell the Hugo to generate page to generate pages based on the style of cv, because the content of the resume is already defined in the data/content.yaml.\nSome Additional Modifications # In fact, with the above modifications, the \u0026ldquo;About\u0026rdquo; page of the website has become a resume of Almeida-CV theme, but I have made some additional modifications. The style of the resume is more in line with my requirements.The main modifications include:\nModify params: swapColumns the file config/_default/config.yaml to true, so that the left and right columns in the resume will exchange positions. Modify the style in the resume. Modify the semi -transparent watermark in the resume. Modify the copyright information at the bottom of the website page and add copyright information of the Almeida-CV theme. No longer repeated the modification of the code here. If you are interested, you can check my GitHub repository https://github.com/jin-li/almeida-cv.\nFinal Effect # For the final effect, please refer to the \u0026ldquo;About\u0026rdquo; page https://jinli.cyou/bout/ on this website.\n","date":"2 September 2023","externalUrl":null,"permalink":"/en/p/merge-two-hugo-themes-use-the-resume-template-of-almeida-cv-theme-in-hugo-theme-stack/","section":"Posts","summary":" Motivation # When setting up this website, I used the hugo theme [hugo-theme-stack] (https://github.com/caijimmy/hugo-theme-stack), you can add personal resumes on the `page.However, the style of this page is not the style of the traditional resume, but the style of a blog post, which is not in line with my requirements. My requirements for the resume template are:\n","title":"Merge Two Hugo Themes: Use the Resume Template of almeida-CV Theme in hugo-theme-Stack","type":"post"},{"content":"","date":"2 September 2023","externalUrl":null,"permalink":"/en/tags/resume-template/","section":"Tags","summary":"","title":"Resume Template","type":"tags"},{"content":"","date":"2023年9月2日","externalUrl":null,"permalink":"/tags/%E7%AE%80%E5%8E%86%E6%A8%A1%E6%9D%BF/","section":"Tags","summary":"","title":"简历模板","type":"tags"},{"content":"","date":"31 August 2023","externalUrl":null,"permalink":"/en/tags/jellyfin/","section":"Tags","summary":"","title":"Jellyfin","type":"tags"},{"content":"","date":"31 August 2023","externalUrl":null,"permalink":"/en/tags/media-server/","section":"Tags","summary":"","title":"Media Server","type":"tags"},{"content":"","date":"31 August 2023","externalUrl":null,"permalink":"/en/tags/personal-media-library/","section":"Tags","summary":"","title":"Personal Media Library","type":"tags"},{"content":" Motivation # I have downloaded a lot of film and television resources before it is placed in the hard disk of the host at home, but it is not convenient to watch it on other places or other equipment.So think of building an online film and television library, you can watch your film and television resources through the Internet anywhere. I saw a lot of methods to build online film and television libraries on the Internet, such as using plex, emby, jellyfin etc. These software are open source and can be built on your own computer or server, and then watch your film and television resources through web or clients.\nMany advanced features of [Plex] (https://www.plex.tv/) and [Emby] (https://emby.Media/) need to be paid to be available, and [Jellyfin] (https: ///Jellyfin.org/) is free of charge. Here I choose to use [jellyfin] (https://jellyfin.org/) to build a personal online film and television library.\nPrerequisites # docker and docker compose Domain name (second-level domain name is OK) acme.sh (generate SSL certificate) nginx (reverse proxy) Preparation # Prepare domain name # For the installation and preparation of the above software and tools, please refer to \u0026ldquo;Building a Personal Cloud Network Disk using Nextcloud and Docker\u0026rdquo;\nInstall Jellyfin # Here we use the docker- compose to install Jellyfin. The specific operation refer to [https://hub.docker.com/r/linuxserver/jellyfin] (https://hub.docker.com/r/linuxServer/jellyfin).\nThe parameters in docker-compose.yml that can be modified are as follows:\nTZ: Time zone ports: Port mapping volumes: Volume mapping Here I choose to put all Jellyfin\u0026rsquo;s configuration files in the Config folder in Jellyfin installation directory.Film and television files are stored in the original hard disk. Here, the folder (such as /Media/Disk8t) that stores film and television files is mapped into the/data/media folder in Jellyfin containers. Then the specific settings about Volumes are set in the docker-compose.yml file as shown below:\n1 2 3 4 5 volumes: - ./config:/config - type: bind source: /media/Disk8T target: /data/media Initialization and Settings # After installing Jellyfin, you can access the website through the browser, and you can enter http: //0.0.0.0: 8081 in the address bar, if Nginx has been set up, or you can also access it through the URL: http:// m.jinli.cyou.\nYou need to set up an administrator account and password for the first time, and then you can enter the management interface of Jellyfin.\nCreate a Media Library # Then you can create a film and television library to add film and television resources stored on the hard disk to the film and television library.Since we mapped the folders that stored film and television resources to the /data/Media folder when creating the Docker container, we can create different folders in the/Media/Disk8t, and then create the corresponding in Jellyfin.Film and television library. For example, we created a folder named Movies and a folder called Movies and a folder in the /Media/Disk8t, then we can create a film and television library called my movies and a film called My Music\u0026rsquo;s music library, and then specify the source path of /data/Media/Movies as the source path of Movies, specify the source path of the source path of `/data/Media/Music/Song.\nThe method of creating a film and television library is:\nLog in to Jellyfin\u0026rsquo;s administrator account, click the icon of the three horizontal lines in the upper left corner, expand the menu bar, and then click dashboard. Click the libraries in the menu bar on the left, and then click the add library in the upper right corner. Select the type of film and television library, such as music, and then click Next. Enter the name of the film and television library, for example, My Music, and then specify the source path of the film and television library, such as/data/Media/Music/Song, and then click OK. Install Plugins # Jellyfin has a lot of useful plug -ins, you can install plugins in the plugins of dashboard.\nYou need to add the plug -in warehouse address to the plug -in warehouse list of Jellyfin before installing the plug -in. Here are two plug -in warehouses:\nhttps://repo.jellyfin.org/releases/plugin/manifest-stable.json https://gitee.com/jjm2473/jellyfin-plugins-repo/raw/master/manifest.json It is recommended to install the following plugins: Among them, TMDB and OMDB plugins can be used to obtain information about film and television, such as posters, profiles, etc., and Meiam.Shooter and Meiam.thunder plug -in can be used to obtain the subtitles of film and television.\nAfter installing the plug -in, you need to restart the Jellyfin container to take effect.\nSolve the problem of garbled code # When the system and docker container lack the font files specified in the subtitle file, some Chinese fonts will display squares. To completely solve this problem, you need to install Chinese fonts in the docker container and set up a backup font for Jellyfin. After the installation is completed, you need to restart the Jellyfin container to make the font take effect.\nSet up a backup font for Jellyfin # The spare fonts in Jellyfin need to use the font of the woff2 format. Here we choose [Microsoft Yahei] (https://www.onLineWebfonts.com/download/e636534076698F5B0EB9BBDC175F77) and [NOTO SANS] (HT TPS: //www.onlinewebfonts.com/download/04219e2F2010127AAA13bb87E6DC50EC) font.\nAfter downloading the font file, you need to put it in the Config folder in the Jellyfin installation directory, and then set the backup font in the Jellyfin\u0026rsquo;s management interface. The specific operation is as follows:\nCreate a folder called Fonts in the Config folder in Jellyfin installation directory, and then place the downloaded font file in the Fonts folder. Enter the Jellyfin setting interface, click Playback on the menu bar on the left, set the Fallback Font Folder Path as /config/fonts. Install Chinese fonts in docker container # First enter the Jellyfin container:\n1 sudo docker exec -it jellyfin /bin/bash Then install the Chinese font:\n1 2 apt update apt install fonts-noto-cjk-extra Install Chinese fonts for the system # If you need to play videos and display subtitles locally on the server computer, it is best to install a Chinese font for the server system.\nThis is relatively simple, just download the Chinese font file in the TFF format, and then double -click to install it.It is recommended to install [Siyuan Black Body] (https://github.com/tellarcn/scp_zh/blob/master/fonts/simheii.ttf).\n","date":"31 August 2023","externalUrl":null,"permalink":"/en/p/use-jellyfin-and-docker-to-build-a-personal-online-media-library/","section":"Posts","summary":" Motivation # I have downloaded a lot of film and television resources before it is placed in the hard disk of the host at home, but it is not convenient to watch it on other places or other equipment.So think of building an online film and television library, you can watch your film and television resources through the Internet anywhere. I saw a lot of methods to build online film and television libraries on the Internet, such as using plex, emby, jellyfin etc. These software are open source and can be built on your own computer or server, and then watch your film and television resources through web or clients.\n","title":"Use Jellyfin and docker to Build a Personal Online Media Library","type":"post"},{"content":" Motivation # I\u0026rsquo;ve previously downloaded some eBooks, all of which are stored on the cloud drive. However, every time I want to read them on different devices, I need to download them first (even though they can be previewed on the cloud drive, I still find it better to download them and use a dedicated e-reader for a better experience). Moreover, with a growing number of eBooks, managing them on the cloud drive becomes quite chaotic. Recently, I came across the tool called \u0026ldquo;calibre-web,\u0026rdquo; which allows you to set up your own eBook library. I feel that the results are quite promising.\nPrerequisites # docker and docker-compose Domain name (second-level domain is sufficient) acme.sh (generating SSL certificate) nginx (reverse proxy) Preparation # Install Docker and Docker Compose # Debian-based systems can directly install docker using apt:\n1 sudo apt-get install docker-ce Install Docker Compose:\n1 sudo apt-get install docker-compose Docker Compose uses a file named docker-compose.yml to \u0026ldquo;synthesize\u0026rdquo; a docker container. docker-compose.yml is the formula for synthesizing this docker container.\nPrepare Domain Name # Create a second-level domain on the website where you purchased your domain name or on your domain management website, and configure domain name resolution. For specific instructions, please refer to \u0026ldquo;Building a Personal Website (Part One): Purchasing a Personal Domain Name and Configuring Dynamic Domain Name Resolution\u0026rdquo;.\nFor example, if you have a primary domain name jinli.cyou, you can create a second-level domain \u0026ldquo;book.jinli.cyou\u0026rdquo; for your personal cloud network disk. Then, bind the domain name and IP address on the Alibaba Cloud domain management platform.\nUse acme.sh to Generate SSL Certificates # Many online tutorials use Let\u0026rsquo;s Encrypt to generate SSL certificates, but since I have already downloaded the acme.sh tool when building this website, I will use acme.sh to generate a certificate for \u0026ldquo;cloud.jinli.cyou\u0026rdquo;. For specific instructions, please refer to \u0026ldquo;Building a Personal Website (Part Two): Building a Personal Website Using the Hugo Framework\u0026rdquo;.\nThe process is as follows:\nFind the Access key for your domain management account. You can use the previous one or obtain a new one and export the Access key as system variables Ali_Key and Ali_Secret.\nUse the following command to generate the certificate:\n1 acme.sh --issue --dns dns_ali -d your.domain.com Use the following command to copy the certificate to your Nextcloud directory. For example, I created a folder named cert in the Nextcloud installation directory to store the certificate files:\n1 2 3 acme.sh --install-cert -d your.domain.com \\ --key-file /media/calibre/cert/key.pem \\ --fullchain-file /media/calibre/cert nginx reverse proxy configuration # Since I installed nginx when building this website, I used nginx as the reverse proxy tool for Nextcloud (although Apache is mainly used for installation instructions on the Nextcloud website).\nCreate a new configuration file named calibre.conf as the reverse proxy configuration file for the cloud drive in the nginx configuration directory, and write the following content:\nserver{\nlisten 80;\nserver_name book.jinli.cyou;\nreturn 301 https://book.jinli.cyou$request_uri;\n}\nserver{\nlisten 443 ssl;\nssl_certificate /media/lijin/book/certs/cert.pem;\nssl_certificate_key /media/lijin/book/certs/key.pem;\nserver_name book.jinli.cyou;\nlocation / {\nproxy_redirect off;\nproxy_pass http://book.jinli.cyou;\nproxy_set_header Host $http_host;\nproxy_set_header X-Real-IP $remote_addr;\nproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\nproxy_set_header X-Forwarded-Proto $scheme;\nclient_max_body_size 1000m;\n}\n}\nHere, I used port 8080 as the service port for calibre-web. The SSL certificate is stored in the location specified in the previous step.\nAfter the configuration is completed, restart the nginx service to make the modification effective:\n1 sudo service nginx restart Install calibre-web # Here we use docker-compose to install calibre-web, please refer to https://hub.docker.com/r/linuxserver/calibre-web for details.\nInitialization and Settings # After installing calibre-web, we can enter the website through the browser. The url is http://0.0.0.0:8080. If nginx is set up, we can also use the domain name http://book.jinli.cyou.\nThe default account name and passcode for the administrator is admin and admin123. We can do the initial settings after logging in.\nFirst we need to download an empty database file metadata.db and move it to the library folder of the installation directory of calibre-web, such as /media/calibre-web/library.\nThen choose /books as the database path in calibre-web. Note that the directory /books in docker is actually the directory /media/calibre-web/library on the server. Here calibre-web may still show error message because of the file access permission problem: .\nTo change the read/write permission, we need to enter the docker environment. The commands are\n1 2 3 4 docker exec -it calibre_web /bin/bash chmod 777 books cd books chmod 766 metadata.db After that we can do some other settings, such as server\u0026rsquo;s port, SSL certificate, etc.\nAdd eBook # After setting up all initial configurations, we can add eBooks to the library. And we can create different shelves for different categories of books. .\n","date":"5 August 2023","externalUrl":null,"permalink":"/en/p/build-a-private-digital-library-with-calibre-web-and-docker/","section":"Posts","summary":" Motivation # I’ve previously downloaded some eBooks, all of which are stored on the cloud drive. However, every time I want to read them on different devices, I need to download them first (even though they can be previewed on the cloud drive, I still find it better to download them and use a dedicated e-reader for a better experience). Moreover, with a growing number of eBooks, managing them on the cloud drive becomes quite chaotic. Recently, I came across the tool called “calibre-web,” which allows you to set up your own eBook library. I feel that the results are quite promising.\n","title":"Build a Private Digital Library with calibre-web and docker","type":"post"},{"content":"","date":"5 August 2023","externalUrl":null,"permalink":"/en/tags/calibre-web/","section":"Tags","summary":"","title":"Calibre-Web","type":"tags"},{"content":"","date":"5 August 2023","externalUrl":null,"permalink":"/en/tags/ebook/","section":"Tags","summary":"","title":"Ebook","type":"tags"},{"content":"","date":"5 August 2023","externalUrl":null,"permalink":"/en/tags/personal-library/","section":"Tags","summary":"","title":"Personal Library","type":"tags"},{"content":" Motivation # Now that you have a private domain name, you can make full use of it and build a private mail server, so that you can create your own mailbox, and all the information in the mail can be completely controlled by yourself, which is more secure.\nThere are multiple deployment methods for private mail servers, because I have installed docker on the server before, and other software tools on the server also use docker for containerized deployment and operation, so the mail server here is also chosen to be more convenient for containerizationtools to deploy.\nI saw Mailu and mailcow as the most used tools for building mail servers on the Internet..At first, I felt that the settings of Mailu were relatively simple and the interface was relatively refreshing, so I tried to do it, but after a lot of effort, I couldn\u0026rsquo;t send and receive emails successfully, so I had to give up.\nmailcow is another open source mail server package that contains the necessary software packages to build a private mail server.Mailcow officially provides an installation guide for docker containerization, which is more convenient to deploy on the server. Then I tested mailcow, and it was able to receive emails successfully, but failed to send emails to the outside world. Later, I found out that it was because the operator I used closed port 25. At this time, I was close to giving up the idea of building my own email server.But later I saw on the Internet that email relay/mail forwarding could be used as an alternative, so I spent a long time exploring, and finally used email relay to solve the problem of external sending.Here is a record of the process of tossing this mail server.\nPrerequisites # docker and docker compose Domain name (secondary domain name is enough) acme.sh (to generate SSL certificates) nginx (reverse proxy) Preparation # Prepare Domain Name # Create a second-level domain name and configure domain name resolution on the website where you purchased the domain name or the management website of your own domain name. The specific process can be found in \u0026ldquo;How This Website Was Built 1 \u0026ndash; Purchase and Configure a Personal Domain Name\u0026rdquo;.\nFor example, I have a primary domain name jinli.cyou, so I created a secondary domain name \u0026ldquo;mail.jinli.cyou\u0026rdquo; for my private email server.Then the domain name and IP address were bound on Alibaba Cloud\u0026rsquo;s domain name management platform.\nGenerate SSL Certificate using acme.sh # Many tutorials on the Internet use Let\u0026rsquo;s Encrypt to generate SSL certificates, but since I have already downloaded the acme.sh tool when building this website, I used acme.sh to generate a certificate for \u0026ldquo;cloud.jinli.cyou\u0026rdquo;. The specific process can be found in How This Website Was Built 2 \u0026ndash; Build a Personal Website using Hugo.\nThe process is as follows:\nFind the Access key of your domain name management account. You can use the previous one or get a new one, and then export the Access key as system variables Ali_Key and Ali_Secret.\nUse the following command to generate the certificate:\n1 acme.sh --issue --dns dns_ali -d your.domain.com Use the following command to copy the certificate to your mailcow directory. For example, I created a folder cert under the mailcow installation directory to store the certificate files:\n1 2 3 acme.sh --install-cert -d your.domain.com \\ --key-file /media/mailcow/cert/key.pem \\ --fullchain-file /media/mailcow/cert/cert.pem nginx Reverse Proxy Configuration # Because I installed nginx when building this website, I used nginx as a reverse proxy tool here.\nCreate a new configuration file mailcow.conf in the configuration directory of nginx as the reverse proxy configuration file of the cloud network disk, write in the file:\nserver{\nlisten 80;\nserver_name mail.jinli.cyou;\nreturn 301 https://mail.jinli.cyou$request_uri;\n}\nserver{\nlisten 443 ssl;\nssl_certificate /media/lijin/mailcow/certs/cert.pem;\nssl_certificate_key /media/lijin/mailcow/certs/key.pem;\nserver_name mail.jinli.cyou;\nlocation / {\nproxy_pass http://127.0.0.1:8080/;\nproxy_set_header Host $http_host;\nproxy_set_header X-Real-IP $remote_addr;\nproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\nproxy_set_header X-Forwarded-Proto $scheme;\nclient_max_body_size 0;\nproxy_buffer_size 128k;\nproxy_buffers 64 512k;\nproxy_busy_buffers_size 512k;\n}\n}\nHere I use port 8080 as the service port of mailcow.The SSL certificate is stored in the location specified in the previous step.\nAfter the configuration is complete, restart the nginx service to make the changes take effect:\n1 sudo service nginx restart Install mailcow # For the method of installing mailcow using docker-compose, please refer to the [tutorial] (https://docs.mailcow.email/) officially provided by mailcow.\nAdd New Users and Other Settings # After logging in to the mailcow management interface as an administrator, click the administrator account avatar in the upper right corner, and you can see the \u0026ldquo;Users\u0026rdquo; option in the drop-down menu. Click this option to add user groups and new users.\nEmail Relay (Email Forwarding) # Many Internet Service Provider(ISP) will block port 25, which will cause emails to fail to be sent from the server.So one option is to contact the ISP and ask the ISP to unblock port 25, but this is not easy.\nAnother solution is to use the mailbox relay (mailbox forwarding) service, that is, the mail that needs to be sent from the self-built mailbox A is handed over to another mailbox server B that has the function of sending mail, and the mail to be sent is sent by the mailbox server B on behalf of.Usually, mailbox server B can be Gmail like this Mail service providers can be commercial companies that provide email forwarding services (such as Mailgun, SendGrid, etc.), some network carriers (such as AT\u0026amp;T) also provide users with mail forwarding services.\nSince Gmail provides mail forwarding service, the sent mail will add Google\u0026rsquo;s signature information, and commercial mail forwarding such as Mailgun and SendGrid needs to be paid to use, so here I use the free mail forwarding service provided by AT\u0026amp;T.\nRelay Mailbox Settings # Different mailboxes may have different setting methods, here we take the mailbox provided by AT\u0026amp;T as an example.\nFind the Mailboxes option in Settings, there is Send-only email address, click Add to add the mailbox you need to be forwarded by the agent.For example, I want to use AT\u0026amp;T mailbox for my Personal mailbox i@jinli.cyou to send emails, I will add a i@jinli.cyou.\nAfter clicking confirm, AT\u0026amp;T will send an email to the forwarded mailbox to verify the identity.After logging into your personal mailbox i@jinli.cyou, follow the instructions in the email and click the link to confirm.\nAfter the mailbox verification is successful, the red prompt of \u0026ldquo;Not verified\u0026rdquo; in the above AT\u0026amp;T settings will disappear.\nmailcow Settings # Follow the steps to set up mailbox forwarding given in the official mailcow document to set up mailbox forwarding for personal mailboxes:https://docs.mailcow.email/manual-guides/Postfix/u_e-postfix-relayhost/。\nIt is mainly divided into three steps:\nLog in to the mailcow administrator interface, find Routing in the Configuration and Details settings, and add your forwarding server, for example, I use AT\u0026amp;T mailbox, use smtp.mail.att.net:587. Then fill in your user name and password. It is recommended to reset a password for external use in the proxy mailbox.\nAfter setting, click the Test button and fill in the email address to be proxied for testing.If everything goes well, you can see the green prompt message of the last successful sending.If you receive a red failure message, it means that the settings are incorrect, and you need to check the previous settings.\nIn the management interface of mailcow, enter Mail Setup and find Domains.Add the proxy mailbox set in Routing to Sender-dependent transports.\nTest # In Mail tester, you can test the effect of the mailbox you built.\nThe score is not high, but it is basically enough.\nIf you want to improve your score, you can look at the points deducted and improve according to the suggestions of Mail tester.\n","date":"17 June 2023","externalUrl":null,"permalink":"/en/p/building-a-private-mail-server-using-mailcow-and-docker/","section":"Posts","summary":" Motivation # Now that you have a private domain name, you can make full use of it and build a private mail server, so that you can create your own mailbox, and all the information in the mail can be completely controlled by yourself, which is more secure.\nThere are multiple deployment methods for private mail servers, because I have installed docker on the server before, and other software tools on the server also use docker for containerized deployment and operation, so the mail server here is also chosen to be more convenient for containerizationtools to deploy.\n","title":"Building a Private Mail Server using Mailcow and Docker","type":"post"},{"content":"","date":"15 April 2023","externalUrl":null,"permalink":"/en/tags/cluster/","section":"Tags","summary":"","title":"Cluster","type":"tags"},{"content":"","date":"15 April 2023","externalUrl":null,"permalink":"/en/tags/hpc/","section":"Tags","summary":"","title":"HPC","type":"tags"},{"content":"","date":"15 April 2023","externalUrl":null,"permalink":"/en/tags/scientific-computing/","section":"Tags","summary":"","title":"Scientific Computing","type":"tags"},{"content":" Introduction # Slurm is a widely used workload manager and job scheduler for Linux and Unix systems. It allows users to submit and manage jobs on a cluster environment.\nOne important aspect of managing jobs on a cluster is job priority control. In this post, we will discuss how to use the \u0026ldquo;\u0026ndash;nice\u0026rdquo; option in Slurm to set the priority of a job.\nMotivation # Consider the following scenario:\nUser A and user B share a computing cluster. User A has submitted a large number of computing tasks that occupy all available computing nodes, and there are also some computing tasks waiting, but the time required for a single computing task is not particularly long. At this time, user B needs to submit a computing task that requires very few computing resources (such as CPU cores and memory). However, because B submitted the task later than A, B\u0026rsquo;s task can only be started after all of A\u0026rsquo;s computing tasks have been completed. In this scenario, although the computing resources required by user B are very small, because the computing task was submitted later, it must wait a long time before it can start computing, which wastes a lot of user B\u0026rsquo;s time.\nIf user B\u0026rsquo;s computing task could be prioritized ahead of the computing tasks that user A is waiting for, then as soon as one of user A\u0026rsquo;s tasks that is running ends, user B\u0026rsquo;s task can immediately start computing. After user B\u0026rsquo;s computation is completed, user A can continue to use the computing resources that user B has freed up. This way, user B can save a lot of waiting time, while the increase in waiting time for user A is minimal.\nTherefore, in this situation, it is very important to give higher priority to the computing task submitted later by user B than to the computing tasks submitted earlier by user A.\nManagement of Job Priority # Job Priority in Slurm # Generally, the jobs submitted through Slurm are prioritized according to the \u0026ldquo;First In, First Out (FIFO)\u0026rdquo; principle.\nHowever, cluster administrators may also calculate job priorities by configuring the \u0026ldquo;multi-factor priority plugin\u0026rdquo;, in which case job priorities depend on factors such as job size, queue time, affinity, and partition.\nMethods for Adjusting Job Priority # For administrators, job priorities can be directly managed by specifying or changing the value of the \u0026ldquo;priority\u0026rdquo; option for the job. For ordinary users, it is generally not allowed to specify or change the \u0026ldquo;priority\u0026rdquo; option for the job directly. However, there is a \u0026ldquo;nice\u0026rdquo; factor among the multiple factors that determine job priority, which ordinary users can specify. The \u0026ldquo;nice\u0026rdquo; factor can be understood as \u0026ldquo;niceness value\u0026rdquo;. The higher the value, the lower the job priority, which means you are more friendly to other users. Usage of the \u0026ldquo;nice\u0026rdquo; Option # Specification and Modification # Users can specify the value of \u0026ldquo;nice\u0026rdquo; when submitting a job:\n1 sbatch --nice=100 your_slurm_script Users can also update the \u0026ldquo;nice\u0026rdquo; value of a submitted job that is waiting to start:\n1 scontrol update JobId=\u0026lt;job_id\u0026gt; Nice=\u0026lt;new_nice_value\u0026gt; Viewing Job Priority # Users can view the priority of a submitted job: 1 scontrol show job=\u0026lt;job_id\u0026gt; | grep Priority Notes # The default \u0026ldquo;nice\u0026rdquo; value is 0. Ordinary users can only specify a positive \u0026ldquo;nice\u0026rdquo; value, while administrators can specify a negative \u0026ldquo;nice\u0026rdquo; value. That is, ordinary users can only be \u0026ldquo;nice\u0026rdquo;, while administrators can be \u0026ldquo;bad\u0026rdquo;. ","date":"15 April 2023","externalUrl":null,"permalink":"/en/p/slurm-how-to-change-jobs-priority-without-admin-privilege/","section":"Posts","summary":" Introduction # Slurm is a widely used workload manager and job scheduler for Linux and Unix systems. It allows users to submit and manage jobs on a cluster environment.\nOne important aspect of managing jobs on a cluster is job priority control. In this post, we will discuss how to use the “–nice” option in Slurm to set the priority of a job.\n","title":"Slurm: How to Change Job's Priority without Admin Privilege","type":"post"},{"content":"","date":"15 April 2023","externalUrl":null,"permalink":"/en/tags/workload-manager/","section":"Tags","summary":"","title":"Workload Manager","type":"tags"},{"content":"","date":"2023年4月15日","externalUrl":null,"permalink":"/tags/%E7%A7%91%E5%AD%A6%E8%AE%A1%E7%AE%97/","section":"Tags","summary":"","title":"科学计算","type":"tags"},{"content":"","date":"2023年4月15日","externalUrl":null,"permalink":"/tags/%E8%AE%A1%E7%AE%97%E4%BC%98%E5%85%88%E7%BA%A7%E7%AE%A1%E7%90%86/","section":"Tags","summary":"","title":"计算优先级管理","type":"tags"},{"content":"","date":"2023年4月15日","externalUrl":null,"permalink":"/tags/%E8%AE%A1%E7%AE%97%E9%9B%86%E7%BE%A4/","section":"Tags","summary":"","title":"计算集群","type":"tags"},{"content":"","date":"2023年4月15日","externalUrl":null,"permalink":"/tags/%E9%AB%98%E6%80%A7%E8%83%BD%E8%AE%A1%E7%AE%97/","section":"Tags","summary":"","title":"高性能计算","type":"tags"},{"content":"","date":"21 February 2023","externalUrl":null,"permalink":"/en/tags/chevereto/","section":"Tags","summary":"","title":"Chevereto","type":"tags"},{"content":"","date":"21 February 2023","externalUrl":null,"permalink":"/en/tags/cloud-photo-album/","section":"Tags","summary":"","title":"Cloud Photo Album","type":"tags"},{"content":"","date":"21 February 2023","externalUrl":null,"permalink":"/en/tags/google-photo-alternative/","section":"Tags","summary":"","title":"Google Photo Alternative","type":"tags"},{"content":" Background and Motivation # Previously, I used Nextcloud to build a personal cloud drive on my own server (see \u0026ldquo;Building a Personal Cloud Drive with Nextcloud and Docker\u0026rdquo;) and backed up my photos from various devices to the cloud drive. Later, I found that the Nextcloud app on mobile devices has an option for automatic uploading, which gave me the idea of using it to replace Google Photos and building a personal cloud photo album.\nLater, a friend told me that our school began limiting the storage of Google Drive to only 250 GB and would further limit it to only 15 GB after graduation. Therefore, I started using Nextcloud to build a cloud photo album to replace Google Photos.\nHow to Use # Install the Nextcloud client on your mobile phone and log in to your Nextcloud personal cloud drive. Create a folder in Nextcloud to store your photo album, e.g., \u0026ldquo;Photos.\u0026rdquo; Open the Nextcloud mobile app, click the three horizontal lines in the upper left corner to enter the menu page, and click \u0026ldquo;Settings\u0026rdquo; to enter the settings page. Click the \u0026ldquo;Auto Upload\u0026rdquo; option to enter the auto-upload settings page. The auto-upload settings page lists all the albums on your phone. To the right of each album, there is a white cloud icon and three dots. Click the three dots and then \u0026ldquo;Config\u0026rdquo; to set the upload path, upload only on Wi-Fi, etc. After configuring, click the white cloud icon with a diagonal line to make it blue, and the auto-upload is set up. Note: Be sure to configure the upload path before enabling the white cloud icon; otherwise, Nextcloud will upload the album to the default folder instead of the folder you specified!\nYou can also do similar settings for other mobile phones, tablets, computers, etc. You can upload all albums to the same folder or upload them to different folders. This way, the albums on multiple devices can be synchronized in the cloud!\n","date":"21 February 2023","externalUrl":null,"permalink":"/en/p/google-photo-alternative-using-nextcloud-to-create-a-personal-cloud-photo-album-that-can-be-synchronized-across-multiple-devices/","section":"Posts","summary":" Background and Motivation # Previously, I used Nextcloud to build a personal cloud drive on my own server (see “Building a Personal Cloud Drive with Nextcloud and Docker”) and backed up my photos from various devices to the cloud drive. Later, I found that the Nextcloud app on mobile devices has an option for automatic uploading, which gave me the idea of using it to replace Google Photos and building a personal cloud photo album.\n","title":"Google Photo Alternative: Using Nextcloud to Create a Personal Cloud Photo Album That Can Be Synchronized Across Multiple Devices","type":"post"},{"content":"","date":"2023年2月21日","externalUrl":null,"permalink":"/tags/google-photo%E6%9B%BF%E4%BB%A3%E5%93%81/","section":"Tags","summary":"","title":"Google Photo替代品","type":"tags"},{"content":"","date":"21 February 2023","externalUrl":null,"permalink":"/en/tags/nextcloud/","section":"Tags","summary":"","title":"Nextcloud","type":"tags"},{"content":" Motivation # When blogging online, it is possible that the same article needs to be uploaded to different platforms. For example, some articles will be placed on this website as well as in the README of the project’s GitHub repository. At this time, if the pictures in the article use local pictures, you need to put a copy on different platforms, which is not easy to update and manage in a unified manner.Hence the idea of building a personal drawing bed To unify the idea of managing these pictures.\nUse the private picture bed, and directly insert the link of the picture in the private picture bed when writing an article to quote the picture, which is convenient for unified management in the future.\nPrerequisites # docker and docker compose Domain name (secondary domain name is enough) acme.sh (to generate SSL certificate) nginx (reverse proxy) For the installation and preparation of the above software and tools, please refer to \u0026ldquo;Building a Personal Cloud Network Disk using Nextcloud and Docker\u0026rdquo;\nInstall Chevereto # Configure docker-compose # Since we have generated the SSL certificate and configured nginx above, we only need to configure Chevereto in docker.\nHere I have created a separate partition on the hard disk of the server as the installation root directory of Chevereto.Create docker-compose.yml file in the root directory of Chevereto installation, write the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 version: \u0026#34;3.2\u0026#34; services: database: container_name: chevereto-free_database image: mariadb:focal networks: - chevereto_network volumes: - ./database:/var/lib/mysql restart: always environment: MYSQL_ROOT_PASSWORD: your_root_password MYSQL_DATABASE: chevereto MYSQL_USER: chevereto MYSQL_PASSWORD: your_user_password chevereto: container_name: chevereto-free_app image: ghcr.io/rodber/chevereto-free:1.6 networks: - chevereto_network volumes: - ./images:/var/www/html/images/ - ./content:/var/www/html/content/ ports: - 8810:80 restart: always environment: CHEVERETO_TAG: \u0026#34;free\u0026#34; CHEVERETO_DB_HOST: database CHEVERETO_DB_USER: chevereto CHEVERETO_DB_PASS: your_chevereto_password CHEVERETO_DB_PORT: 3306 CHEVERETO_DB_NAME: chevereto CHEVERETO_DISABLE_UPDATE_HTTP: 1 CHEVERETO_DISABLE_UPDATE_CLI: 1 CHEVERETO_HTTPS: 0 networks: chevereto_network: Here you need to change \u0026ldquo;MYSQL_ROOT_PASSSWORD\u0026rdquo; and \u0026ldquo;MYSQL_PASSWORD\u0026rdquo; to the database password you set yourself, and you can also change the names of \u0026ldquo;MYSQL_DATABASE\u0026rdquo; and \u0026ldquo;MYSQL_USER\u0026rdquo;.\nBuild containers with docker-compose # Execute the following command in the Nextcloud installation root directory to build the container:\n1 sudo docker-compose up -d Domain Certificates and Reverse Proxy # Generate a free domain name certificate with acme.sh # The specific process can be found inHow This Website Was Built 2 \u0026ndash; Hugo Framework for Personal Blog Website。\nThe main steps are as follows:\nGenerate the certificate using the following command:\n1 acme.sh --issue --dns dns_ali -d your.domain.name Use the following command to copy the certificate to your nextcloud directory. For example, I created a new folder cert in the nextcloud installation directory to store the certificate file:\n1 2 3 acme.sh --install-cert -d your.domain.name \\ --key-file your_cert_path/key.pem \\ --fullchain-file your_cert_path/cert.pem Set up a reverse proxy with nginx # Create a new configuration file nextcloud.conf in the configuration directory of nginx as the reverse proxy configuration file of the cloud network disk, write in the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 upstream your.domain.name { server 127.0.0.1:8810; } server { listen 80; server_name your.domain.name; return 301 https://your.domain.name$request_uri; } server { listen 443 ssl; server_name your.domain.name; gzip on; # ssl 配置 ssl_certificate your_cert_path/cert.pem; ssl_certificate_key your_cert_path/key.pem; location / { proxy_redirect off; proxy_pass http://your.domain.name; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Ssl on; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Frame-Options SAMEORIGIN; client_max_body_size 100m; client_body_buffer_size 128k; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; } } Here I use port 8810 as the service port of Chevereto, and limit users to upload files up to 100 M.The SSL certificate is stored in the location specified in the previous step.\nAfter the configuration is complete, restart the nginx service to make the changes take effect:\n1 sudo service nginx restart Initialization and Subsequent Setup # After completing the above steps, you can access your private picture bed from the browser!Enter the URL of the picture bed in the browser, for example, mine is https://img.jinli.io.You will enter the following page:\nI forgot whether I need to do additional settings when entering this page for the first time, but it should be easy to complete as long as you follow the prompts.\nThen follow the prompts to create a user name and password, and you can log in!\n","date":"21 February 2023","externalUrl":null,"permalink":"/en/p/use-chevereto-free-to-build-a-private-image-bed/","section":"Posts","summary":" Motivation # When blogging online, it is possible that the same article needs to be uploaded to different platforms. For example, some articles will be placed on this website as well as in the README of the project’s GitHub repository. At this time, if the pictures in the article use local pictures, you need to put a copy on different platforms, which is not easy to update and manage in a unified manner.Hence the idea of building a personal drawing bed To unify the idea of managing these pictures.\n","title":"Use Chevereto-free to Build a Private Image Bed","type":"post"},{"content":"","date":"2023年2月21日","externalUrl":null,"permalink":"/tags/%E4%BA%91%E7%9B%B8%E5%86%8C/","section":"Tags","summary":"","title":"云相册","type":"tags"},{"content":"","date":"23 January 2023","externalUrl":null,"permalink":"/en/tags/barrier/","section":"Tags","summary":"","title":"Barrier","type":"tags"},{"content":"","date":"2023年1月23日","externalUrl":null,"permalink":"/tags/deskflow/","section":"Tags","summary":"","title":"Deskflow","type":"tags"},{"content":"","date":"23 January 2023","externalUrl":null,"permalink":"/en/tags/inputleap/","section":"Tags","summary":"","title":"InputLeap","type":"tags"},{"content":" Motivation # It is often necessary to use more than one computer at work, for example, there is a desktop computer with Linux system in the office, and another laptop computer with macOS For classes or meetings.Sometimes it is necessary to use these two computers at the same time to complete a certain task. At this time, switching back and forth with two sets of mice and keyboards will become a problem. It is very troublesome, Barrier is a free software that can share mouse and keyboard across multiple computers.\nBarrier已于2021年停止维护，现推荐使用InputLeap。InputLeap和Barrier是同源的，功能和使用方法与Barrier几乎一致。本文所讲的Barrier的安装和设置基本也都适用于InputLeap。 Update: In 2021, Barrier stopped maintaining, now I recommend to use InputLeap as an alternative. InputLeap and Barrier are of the same source, and the functions and usage methods are almost the same as Barrier. The installation and settings of Barrier mentioned in this article are basically applicable to InputLeap. On Fedora 42 system, InputLeap 3.0.2 version will crash when set as server, while 3.0.3 version works fine.\nIntroduction # In fact, the keyboard and mouse sharing software is not limited to the barrier, there are other Mouse Without Borders, Synergy, etc., but Mouse Without Borders is only available for Windows systems, while Synergy is a paid software.\nBarrier is a software with the same source as Synergy.Originally Synergy was written based on Chris Schoeneman A free software developed by CosmoSynergy, Synergy began to charge after iterating several versions, so some teams started based on Synergy\u0026rsquo;s open source kernel has again developed a free version of the barrier.\nBarrier is a free and open source cross-system keyboard and mouse sharing software. The main features are:\nShared mouse and shared keyboard shared clipboard Across systems. Available for Windows, macOS and Linux systems Free, open source Installation # Windows and macOS # There are already packaged installation packages for Windows and macOS systems, which can be downloaded from the release page of the Barrier open source GitHub repository: https://github.com/debauchee/barrier/releases.Select the installation package with the suffix exe for the Windows system, and select the installation package with the suffix dmg for the macOS system.\nWindows # Just double-click the installation package and install it step by step under Windows.\nmacOS # The installation under macOS is relatively more complicated, mainly because the publisher of Barrier or InputLeap is not certified, so there will be some security restrictions during installation.Therefore, we need to manually bypass these settings.\nWhen downloading the installation package under macOS, you need to pay attention to whether your computer is an Intel chip or an Apple chip————Download the x86 version of the dmg installation package for Intel chips, and download the AppleSilicon version of the installation package for Apple chips.In addition, I use macOS version Sequoia 15.5, and after using InputLeap 3.0.3 version of the installation package, it will show that it cannot be installed, while InputLeap 3.0.2 version is fine, so here we use InputLeap 3.0.2 version.\nAfter double-clicking the installation package, it will show that it cannot be installed for security reasons. Click the question mark icon in the upper right corner, and the following window will open: Then follow the prompts to enable security settings, and then install Barrier or InputLeap.\nNote: After opening the \u0026ldquo;Security\u0026rdquo; settings by clicking the link in the above figure, you may not see the option to allow Barrier or InputLeap to be installed. At this time, you need to try to close and re-open Barrier or InputLeap installation package, then you will see \u0026ldquo;Open Anyway\u0026rdquo; option.\nLinux # For Debian-based Linux systems (such as Ubuntu, etc.), you can directly use the package management tool apt to install:\n1 sudo apt install barrier Other Linux systems can be installed through the package management tool snap:\n1 sudo snap install barrier If the system does not come with snap, you need to install snap first.\nConfiguration and Usage # Barrier settings are divided into server and client.\nServer # The computer directly connected to the keyboard and mouse is the server.\nOpen Barrier on the server computer and check Server. Write down the IP address of the server (usually the LAN IP). Tick Manually configure the server, and then click \u0026ldquo;Configure Server\u0026rdquo;. Drag the computer icon on the upper right side of the configuration interface to the lower grid, double-click the computer icon, and change the computer name to the name displayed on the client computer (you can find it in the \u0026ldquo;screen name\u0026rdquo; column of the client barrier software interface).The relative position of the two computer icons in the grid corresponds to the screen position of the actual computer. Client # The computer to use the keyboard and mouse of the server computer is the client.\nOpen Barrier on the client computer and check Client. Fill in the IP address of the server in the server IP column (Server IP).If this column is not editable, uncheck \u0026ldquo;Auto config\u0026rdquo;. Click \u0026ldquo;Barrier\u0026rdquo; on the Barrier menu bar, select \u0026ldquo;Change Settings\u0026rdquo;, make sure the network port (Port) used is 24800, and check \u0026ldquo;Enable SSL\u0026rdquo;. Restart or reload (Reload) Barrier. Common Issues # Make sure the server and client computers are on the same network.\nIf the mouse and keyboard cannot be shared successfully, you can check the above settings and restart the Barrier software.\nIf it still doesn\u0026rsquo;t work, you can click \u0026ldquo;Show Log\u0026rdquo; in the menu bar to view the error message in the log.\n","date":"23 January 2023","externalUrl":null,"permalink":"/en/p/installation-and-use-of-free-keyboard-and-mouse-sharing-software-barrier/inputleap-for-all-platforms/","section":"Posts","summary":" Motivation # It is often necessary to use more than one computer at work, for example, there is a desktop computer with Linux system in the office, and another laptop computer with macOS For classes or meetings.Sometimes it is necessary to use these two computers at the same time to complete a certain task. At this time, switching back and forth with two sets of mice and keyboards will become a problem. It is very troublesome, Barrier is a free software that can share mouse and keyboard across multiple computers.\n","title":"Installation and Use of Free Keyboard and Mouse Sharing Software Barrier/InputLeap for All Platforms","type":"post"},{"content":"","date":"23 January 2023","externalUrl":null,"permalink":"/en/tags/kvm-software/","section":"Tags","summary":"","title":"KVM Software","type":"tags"},{"content":"","date":"2023年1月23日","externalUrl":null,"permalink":"/tags/kvm%E8%BD%AF%E4%BB%B6/","section":"Tags","summary":"","title":"KVM软件","type":"tags"},{"content":" Introduction # Nextcloud is an open-source network disk tool that allows users to build their own cloud network disk on their private server.\nNextcloud provides both server-side and client-side software. It supports desktop-level operating systems such as Windows, Linux, and macOS, as well as mobile-level operating systems such as Android and iOS. Therefore, personal users can easily use their private cloud network disk after building it.\nDocker is a tool that creates containers that can virtualize the operating system layer, isolating a separate environment for running one or more software applications. It avoids dependency and runtime environment conflicts for the software and allows for easy deployment of certain software on a server.\nPrerequisites # docker and docker-compose Domain name (second-level domain is sufficient) acme.sh (generating SSL certificate) nginx (reverse proxy) Preparation # Install Docker and Docker Compose # Debian-based systems can directly install docker using apt:\n1 sudo apt-get install docker-ce Install Docker Compose:\n1 sudo apt-get install docker-compose Docker Compose uses a file named docker-compose.yml to \u0026ldquo;synthesize\u0026rdquo; a docker container. docker-compose.yml is the formula for synthesizing this docker container.\nPrepare Domain Name # Create a second-level domain on the website where you purchased your domain name or on your domain management website, and configure domain name resolution. For specific instructions, please refer to \u0026ldquo;Building a Personal Website (Part One): Purchasing a Personal Domain Name and Configuring Dynamic Domain Name Resolution\u0026rdquo;.\nFor example, if you have a primary domain name jinli.cyou, you can create a second-level domain \u0026ldquo;cloud.jinli.cyou\u0026rdquo; for your personal cloud network disk. Then, bind the domain name and IP address on the Alibaba Cloud domain management platform.\nUse acme.sh to Generate SSL Certificates # Many online tutorials use Let\u0026rsquo;s Encrypt to generate SSL certificates, but since I have already downloaded the acme.sh tool when building this website, I will use acme.sh to generate a certificate for \u0026ldquo;cloud.jinli.cyou\u0026rdquo;. For specific instructions, please refer to \u0026ldquo;Building a Personal Website (Part Two): Building a Personal Website Using the Hugo Framework\u0026rdquo;.\nThe process is as follows:\nFind the Access key for your domain management account. You can use the previous one or obtain a new one and export the Access key as system variables Ali_Key and Ali_Secret.\nUse the following command to generate the certificate:\n1 acme.sh --issue --dns dns_ali -d your.domain.com Use the following command to copy the certificate to your Nextcloud directory. For example, I created a folder named cert in the Nextcloud installation directory to store the certificate files:\n1 2 3 acme.sh --install-cert -d your.domain.com \\ --key-file /media/nextcloud/cert/key.pem \\ --fullchain-file /media/nextcloud/cert nginx reverse proxy configuration # Since I installed nginx when building this website, I used nginx as the reverse proxy tool for Nextcloud (although Apache is mainly used for installation instructions on the Nextcloud website).\nCreate a new configuration file named nextcloud.conf as the reverse proxy configuration file for the cloud drive in the nginx configuration directory, and write the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 server { listen 443 ssl http2; listen [::]:443 ssl http2; server_name cloud.jinli.cyou; ssl_certificate /media/nextcloud/cert/cert.pem; ssl_certificate_key /media/nextcloud/cert/key.pem; client_max_body_size 0; underscores_in_headers on; location ~ { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; add_header Front-End-Https on; proxy_headers_hash_max_size 512; proxy_headers_hash_bucket_size 64; proxy_buffering off; proxy_redirect off; proxy_max_temp_file_size 0; proxy_pass http://127.0.0.1:7080; } } Here, I used port 7080 as the service port for Nextcloud and did not limit the size of uploaded files for users. The SSL certificate is stored in the location specified in the previous step.\nAfter the configuration is completed, restart the nginx service to make the modification effective:\n1 sudo service nginx restart Install Nextcloud # Configure docker-compose # Since we have generated SSL certificates and configured nginx above, we only need to configure two containers, nextcloud and database, in docker. Here we choose to use MariaDB as the database.\nCreate a docker-compose.yml file in the Nextcloud installation root directory and write the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 version: \u0026#39;3\u0026#39; services: db: image: mariadb container_name: nextcloud-mariadb networks: - nextcloud_network volumes: - ./db:/var/lib/mysql - /etc/localtime:/etc/localtime:ro environment: - MYSQL_ROOT_PASSWORD=PASSWORD1 - MYSQL_PASSWORD=PASSSWORD2 - MYSQL_DATABASE=nextcloud - MYSQL_USER=nextcloud restart: unless-stopped app: image: nextcloud:latest container_name: nextcloud-app networks: - nextcloud_network ports: - 7080:80 depends_on: - db volumes: - ./nextcloud:/var/www/html - ./app/config:/var/www/html/config - ./app/custom_apps:/var/www/html/custom_apps - ./app/data:/var/www/html/data - ./app/themes:/var/www/html/themes - /etc/localtime:/etc/localtime:ro environment: - VIRTUAL_HOST=your.cloud.domain.name restart: unless-stopped networks: nextcloud_network: Here you need to:\nChange \u0026ldquo;MYSQL_ROOT_PASSSWORD\u0026rdquo; and \u0026ldquo;MYSQL_PASSWORD\u0026rdquo; to the database password you set, or change the names of \u0026ldquo;MYSQL_DATABASE\u0026rdquo; and \u0026ldquo;MYSQL_USER\u0026rdquo;. Change the domain name in \u0026ldquo;VIRTUAL_HOST\u0026rdquo; to your own cloud drive domain name. Generate containers using docker-compose # Execute the following command in the Nextcloud installation root directory to generate containers:\n1 sudo docker-compose up -d Install and initialize Nextcloud # After completing the above steps, you can access your cloud drive from the browser! Enter the cloud drive URL in the browser, for example, mine is https://cloud.jinli.cyou. Then you will see the following page:\nHere you need to:\nCreate an administrator account and remember the username and password. Click \u0026ldquo;Storage \u0026amp; databases\u0026rdquo;, change the default database \u0026ldquo;SQLite\u0026rdquo; to \u0026ldquo;MySQL/MariaDB\u0026rdquo;, enter the user, name, and password of the database, and the last field (default \u0026ldquo;localhost\u0026rdquo;) should be the address of the database. Since we defined the database container service as db in the docker-compose.yml file above and placed it in the same network as the Nextcloud container, the address here should be db, so that the Nextcloud container can access the database container by this name. If your docker-compose.yml file defines a different name for the database container service, you need to fill in that name here. Finally, click \u0026ldquo;Finish Setup\u0026rdquo; to install Nextcloud, which may take several minutes. After installation, you can log in to Nextcloud with the administrator account!\nAdd new users and other settings # After logging into Nextcloud as an administrator, click the administrator account avatar in the upper right corner and you will see the \u0026ldquo;Users\u0026rdquo; option in the dropdown menu. Click this option to add user groups and new users.\nTroubleshooting # \u0026ldquo;file/directory is locked\u0026rdquo; error # After a failed file upload, I wanted to re-upload the file but it kept failing. I tried to delete the entire folder but received an error message saying \u0026ldquo;file/directory is locked\u0026rdquo; and could not delete it.\nTo resolve this issue, you can try the following steps:\nLog in to the Nextcloud container:\n1 docker exec -it nextcloud-app bash You should be in /var/www/html directory. And you can find an executable file named occ in this directory, which is the command-line tool for Nextcloud.\nEnter the maintenance mode:\n1 ./occ maintenance:mode --on Install MySQL client:\n1 apt-get update \u0026amp;\u0026amp; apt-get install -y mariadb-client Connect to the database:\n1 mysql -h db -u ${MYSQL_USER} -p${MYSQL_PASSWORD} ${MYSQL_DATABASE} Here, ${MYSQL_USER} and ${MYSQL_PASSWORD} are the database user and password you set in the docker-compose.yml file.\nExecute the following SQL command to unlock the file/directory:\n1 DELETE FROM oc_file_locks; Exit the MySQL client:\n1 exit; Exit the maintenance mode:\n1 ./occ maintenance:mode --off Finally go back to the Nextcloud web interface and try to delete the file or directory again. It should work now.\nNote: There is another solution online, which is to enter the Nextcloud container and use the command ./occ files:scan --all to scan all files. This command can indeed remove the lock, but it will refresh the last modified time of all files, causing all files\u0026rsquo; modification times to change to the current time. If you want to keep the original modification times of the files, do not use this command.\n","date":"23 September 2022","externalUrl":null,"permalink":"/en/p/building-a-personal-cloud-network-disk-using-nextcloud-and-docker/","section":"Posts","summary":" Introduction # Nextcloud is an open-source network disk tool that allows users to build their own cloud network disk on their private server.\nNextcloud provides both server-side and client-side software. It supports desktop-level operating systems such as Windows, Linux, and macOS, as well as mobile-level operating systems such as Android and iOS. Therefore, personal users can easily use their private cloud network disk after building it.\n","title":"Building a Personal Cloud Network Disk using Nextcloud and Docker","type":"post"},{"content":"","date":"22 September 2022","externalUrl":null,"permalink":"/en/tags/environment-modules/","section":"Tags","summary":"","title":"Environment Modules","type":"tags"},{"content":"","date":"22 September 2022","externalUrl":null,"permalink":"/en/tags/environment-variables/","section":"Tags","summary":"","title":"Environment Variables","type":"tags"},{"content":" Motivation # When using the university\u0026rsquo;s computing cluster, it is very convenient to use the module load command to load software packages such as the Intel compiler and Python Anaconda. These packages can also be easily discarded when they are not needed, which makes it convenient to select and enable certain environment variables, and prevent certain environment variables from contaminating the compilation environment of other software.\nOn one\u0026rsquo;s own computer, in order to install and compile various programs, it is sometimes necessary to download and install many compilers and software packages. As a result of the impact of environment variables, compilers or programs often report errors during compilation or execution. Therefore, I plan to install modules on my own computer to manage these software packages and environment variables.\nInstallation and Working Mechanism of module Tool # Installation # For Debian-based systems such as Ubuntu, it can be directly installed using apt:\n1 sudo apt install environment-modules Generally, it will be installed by default under /usr/share/modules. The modulefiles folder under this path will store some default module files:\ndot module-git module-info modules null use.own\nUsers can view them by using the module avail command.\nIf you find that the module command cannot be used, it is because it has not been initialized yet. Under /usr/share/modules/init, there are initialization tools for various shells such as bash, ksh, fish, tcsh, and zsh. For example, if I use the module tool in zsh, I need to run the following command to complete the initialization:\n1 source /usr/share/modules/init/zsh Since the above command needs to be re-entered every time the system or zsh is re-logged in, for convenience, it can be added to \u0026ldquo;~/.zshrc\u0026rdquo;.\nWorking Mechanism # The module tool lists and loads user-specified software packages and environment variables by recognizing module files. A typical module file content is shown below:\n#%Module\nproc ModulesHelp { } { puts stderr \u0026ldquo;This module adds solar to your path\u0026rdquo; }\nmodule-whatis \u0026ldquo;This module adds solar to your path\\n\u0026rdquo;\nset basedir \u0026ldquo;/home/aturing/software/solar-1.2\u0026rdquo;\nprepend-path PATH \u0026ldquo;${basedir}/bin\u0026rdquo;\nprepend-path LD_LIBRARY_PATH \u0026ldquo;${basedir}/lib64\u0026rdquo;\nmodule load intel/19.1/64/19.1.1.217\nmodule load intel-mpi/intel/2019.7/64\nAfter installing module, it will look for modulefiles in the default path /usr/share/modules/modulefiles. When users use the module avail command to view available modules, the module tool will display the modulefiles under the /usr/share/modules/modulefiles path. After installing the module tool, the default modules are usually displayed as follows:\n\u0026mdash;\u0026mdash;- /usr/share/modules/modulefiles \u0026mdash;\u0026mdash;\u0026ndash;\ndot module-git module-info modules null use.own\nIf the user uses the module load use.own command to load use.own, the module tool will create a privatemodules folder in the user\u0026rsquo;s home directory, and the user can place custom modulefiles in this folder. After that, modulefiles located in the ~/privatemodules directory will also be detected by the module tool.\nInstalling Packages and Managing with module # Installing Intel\u0026rsquo;s C++ compiler icc # Intel provides its own C++ compiler icc and Fortran compiler ifort for free to users of Intel chips, which can be downloaded and installed on the official website of Intel. Intel offers multiple installation methods, including oneAPI which integrates multiple Intel compilers and tools, as well as standalone versions of these compilers and tools. The full version of oneAPI toolkit is very large, but since I only need icc and ifort compilers at present, I only downloaded and installed these two standalone compilers.\nDownload the installation package\nThe standalone icc compiler installation package can be downloaded from the Intel website at the following URL: https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#dpcpp-cpp. For each of the Linux/Windows/macOS systems, there are both online and offline installation packages available. You can choose the online version of the installation package. For example, the name of the installation package I downloaded is l_dpcpp-cpp-compiler_p_2022.1.0.137.sh.\nAlternatively, you can download the package using the command line:\n1 wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18717/l_dpcpp-cpp-compiler_p_2022.1.0.137.sh Run the installation package\nAfter downloading, navigate to the directory where the installation package is saved and use the following command to install:\n1 sudo sh ./l_dpcpp-cpp-compiler_p_2022.1.0.137.sh -a -s --eula accept Here, -a indicates the use of command-line parameters; -s indicates a silent installation, i.e., no installation window is displayed; --eula accept indicates that you accept the user agreement. For more details, please refer to the Intel installation guide.\nThe compiler is installed by default in the /opt/intel/oneapi directory.\nConfigure Modulefiles\nIntel has integrated a script in the installation package that can automatically generate modulefiles. Users only need to run the script, and the installed compiler or toolkit will be automatically written into modulefiles that can be used by the module tool.\nThe configuration script is located in the installation directory /opt/intel/oneapi, and the modulefiles-setup.sh file is used. When running this script, users can specify the folder where the generated modulefiles are placed. Here, I choose to put the generated modulefiles in the ~/privatemodules directory:\n1 /opt/intel/oneapi/modulefiles-setup.sh --output-dir=$HOME/privatemodules View and Load the Compiler\nAfter completing the above steps, you can use the module avail command to view the icc compiler that was just installed. After running the module avail command, you will see output similar to the following:\n1 2 3 4 5 6 7 8 ------------------------------------------------ /usr/share/modules/modulefiles ------------------------------------------------ dot module-git module-info modules null use.own ------------------------------------------------- /home/lijin/privatemodules --------------------------------------------------- compiler-rt/2022.1.0 compiler/2022.1.0 debugger/2021.6.0 icc/2022.1.0 init_opencl/2022.1.0 tbb/2021.6.0 compiler-rt/latest compiler/latest debugger/latest icc/latest init_opencl/latest tbb/latest compiler-rt32/2022.1.0 compiler32/2022.1.0 dev-utilities/2021.6.0 icc32/2022.1.0 oclfpga/2022.1.0 tbb32/2021.6.0 compiler-rt32/latest compiler32/latest dev-utilities/latest icc32/latest oclfpga/latest tbb32/latest Since I have also installed some other tools, in addition to icc, other modules are also displayed.\nThen, use module load icc/latest to load the latest installed version of the icc compiler!\nInstalling Intel Fortran Compiler ifort # The steps to install the ifort compiler are similar to the above steps, and will not be repeated here. The default installation directory for the ifort compiler is also /opt/intel/oneapi. After installation, run the following command again to update the modulefiles in the ~/privatemodules directory:\n1 /opt/intel/oneapi/modulefiles-setup.sh --output-dir=$HOME/privatemodules ","date":"22 September 2022","externalUrl":null,"permalink":"/en/p/use-environment-module-to-manage-software-packages-and-environment-variables-in-linux/","section":"Posts","summary":" Motivation # When using the university’s computing cluster, it is very convenient to use the module load command to load software packages such as the Intel compiler and Python Anaconda. These packages can also be easily discarded when they are not needed, which makes it convenient to select and enable certain environment variables, and prevent certain environment variables from contaminating the compilation environment of other software.\n","title":"Use Environment Module to Manage Software Packages and Environment Variables in Linux","type":"post"},{"content":" Build Dakota # Prerequisites # Dakota relies on some other tools and libraries. Please refer to the Dakota\u0026rsquo;s official website. Most of the tools (such as CMake, Python) are already installed. The ones that need to take care of are:\nLinear Algebra libraries: BLAS and LAPACK Boost System information of my case:\nSystem: Red Hat 8 Conda package: Mambaforge, moose environment Install Prerequisites # The linear algebra libraries BLAS and LAPACK are already in the moose conda environment. Please refer to \u0026ldquo;Build SAM from Source\u0026rdquo; for how to install moose conda environment using Mambaforge.\nInstall Boost 1.69.0. I also tried Boost 1.80, but it didn\u0026rsquo;t work.\nDownload Boost 1.69.0 source code from https://boostorg.jfrog.io/artifactory/main/release/1.69.0/source/boost_1_69_0.tar.gz. Or use following command: 1 wget https://boostorg.jfrog.io/artifactory/main/release/1.69.0/source/boost_1_69_0.tar.gz Extract the source code: 1 tar xzf boost_1_69_0.tar.gz Configure and install. --prefix option specifies the installation location: 1 2 3 cd boost_1_69_0 ./bootstrap.sh --prefix=${HOME}/local/boost/1.69 ./b2 -j4 install Setup Dakota compiling environment # Activate moose conda environment:\n1 mamba activate moose Add Boost dynamic linked library to the LD_LIBRARY_PATH:\n1 export LD_LIBRARY_PATH=${HOME}/local/boost/1.69:$LD_LIBRARY_PATH Or if you use Environment Modules to manage your libraries, you can make a module file for Boost and load the Boost module. Please refer to \u0026ldquo;Use Environment Module to Manage Software Packages and Environment Variables in Linux\u0026rdquo; for how to use Environment Modules.\nBuild Dakota from Source # Build Dakota # Download Dakota source code. I downloaded the Dakota version 1.16.0, Source (Unix/OS X) from https://dakota.sandia.gov/sites/default/files/distributions/public/dakota-6.16.0-public-src-cli.tar.gz. Or you can download it via command:\n1 wget https://dakota.sandia.gov/sites/default/files/distributions/public/dakota-6.16.0-public-src-cli.tar.gz Extract the source code:\n1 tar xzf dakota-6.16.0-public-src-cli.tar.gz Make a build directory. Usually we can make the build directory inside the source code directory:\n1 2 cd dakota-6.16.0-public-src-cli mkdir build Configure in the build directory:\n1 2 cd build cmake -DCMAKE_INSTALL_PREFIX=${HOME}/local/dakota/6.16.0 .. Here we specify ${HOME}/local/dakota/6.16.0 as the Dakota installation directory.\nBuild Dakota:\n1 make -j 4 It may take a while.\nTest Dakota # Test the Dakota build:\n1 2 cd test ctest -j 4 -L Accept If all the tests are passed, it shows the Dakota build was successful.\nInstall Dakota # Finally you can install Dakota:\n1 2 cd .. make install The installation location is ${HOME}/local/dakota/6.16.0. You should add the Dakota binary executable to the system PATH:\n1 export PATH=${HOME}/local/dakota/6.16.0/bin:$PATH so that the system can find it. Or you can make a symbolic link for it to link it to you executable directory. For example, I have all my user\u0026rsquo;s exectable or tools in $HOME/bin, so I linked Dakota binary executable to this directory:\n1 ln -s ${HOME}/local/dakota/6.16.0/bin/dakota $HOME/bin You can check whether Dakota is recongnized by the system using:\n1 dakota -v ","date":"20 September 2022","externalUrl":null,"permalink":"/en/p/build-dakota-from-source/","section":"Posts","summary":" Build Dakota # Prerequisites # Dakota relies on some other tools and libraries. Please refer to the Dakota’s official website. Most of the tools (such as CMake, Python) are already installed. The ones that need to take care of are:\nLinear Algebra libraries: BLAS and LAPACK Boost System information of my case:\n","title":"Build Dakota from Source","type":"post"},{"content":"","date":"20 September 2022","externalUrl":null,"permalink":"/en/tags/dakota/","section":"Tags","summary":"","title":"Dakota","type":"tags"},{"content":"","date":"20 September 2022","externalUrl":null,"permalink":"/en/categories/research/","section":"Categories","summary":"","title":"Research","type":"categories"},{"content":" Build SAM # Prerequisites # SAM relies on the MOOSE framework. We build MOOSE by following the instructions on the MOOSE website first.\nInstall Mambaforge3 # For Linux users:\n1 2 curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh bash Mambaforge-Linux-x86_64.sh -b -p ~/mambaforge3 After installing, put Mambaforge\u0026rsquo;s path to system environment PATH:\n1 export PATH=$HOME/mambaforge3/bin:$PATH so that the system recognizes the mamba command.\nThen initialize mambaforge by\n1 mamba init This command will append several mamba initialization to your ~/.bashrc file. Since I use zsh, I moved those lines from ~/.bashrc to ~/.zshrc.\nThen restart the terminal.\nInstall MOOSE # Add the INL public channel to Conda:\n1 conda config --add channels https://conda.software.inl.gov/public Create moose environment in Conda\n1 mamba create -n moose moose-dev Activate the environment\n1 mamba activate moose Clone SAM and Build # Clone SAM from ANL\u0026rsquo;s GitLab # Get an access token to the GitLab repository.\nGitLab -\u0026gt; Settings -\u0026gt; Access Tokens\nClone the SAM repository to your computer or cluster\n1 2 3 mkdir ~/Documents cd ~/Documents git clone https://git-nse.egs.anl.gov/*YourUserName_or_SAM*/SAM.git It will ask you for the GitLab username and password. Use the access token you generated for the password.\nBuild libmesh # Get moose submodule\n1 2 3 cd SAM git submodule init git submodule update --recursive Build SAM # Build SAM\n1 2 cd ~/Documents/SAM make -j4 Run the tests\n1 ./run_tests -j4 If SAM was built correctly, all the tests will pass.\n","date":"9 September 2022","externalUrl":null,"permalink":"/en/p/build-sam-from-source/","section":"Posts","summary":" Build SAM # Prerequisites # SAM relies on the MOOSE framework. We build MOOSE by following the instructions on the MOOSE website first.\nInstall Mambaforge3 # For Linux users:\n1 2 curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh bash Mambaforge-Linux-x86_64.sh -b -p ~/mambaforge3 After installing, put Mambaforge’s path to system environment PATH:\n","title":"Build SAM from Source","type":"post"},{"content":"","date":"9 September 2022","externalUrl":null,"permalink":"/en/tags/sam/","section":"Tags","summary":"","title":"SAM","type":"tags"},{"content":" Motivation # The default shell on the school\u0026rsquo;s computing cluster is bash, and my commonly used zsh is not available. It feels very uncomfortable without auto-completion and syntax highlighting. Moreover, ordinary accounts do not have administrative privileges and cannot use sudo to install zsh directly through package management tools. Therefore, it is necessary to compile and install zsh from source.\nInstall Dependency ncurse # The ncurse library (new curse) is an interface library that supports text-based user interfaces, and zsh depends on this library at runtime. Therefore, it is necessary to install the ncurse library before installing zsh.\nDownload ncurse version 6.1 (the latest version at the moment)\n1 wget https://ftp.gnu.org/pub/gnu/ncurses/ncurses-6.1.tar.gz --no-check-certificate Extract and enter the extracted directory\n1 2 tar xvfz ncurses-6.1.tar.gz cd ncurses-6.1 Configure the build\n1 ./configure --prefix=\u0026#34;$HOME/software\u0026#34; CXXFLAGS=\u0026#34;-fPIC\u0026#34; CFLAGS=\u0026#34;-fPIC\u0026#34; Here, the installation directory is specified as the software folder under the user\u0026rsquo;s home directory.\nCompile and install\n1 make \u0026amp;\u0026amp; make install Install zsh from Source # Download the latest version of zsh\n1 wget -O zsh.tar.xz https://sourceforge.net/projects/zsh/files/latest/download --no-check-certificate Extract and enter the extracted directory\n1 2 tar xvfz -C zsh zsh.tar.xz cd zsh Configure the build\n1 ./configure --prefix=\u0026#34;$HOME/software\u0026#34; CPPFLAGS=\u0026#34;-I$HOME/software/include\u0026#34; LDFLAGS=\u0026#34;-L$HOME/software/lib\u0026#34; Here, the installation directory is specified as the software folder under the user\u0026rsquo;s home directory.\nCompile and install\n1 make \u0026amp;\u0026amp; make install Finally, add the zsh binary folder to the system environment\n1 export PATH=$HOME/software/bin:$PATH You can also add this line to the bash configuration file .bashrc.\nInstall and Configure oh-my-zsh # Please refer to \u0026ldquo;oh-my-zsh Configuration for Command Line Users\u0026rdquo;.\n","date":"26 August 2022","externalUrl":null,"permalink":"/en/p/compile-and-install-zsh-from-source-without-sudo-permission/","section":"Posts","summary":" Motivation # The default shell on the school’s computing cluster is bash, and my commonly used zsh is not available. It feels very uncomfortable without auto-completion and syntax highlighting. Moreover, ordinary accounts do not have administrative privileges and cannot use sudo to install zsh directly through package management tools. Therefore, it is necessary to compile and install zsh from source.\n","title":"Compile and Install zsh from Source without sudo Permission","type":"post"},{"content":"","date":"2022年8月18日","externalUrl":null,"permalink":"/tags/module/","section":"Tags","summary":"","title":"Module","type":"tags"},{"content":"","date":"2022年8月18日","externalUrl":null,"permalink":"/tags/%E7%8E%AF%E5%A2%83%E5%8F%98%E9%87%8F/","section":"Tags","summary":"","title":"环境变量","type":"tags"},{"content":" Linux发行版 # Linux系统是一个自由软件的操作系统，它的内核是Linux内核，由芬兰计算机科学家Linus Torvalds在1991年发布。Linux系统的内核是自由软件，因此任何人都可以基于Linux内核发布自己的操作系统，这些操作系统就是Linux发行版。\n文件系统 # 文件系统在任何一个计算机系统中都是核心功能之一，不同操作系统会使用不同的文件系统，这里我们通过对比Windows系统和Linux系统，来简单了解一下Linux系统中的文件结构。\n目录树 # 计算机中的各种工具、软件程序、数据等都是以文件方式存储的。这些文件大部分存储在计算机的硬盘上，通常以树状结构组织起来，例如在Windows系统中，文件系统的目录树可能如下所示：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 此电脑 ├── 磁盘C │ ├── $Recycle.Bin │ ├── PerfLogs │ ├── Program Files │ ├── Program Files (x86) │ ├── ProgramData │ ├── Users │ │ ├── All Users -\u0026gt; /mnt/c/ProgramData │ │ ├── Default │ │ ├── Guest │ │ ├── Public │ │ ├── Visitor │ │ ├── desktop.ini │ │ └── lijin │ │ ├── .ssh │ │ ├── AppData │ │ ├── Desktop │ │ ├── Documents │ │ ├── Downloads │ │ ├── Music │ │ ├── Pictures │ │ ├── Videos │ │ └── Favorites │ └── Windows ├── CD-ROM │ └── 我的光盘 │ └── 磁盘F 以下是Windows系统中一些比较重要的目录：\n用户根目录\nWindows系统的用户根目录在C:\\Users\\文件夹下，每个用户有一个文件夹，例如我的用户根目录是C:\\Users\\lijin。在命令行中，用户根目录可以用~表示。\n用户根目录下一般存储用户的个人文件，例如个人的文档、照片、音乐、电影等可以分类存储到用户根目录下的Documents、Pictures、Music、Videos等文件夹下。\n只限当前用户使用的程序一般安装到用户根目录的AppData文件夹内，这个文件夹系统默认是隐藏的，用户在一般情况下不应该动这个文件夹。\nWindows系统目录\nWindows系统的系统根目录是C:\\Windows，操作系统和系统级的软件工具都在这个目录下，用户不应该轻易动这个文件夹，否则可能会损坏系统。\n应用程序目录\n用户一般可以将程序安装到C:\\Program Files (x86)和C:\\Program Files下，其中C:\\Program Files (x86)一般安装32位程序，C:\\Program Files安装64位程序。\n安装到这两个目录下的程序对于当前计算机的所有用户都是可见的。但有些商用软件可能会限制使用者，未注册的用户虽然可以看到该软件但却有可能无法运行使用。\n在Linux系统中，文件系统的目录树可能如下所示：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 系统根目录 ├── bin -\u0026gt; usr/bin ├── boot ├── dev ├── etc ├── home │ ├── Visitor │ ├── otherusers │ └── lijin │ ├── .ssh │ ├── AppData │ ├── Desktop │ ├── Documents │ ├── Downloads │ ├── Music │ ├── Pictures │ ├── Videos │ └── Favorites ├── lib -\u0026gt; usr/lib ├── lost+found ├── media ├── mnt │ ├── 硬盘F │ ├── 虚拟光驱 │ └── 我的U盘 ├── opt ├── root ├── run ├── snap ├── sys ├── tmp ├── usr └── var 以下是Linux系统中一些比较重要的目录：\n用户根目录\nLinux系统的用户根目录在/home/文件夹下，每个用户有一个文件夹，例如我的用户根目录是/home/lijin。\n用户根目录下一般存储用户的个人文件，例如个人的文档、照片、音乐、电影等可以分类存储到用户根目录下的Documents、Pictures、Music、Videos等文件夹下。\n只限当前用户使用的程序一般安装到用户根目录的AppData文件夹内，这个文件夹系统默认是隐藏的，用户在一般情况下不应该动这个文件夹。\nLinux系统目录\nLinux系统的系统根目录是/sys，操作系统在这个目录下，用户不应该轻易动这个文件夹，否则可能会损坏系统。\n应用程序目录\n用户一般可以将程序安装到/opt下。\n库目录\n系统默认的一些运行库和工具软件一般在/usr下，用户在一般情况下不应该动这个文件夹。\n注意：\n在Windows系统中，路径是带盘符的，例如C:表示磁盘C下的路径；而Linux系统中一般不用字母表示的盘符。 在Windows系统中，目录分隔符是反斜杠\\；在Linux系统中，目录分隔符则是斜杠/。 磁盘分区 # 对于机械硬盘而言，存储空间的起点在圆盘的外圈，磁头读取数据时也是从外圈开始寻址，位于磁盘外圈的数据访问速度较快。所以我们往往将硬盘进行分区，位于外层的分区用来安装操作系统和软件，内层的分区用来放数据文件，这样可以让系统和软件运行比较流畅。\n而对于固态硬盘而言，硬盘各部位的数据访问速度都差不多，因此单就访问速度而言，固态硬盘一般没有必要进行磁盘分区。\n除了区分访问速度的不同，硬盘分区的另一个作用是隔离各个分区的数据。例如将操作系统和用户的电影音乐等文件放置在不同分区，这样当需要重新安装操作系统时，只需要格式化操作系统所在的分区，用户的个人文件不会受到影响。\n挂载点 # 在Linux系统中，存储设备需要挂载（mount）到文件目录树的某个位置，这个位置就是挂载点。例如在手动分区并安装Linux系统时，我们往往会创建home、swap、usr等分区，这些分区将被自动挂载到系统根目录/下。如果我们给计算机外接一个硬盘或U盘，则这个硬盘或U盘一般会被挂载到/mnt/下。\n需要注意的是，挂载的存储设备不一定非得是一块真实的硬盘，也可以是网络云盘、远程的计算机，甚至是本电脑上的一个虚拟的光盘文件！例如在数年前，不少计算机软件或操作系统都存储在光盘上，用户可以购买存储有某个软件的光盘，然后用电脑上的光盘驱动器读取安装软件或操作系统。然而现在许多笔记本电脑为了做得轻便，都取消了光盘驱动器，用户也基本不再购买软件光盘了，取而代之的是在网上下载安装包，而有不少安装包依旧以光盘文件的格式来存储软件，这时用户就可以通过虚拟光驱将这个“虚拟的光盘”挂载到电脑上。\n软件安装与管理 # 我们选择使用某个操作系统的原因可能很多，但有一条原因往往是普遍的，那就是可以方便地使用这个操作系统上的一些特色软件。在安装了某个操作系统之后，我们往往还需要安装各种软件才能真正让这个操作系统发挥它的作用，成为生产力工具。\n在大多数用户都熟悉的Windows系统上，我们可以在一些软件应用商店安装某个软件，或者到某个软件的官网下载安装包，双击安装包来安装。但这些都需要在图形界面上点击来完成，但对于某些Linux（例如在Windows上安装的WSL，或者学校的计算集群）而言，我们可能并没有一个可用的图形界面，这时我们就需要熟悉在命令行界面下安装软件的技巧。这里我们介绍在Linux命令行下三种常见的软件安装方法：\n使用包管理工具 手动下载安装包 从源码编译安装 使用包管理工具 # 图形化的操作系统上往往会配备“应用中心”等图形化的软件管理工具，在命令行化的界面上，我们也可以有类似“应用中心”的软件包管理工具。下面以Debian系（包括Debian、Ubuntu、Deepin等系统）常用的包管理工具APT来简单介绍一下在Linux命令行界面内安装软件的方法。\nAPT简介 # APT是Advanced Packaging Tools（高级打包工具）的缩写，是在Debian系操作系统中最常用的包管理工具，它可以自动下载、配置以及安装很多软件包。通常情况下，我们在命令行内通过apt或apt-get等命令来使用APT，但需要注意的是，除此之外，APT还可以通过aptitude、Synaptic、GNOME软件商店等前端来使用。这里我们只介绍通过apt命令使用APT的方法。\n使用apt命令管理软件 # 搜索软件包\n1 apt search software_name 显示软件包详情\n1 apt show software_name 安装软件包\n1 apt install software_name 卸载软件包\n1 apt remove software_name 卸载软件包并删除配置文件\n1 apt purge software_name 自动删除不需要的软件包\n1 apt autoremove 更新软件仓库\n1 apt update 升级软件仓库\n1 apt upgrade 使用以上命令安装、更新或卸载软件时可能需要sudo权限。\n手动下载安装包 # 有些软件可能不在APT仓库里，无法直接通过APT安装。不过开发者可能提供.deb或.rpm格式的安装包，.deb格式的安装包一般在Debian系系统中使用，.rpm格式的安装包一般在Red Hat系系统（Red Hat、CentOS、Fedora等）中使用。\n例如文本编辑器Visual Studio Code （VS code）就不在APT仓库里，想要在Linux下安装，我们就需要下载它的安装包。\n下载完成后，进入存放安装包的文件夹，执行下面的命令就可以安装了：\n1 sudo dpkg -i package_name.deb 从源码编译安装 # 从源码编译安装的情形一般有3种：\n有些软件可能连安装包都不提供，开发者只提供源代码，这是我们就需要从源码编译安装。 通过APT和安装包安装都需要sudo权限，而有时候我们并没有sudo权限。例如在学校的计算集群上时，我们的账户只是普通用户，没有管理员权限，所以无法使用sudo。 我们需要对软件做一些改动以满足自己具体的需求，这时需要先更改源代码，再自己编译安装。 例如很多Linux用户喜欢使用zsh，而在学校的计算集群上往往只提供bash，且用户没有管理员权限无法使用包管理工具或者安装包直接安装zsh。这时如果依旧想要使用zsh，就需要用户自己从源码编译安装。\n图形界面GUI # 在Windows系统和macOS系统中，我们通常通过鼠标点击来操作电脑，这种操作方式叫做图形用户界面（Graphical User Interface，GUI），图形界面可以说是Windows和macOS系统的标志，也为计算机在普通人中的普及起到了巨大的作用。\n而在Linux系统中，图形界面长久以来未受到足够的重视，这也是Linux系统在普通人中难以普及的原因之一。尽管如此，近年来Linux系统的图形界面还是取得了不小的进步，现在的Linux系统图形界面也越来越美观、易用。\n在Windows系统和macOS系统中，图形界面是固定的，也是默认开启的，用户可以开箱即用。而在Linux系统中，图形界面不是必须的，用户可以选择是否安装图形界面，也可以选择安装不止一个图形界面。我们接下来简单介绍一下Linux系统中的图形界面。\nLinux图形界面——堆叠式窗口管理器和平铺式窗口管理器 # 按照操作习惯来分，Linux系统中的图形界面可以分为两大类：堆叠式窗口管理器（Stacking Window Manager）和平铺式窗口管理器（Tiling Window Manager）。\n堆叠式窗口管理器 # 堆叠式窗口管理器是最常见的窗口管理器，它的窗口可以重叠在一起，用户可以通过鼠标点击来切换窗口。常见的堆叠式窗口管理器有GNOME、KDE、Xfce等。\n堆叠式窗口管理器也是最常见的桌面环境，包括Windows和macOS系统的图形界面都使用堆叠式窗口管理器。\nGNOME\nGNOME是一个自由软件的桌面环境，它的设计理念是简洁、易用。GNOME的界面风格简洁大方，用户可以通过GNOME的扩展来定制自己的桌面环境。\nGNome是很多发行版的默认桌面环境，例如Ubuntu、Fedora等。\nKDE\nKDE是另一个自由软件的桌面环境，它的设计理念是强大、灵活、可定制性强。KDE的界面风格更加华丽，用户可以通过KDE的设置来定制自己的桌面环境。\nKDE也是很多发行版的默认桌面环境，例如Kubuntu、openSUSE等。\n平铺式窗口管理器 # 平铺式窗口管理器的窗口一般不重叠在一起，而是平铺在屏幕上，用户可以通过快捷键来切换窗口。常见的平铺式窗口管理器有i3、dwm、bspwm、Hyprland等。\n平铺式窗口管理器并不是很常见，一般只有一些专业用户或者极客用户会使用。其特点是很多操作都可以通过快捷键来完成，而且几乎所有设置都可以由用户自定义，因此用户可以高度定制，排除不需要的功能，保持系统的简洁。并且几乎所有操作都可以按照用户自己习惯的方式来配置，因此可以大大提高熟练用户的工作效率。\ni3\ni3是一个自由软件的平铺式窗口管理器，它的设计理念是简洁、高效。i3的界面风格简洁大方，用户可以通过i3的配置文件来定制自己的桌面环境。\nHyprland\nHyprland是一个自由软件的平铺式窗口管理器，基于Wayland。它的设计理念是简洁、高效。Hyprland的界面风格简洁大方，内置的窗口切换动画非常优雅流畅。目前Hyprland还在开发中，但已经受到了很多极客用户的喜爱。\nLinux图形界面——X Window系统和Wayland # 在Linux系统中，图形界面的实现主要有两种技术：X Window系统和Wayland。要想知道自己的Linux系统使用的是哪种技术，可以在终端输入echo $XDG_SESSION_TYPE，如果输出x11，则表示使用的是X Window系统；如果输出wayland，则表示使用的是Wayland。\nX Window系统 # X Window系统是一个自由软件的图形窗口系统，它是Linux系统中最常用的图形窗口系统。X Window系统的设计理念是基于服务端-客户端的架构，服务端负责管理硬件设备，客户端负责绘制图形界面。\n需要注意的是，X Window系统中客户端和服务端的定义与传统意义上的客户端和服务端不同。例如如果你有一台电脑A在远程，然后你在本地的电脑B上使用ssh登录到电脑A，而且在电脑B上显示了电脑A的图形界面。 这时对于ssh服务来说，电脑A是服务端，电脑B是客户端；但对于X Window系统来说，电脑A是客户端，电脑B是服务端。 因为对X Window系统来说，你在电脑B上起了一个X Window服务，而电脑A上的程序是客户端，通过这个服务来显示图形界面。\nX Window系统是在1984年发布的，而后快速迭代，不断发布新的版本。到1987年，X Window系统的第11个版本发布，自此版本之后，X Window系统的标准就基本固定下来了，因此我们通常称X Window系统为X11。\nXorg是X11的一个实现，也是Linux系统中最常用的X11实现。\nWayland # 随着硬件的发展和用户需求的变化，X Window系统的一些设计理念逐渐显得过时，例如X Window系统的设计理念是基于服务端-客户端的架构，这种架构在现代硬件上显得有些过时，因此一些开发者开始尝试设计新的图形窗口系统。\nWayland是一个自由软件的图形窗口系统，它的设计理念是简单、高效。Wayland的设计理念是基于内核-客户端的架构，内核负责管理硬件设备，客户端负责绘制图形界面。\nWayland的设计理念更加现代化，因此在一些新的Linux发行版中，例如Fedora、Arch Linux等，Wayland已经取代了X Window系统成为默认的图形窗口系统。\n而且Wayland对高分辨率屏幕（HiDPI）的支持更好，有些基于X11的软件在高分辨率屏幕下的显示过小，而如果通过缩放来解决这个问题，会导致图形界面模糊，而Wayland则可以更好地支持高分辨率屏幕。\n","date":"2022年8月8日","externalUrl":null,"permalink":"/p/linux%E7%B3%BB%E7%BB%9F%E5%85%A5%E9%97%A8%E7%9F%A5%E8%AF%86%E7%AE%80%E4%BB%8B/","section":"Posts","summary":" Linux发行版 # Linux系统是一个自由软件的操作系统，它的内核是Linux内核，由芬兰计算机科学家Linus Torvalds在1991年发布。Linux系统的内核是自由软件，因此任何人都可以基于Linux内核发布自己的操作系统，这些操作系统就是Linux发行版。\n","title":"Linux系统入门知识简介","type":"post"},{"content":"","date":"2022年8月8日","externalUrl":null,"permalink":"/tags/wsl/","section":"Tags","summary":"","title":"WSL","type":"tags"},{"content":"","date":"2 August 2022","externalUrl":null,"permalink":"/en/tags/color-scheme/","section":"Tags","summary":"","title":"Color Scheme","type":"tags"},{"content":" Motivation # Sometimes the default MatLab colors for plottings are not satisfactory and I would like to define and use my own color schemes. When I see a beautiful color scheme, I also would like to save it to my color scheme library. But when I want to use a certain color scheme, it might be difficult to find it in my library. So here I created this project to save, preview and use customized color schemes in MatLab.\nFeatures # Preview a certain color scheme, or preview all color schemes Get and use a color scheme Define and save a color scheme Create a color scheme by sampling points on a image/picture Prerequisite # MatLab version $\\geq$ 2021a\nOptional. Function tiledlayout(), which was introduced to MatLab in vewrsion 2021a, is used in jlShowColor.m. If your MatLab version is older than 2021a, you need to modify the source code in the jlShowColor.m file and change the tiledlayout() to some equivalent functions, such as subplot().\nImage Processing Toolbox\nOptional. This MatLab toolbox is needed when you want to get colors from a picture to define customized color schemes.\nInstall # By default, the user-defined scripts, functions and tools are placed in directory ~/Documents/MATLAB. Here I suggest you should also clone this jlcolor library to this directory:\n1 git clone https://github.com/jin-li/jlcolor.git ~/Documents/MATLAB/jlcolor Then add the path to MATLAB。 You can do it with commands, too. Open MatLab and run following commands in the MatLab command window:\nFor Windows users 1 2 addpath([getenv(\u0026#39;USERPROFILE\u0026#39;),\u0026#39;\\Documents\\MATLAB\\jlcolor\u0026#39;]); savepath; For Linux or macOS users 1 2 addpath([getenv(\u0026#39;HOME\u0026#39;),\u0026#39;/Documents/MATLAB/jlcolor\u0026#39;]); savepath; Usage # Preview a Color Scheme\nOpen MatLab and run following command to preview all the color schemes in the library: 1 jlShowColor(\u0026#39;all\u0026#39;) Then all the color schemes will show up with their names: You can also preview a single color scheme. For example, you can use following command to preview the color scheme named \u0026ldquo;rainbow\u0026rdquo;: 1 jlShowColor(\u0026#39;rainbow\u0026#39;) Get and Use a Color Scheme\nYou can get a color scheme in the library with function jlGetColor(). For example, using following command 1 c = jlGetColor(\u0026#39;rainbow\u0026#39;) you can export the seven colors in color scheme \u0026ldquo;rainbow\u0026rdquo; to variable c. Then you can use the seven colors when you make a plot. For example, 1 2 3 4 5 6 c = jlGetColor(\u0026#39;rainbow\u0026#39;); % get color scheme figure; hold on; x=0:0.01:1; for i=1:length(c) plot(x,sin(2*pi*x-pi*i/12),\u0026#39;color\u0026#39;,c(i),\u0026#39;linewidth\u0026#39;,5); end Here is how the plot looks like: Self-define and Save a Color Scheme\nThe color schemes are stored in file jlColorLib.m. Users can add color schemes to the file. For example, the default color scheme in MatLab is: One can add the hex code of the seven colors to jlColorLib.m by mimicing the existing color schemes in the file. So you can add another line in jlColorLib.m:\n1 jlColor(\u0026#39;matlab\u0026#39;) = [\u0026#39;#0072BD\u0026#39;;\u0026#39;#D95319\u0026#39;;\u0026#39;#EDB120\u0026#39;;\u0026#39;#7E2F8E\u0026#39;;\u0026#39;#77AC30\u0026#39;;\u0026#39;#4DBEEE\u0026#39;;\u0026#39;#A2142F\u0026#39;]; Then the default MatLab color scheme is saved in the library with a name \u0026ldquo;matlab\u0026rdquo;.\nOr you can use function jlAddColor() to add a color scheme to the library. For example, you can run following commands in MatLab\u0026rsquo;s command window:\n1 2 colors = [\u0026#34;#0072BD\u0026#34;,\u0026#34;#D95319\u0026#34;,\u0026#34;#EDB120\u0026#34;,\u0026#34;#7E2F8E\u0026#34;,\u0026#34;#77AC30\u0026#34;,\u0026#34;#4DBEEE\u0026#34;,\u0026#34;#A2142F\u0026#34;]; jlAddColor(colors, \u0026#34;matlab\u0026#34;); Create a Color Scheme by Sampling Points on a Picture\nYou can create you own color scheme conveniently by sampling points on a picture, with the help of function jlAddColorFromImage(). For example, the color scheme \u0026ldquo;rainbow\u0026rdquo; used above and the color scheme \u0026ldquo;sunset\u0026rdquo; in the library are all generated from pictures. Suppose you have the picture file \u0026ldquo;sunset.jpeg\u0026rdquo;, then you can run following command in MatLab\u0026rsquo;s command window:\n1 jlAddColorFromImage(\u0026#39;sunset.jpeg\u0026#39;,\u0026#39;sunset\u0026#39;); Then a window like below will pop out. And you can click on it with your mouse for the colors you need: Press Enter key when you are done selecting the points. Then another window showing the color scheme being created successfully will pop out! Acknowledgement # The function that converts RGB to hex code used in the library is from the Mathworks community user Chad Greene\u0026rsquo;s script. Some of the color schemes in this library are from Zhihu user CoderMan\u0026rsquo;s post 【2.1.1 颜色模板】科研绘图必备的60套颜色模板. ","date":"2 August 2022","externalUrl":null,"permalink":"/en/p/customizable-matlab-color-library-jlcolor/","section":"Posts","summary":" Motivation # Sometimes the default MatLab colors for plottings are not satisfactory and I would like to define and use my own color schemes. When I see a beautiful color scheme, I also would like to save it to my color scheme library. But when I want to use a certain color scheme, it might be difficult to find it in my library. So here I created this project to save, preview and use customized color schemes in MatLab.\n","title":"Customizable MatLab Color Library jlcolor","type":"post"},{"content":"","date":"2 August 2022","externalUrl":null,"permalink":"/en/tags/matlab/","section":"Tags","summary":"","title":"MatLab","type":"tags"},{"content":"","date":"2 August 2022","externalUrl":null,"permalink":"/en/tags/oh-my-zsh/","section":"Tags","summary":"","title":"Oh-My-Zsh","type":"tags"},{"content":" Shell Configuration # Configurations for Linux shell and Neovim\nLinux Shell Configuration # Features # Shell: oh-my-zsh Theme: powerlevel10k Plugins: zsh-autosuggestions, zsh-syntax-highlighting Notice # The powerlevel10k theme is no longer maintained as of May 2024, but it should still work fine. The author, romkatv, believes that powerlevel10k is mature and feature-complete. Most issues are either personalized requests or involve underlying changes that are unnecessary for 99% of users. If you are not in that 99%, you need to be able to modify the code to meet your needs. Therefore, romkatv has decided to stop maintaining the powerlevel10k theme. Here are the relevant discussions:\npowerlevel10k issue #2690 reddit discussion This post is now deprecated as the primary installation guide. The P10k-based configuration is kept here for reference, but new installations should use the successor post and its Oh My Posh-based configuration. The new setup uses one theme across Zsh and PowerShell and separates shared configuration from machine-local settings.\nIf you still need the legacy P10k setup, continue with the instructions below. Otherwise, follow the Oh My Posh migration guide.\nOther alternatives include:\nstarship: A cross-platform command line prompt that supports various shells. Compared to powerlevel10k, starship is easier to configure. oh-my-posh: A cross-platform command line prompt that was originally designed for PowerShell but now supports various shells, including zsh. oh-my-posh is also easy to configure. Install # Method 1: Use Installation Script (Recommend) # Download the installation script:\n1 curl -s https://raw.githubusercontent.com/jin-li/ShellConfig/main/install_OhMyZsh_p10k.sh -o install_OhMyZsh_p10k.sh Run the script:\nDo NOT run it in root NOR using sudo, just run it as a normal user. When it prompts for password, then enter the root or sudo password.\n1 2 chmod +x install_OhMyZsh_p10k.sh ./install_OhMyZsh_p10k.sh Method 2: Manual Install # Install Meslo Nerd Font\nDownload these four ttf files:\nMesloLGS NF Regular.ttf MesloLGS NF Bold.ttf MesloLGS NF Italic.ttf MesloLGS NF Bold Italic.ttf Double-click on each file and click \u0026ldquo;Install\u0026rdquo;. This will make MesloLGS NF font available to all applications on your system.\nFor setting up the font in the preference of the terminals, please refer to p10k Font.\nInstall zsh\nUbuntu, Debian \u0026amp; derivatives\nUpdate apt sources\n1 sudo apt update Install prerequisite tools (git, wget, python3, pip3)\n1 sudo apt install wget git python3-dev python3-pip python3-setuptools Install zsh\n1 sudo apt install zsh Arch Linux or Manjaro\n1 sudo pacman -S zsh Fedora\n1 sudo dnf install zsh Install oh-my-zsh via wget\n1 sh -c \u0026#34;$(wget -O- https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\u0026#34; Install powerlevel10k\n1 git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k Install plugins and tools\nzsh-autosuggestions 1 git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions fast-syntax-highlighting 1 git clone https://github.com/zdharma-continuum/fast-syntax-highlighting.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting Download configuration files and link to home directory\nDownload ShellConfig from GitHub 1 2 3 mkdir -p ~/Documents/GitHub cd ~/Documents/GitHub git clone https://github.com/jin-li/ShellConfig.git Backup old configuration files 1 2 mv ~/.bashrc ~/.bashrc_bak mv ~/.zshrc ~/.zshrc_bak Link configuration files to home directory 1 2 3 ln ~/Documents/GitHub/ShellConfig/.bashrc ~/.bashrc ln ~/Documents/GitHub/ShellConfig/.zshrc ~/.zshrc ln ~/Documents/GitHub/ShellConfig/.p10k.zsh ~/.p10k.zsh Restart terminal\nVim Configuration # Motivation # I migrated from vim to Neovim because of the better performance and more features.\nLazyVim is a starter-friendly configuration for Neovim with the commonly used plugins. It is easy to install and use.\nMethod 1: Use Installation Script (Recommend) # Download the installation script:\n1 curl -s https://raw.githubusercontent.com/jin-li/ShellConfig/main/install_LazyVim.sh -o install_LazyVim.sh Run the script:\n1 2 chmod +x install_LazyVim.sh ./install_LazyVim.sh Method 2: Manual Install # Remove vim-tiny or vim-minimal\nThe default vim in Debian OS is vim-tiny (in Fedora is vim-minimal), which does not support the plugins. Use following command to check your vim version first before using this configuration.\n1 vi --version If there is \u0026ldquo;Small version without GUI\u0026rdquo; in the output, it means your vim is vim-tiny or vim-minimal. You can remove it:\nDebian / Ubuntu 1 sudo apt remove vim-tiny Fedora 1 sudo dnf remove vim-minimal Install Neovim\nThen install Neovim.\nDebian / Ubuntu\nSince LazyVim requires Neovim 0.8 or later, in Ubuntu 22.04, the neovim version is 0.6.1. Therefore, We need to install the newer version of neovim from the official website.\nDownload the pre-built binaries of Neovim from the official website 1 curl -LO https://github.com/neovim/neovim/releases/latest/download/nvim-linux64.tar.gz Extract the downloaded file 1 sudo tar -C /opt -xzf nvim-linux64.tar.gz This will install Neovim to /opt/nvim-linux64 directory. Create a symbolic link to the Neovim executable 1 sudo ln -s /opt/nvim-linux64/bin/nvim /usr/bin/nvim Fedora\nIn Fedora 40, the Neovim version is 0.9.5, which is new enough to use LazyVim. So we can install it using dnf directly.\n1 sudo dnf install neovim Install LazyVim\nRefer to the official website, we can install LazyVim using the following command:\nInstall LazyVim\n1 git clone https://github.com/LazyVim/starter ~/.config/nvim Remove the .git folder, so you can add it to your own repo later\n1 rm -rf ~/.config/nvim/.git First time run nvim will install the plugins automatically. It may take a while to finish.\n1 nvim Replace macOS Terminal # After installing LazyVim, I found that the display effect of LazyVim in macOS Terminal is very poor, with dim colors and low contrast:\nThe reason is that the macOS Terminal\u0026rsquo;s color scheme does not support 256 true colors, while LazyVim defaults to a 256-color color scheme. I heard that iTerm2 supports 256 true colors, so I decided to migrate from the macOS built-in terminal to iTerm2. After installing iTerm2, you need to set iTerm2 to use the Meslo Nerd font you downloaded and installed earlier. Then open LazyVim in iTerm2, it looks much better now:\nIssues # On Fedora 42, after running the above script, when opening nvim, the nvim interface shows the following error:\n1 No C compiler found! \u0026#34;cc\u0026#34;, \u0026#34;gcc\u0026#34;, \u0026#34;clang\u0026#34;, \u0026#34;cl\u0026#34;, \u0026#34;zig\u0026#34; are not executable. It\u0026rsquo;s because some plugins in LazyVim need to be compiled, but there is no C compiler installed on the system. You can install a C compiler, such as gcc or clang, to resolve this issue. For me, I installed zig, which is a lightweight compiler, also supports C:\n1 sudo dnf install zig After installing zig, restart nvim, and the error should be resolved.\n","date":"2 August 2022","externalUrl":null,"permalink":"/en/p/oh-my-zsh-shell-and-neovim-configuration-for-commandline-users/","section":"Posts","summary":" Shell Configuration # Configurations for Linux shell and Neovim\nLinux Shell Configuration # Features # Shell: oh-my-zsh Theme: powerlevel10k Plugins: zsh-autosuggestions, zsh-syntax-highlighting Notice # The powerlevel10k theme is no longer maintained as of May 2024, but it should still work fine. The author, romkatv, believes that powerlevel10k is mature and feature-complete. Most issues are either personalized requests or involve underlying changes that are unnecessary for 99% of users. If you are not in that 99%, you need to be able to modify the code to meet your needs. Therefore, romkatv has decided to stop maintaining the powerlevel10k theme. Here are the relevant discussions:\n","title":"oh-my-zsh Shell and Neovim Configuration for Commandline Users","type":"post"},{"content":"","date":"2 August 2022","externalUrl":null,"permalink":"/en/tags/plotting/","section":"Tags","summary":"","title":"Plotting","type":"tags"},{"content":" 封面 # 内容概览 # 目录 # 计算机简介\n计算机发展简史 手动式计算机 机械式计算机 机电式计算机 电子计算机 现代电子计算机 计算机的分类 按运行原理分 按通用性分 按适用范围分 第1章习题 现代数字计算机基础数学物理知识\n二进制 数的进制 进制转换 编码——万物皆数 逻辑代数 简单逻辑运算 运算律 逻辑函数与化简 逻辑电路 逻辑门 组合逻辑 可编程逻辑 第2章习题 现代计算机基本原理简介\n图灵机模型 图灵机 可计算性 图灵完备 计算机体系结构简介 计算机体系结构 冯·诺伊曼结构 哈佛结构 改进型哈佛结构 第3章习题 计算机基本硬件简介\n处理器 CPU发展简史 CPU的分类 CPU的结构 CPU制造工艺简介 显卡简介 存储设备 寄存器 缓存 主存 硬盘 输入输出 输入设备 输出设备 主板和外设 主板 外设 第4章习题 操作系统简介\n操作系统基本结构 驱动 内核 接口库 壳 操作系统分类 按内核结构分 按应用平台分 第5章习题 计算机软件简介\n计算机的启动过程简介 上电 BIOS 加载启动项 启动操作系统 编程语言简介 按照等级分 按照运行方式分 算法 算法基本概念和表示方法 算法复杂度简介 数据结构简介 软件工程简介 编译过程简介 软件开发流程 第6章习题 附录\n国际单位制词头 摩尔定律发展图 ASCII编码表 英特尔和AMD桌面级CPU世代表 索引\n名词索引 人名索引 下载PDF # 可以在本书GitHub仓库的发布页面(Releases)下载本书的PDF。\n仓库暂不提供本书的LaTeX源码 本书后续若有修订或更新，亦会同步到仓库的发布页面 发布页面提供压缩版和未压缩版两种格式 文件名带有_compressed的为压缩版，文件体积小，但图片质量差 另一个为未压缩版，文件体积较大，但图片清晰度高 版权 # 本书使用CC-BY-NC-ND 4.0协议。\n作者保留本书一切权利 允许在各个互联网平台上复制、下载、分享、传播本书，但须注明作者和出处(出处即本仓库，地址：https://github.com/jin-li/ComputerIntro_release) BY：必须按照作者或者许可人指定的方式对作品进行署名 NC：不得为商业目的而使用本书 ND：不得改变、转变或更改本书 鸣谢 # 本书使用了ElegantLaTeX项目制作的ElegantBook书籍模板。\n","date":"2022年7月22日","externalUrl":null,"permalink":"/p/%E7%8E%B0%E4%BB%A3%E8%AE%A1%E7%AE%97%E6%9C%BA%E8%BD%AF%E7%A1%AC%E4%BB%B6%E7%AE%80%E4%BB%8B%E5%BD%93%E4%BB%A3%E7%A0%81%E9%81%87%E4%B8%8A%E7%94%B5%E8%B7%AF/","section":"Posts","summary":" 封面 # 内容概览 # 目录 # 计算机简介\n","title":"《现代计算机软硬件简介——当代码遇上电路》","type":"post"},{"content":"","date":"2022年7月22日","externalUrl":null,"permalink":"/tags/%E7%BB%98%E5%9B%BE/","section":"Tags","summary":"","title":"绘图","type":"tags"},{"content":"","date":"2022年7月22日","externalUrl":null,"permalink":"/tags/%E8%AE%A1%E7%AE%97%E6%9C%BA%E5%9F%BA%E7%A1%80/","section":"Tags","summary":"","title":"计算机基础","type":"tags"},{"content":"","date":"2022年7月22日","externalUrl":null,"permalink":"/tags/%E8%AE%A1%E7%AE%97%E6%9C%BA%E7%A1%AC%E4%BB%B6/","section":"Tags","summary":"","title":"计算机硬件","type":"tags"},{"content":"","date":"2022年7月22日","externalUrl":null,"permalink":"/tags/%E8%AE%A1%E7%AE%97%E6%9C%BA%E8%BD%AF%E4%BB%B6/","section":"Tags","summary":"","title":"计算机软件","type":"tags"},{"content":"","date":"2022年7月22日","externalUrl":null,"permalink":"/tags/%E9%85%8D%E8%89%B2/","section":"Tags","summary":"","title":"配色","type":"tags"},{"content":"","date":"2022年4月1日","externalUrl":null,"permalink":"/tags/bash/","section":"Tags","summary":"","title":"Bash","type":"tags"},{"content":"","date":"2022年4月1日","externalUrl":null,"permalink":"/tags/linux%E5%AD%90%E7%B3%BB%E7%BB%9F/","section":"Tags","summary":"","title":"Linux子系统","type":"tags"},{"content":" 简介 # Windows的Linux子系统（Windows Subsystem for Linux），简称WSL，是微软公司开发的一个可以在Windows 10或Windows 11上运行原生Linux二进制程序的兼容层。借助WSL，Windows用户不再需要安装Linux系统就可以使用Linux上的工具和程序，而且可以在WSL中直接使用Windows系统的文件系统，用户在两个系统间的切换也更加流畅和方便。\nWSL的初始版本，或WSL1在2016年8月发布。WSL1是一个Linux兼容层，用户在WSL1中输入的Linux指令由微软实现的接口转换为对于Windows的调用，这种模式具有不少的局限性，例如它只能运行一部分Linux软件，缺乏音频支持和GPU支持等。\n微软在2019年6月发布了WSL的第二个版本，也是目前最常用的一个版本WSL2。WSL2使用了虚拟化的技术，使得在Windows上运行Linux内核和各种发行版成为可能，同时WSL2对于图形化的支持更好，利用gWSL可方便地运行一些图形化Linux程序。\n因此这里我们推荐安装WSL2，本文中若无特殊说明，提到WSL时均指WSL2。\n前提 # Windows 10版本号高于或等于1903（2019年春季更新），最好高于或等于2004（2020年春季更新） 安装 # 方法一（Windows 10版本高于或等于2004） # 在开始菜单搜索\u0026quot;Windows Powershell\u0026quot;，然后右击选择“以管理员身份运行”，然后在打开的命令行窗口输入如下命令\n1 wsl.exe --install 然后回车运行。等待数分钟后WSL2就会自动完成安装，然后需要重启Windows，在WSL命令窗口根据提示设置用户名和密码。\n方法二（Windows 10版本高于或等于1903，但低于2004） # 开启WSL。 在开始菜单搜索\u0026quot;Windows Powershell\u0026quot;，然后右击选择“以管理员身份运行”，然后在打开的命令行窗口输入如下命令并运行\n1 dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart 开启Windows 10自带的虚拟机平台。 继续在Powershell的命令行窗口（管理员身份运行）输入如下命令并运行\n1 Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart 将WSL默认版本设置为WSL2。 继续在Powershell的命令行窗口（管理员身份运行）输入如下命令并运行\n1 wsl --set-default-version 2 安装Linux发行版。 然后你可以在微软应用商店搜索并安装支持的Linux发行版。这里我们推荐安装Ubuntu。安装好之后打开你安装的Linux发行版，根据提示设置用户名和密码。\n","date":"2022年4月1日","externalUrl":null,"permalink":"/p/windows%E4%B8%8B%E7%9A%84%E5%8E%9F%E7%94%9Flinux%E5%AD%90%E7%B3%BB%E7%BB%9Fwsl%E7%9A%84%E5%AE%89%E8%A3%85%E4%B8%8E%E9%85%8D%E7%BD%AE/","section":"Posts","summary":" 简介 # Windows的Linux子系统（Windows Subsystem for Linux），简称WSL，是微软公司开发的一个可以在Windows 10或Windows 11上运行原生Linux二进制程序的兼容层。借助WSL，Windows用户不再需要安装Linux系统就可以使用Linux上的工具和程序，而且可以在WSL中直接使用Windows系统的文件系统，用户在两个系统间的切换也更加流畅和方便。\n","title":"Windows下的原生Linux子系统WSL的安装与配置","type":"post"},{"content":"This website was built with Hugo framework. The theme \u0026ldquo;stack\u0026rdquo; used by this website supports to generate labels based on the Open Graph Protocol (OG). Here is how to do the Search Engine Optimization (SEO) for a website built with Hugo framework.\nIntro # By the 3rd article of this series, we have built a bi-lingual website that can be visited via domain name and HTTPS protocol. However, if we want to search our website on a search engine (such as Google/Bing/Baidu), we will find nothing related to our website.\nThe reason is that our website hasn\u0026rsquo;t been included by the search engines nor been linked to other websites. Therefore, if we want others to be able to find our website on a search engine, we need to submit our website to the search engine. At the same time, we could also do some seach engine optimization (SEO), so that the search engines could obtain more detailed information of our website. Thus our website could be more easily found by others.\nTurn on Open Graph # Open Graph protocol (OG protocol) is a protocol for labelling the webpage contents, released by Facebook in 2010. OG protocol contains some labels in the head of a webpage, which mark the title, description, etc. of a webpage. OG labels make a webpage a \u0026ldquo;riched-media-object\u0026rdquo;, which can be cited by other social websites.\nMost search engines support OG protocol. Hence using OG labels in a webpage can improve the ranking of our website in a search engine.\nThe OG labels in a webpage often look like in following format:\n1 2 3 4 \u0026lt;meta property=\u0026#34;og:title\u0026#34; content=\u0026#34;The Rock\u0026#34; /\u0026gt; \u0026lt;meta property=\u0026#34;og:type\u0026#34; content=\u0026#34;video.movie\u0026#34; /\u0026gt; \u0026lt;meta property=\u0026#34;og:url\u0026#34; content=\u0026#34;https://www.imdb.com/title/tt0117500/\u0026#34; /\u0026gt; \u0026lt;meta property=\u0026#34;og:image\u0026#34; content=\u0026#34;https://ia.media-imdb.com/images/rock.jpg\u0026#34; /\u0026gt; The theme \u0026ldquo;stack\u0026rdquo; supports OG protocol. All we need is turning on the OG option in the configuration files config/_default/params.en.yaml and config/_default/params.zh-cn.yaml in the website root directory:\n1 2 3 4 5 6 7 opengraph: twitter: # Your Twitter username site: JinliCyou # Available values: summary, summary_large_image card: summary_large_image Thus, the OG labels will be generated and inserted to the HTML files when deploying our website using Hugo.\nOptimization for Google Search Engine # When submitting our website information to Google, Google needs to verify our ownership of the website. There are several verifying methods:\nPlace a verification file that generated by Google in the website public directory Insert a specific label into the head of the HTML file of our webpage Use the Tracking ID (or Measurement ID) of Google Analytics Since the theme stack integrates the supporting of Google Analytics, we use the 3rd method for verification.\nTurn on Google Analytics # Google Analytics is a tool that help the website owner for analyses of the website data.\nGo to the official website of Google Analytics and sign up. You can also sign in with your existing Google account. Currently, the widely-used version of Google Analytics is Google Analytics 4, in which version Measurement ID instead of Tracking ID is used to track a website.\nObtain the Measurement ID. For details, please refer to the Help Documents of Google Analytics. Here is how I did it:\nFind the \u0026ldquo;Admin\u0026rdquo; option in the user\u0026rsquo;s frontpage. New a \u0026ldquo;Property\u0026rdquo; and fill in necessary information as instructed. Click on the \u0026ldquo;Data Stream\u0026rdquo; option in the \u0026ldquo;Property\u0026rdquo; column. Click on \u0026ldquo;Add stream\u0026rdquo; and select the \u0026ldquo;Web\u0026rdquo;. Fill in your domain name and your website\u0026rsquo;s name. In the \u0026ldquo;Property\u0026rdquo; page, click on the stream that you added just now. You will see a Measurement ID starting with \u0026ldquo;G-\u0026rdquo;. Write down the Measurement ID of your website. Find the \u0026ldquo;googleAnalytics\u0026rdquo; option in the configuration file config/_default/config.yaml under the website root directory. Fill in your Measurement ID:\n1 2 # GA Tracking ID googleAnalytics: G-measuremntID Submit the site maps # A site map is a XML data file that stores some webpage information of our website, usually named sitemap.xml. Submit it to the search engine, then the search engine could access the webpage information of our website.\nHugo generates sitemap.xml in the public directory automatically when deploying the website.\nFor a detailed instruction about how to submit our site maps to Google Search, please refer to the instructions. Here is how I did it:\nLog in to the Google Search Console. We can use the account that registered on Google Analytics.\nClick on the \u0026ldquo;Add property\u0026rdquo; at the top left menu. Select the \u0026ldquo;URL prefix\u0026rdquo;. Then enter our website address starting with \u0026ldquo;https\u0026rdquo;. Select \u0026ldquo;Google Analytics\u0026rdquo; for the ownership verification. Click on \u0026ldquo;Verify\u0026rdquo;. If the website was deployed again after turing on the Google Analytics in the last step, we will pass the verification.\nSubmit the sitemap.xml files. In the left sidebar menu, click on \u0026ldquo;Sitemaps\u0026rdquo;, then enter the URL of the sitemap.xml file. For example, this website is a bi-lingual website, Hugo generates 3 sitemap.xml files, which are /public/sitemap.xml, /public/zh-cn/sitemap.xml and /public/en/sitemap.xml.\nNote, DO NOT miss the / at the beginning of the paths when adding the site maps. Even if there exsits a slash at the end of the website domain name, it can NOT be omitted. See the picture below: After submitting it successfully, it will display \u0026ldquo;success\u0026rdquo; for the \u0026ldquo;status\u0026rdquo;.\nAmong the three site map files generated by Hugo, the /public/sitemap.xml is actually pointing to /public/zh-cn/sitemap.xml and /public/en/sitemap.xml. Therefore, only submitting the /public/sitemap.xml is fine.\nUsually one or two days after we submitted the site maps successfully, we could find our website in Google\u0026rsquo;s records. You can check whether a certain website is included by Google by type site:xxx.com in Google\u0026rsquo;s searching box.\nOptimization for Baidu Search Engine # The optimizations for Baidu Search Engine can be done on Baidu Resource Searching Platform.\nGo to Baidu Resource Searching Platform. Sign in with your Baidu account.\nClick on \u0026ldquo;Submitting a link\u0026rdquo;. Then click on \u0026ldquo;Add a site\u0026rdquo;. Enter the domain name of your website. It also requires to verify your ownership of your website. You can download the verification file and place it in the public directory. Then click on \u0026ldquo;Verify\u0026rdquo;.\nClick on the \u0026ldquo;Ordinary Collection\u0026rdquo; in the \u0026ldquo;Resource Submission\u0026rdquo; of the left sidebar menu. Select \u0026ldquo;sitemap\u0026rdquo; and enter the URL of the sitemap.xml files.\nThere are two restrictions for submitting site maps to Baidu:\nIndexing site maps are not allowed Submitting one site map file only per day for a new account ","date":"24 February 2022","externalUrl":null,"permalink":"/en/p/how-this-website-was-built-4--search-engine-optimization-seo/","section":"Posts","summary":"This website was built with Hugo framework. The theme “stack” used by this website supports to generate labels based on the Open Graph Protocol (OG). Here is how to do the Search Engine Optimization (SEO) for a website built with Hugo framework.\nIntro # By the 3rd article of this series, we have built a bi-lingual website that can be visited via domain name and HTTPS protocol. However, if we want to search our website on a search engine (such as Google/Bing/Baidu), we will find nothing related to our website.\n","title":"How This Website Was Built 4 -- Search Engine Optimization (SEO)","type":"post"},{"content":"","date":"24 February 2022","externalUrl":null,"permalink":"/en/tags/seo/","section":"Tags","summary":"","title":"SEO","type":"tags"},{"content":"","date":"2022年2月24日","externalUrl":null,"permalink":"/tags/%E6%90%9C%E7%B4%A2%E5%BC%95%E6%93%8E%E4%BC%98%E5%8C%96/","section":"Tags","summary":"","title":"搜索引擎优化","type":"tags"},{"content":"This website uses Hugo theme \u0026ldquo;stack\u0026rdquo;. This post shows how to apply different Hugo themes, introduces the features of Hugo theme \u0026ldquo;stack\u0026rdquo; and some customization of \u0026ldquo;stack\u0026rdquo;.\nMotivation # Initially when I built a personal website, I tried Hexo Framework. I used several Hexo themes, including hexo-theme-yilia, hexo-theme-yilia-plus and hexo-theme-next. I also tried a theme https://leirock.github.io/blog/, which is a theme based on hexo-theme-next and was created by a GitHub user called \u0026ldquo;leirock\u0026rdquo;. This theme was my favorate and its GitHub repo is https://github.com/leirock/blog. And it looks like this:\nHowever, since the multi-lingual configurations on Hexo was complicated and annoying, I finally decided to give up using Hexo and turned to Hugo framework. I didn\u0026rsquo;t try too many Hugo themes, I was using hugo-theme-stack since then. I found that some useful features were missed when I was applying this theme. So I modified it a little bit.\nPrerequisite # The domain name was configured\nHugo framework was installed\nIf these two conditions are not satisfied, please refer to the previous articles of this series:\n\u0026ldquo;How This Website Was Built 1 \u0026ndash; Purchase and Configure a Personal Domain Name\u0026rdquo; \u0026ldquo;How This Website Was Built 2 \u0026ndash; Hugo Framework for Personal Blog Website\u0026rdquo; Install Hugo Theme # We call the directory of the website on our local server as \u0026ldquo;Website root directory\u0026rdquo;. For example, the website root directory used in this article is ~/Documents/www/website1. We call the theme directory in the website root directory as \u0026ldquo;theme root directory\u0026rdquo;. For example, the theme root directory used in this article is ~/Documents/www/website1/themes/hugo-theme-stack. First, enter the website root directory. Then use following command to download theme stack and put it in the themes folder:\n1 2 cd ~/Documents/www/website1 git clone https://github.com/CaiJimmy/hugo-theme-stack/ themes/hugo-theme-stack Apply the example site. There is an example site in theme stack, which is in the exampleSite folder in the theme root directory. Once the files in exampleSite folder are copied to the website root directory, we can apply the example site. Do note that this process will override the existing files with the same names. If you don\u0026rsquo;t want them to be overridden, please back up them before running the following commands.\nCopy the files in exampleSite under the stack theme root directory to the website root directory:\n1 cp -r themes/hugo-theme-stack/exampleSite/* ./ Then we need to delete the config.toml file generated automatically by Hugo in the website root directory. So that to avoid the conflict between it and the config.yaml copied from exampleSite folder.\n1 rm config.toml Preview the example site. Run the following command in the website root directory:\n1 hugo server Then open a browser in your local server. Enter http://localhost:1313 and you will see the example site. You can refer to the cover image of this article for the actual effect of the example site. Or you can go to the example site provided by the author of theme stack https://demo.stack.jimmycai.com/.\nInstruction for Basic Usage # File structure of a Hugo website # The file and directory tree of a website root directory is shown as following:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 . ├── archetypes │ └── default.md ├── config.yaml # Website configuration file ├── content # all the contents for this website │ ├── categories # Mainpage of the \u0026#34;Categories\u0026#34; page │ │ └── Test # One category called \u0026#34;Test\u0026#34; under the \u0026#34;Categories\u0026#34; page │ ├── page # All the menu options shown in the left sidebar of the website mainpage │ │ ├── about # The \u0026#34;About\u0026#34; page in the left sidebar │ │ ├── archives # The \u0026#34;Archives\u0026#34; page in the left sidebar │ │ ├── links # The \u0026#34;Links\u0026#34; page in the left sidebar │ │ └── search # The \u0026#34;Search\u0026#34; page in the left sidebar │ └── post # The posts written by the user are all in this directory. Each folder for one post. │ ├── chinese-test │ ├── emoji-support │ ├── markdown-syntax │ ├── math-typesetting │ ├── placeholder-text │ └── rich-content ├── data ├── layouts ├── LICENSE ├── README.md ├── resources │ └── _gen │ ├── assets │ └── images ├── static # For user-defined fonts, user\u0026#39;s avatar, website icon, etc. └── themes # For Hugo themes └── hugo-theme-stack # Theme stack ├── archetypes ├── assets ├── config.yaml ├── data ├── debug.sh ├── exampleSite ├── go.mod ├── i18n ├── images ├── layouts ├── LICENSE ├── netlify.toml ├── README.md └── theme.toml Besides, the private and public directory in the website root directory are not shown.\npublic directory will be generated after deploying the website using hugo -D command. public directory is exposed to public. All the HTML files for this webnsite pages will be generated by Hugo and be put in the public directory. DO NOT place any sensitive files, such as files that contain user\u0026rsquo;s information, in the public directory! private directory can be created by the user. Here we use it for storing the SSL certificate of this website. For detailed explaination of the directories in the website root directory, please refer to the official documents of Hugo https://gohugo.io/getting-started/directory-structure/.\nConfiguration directory config # Hugo supports two kinds of configurations:\n.config file in the website root directory Creating a config directory in the website root directory and put the configuration files in config directory For the 2nd configuration method, we can put different configurations in different files, which is more logical and clear. So here we use the 2nd configuration method. The structure of the config directory for this website is shown below:\n1 2 3 4 5 6 7 8 config └── _default ├── config.yaml ├── languages.yaml ├── menu.en.yaml ├── menu.zh-cn.yaml ├── params.en.yaml └── params.zh-cn.yaml For more detailed instructions of the configurations in Hugo, please refer to the official documents https://gohugo.io/getting-started/configuration/.\nModify and Customize Theme Stack # I made two modifications on the original theme stack:\nAdd new user-defined icons, so that the left sidebar can show the icon of Douban, Zhihu, Gitee, etc. Add a multi-language switcher button at the bottom of the life sidebar (main page), or at the top of the right sidebar (article page). User-defined icons # Since the original theme stack only supports several limited social icons, including GitHub, Twitter, RSS etc. While for the Chinese users, Weibo, Zhihu etc. are more generally used, but the icons are not supported. So I decided to replace the SVG icons in the original theme stack, and introduce more icons including Weibo, Zhihu, etc.\nWhen I used Hexo framework, I found some social icons in theme yilia and theme yilia-plus. So I introduced the icons from them to theme stack. Here is how I did it.\nObtain the fonts from Hexo theme yilia-plus. Go to the GitHub repo of theme yilia-plus https://github.com/JoeyBling/hexo-theme-yilia-plus. Download the fonts folder under the directory source-src/css. And copy this folder to the static directory in the website root directory.\nModify the menu style in the theme stack. Find the file assets/scss/partial/menu.scss in the theme root directory of theme stack, which is the style file of the left sidabar menu in theme stack. Replace the code block which defines the .social-menu at the bottom of the file with following code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 .social-menu { list-style: none; padding: 0%; display: flex; flex-direction: row; gap: 0px; a { border-radius:50%; display:-moz-inline-stack; display:inline-block; vertical-align:middle; *vertical-align:auto; zoom:1; *display:inline; margin:0 8px 15px 8px; transition:0.3s; text-align: center; color: #fff; opacity: 0.7; width: 28px; height: 28px; line-height: 26px; \u0026amp;:hover { opacity:1 } } a.weibo { background: #aaaaff; border:1px solid #aaaaff; \u0026amp;:hover { border:1px solid #aaaaff; } } a.segmentfault { background: #009a61; border:1px solid #009a61; \u0026amp;:hover { border:1px solid #009a61; } } a.rss { background: #ef7522; border:1px solid #ef7522; \u0026amp;:hover { border:1px solid #cf5d0f; } } a.github { background: #afb6ca; border:1px solid #afb6ca; \u0026amp;:hover { border:1px solid #909ab6; } } a.gitee { background: #c8171e; border:1px solid #c8171e; \u0026amp;:hover { border:1px solid #c8171e; } } a.facebook { background: #3b5998; border:1px solid #3b5998; \u0026amp;:hover { border:1px solid #2d4373; } } a.google { background: #4086f4; border:1px solid #4086f4; \u0026amp;:hover { border:1px solid #4086f4; } } a.twitter { background: #55cff8; border:1px solid #55cff8; \u0026amp;:hover { border:1px solid #24c1f6; } } a.linkedin { background: #005a87; border:1px solid #005a87; \u0026amp;:hover { border:1px solid #006b98; } } a.acfun { background: #fd4c5d; border:1px solid #fd4c5d; \u0026amp;:hover { border:1px solid #fd4c5d; } } a.bilibili { background: #e15280; border:1px solid #e15280; \u0026amp;:hover { border:1px solid #e15280; } } a.zhihu { background: #0078d8; border:1px solid #0078d8; \u0026amp;:hover { border:1px solid #0078d8; } } a.douban { background: #06c611; border:1px solid #06c611; \u0026amp;:hover { border:1px solid #06c611; } } a.mail { background: #005a87; border:1px solid #005a87; \u0026amp;:hover { border:1px solid #006b98; } } a.jianshu { background: #ff5722; border:1px solid #ff5722; \u0026amp;:hover { border:1px solid #ff5722; } } a.weixin { background: #4caf50; border:1px solid #4caf50; \u0026amp;:hover { border:1px solid #4caf50; } } a.qq { background: #34baad; border:1px solid #34baad; \u0026amp;:hover { border:1px solid #34baad; } } a.psn { background: #086ef6; border:1px solid #086ef6; \u0026amp;:hover { border:1px solid #086ef6; } } } Add index for the fonts. The index can be treated as the user-defined customizable style file. Go to the GitHub repo of theme yilia-plus https://github.com/JoeyBling/hexo-theme-yilia-plus. Download the source-src/css/fonts.scss file, and copy all the contents into the assets/scss/custom.scss file in stack\u0026rsquo;s theme root directory.\nopen the file layouts/partials/sidebar/left.html under stack\u0026rsquo;s theme root directory. Find the row that containing {{- with .Site.Menus.social -}}. Replace all the contents between this line and its corresponding line {{- end -}} with following contents:\n1 2 3 4 5 6 7 8 9 10 11 12 13 \u0026lt;ol class=\u0026#34;social-menu\u0026#34;\u0026gt; {{ range . }} \u0026lt;li\u0026gt; \u0026lt;a class=\u0026#34;{{ .Identifier }}\u0026#34; href=\u0026#34;{{ .URL }}\u0026#34; {{ with .Name }}title=\u0026#34;{{ . }}\u0026#34;{{ end }} {{ if eq (default true .Params.newTab) true }}target=\u0026#34;_blank\u0026#34;{{ end }} \u0026gt; \u0026lt;i class=\u0026#34;icon-{{ .Params.Icon }}\u0026#34;\u0026gt;\u0026lt;/i\u0026gt; \u0026lt;/a\u0026gt; \u0026lt;/li\u0026gt; {{ end }} \u0026lt;/ol\u0026gt; If you need the social icons to display below your avatar at the left sidebar, you need to add the corresponding options in the configuration file. For example, if you want the GitHub and the email to be displayed, you can add following contents in the configuration file config/_default/menu.en.yaml and config/_default/menu.zh-cn.yaml:\n1 2 3 4 5 6 7 8 9 10 11 12 social: - identifier: github name: GitHub url: https://github.com/jin-li/ params: icon: github - identifier: mail name: Email url: \u0026#34;mailto:i@jinli.cyou\u0026#34; params: icon: mail For the effect, please refer to the left sidebar of this website.\nLanguage switch button (Multi-lingual switcher) # Although Hugo framework natively supports multi-lingual, the theme stack doesn\u0026rsquo;t provide a language switch button. I read some discussions on the Hugo forum and then modified the code in theme stack. So that the language switch button is added to the theme.\nThere are mainly two functions of the button:\nAdding a language switch button below the \u0026ldquo;Dark Mode\u0026rdquo; button at the left sidebar of the main page, for the language switching of the whole website. When the current website language is Chinese, the website language will switch to English after clicking the button. When the current website language is English, the website language will switch to Chinese after clicking the button. Adding a language switch button on the top of the \u0026ldquo;Contents\u0026rdquo; at the right sidebar of the article page, for the language switching of a single page. When there is a corresponding bi-lingual page of the current page, the button will display When there is no corresponding bi-lingual page of the current page, the button will not display For the effect of this language switch button, please refer to this website. Here is how I did it.\nDownload a SVG icon for the translation button. And place it in the assets/icons directory under stack\u0026rsquo;s theme root directory.\nFind the assets/scss/partials/sidebar.scss file in stack\u0026rsquo;s theme root directory, which is the style file that defines the style of the sidebar of the website. Add following contents at the end of the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 .language-select{ align-self: stretch; color: var(--body-text-color); align-items: center; } .language-select #selected{ font-weight: 500; display: flex; margin-top: 0px; margin-bottom: 0px; } .language-select #to-select{ font-weight: 100; display: flex; a { margin-left: 60px; } margin-top: 0px; margin-bottom: 0px; } Find the assets/scss/partials/layout/article.scss file in stack\u0026rsquo;s theme root directory. Add following contents at the end of the file:\n1 2 3 4 5 6 7 8 9 .widget--translation { background-color: var(--card-background); border-radius: var(--card-border-radius); box-shadow: var(--shadow-l1); display: flex; flex-direction: column; color: var(--card-text-color-main); overflow: hidden; } Find the layouts/partials/sidebar/left.html file in stack\u0026rsquo;s theme root directory. Add following contents at 3rd line from the bottom of the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 {{ if .Site.IsMultiLingual }} {{ $siteLanguages := .Site.Languages}} {{ $pageLang := .Page.Lang}} {{ range .Page.AllTranslations }} {{ $translation := .}} {{ range $siteLanguages }} {{ if eq $translation.Lang .Lang }} {{ $selected := false }} {{ if eq $pageLang .Lang}} {{ else }} \u0026lt;div class=\u0026#34;language-select\u0026#34;\u0026gt; \u0026lt;li id=\u0026#34;selected\u0026#34;\u0026gt;{{ partial \u0026#34;helper/icon\u0026#34; \u0026#34;trans\u0026#34; }}\u0026lt;a href=\u0026#34;{{ $translation.RelPermalink }}\u0026#34;\u0026gt;\u0026lt;span\u0026gt;{{ .LanguageName }}\u0026lt;/span\u0026gt;\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;/div\u0026gt; {{ end }} {{ end }} {{ end }} {{ end }} {{ end }} Open the layouts/partials/sidebar/left.html file in stack\u0026rsquo;s theme root directory. Find the last code block that defines the right sidebar in the file and replace it with following contents:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 {{ if .Site.IsMultiLingual }} {{ $siteLanguages := .Site.Languages}} {{ $pageLang := .Page.Lang}} {{ range .Page.AllTranslations }} {{ $translation := .}} {{ range $siteLanguages }} {{ if eq $translation.Lang .Lang }} {{ $selected := false }} {{ if eq $pageLang .Lang}} {{ else }} \u0026lt;div class=\u0026#34;language-select\u0026#34;\u0026gt; \u0026lt;li id=\u0026#34;selected\u0026#34;\u0026gt;{{ partial \u0026#34;helper/icon\u0026#34; \u0026#34;trans\u0026#34; }}\u0026lt;a href=\u0026#34;{{ $translation.RelPermalink }}\u0026#34;\u0026gt;\u0026lt;span\u0026gt;{{ .LanguageName }}\u0026lt;/span\u0026gt;\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; \u0026lt;/div\u0026gt; {{ end }} {{ end }} {{ end }} {{ end }} {{ end }} Add the language configuration file languages.yaml in the configuration directory config/_default in the website root directory. Add following contents:\n1 2 3 4 5 6 7 en: languageName: \u0026#34;English\u0026#34; languageNameShort: \u0026#34;en\u0026#34; zh-cn: languageName: \u0026#34;简体中文\u0026#34; languageNameShort: \u0026#34;zh\u0026#34; Finally, name the articles for the two different languages in the post folder as index.md and index.en.md. Then when you deploy the website with Hugo, the bi-lingual pages will be generated automatically and the language switch button will show automatically.\n","date":"22 February 2022","externalUrl":null,"permalink":"/en/p/how-this-website-was-built-3--usage-and-customization-of-hugo-theme-stack/","section":"Posts","summary":"This website uses Hugo theme “stack”. This post shows how to apply different Hugo themes, introduces the features of Hugo theme “stack” and some customization of “stack”.\nMotivation # Initially when I built a personal website, I tried Hexo Framework. I used several Hexo themes, including hexo-theme-yilia, hexo-theme-yilia-plus and hexo-theme-next. I also tried a theme https://leirock.github.io/blog/, which is a theme based on hexo-theme-next and was created by a GitHub user called “leirock”. This theme was my favorate and its GitHub repo is https://github.com/leirock/blog. And it looks like this:\n","title":"How This Website Was Built 3 -- Usage and Customization of Hugo Theme Stack","type":"post"},{"content":"","date":"22 February 2022","externalUrl":null,"permalink":"/en/tags/hugo-themes/","section":"Tags","summary":"","title":"Hugo Themes","type":"tags"},{"content":"","date":"22 February 2022","externalUrl":null,"permalink":"/en/tags/multi-lingual/","section":"Tags","summary":"","title":"Multi-Lingual","type":"tags"},{"content":"","date":"2022年2月22日","externalUrl":null,"permalink":"/tags/%E5%A4%9A%E8%AF%AD%E7%A7%8D/","section":"Tags","summary":"","title":"多语种","type":"tags"},{"content":"This website is running on my personal computer. Here is how this website was built.\nMotivations # After purchasing a domain name, I was considering to build a personal website using the domain name. At first I tried using Hexo as a static website generator. I would like to build a bilingual website (Chinese and English). However, Hexo does support multi-lingual natively. I have to use some extensions such as hexo-generator-i18n, etc. And I met tons of issues when building the bilingual website. Therefore I gave up and turned to Hugo.\nHugo is a static website generator developed in Go language. It is famous for its amazing speed. But I chose it because of its native support for multiple languages. For more details about Hugo\u0026rsquo;s feature, you can visit its official website https://gohugo.io. There is also a brief introduction to Hugo in the \u0026ldquo;About\u0026rdquo; page in this website.\nPrerequisite # A Linux computer (as private server) nginx (as HTTP server) Go language runtime environment Hugo (a static blog framework) A personal domain (refer to the 1st post of this series \u0026ldquo;How This Website Was Built 1 \u0026ndash; Purchase and Configure a Personal Domain Name\u0026rdquo; for details) Website Server Configurations # This website uses nginx as HTTP server. The serving computer uses deepin OS and the blog framework uses Hugo。\nInstall and Configuring nginx # Install nginx 1 sudo apt install nginx Enable nginx Service 1 sudo systemctl enable nginx Install and Configure Hugo # After using apt or snap to install Hugo, an error \u0026ldquo;port 1313 already in use\u0026rdquo; occurred. So I turned to use \u0026ldquo;deb\u0026rdquo; package to install it directly. First, download the latest \u0026ldquo;deb\u0026rdquo; package that applicable for Debian-based OS on the Release page of Hugo\u0026rsquo;s GitHub repo. It\u0026rsquo;s better to download the \u0026ldquo;extended\u0026rdquo; version. For example, I downloaded hugo_extended_0.92.2_Linux-64bit.deb. Then install it using dpkg package manager: 1 sudo dpkg -i hugo_extended_0.92.2_Linux-64bit.deb Using following command in the terminal to check the installation 1 hugo version Initialize the website. Create a directory in your local serving computer to store your website. For example, the path to the directory can be ~/Documents/www/ 1 mkdir -p ~/Documents/www Then new a site with Hugo. Say it\u0026rsquo;s called website1: 1 2 cd ~/Documents/www hugo new site website1 With above command, Hugo will create a website template under directory ~/Documents/www/website1. Local browser test. Run following command in terminal: 1 hugo server Then open a browser on you local computer. Enter the default url http://localhost:1313. If the default website displays in your browser, it means the configurations are correct. Binding the Website with a Domain Name # So far, your website can only be reached through http://localhost:1313 on your local computer. If you want it to be reached by anyone in the world, you need to assign it a domain name.\nCreate a configuration file for nginx to bind the local files of the website to a domain name. To avoid contaminating nginx\u0026rsquo;s original configurations, you can create a new configuration file:\n1 2 3 cd /etc/nginx sudo mkdir vhost sudo touch blog.conf Above commands will create a configuration file blog.conf for our website in /etc/nginx/vhost directory. Then enter following content in blog.conf (Note that you need sudo privilege to edit this file):\nserver{\nlisten 80;\nroot /home/YourUserName/Documents/www/website1/public;\nserver_name www.your.domain.name;\nlocation /{\n}\n}\nNote that above configurations will point your domain name to the public folder in directory ~/Documents/www/website1, instead of the whole website1 directory. It\u0026rsquo;s because that you would not like to expose some private files.\nIf you would like to assign a domain name to your other websites, you can do it in a similar way.\nTo assure that nginx can find the configuration file /etc/nginx/vhost/blog.conf you created, you need to include it in nginx\u0026rsquo;s default configuration file /etc/nginx/nginx.conf. Find the block for http settings in /etc/nginx/nginx.conf. And insert the following line in the block:\ninclude /etc/nginx/vhost/*.conf\nThis will let nginx read the configuration file in /etc/nginx/vhost automatically. After the nginx service is reloaded, your domain name will be pointed to the website directory.\nReload nginx service and deploy the website using Hugo:\n1 2 3 sudo nginx -s reload cd ~/Documents/www/website1 hugo -D Thus, you can use your domain name (for example, http://www.your.domain.name) to visit your website!\nConfiguring the SSL Certificate # After above configurations, your website can be accessed via HTTP protocol by the users from worldwide. HTTP is an abbreviation of Hyper Text Transfer Protocol. The content transferred via HTTP is in plaintext, which causes some security issues. Currently most websites are using the safer HTTPS protocol. And the website using HTTP might be restricted by many browsers. When a user visits your website, he/she might receive some safety warnings from the browser. And the ranking of your website will be lowered by the search engines. Therefore, it\u0026rsquo;s necessary to upgrade your website to HTTPS.\nHTTPS protocol depends on SSL encryption, which requires an SSL certifiacte. Many SSL certificate application are not free. But there\u0026rsquo;re still some ways to get free SSL certificate. For example, ZeroSSL can provide free SSL certificate.\nCreate a ZeroSSL Account # Go to ZeroSSL\u0026rsquo;s official websitehttps://zerossl.com and create a personal account.\nUse acme.sh Generate and Periodically Update SSL Certificate # The SSL certificate generated directly on ZeroSSL usually expires in 60 to 90 days. Therefore, if you generate an SSL certificate on their website, you need to regenerate after some time. The good news is that some developers have developed some tools to apply and update SSL certificate automatically from ZeroSSL. The tool is acme.sh. Referring to the instructions, you can install an SSL certificate for your local server.\nPlease refer to the instructions of acme.sh for detailed instructions. Here we only show what we need for this website.\nInstall acme.sh\n1 curl https://get.acme.sh | sh -s email=my@example.com Since we are using Alibaba Cloud\u0026rsquo;s domain name, we choose to generate SSL certificate through Aliyun domain API. The detailed instruction can be found in the wiki of acme.sh. To be short, we need to use the API provided by Alibaba Cloud (i.e., the Access Key we mentioned in the 1st post of this series \u0026ldquo;How This Website Was Built 1 \u0026ndash; Purchase and Configure a Personal Domain Name\u0026rdquo;). You can login to Alibaba Cloud and get a new Access Key, or you can use the one you obtained previously.\nExport the Access key as the system variable Ali_Key and Ali_Secret using following command (Again, note that do NOT leak the information to others):\n1 2 export Ali_Key=\u0026#34;sdfsdfsdfljlbjkljlkjsdfoiwje\u0026#34; export Ali_Secret=\u0026#34;jlsdflanljkljlfdsaklkjflsa\u0026#34; Then use following command to generate the SSL certificate:\n1 acme.sh --issue --dns dns_ali -d your.domain.com -d www.your.domain.com Copy/Install the SSL certificate to where you need to use it. By default, the certificates generated by acme.sh are located at ~/.acme.sh/. But usually you need to copy them to a another destination directory. For example, you can copy them to the private folder of your website:\n1 2 3 4 acme.sh --install-cert -d your.domain.com -d www.your.domain.com \\ --key-file /home/YourUserName/Documents/www/website1/private/key.pem \\ --fullchain-file /home/YourUserName/Documents/www/website1/private/cert.pem \\ --reloadcmd \u0026#34;sudo service nginx force-reload\u0026#34; After copying the certificate using above command, the nginx service will be reloaded to make HTTPS valid. It requires your sudo password.\nModify the configuration file of nginx to support HTTPS protocol. You can add following content in the nginx configuration file /etc/nginx/vhost/blog.conf (note that you need sudo privilege to edit the file):\nserver{\nlisten 443;\nssl on;\nssl_certificate /home/YourUserName/Documents/www/website1/private/cert.pem;\nssl_certificate_key /home/YourUserName/Documents/www/website1/private/key.pem;\nroot /home/YourUserName/Documents/www/website1/public;\nserver_name www.your.domain.name;\nlocation /{\n}\n}\nReload nginx Service\n1 sudo nginx -s reload Then you will be able to use HTTPS protocol to visit your website. You can try to enter your domain name starting with https to visit your website on other computers (For example, https://www.your.domain.name)!\n","date":"17 December 2021","externalUrl":null,"permalink":"/en/p/how-this-website-was-built-2--hugo-framework-for-personal-blog-website/","section":"Posts","summary":"This website is running on my personal computer. Here is how this website was built.\nMotivations # After purchasing a domain name, I was considering to build a personal website using the domain name. At first I tried using Hexo as a static website generator. I would like to build a bilingual website (Chinese and English). However, Hexo does support multi-lingual natively. I have to use some extensions such as hexo-generator-i18n, etc. And I met tons of issues when building the bilingual website. Therefore I gave up and turned to Hugo.\n","title":"How This Website Was Built 2 -- Hugo Framework for Personal Blog Website","type":"post"},{"content":"This website uses my personal domain name. Here is how the domain name was purchased and the configurations of the domain name, including setting up a dynamically domain name system (DDNS).\nMotivation # I need to use ssh to login to the computers in my home and office frequently for some working reasons. However, there\u0026rsquo;s no fixed IP addresses for those computers. After the IP addresses changed, I lost the connection to the computers and could not re-connect immediately (because I don\u0026rsquo;t know the new IP address).\nSo I was considering to use domain name instead of IP address for connecting those computers. Then I looked at the prices of the domain names, finding that there\u0026rsquo;re some cheap ones (Although the domain names with .com are usually very expensive, the ones with .xyz, .top, .cyou, etc are much cheaper).\nPurchase Domain Name from Alibaba Cloud # Register an account for Alibaba Cloud\nGo to Alibaba Cloud International Website, and register an international account (no need for real-name authentication!)。\nLook up and purchase a domain name\nGo to Alibaba International Domain Name Market, and search for the secondary domain name you want, for example, I searched for “jinli”.\nThe result page will show all domain names that are available： Purchase the domain name you need with the Alibaba account you registered in the last step. For example, I purchased “jinli.cyou”.\nResolving Domain Name to Server\u0026rsquo;s IP Address (DNS) # The domain name your purchased needs to be resolved to your server\u0026rsquo;s IP address before you can use domain name to visit your computer. If you have a fixed IP address for your server (computer), you can assign the IP address to your domain name on Alibaba Cloud. If you don\u0026rsquo;t have a fixed IP address for your server (computer), you need to dynamically resolve the domain name to your server\u0026rsquo;s IP address. Having a fixed IP address \u0026ndash; assign it to your domain name directly # Obtain your IP address\nGo to IPv6 Testing to check whether your network supports IPv6： On a Unix-like OS, you can use following command to get your IPv4 address:\n1 curl ipinfo.io/ip and use following command for IPv6 address:\n1 curl ipv6.icanhazip.com Assign your domain name with your IP address\nLogin to International Alibaba Cloud, click on the user\u0026rsquo;s avatar and select \u0026ldquo;Account Management\u0026rdquo; in the menu. Moving the mouse cursor to the upper left menu, you will see a drop-down menu. Click \u0026ldquo;Domains\u0026rdquo;. Click on the domain name you purchased in the domain name list. Click on the \u0026ldquo;DNS Settings\u0026rdquo; in the right-side menu. Click on \u0026ldquo;Add Record\u0026rdquo; to assign an IP address to your domain name: Type：For IPv4 address, choose \u0026ldquo;A\u0026rdquo;; for IPv6 address, choose \u0026ldquo;AAAA\u0026rdquo;. Host：Enter the third-level domain name you need. For example, if you need to set up an IP address for \u0026ldquo;surface.jinli.cyou\u0026rdquo;, you need to enter \u0026ldquo;surface\u0026rdquo;. For more instructions about the domain name settings, you can click on the question mark for details. ISP Line：Choose \u0026ldquo;default\u0026rdquo;. Value：Enter the IP address your obtained from the last step. (IPv4 address for \u0026ldquo;A\u0026rdquo; and IPv6 address for \u0026ldquo;AAAA\u0026rdquo;) TTL：Time interval for IP address updating, 10 minutes in the default is fine. You can add multiple domain name - IP address pairs, for both IPv4 and IPv6 addresses. Check After adding the record, you may need to wait for 10 minutes for Alibaba Cloud to update the info in their DNS server (usually it will be updated once you submitted the record). Then you can check whether the record is added successfully.\nUse ping command to check it in another network environment 1 ping surface.jinli.cyou If the ping is successful in a public network environment, it means you got the domain name bounded with your IP address successfully. Done! After binding your domain name with your IP address, you can login to your server with the domain name. For example, if you need to use ssh to login to your remote server, you can ssh via domain name:\n1 ssh lijin@surface.jinli.cyou -X You can also build a NAS (Network Attached Stordage), etc on your server, and visit it via domain name.\nDon\u0026rsquo;t have a fixed IP address —— Add record and dynamically DNS resolving (DDNS) # If your computer or server don\u0026rsquo;t have a fixed IP address, you need to add a record using your current IP address on Alibaba Cloud following the instructions above. Then you need to watch the IP address on your computer, once it changes, the record on Alibaba Cloud DNS server can be updated using the API provided by Alibaba Cloud. Basically, here are what we are doing:\nUse a Python3 script to do dynamical DNS (DDNS) The Python3 script uses Alibaba Cloud API It needs the Access key generated by your Alibaba Cloud account for password-free login For the security of your account, Do NOT leak your Access key. Once it leaks, login to your Alibaba Cloud account and delete the Access key immediately. Download the DDNS script The DDNS(Dynamic Domain Name System) script was adapted from the aliyun-ddns-client repo on GitHub. There\u0026rsquo;re two problems to use it directly in USA: The website for obtaining IP address is unavailable in USA Some problem for IPv6 address support So I forked this repo and fixed above problems. My repo is on https://github.com/jin-li/aliyun-ddns-client The DDNS script uses Python3 Use following command to clone the repo. Placing it in ~/.config folder is recommended (If there\u0026rsquo;s no .config folder in home directory, you should create the directory dirst): 1 2 cd ~/.config git clone https://github.com/jin-li/aliyun-ddns-client Install the dependency Python package requests and netifaces If you use Anaconda as your Python package manager, you can use following command: 1 2 3 conda activate conda install requests conda install netifaces If you usepip3as your Python package manager, you can use following command: 1 2 pip3 install requests pip3 install netifaces Generate Access key in your Alibaba Cloud Login to your Alibaba Cloud account, click on the menu at the upper left corner and find Resource Access Management in Management and Governance: After entering the RAM page, click on the Users in Identities in the left side menu. Then click on Create User to create a new user. Enter \u0026ldquo;Logon Name\u0026rdquo; and \u0026ldquo;Display Name\u0026rdquo;, check the \u0026ldquo;API Call-based Access\u0026rdquo; so that you can login to the account via API: After creating a new user, return the \u0026ldquo;RAM/Users\u0026rdquo; page. Click on the user name you created, and enter \u0026ldquo;User\u0026rdquo; setting page. Go to the bottom of the page, click on the Create AccessKey Pair button in User AccessKeys to generate an AccessKey. Write down the \u0026ldquo;Access ID\u0026rdquo; and \u0026ldquo;Access Key\u0026rdquo; poped out. Note that the Access Key appears only once. Once you close the poped-out page, it will disappear. But you can re-create one. Configure and run the script Go to the directory where you placed the aliyun-ddns-client repo. Finding the file ddns.conf.example, rename it as ddns-conf. Open ddns.conf with a text editor. Fill in the \u0026ldquo;Access ID\u0026rdquo; and \u0026ldquo;Access Key\u0026rdquo; you obtained from the last step. Then fill in your domain name and the corresponding IP address (The correctness of the IP address doesn\u0026rsquo;t matter since if it\u0026rsquo;s wrong, the script will correct it during running. But the domain name must be correct). If you have multiple domain names for resolving, you can add more entries ddns.conf imitating the existing examples. It supports both IPv4 and IPv6 addresses. Note that the domain name must be those were already created on Alibaba Cloud . Then you can run the script using Python3: 1 python3 ddns.py The script will check whether the current IP address matches what is on the Alibaba Cloud DNS server on each of its running. If it is not the same as on the Alibaba Cloud DNS server, it will update the record on the server. Setup a regular timed task for DDNS If your computer or server don\u0026rsquo;t have a fixed IP address, the IP address will change after some time. You need to update the record on Alibaba Cloud DNS server as soon as possible. To watch the IP address change manually is not convenient, you can setup a timed task and let the OS run the DDNS script in a certain time interval (for example, every hour). Then you can assure that the record can be updated in one hour after the IP address changes. In a Unix-like OS, the timed task can be setup using a system tool called crontab. You can find some instructions for how to setup crontab at https://crontab.guru/#. For example, if you want the DDNS script to run at the 42th minute every hour, you can use following command: 1 crontab -e which will open a configuration file containing crontab timed tasks, you can append one following line at the end of the file: 1 42 * * * * cd ~/.config/aliyun-ddns-client ipython3 ddns.py Then you can use following command to check whether the timed task is running: 1 crontab -l ","date":"15 August 2021","externalUrl":null,"permalink":"/en/p/how-this-website-was-built-1--purchase-and-configure-a-personal-domain-name/","section":"Posts","summary":"This website uses my personal domain name. Here is how the domain name was purchased and the configurations of the domain name, including setting up a dynamically domain name system (DDNS).\nMotivation # I need to use ssh to login to the computers in my home and office frequently for some working reasons. However, there’s no fixed IP addresses for those computers. After the IP addresses changed, I lost the connection to the computers and could not re-connect immediately (because I don’t know the new IP address).\n","title":"How This Website Was Built 1 -- Purchase and Configure a Personal Domain Name","type":"post"},{"content":"","date":"28 May 2019","externalUrl":null,"permalink":"/en/archives/","section":"Pages","summary":"","title":"Archives","type":"list"},{"content":"","externalUrl":null,"permalink":"/en/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"JinLi Go — Jin Li\u0026rsquo;s online Go game server.\nJinLi Travel — Jin Li\u0026rsquo;s online travel planning service.\nJinLi Twitch+ — A Twitch live broadcast recommendation platform based on user preferences.\nJinLi Cloud — Jin Li\u0026rsquo;s private cloud disk.\nJinLi Git — Jin Li\u0026rsquo;s private GitLab server.\nJinLi Image — Jin Li\u0026rsquo;s private image hosting server.\nJinLi Mail — Jin Li\u0026rsquo;s private email server.\nJinLi Overleaf — Jin Li\u0026rsquo;s private LaTeX repository.\nJinLi Book — Jin Li\u0026rsquo;s private ebook library.\nJinLi Media — Jin Li\u0026rsquo;s private media library.\nJin Li GitHub — Jin Li\u0026rsquo;s GitHub homepage.\n","externalUrl":null,"permalink":"/en/links/","section":"Pages","summary":"JinLi Go — Jin Li’s online Go game server.\nJinLi Travel — Jin Li’s online travel planning service.\nJinLi Twitch+ — A Twitch live broadcast recommendation platform based on user preferences.\nJinLi Cloud — Jin Li’s private cloud disk.\nJinLi Git — Jin Li’s private GitLab server.\nJinLi Image — Jin Li’s private image hosting server.\nJinLi Mail — Jin Li’s private email server.\nJinLi Overleaf — Jin Li’s private LaTeX repository.\n","title":"Links","type":"page"}]