空垠尘

DevOps Troubleshooting and Debugging Playbook

· 779 words · 4 min read
/ /
#DevOps #Troubleshooting #Linux #MacOS #System Tuning #Debugging

Introduction: A Systematic Mindset for DevOps Debugging

In Linux and macOS operations, weird system glitches inevitably emerge:

  • The disk shows gigabytes of free space, yet writes fail with No space left on device.
  • A port is locked, but no listening process is found.
  • Containers or backend services vanish silently without application crash logs.
  • Sudden connection spikes crash systems with Too many open files.

Effective troubleshooting is a systematic deductive process. This playbook collects 6 high-impact real-world issues with battle-tested root-cause fixes.

flowchart TD
    Issue["System Anomalies & Alerts"] --> A["1. File Descriptor Limits<br/>(Too Many Open Files)"]
    Issue --> B["2. Phantom Full Disks<br/>(Inode Exhaustion / Unreleased FD)"]
    Issue --> C["3. Port Clashes & Zombies<br/>(Address in use / Defunct)"]
    Issue --> D["4. Silent Process Termination<br/>(Kernel OOM Killer)"]
    Issue --> E["5. Clock Skew & Auth Failures<br/>(NTP Synchronization)"]
    Issue --> F["6. macOS Dev Bottlenecks<br/>(Homebrew Mirroring & Network)"]

1. Resolving Too Many Open Files (Descriptor Exhaustion)

In Unix, “everything is a file” (sockets, pipes, disk files). When concurrent connections exceed system thresholds, the OS denies new descriptors.

1. Permanent Linux Production Tuning

Running ulimit -n 65535 only affects the active terminal session. Make it permanent:

① Modify System Limits (/etc/security/limits.conf)

1* soft nofile 655350
2* hard nofile 655350
3root soft nofile 655350
4root hard nofile 655350

② Modify Systemd Global Daemon Limits (/etc/systemd/system.conf)

1[Manager]
2DefaultLimitNOFILE=655350
3DefaultLimitNPROC=655350

③ Check Open Descriptors by Process

1ls /proc/<PID>/fd | wc -l

2. macOS Developer Environment Tuning

1# Check current limit
2launchctl limit maxfiles
3
4# Temporarily raise limit for active terminal
5sudo launchctl limit maxfiles 65536 200000
6ulimit -n 65536

2. Phantom Full Disks & Inode Exhaustion (No space left on device)

sequenceDiagram
    autonumber
    participant App as Application (Nginx / Java)
    participant Disk as Disk File (/var/log/app.log)
    participant Admin as DevOps Engineer

    App->>Disk: Writes heavy logs (Holds open FD)
    Admin->>Disk: rm /var/log/app.log (Removes directory entry)
    Note over Disk: Space NOT freed! Open FD still referenced
    Admin->>App: Locates unreleased FD (lsof | grep deleted) & truncates
    Note over Disk: Disk space reclaimed immediately!

Case A: File Deleted by rm, but Process Holds Open Descriptor

When a large log file is removed while an application is actively writing to it, Linux unlinks the directory name, but disk blocks remain allocated until the process releases the descriptor.

1# 1. Find deleted files held by active processes
2sudo lsof | grep -i deleted
3
4# 2. Free space gracefully without restarting the service
5sudo truncate -s 0 /proc/<PID>/fd/<FD>
6# Or reload the service: sudo systemctl reload nginx

Case B: Available Space but Exhausted Inodes

Millions of tiny cache or session files can consume all filesystem inodes:

1# 1. Check inode utilization
2df -i
3
4# 2. Count directories containing the highest number of files
5find /var/spool/ -type d -exec sh -c "echo -n '{}: '; ls -1 '{}' | wc -l" \; | sort -n -k 2

3. Port Conflicts & Zombie Process Cleanup

1. Instant Port Identification & Termination

1# Locate process on port 8080
2sudo lsof -i :8080
3# Or
4sudo ss -tulnp | grep :8080
5
6# Instantly kill whatever is holding the port
7sudo fuser -k 8080/tcp

2. Diagnosing Zombie Processes (Z / Defunct)

Zombie processes are already dead, so kill -9 <ZOMBIE_PID> does nothing. You must target the Parent Process (PPID):

1# 1. List zombie processes
2ps aux | grep 'Z'
3
4# 2. Find the Parent PID (PPID)
5ps -o ppid= -p <ZOMBIE_PID>
6
7# 3. Reload or terminate the parent to let systemd (PID 1) reap the zombie
8sudo kill -HUP <PARENT_PID>

4. Silent Process Termination: Linux Kernel OOM Killer

When system memory runs out, the Linux kernel invokes the Out of Memory (OOM) Killer to terminate resource-heavy processes.

1# Check kernel ring buffer for OOM events
2sudo dmesg -T | grep -i -E "oom|out of memory|killed process"

Remediation:

  1. Add Swap Space (vital for 1–2GB cloud instances & Raspberry Pis):
    1sudo fallocate -l 2G /swapfile
2sudo chmod 600 /swapfile
3sudo mkswap /swapfile
4sudo swapon /swapfile
5echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
  2. Configure Memory Limits in Docker / Kubernetes.

5. Clock Skew Causing TLS & Authentication Failures

If system time drifts, HTTPS certificates fail verification and S3/JWT signatures are rejected.

1# 1. Inspect time and NTP sync status
2timedatectl status
3
4# 2. Set timezone and enable automatic NTP synchronization
5sudo timedatectl set-timezone UTC
6sudo timedatectl set-ntp true

6. macOS Homebrew Mirror Acceleration & Reset

 1# 1. Switch to fast mirror
 2export HOMEBREW_API_DOMAIN="https://mirrors.tuna.tsinghua.edu.cn/homebrew-bottles/api"
 3export HOMEBREW_BOTTLE_DOMAIN="https://mirrors.tuna.tsinghua.edu.cn/homebrew-bottles"
 4export HOMEBREW_BREW_GIT_REMOTE="https://mirrors.tuna.tsinghua.edu.cn/git/homebrew/brew.git"
 5export HOMEBREW_CORE_GIT_REMOTE="https://mirrors.tuna.tsinghua.edu.cn/git/homebrew/homebrew-core.git"
 6
 7# 2. Reset back to official defaults
 8unset HOMEBREW_API_DOMAIN
 9unset HOMEBREW_BOTTLE_DOMAIN
10export HOMEBREW_BREW_GIT_REMOTE="https://github.com/Homebrew/brew"
11export HOMEBREW_CORE_GIT_REMOTE="https://github.com/Homebrew/homebrew-core"
12brew update

Conclusion

Understanding fundamental OS mechanisms transforms troubleshooting from frustrating guesswork into precise, surgical problem-solving.

Related Posts

Table of Contents