Essential Linux Utilities Every DevOps Engineer Should Know
Introduction: As DevOps and System Engineers, one common question we often encounter during interviews is how to diagnose mysterious processes occupying much CPU, memory, or IO resources in a Linux environment.
Luckily, Linux offers powerful utilities to help us identify these mysteries and optimize system performance. In this guide, we’ll explore some essential Linux utilities and their real-world applications.
1️⃣ iostat: When it comes to gaining insights into CPU and input/output statistics, iostat comes into the picture. This tool provides valuable data on system health and resource utilization, allowing us to identify performance bottlenecks and optimize system performance.
For example:
$ iostat -x
Linux 5.4.0-80-generic (hostname) 09/20/22 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.04 0.00 0.02 0.04 0.00 99.90
Device rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.002️⃣ netstat: Networking issues can often be challenging to diagnose, but netstat simplifies the process by offering detailed statistical data on IP, TCP, UDP, and ICMP protocols. With netstat, we can quickly identify network-related issues such as connection failures, excessive packet loss, or network congestion.
$ netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 :::80 :::* LISTEN
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp 0 0 0.0.0.0:5353 0.0.0.0:*3️⃣ lsof: Ever wondered which files are open in your system and which processes are accessing them? lsof lists them all. This powerful utility lists all open files, directories, and network connections, providing valuable insights into process activity and resource utilization. By analyzing lsof output, we can identify processes causing file contention or resource bottlenecks, allowing us to take appropriate action to optimize system performance and enhance reliability.
$ lsof -i :22
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1074 root 3u IPv4 2251 0t0 TCP *:ssh (LISTEN)4️⃣ pidstat: Monitoring system resource utilization by specific processes is essential for optimizing performance and ensuring stability. pidstat comes to the rescue by providing comprehensive statistics on CPU, memory, device IO, and thread activity for individual processes. For example, pidstat can help us identify CPU-intensive processes or memory leaks, enabling us to fine-tune system configurations and improve overall system efficiency.
$ pidstat -p 12345
Linux 5.4.0-80-generic (hostname) 09/20/22 _x86_64_ (8 CPU)
11:23:35 PM UID PID %usr %system %guest %wait %CPU CPU Command
11:23:36 PM 1000 12345 0.00 0.00 0.00 0.00 0.00 0 sshd5️⃣ SAR (System Activity Reporter): Discussion of Linux performance analysis tools would only be complete by mentioning SAR. This invaluable utility collects, reports, and analyzes system activity data over time, providing a comprehensive view of system performance trends and patterns. SAR allows us to track system resource usage, identify performance anomalies, and predict future resource requirements.
$ sar
Linux 5.4.0-80-generic (hostname) 09/20/22 _x86_64_ (8 CPU)
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 0.04 0.00 0.02 0.04 0.00 99.90Conclusion:
By leveraging essential Linux utilities such as iostat, netstat, lsof, pidstat, and SAR, we can gain valuable insights into system behaviour, diagnose performance issues, and implement effective solutions. With these tools and techniques, we can ensure the smooth operation of our systems and deliver exceptional user experiences.
🔔 Follow Muhammad Usama Khan here and on LinkedIn for more insightful content on DevOps, MLOps, Data, Cloud Computing and Cloud Native.
