How to Investigate High Disk I/O in Linux (Deep Dive)

High disk I/O is one of the most common and most misunderstood causes of slow Linux servers.

This guide is written the way real incidents are handled — from symptom to root cause, with clear interpretation at every step.

---

What “High Disk I/O” Actually Means

It does NOT just mean “disk is busy”.

It can mean:

• Too many reads/writes
• Slow storage latency
• Queue buildup (requests waiting)
• Blocking processes (D-state)

👉 Your goal is to identify which one.

---

Step 0: Confirm the Symptom (from workflow)

Usually you arrive here after:

• High load
• Low CPU usage
• Slow responses

---

Step 1: iostat (Primary Tool)

Install if needed:

# Linux (RHEL/CentOS)
yum install sysstat -y
# Linux (Debian/Ubuntu)
apt install sysstat -y

Run:

iostat -x 1

Key Columns (MUST understand)

Field	Meaning
%util	Disk usage percentage
await	Average wait time (ms)
r/s, w/s	Reads/Writes per second
avgqu-sz	Queue size

---

How to Interpret iostat (THIS is where expertise is)

Case A: %util ~100%

Disk is saturated

---

Case B: High await (>50ms HDD, >10ms SSD)

Latency problem

---

Case C: High avgqu-sz

Requests are piling up (queue bottleneck)

---

Step 2: iotop (Who is Causing I/O)

iotop

Note: Requires root privileges on Linux. On macOS, iotop is not native.

Look for:

• High read/write processes

---

Step 3: vmstat (System View)

vmstat 1

Look at:

• wa → I/O wait

Interpretation

• High wa → CPU waiting on disk

---

Step 4: Check D-State Processes

ps -eo pid,stat,cmd | grep D

Confirms blocking I/O

---

Step 5: Identify Files Causing I/O

lsof -p PID

Look for:

• Logs
• Databases
• Temporary files

---

Real Production Case (Step-by-Step)

Situation

• Website slow
• Load = 15
• CPU idle high

---

Investigation

1. iostat → %util = 100%
2. vmstat → wa high
3. iotop → MySQL heavy writes

---

Root Cause

Database writing too frequently to disk

---

Fix Options

• Optimize queries
• Add caching
• Move to faster storage (SSD/NVMe)

---

Common Causes of High Disk I/O

• Database overload
• Log files growing rapidly
• Backup processes
• Malware / abuse scripts
• Slow or failing disks

---

Time-Based Thinking

Sudden spike

• Traffic surge
• Backup job started

Gradual increase

• Database growth
• Log accumulation

---

Common Mistakes

❌ Looking only at %util

Latency (await) is equally important

---

❌ Killing processes blindly

Fix cause, not symptom

---

When This Matters in Production

Disk I/O issues affect:

• Databases
• File servers
• Streaming workloads

Infrastructure options:

• Offshore Streaming Servers
• Offshore Cloud Servers

---

Related Linux Guides

• How to Investigate a Slow Linux Server
• Linux Process States Explained
• How to Monitor Processes with htop

---

Final Takeaway

High disk I/O is not just a metric.

It is a signal of system pressure.

An expert does not just see 100% usage —

👉 They understand why the disk is struggling.