Linux high iowait CPU causing app slowdown

Application response times jumped from 50ms to 3-5 seconds suddenly. CPU usage looked "normal" — 30% idle. But top showed something odd: wa (iowait) was at 65%.

Understanding iowait:

iowait shows the percentage of time the CPU is idle waiting for disk I/O to complete. High iowait means the disk is the bottleneck, not the CPU. The CPU has work to do but is waiting for disk reads/writes.

Finding what's causing it:

bash

# Show disk I/O stats per device
iostat -x 1 5
# Look for:
# %util: if this is near 100%, the disk is saturated
# await: average time (ms) for I/O requests to complete
 
# Find which processes are doing the most I/O
iotop -o  # -o shows only processes currently doing I/O
 
# Check if a specific process is the culprit
pidstat -d 1  # I/O stats per PID every 1 second

In my case, iotop showed the PostgreSQL process was doing 250MB/s of writes. A batch job had triggered a massive UPDATE that was writing to WAL and updating millions of rows.

What I did:

Identified the runaway query:

bash

# In psql
SELECT pid, query, query_start, state FROM pg_stat_activity WHERE state='active';

Killed the runaway query:

bash

SELECT pg_cancel_backend(pid_of_query);

Response times dropped back to normal in 30 seconds.
Added rate limiting to the batch job so it processes in smaller chunks with delays between them.

Longer-term fixes:

Move PostgreSQL to an NVMe SSD (our instance had SATA SSD — 4x slower for random writes)
Use ionice to lower I/O priority for non-critical processes:

bash

ionice -c 3 -p $(pgrep batch-job)  # idle I/O class — only runs when nothing else needs disk

Lesson: When your app slows down unexpectedly and CPU looks fine, always check wa in top/htop. High iowait = disk bottleneck. Use iotop to find the offending process immediately.