A Live Server Hit 94% Disk and Nothing Was Wrong With the Site. It Was One Log File and Some Old Backups.

A server running a live business website reached 94% disk usage. There was no attack and no traffic spike. The site worked normally. If nobody had looked, it would have kept working, right up until the disk was full.

Cause one: a 3.2 GB error log

One file had grown to 3.2 GB: the PHP error log. The application was older than the PHP version the server now ran. Newer PHP versions are stricter, and the old code produced small, harmless warnings on every page load.

Each warning is one line and means nothing on its own. On a site that serves pages all day, they add up to gigabytes, and nobody reads them, so nobody notices.

Cause two: backups that were never deleted

The second cause was the backup script. It made a full site backup of 2 to 4 GB plus a database dump, on schedule, exactly as intended. But nothing ever deleted the old ones. Good intentions, no cleanup.

A backup that sits on the same disk as the site also protects less than it seems. If the server is lost, the backups go with it. Copies belong somewhere else, and the local ones only need to cover the last few days.

What we did

1. Measured first. Listed the largest files and folders before deleting anything, so the cleanup was aimed at the real causes.
2. Emptied the error log instead of deleting it. The running PHP processes keep writing to the same file, so it is truncated to zero, not removed.
3. Deleted the older local backups after confirming the server had separate full-machine snapshots elsewhere, keeping the newest site backup and the last three database dumps.
4. Cleared the routine leftovers: the package manager's download cache and old rotated system logs.
5. Checked the site and its email after the cleanup. Pages, logins and mail delivery all kept working.

That freed about 11 GB and brought the disk down from 94% to 73%, with no effect on the site or its mail.

Making sure it cannot creep back

We added a small job that runs every hour and empties any error log larger than 100 MB. In plain Linux it is one line; adjust the folder to where your sites live:

# every hour: empty any error_log over 100 MB (keeps the file, frees the space)
find /home -xdev -type f -name error_log -size +100M -exec truncate -s 0 {} \;

That job is a seatbelt, not a repair. The real fix is clearing the warnings in the code, so the log only contains things worth reading. For backups, the equivalent is a retention rule: keep the last few local copies, delete older ones automatically, and keep the long-term copies off the server.

How to find what is filling your disk

If you have shell access, these two commands answer most of the question in under a minute:

# overall usage per disk
df -h

# the 20 largest files over 500 MB on the server
find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null | sort -k5 -h | tail -20

On shared hosting with a control panel such as cPanel, the Disk Usage page shows the same thing by folder. The usual suspects are error logs, backup archives, database dumps, cache folders and old copies of the site left behind after a migration.

Three questions for any server you are responsible for

• What is the biggest file on it right now? If you do not know, look today. It is often a surprise.
• Does anything delete old backups? If the answer is "I think so", check the folder.
• Would someone get an alert at 80%? A full disk is easy to fix at 80% and an outage at 100%.