Today we had a customer who had an interesting problem. They were getting the message ‘disk is full’ despite having plenty of free space. Luckily for him, my first thought was ‘inodes?’
I logged in and checked his inode usage

root@askdev:# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 525312 524844 468 100% /
varrun 65579 27 65552 1% /var/run
varlock 65579 2 65577 1% /var/lock
udev 65579 2696 62883 5% /dev
devshm 65579 1 65578 1% /dev/shm

This shows that all the inodes on the disk itself are full.
I used the following script to determine where the inode usage was most

root@askdev:/# for i in `ls -1A`; do echo “`find $i | sort -u | wc -l` $i”; done | sort -rn | head -5
468388 var
49844 usr
18741 proc
5187 sys
5026 root

I tracked it down to /var/lib/php5/ and all the session files in there.

I used a find to then find any that were older than 10 days
root@askdev:/var/lib/php5# find ./ -type f -mtime +10 | wc -l
111041

High inode usage is usually caused by a massive number of small files. In this case the session files are normally stored somewhere temporary and removed when not in use. Either there could have been a bug in the code not removing them or it was a higher traffic website.

You can delete files older than 10 days if you want with the following command
cd /dir/of/inodes
find ./ -type f -mtime +10 | xargs rm