"No space left on device" errors when there is enough space left on the device due to a lack of free Inodes

Overview

You may encounter an error message that indicates "No space left on device" even if, upon further investigation, you determine that there is more than enough free space.

The main goal of this article is to discuss that the error message can be misleading considering that the issue may be related to a lack of available inodes on the disk, rather than actual disk space. Inodes are data structures used by Linux file systems to store metadata information about files and directories, such as permissions, ownership, and timestamps. Each file or directory on a Linux file system consumes one inode. If the file system runs out of available inodes, it can cause errors like the one you encountered.

Diagnosis

While this article's overview assumes you already found the underlying issue, here are some commands that might help you. We will assume that the error is related to the /data volume so that more specific commands can be shared.

Verify disk space usage

df -h /data

Check inode usage

df -i /data

Display the inode count and directory path for the 10 directories with the highest inode counts sorted in descending order.

du -i /data | sort -rn | head -n 10

Solution

You needed to identify and remove files or directories that were consuming excessive inodes on the disk. Once identified, you can remove unnecessary files or directories using commands like rm or rmdir.

Also, cache files are often the culprit of unexpected Inode overloads. So, for example, if there is an issue with the cache used by a Rails application, the Rails.cache.clear command can be used to clear the errors while you investigate the application.

Alternatively, you may consider increasing the instance's volume since space increments will also increase the number of available Inodes. However, this should not be your first resource.