When using the ext4 filesystem, have you ever encountered a situation where free space ran very low on your system, yet processes didn't crash? Let's explore the built-in safety mechanism ext4 uses to keep your system safe from a runaway process consuming disk space.
Reserved Blocks
If you've ever logged into a server to find the following situation:
Panic probably quickly ensues as you realize that vital system processes are likely crashing when they fail to write to this 100% full disk. But wait, upon closer inspection, it seems like your system is running normally, even though it's out of free space. This is because ext4 reserves a percentage of blocks that only root is allowed to write to. When tools like df request information about the remaining free space, these reserved blocks are not included in the total; hence these tools will report "100% used" even when some of the reserved blocks are still available (and you can see this with the discrepency between the Size and Used columns above). Moreover, unprivileged processes attempting to write to the disk will be denied. This is described in more detail in the tune2fs manpage:
1Set the percentage of the filesystem which may only be allocated by privileged
2processes. Reserving some number of filesystem blocks for use by privileged
3processes is done to avoid filesystem fragmentation, and to allow system
4daemons, such as syslogd(8), to continue to function correctly after
5non-privileged processes are prevented from writing to the filesystem. Normally,
6the default percentage of reserved blocks is 5%.
The current reserved space can be easily inspected with tune2fs:
1# tune2fs -l /dev/sda2 | grep "Reserved block"
2Reserved block count: 129778
3Reserved blocks uid: 0 (user root)
4Reserved blocks gid: 0 (group root)
You can also see which user (in this case, root) is allowed to write to the reserved blocks. This safety feature gives you a chance to go in as the superuser and correct the problem before a complete system crash occurs.
For large filesystems, it may be desirable to reduce the amount of reserved space (since 5% of a large disk may be an excessively-large reservation). For example, you could reduce it to 1%:
1# tune2fs -m 1 /dev/sda2
2tune2fs 1.46.5 (30-Dec-2021)
3Setting reserved blocks percentage to 1% (25955 blocks)
Available Inodes
It's important to also understand that data blocks aren't the only thing that can fill up your filesystem; running out of inodes is also a concern. Each file has an inode that contains metadata about the file (e.g. where its data blocks are located). When creating an ext4 filesystem, a certain number of inodes are allocated (this number can be customized when creating the filesystem with mkfs.ext4's -N flag). By creating a very large number of files on your system, it's possible to consume all inodes even though there's still plenty of free space:
1# df -i /
2Filesystem Inodes IUsed IFree IUse% Mounted on
3/dev/sda2 1278720 1278720 0 100% /
4
5# df -h /
6Filesystem Size Used Avail Use% Mounted on
7/dev/sda2 9.6G 7.1G 2.0G 79% /
Moreover, the reserved blocks don't help here, as unprivileged users can consume all inodes. Therefore, it's important to check both free space and free inodes when debugging low disk space warnings.
Conclusion
The ext4 filesystem is used by countless devices around the world due to its robust design, performance, and reliability. Reserved blocks are a critical safety feature that ext4 provides to protect your system from a process quickly consuming all free disk space, and it's important to be aware that this feature exists in case you find yourself low on space.