Large directory feature not enabled on this filesystem
Today I learned (2020-08-30)
TIL Firefox & Chromium will keep regenerating .cache/fontconfig until your filesystem blows up!
My wife's computer had been acting really strangely the last couple of days, and today in culminated in extremely bad performance: the computer was usable for a couple seconds, then completely unresponsive for about 2 seconds, then usable again.
top proved useless, because during the unresponsive periods their
UI would freeze, so whatever application was causing the staggering was not
Dmesg showed what was going on. The following log message was appearing with a regularity that corresponded to the 'hiccups'.
[1805.005848] EXT4-fs warning (device dm-3): ext4_dx_add_entry:2357: Large directory feature is not enabled on this fileystem [1805.005340] EXT4-fs warning (device dm-3): ext4_dx_add_entry:2352: Directory (ino: 15337216) index full, reach max htree level :2
Some internet spelunking revealed that it's possible to enable this 'Large directory' feature on a mounted device, and the invocation turned out to be:
tune2fs -O large_dir /dev/mapper/pool-abcd
(Note: this is on a LUKS-encrypted partition). Immediately after enabling this feature, the system stopped hiccupping.
Finding the culprit
Now, I was still bothered by which directory was so large that it hit against filesystem limits.
find / -inum 15337216
find command revealed that the directory in question was
$HOME/.cache/fontconfig. I tried to
ls inside it, but the
seemed to hang forever. Instead, I ran:
find . | wc -l
and this revealed that the target directory had almost 15 million files inside! Deleting them took ~2 hours with the following command:
cd $HOME/.cache/fontconfig find . -delete
The entire thing turned out to be caused by a bug in the interaction of Firefox
and Chromium with fontconfig. Running Firefox once caused
480 new cache files
to appear. Running Chrome subsequently added a couple hundred more
files. Another run of Firefox again added a bunch of files.
To fix this temporarily I issued the following commands:
cd $HOME/.cache rm -rf fontconfig touch fontconfig
Now, there is no directory for the browsers to fight over, and no performance issues so far.