I fought the cache and the cache won
(for now)
So a week after my Mastodon instance ran out of disk space, and after moving the cache to a separate volume, that volume also filled up completely. Oh well. At least the instance kept on running this time. It just couldn’t, well, cache things any more.
I haven’t had a chance to look into moving the cache to a cloud provider. And by the rate things are going, I’m not sure if I really want to, since I have to wonder if this could potentially ruin me with excessive additional costs.
As a short-term measure, I increased the available disk space some more; and then some more again. Good thing that these days all of these things are virtual and can be scaled easily with a few clicks. And then I started looking into understanding what was actually taking up so much space and how to possibly keep it at bay.
But first, a mysterious error message
But first, I had to figure out another problem that had been popping up with that additional volume. Mastodon’s “bundle” task would complain about the new volume being a read-only file system, even though it wasn’t and the Mastodon user could create and delete files and directories without a problem when logged in directly. I got it working somehow two or three times, but the problem would always reappear whenever I fiddled with some other configuration option.
Eventually I found an unrelated, yet helpful forum post that pointed me in the right direction: Mastodon’s config files in /etc/systemd/system/mastodon-* all have an entry ReadWritePaths which by default only points to the live directory in the Mastodon user’s home directory. Adding the new volume as an additional item to these lists finally got rid of the problem.
Taming the cache
Back to the cache. What, exactly, is taking up so much space? There are four sub directories in the cache directory and the culprit can easily be identified: It’s the media_attachments.
In my case, after a week, that directory alone took up 19 GB and therefore the majority of the 25 GB I had allocated solely for the cache. Forget about the preview_cards (the other directory that is regularly cleaned up by a cronjob), they are only a few hundred MB. That’s nothing in comparison.
My solution, for now at least, is to only keep the media attachments for 2 days (and make sure the cronjob actually runs):
RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --days 2
Disk usage of the cache directory has now been stable over the Christmas weekend and I hope it will stay this way while I consider my next steps.