And then it came to a grinding halt
After running just fine for 14 days, my #Mastodon instance stopped working. All it showed was the cute animation of a mastodon banging the keyboard of a computer. My iOS app had also stopped updating my timeline. The server was quite obviously down. What happened?
Logging into it, the Mastodon logfile only showed some nondescriptive error message regarding Postgres. But I had missed the more obvious reason: The server had run out of disk space. I quickly concluded that this was due to me not having activated one of the recommended cleanup tasks (mostly because I misunderstood what it actually does). So I deleted and compressed some logfiles until there was enough disk space available again for the instance to come back to life. Then I ran both cleanup tasks. That brought back about 1.5 GB. Phew? To my horror, I had to watch them quickly fill up again. Something else was going on.
Poking around, I noticed it was the cache directory, live/public/system/cache, that kept on growing. Eventually, it hit me: It was the relay I had subscribed to.
What’s a relay?
Quick excursion: When you are running a small instance with only a few users, you will notice that your federated timeline won’t have many posts you haven’t already seen, that your list of hashtags is not very extensive, and that search just doesn’t seem to return what you are looking for. This is because your Mastodon instance isn’t pulling in a lot of what is out there. Effectively, it is only following the accounts that your local users are following. That will only bring in a small portion of the #Fediverse and you won’t see much beyond that.
One way to fix this is to have more users. The other is to subscribe to a relay. That will effectively pull in posts from other instances, thus enriching your federated timeline, your hashtags, and your search.
But, and that was my problem here, it also means that your instance will have to cache all those posts, including their media attachments and their preview cards. And those will take up disk space.
Solution?
Since my disk space was filling up rapidly, I unsubscribed from the relay and ran the cleanup tasks again. That fixed the immediate problem but was asking for a long-term solution. I considered my options:
- live without a relay
- get more users
- get more disk space
Running a somewhat disconnected instance isn’t fun. It’s not what I joined the Fediverse for in the first place. And while I’m not opposed to getting a few more users on this instance eventually, this was not a short-term solution. So, I had to get back to the relay and, therefore, get more disk space.
There are two ways to do that: Get more “local” disk space or get “disk” space in the cloud. I’m a bit reluctant hooking up with some cloud provider, as this is all new to me and I need to do some research first. So I chose the other option. Since the instance is running on a highly configurable vServer anyway, getting a scalable volume of additional disk space was only a few clicks away. I moved the cache directory over to the newly attached volume, restarted Mastodon and it’s been running off of that happily since.
Current status
The immediate danger of the instance grinding to a halt again is over. I will have to watch the disk space on the new volume, making adjustments as needed. And I will research the cloud option, to see which is more cost effective.