A quick restart of the server didn’t fix the problem. Since I didn’t have the time to dig into it, I left it at that for a few more days.

ffmpeg gone missing

Eventually, I got around to look into it. A web search brought me to an older issue on the Mastodon bugtracker. It’s for an older Mastodon release and has since been closed, so shouldn’t really affect my up-to-date instance.

I was about to give the suggested “fix” a try when something rang in the back of my mind …

Mastodon uses ffmpeg for videos and animated GIFs. I vaguely remembered – but couldn’t find it again, of course – something I’ve read a while ago about #Ubuntu randomly uninstalling ffmpeg for no obvious reason.

Lo and behold, ffmpeg was no longer on the server. A quick apt install ffmpeg brought it back – and videos started showing up and playing again.

For the record: The server is running Ubuntu 22.04 LTS.

#MastodonAdventures #RunningMastodon #ffmpeg

Logging into it, the Mastodon logfile only showed some nondescriptive error message regarding Postgres. But I had missed the more obvious reason: The server had run out of disk space. I quickly concluded that this was due to me not having activated one of the recommended cleanup tasks (mostly because I misunderstood what it actually does). So I deleted and compressed some logfiles until there was enough disk space available again for the instance to come back to life. Then I ran both cleanup tasks. That brought back about 1.5 GB. Phew? To my horror, I had to watch them quickly fill up again. Something else was going on.

Poking around, I noticed it was the cache directory, live/public/system/cache, that kept on growing. Eventually, it hit me: It was the relay I had subscribed to.

What’s a relay?

Quick excursion: When you are running a small instance with only a few users, you will notice that your federated timeline won’t have many posts you haven’t already seen, that your list of hashtags is not very extensive, and that search just doesn’t seem to return what you are looking for. This is because your Mastodon instance isn’t pulling in a lot of what is out there. Effectively, it is only following the accounts that your local users are following. That will only bring in a small portion of the #Fediverse and you won’t see much beyond that.

One way to fix this is to have more users. The other is to subscribe to a relay. That will effectively pull in posts from other instances, thus enriching your federated timeline, your hashtags, and your search.

But, and that was my problem here, it also means that your instance will have to cache all those posts, including their media attachments and their preview cards. And those will take up disk space.

Solution?

Since my disk space was filling up rapidly, I unsubscribed from the relay and ran the cleanup tasks again. That fixed the immediate problem but was asking for a long-term solution. I considered my options:

• live without a relay
• get more users
• get more disk space

Running a somewhat disconnected instance isn’t fun. It’s not what I joined the Fediverse for in the first place. And while I’m not opposed to getting a few more users on this instance eventually, this was not a short-term solution. So, I had to get back to the relay and, therefore, get more disk space.

There are two ways to do that: Get more “local” disk space or get “disk” space in the cloud. I’m a bit reluctant hooking up with some cloud provider, as this is all new to me and I need to do some research first. So I chose the other option. Since the instance is running on a highly configurable vServer anyway, getting a scalable volume of additional disk space was only a few clicks away. I moved the cache directory over to the newly attached volume, restarted Mastodon and it’s been running off of that happily since.

Current status

The immediate danger of the instance grinding to a halt again is over. I will have to watch the disk space on the new volume, making adjustments as needed. And I will research the cloud option, to see which is more cost effective.

#MastodonAdventures #RunningMastodon

It has been a while since I administered a webserver. That was back in the Apache 1.3 days, long before Docker and all this new-fangled stuff (which I still don’t understand to this day) came along. So I got myself the smallest vServer I could find and gave the instructions a try. What can I say – it worked (almost) without a flaw.

Now, those instructions are not for the faint of heart. But if you know where to look (and what to do) when, at the end of the day, nginx reports a permission issue trying to serve a CSS file, then you can do it.

Setting up a Mastodon instance is one thing. Running an instance is another, which we’ll explore in another post …

#MastodonAdventures #InstallingMastodon