A quick restart of the server didn’t fix the problem. Since I didn’t have the time to dig into it, I left it at that for a few more days.

ffmpeg gone missing

Eventually, I got around to look into it. A web search brought me to an older issue on the Mastodon bugtracker. It’s for an older Mastodon release and has since been closed, so shouldn’t really affect my up-to-date instance.

I was about to give the suggested “fix” a try when something rang in the back of my mind …

Mastodon uses ffmpeg for videos and animated GIFs. I vaguely remembered – but couldn’t find it again, of course – something I’ve read a while ago about #Ubuntu randomly uninstalling ffmpeg for no obvious reason.

Lo and behold, ffmpeg was no longer on the server. A quick apt install ffmpeg brought it back – and videos started showing up and playing again.

For the record: The server is running Ubuntu 22.04 LTS.

#MastodonAdventures #RunningMastodon #ffmpeg

Picking a Cloud Service

Initially, I was skeptical about the cloud option, mainly because I wasn’t sure what it would cost me. Since I now have a better idea how much space this instance needs and since it turned out that cloud storage is not so expensive to begin with, I was ready to take the plunge.

In the end, I simply went with the instructions by Thomas Leister and the cloud service he was using, Scaleway. Details will differ between cloud services, whereas the costs are comparable. So if you’re looking for an alternative, try the instructions by Stanislas for Wasabi.

I don’t have much to add to Thomas’ instructions. If you’re looking for a cloud service based in Europe, Scaleway seems like a good option. You can also pick the location of the data center. Their web interface can be a bit overwhelming – just make sure you’re booking the Object Storage. That’s the one you want – ignore all the others.

Then just follow Thomas’ instructions (which sometimes link back to Scaleway’s help pages). Also, make extra sure you replace the default names that Thomas uses with the ones you chose for your instance and your server location. For example, I overlooked the line:

S3_BUCKET=instance-media

Things seemed to work, but all images in posts were missing and instead displayed an “not available” message. So if something is not working as expected, go over all these names, URLs, and keys again carefully. And don’t forget to restart nginx and/or Mastodon itself.

One important bit I learned: You still want to maintain a (small) local cache, since outgoing traffic (i.e. out of the cloud service) costs money. You can avoid having to serve (and therefore pay) for the same file being sent over and over by configuring nginx with a local proxy. Thankfully, both Thomas and Stanislas explain how to do that in their instructions.

Thomas’ instructions cut off a bit abruptly and left me with a “now what?” For that part, switch over to Stanislas, who shows how to check if everything is working using curl. And then concludes: You can now remove your public/system folder on your server! :D

This actually gave me pause. What I had failed to realise was that I was not only moving the cache to The Cloud but also any media that are being uploaded directly onto my instance. This felt a little like losing control over these assets. But then again, I can still download them via the cloud provider’s web interface, if need be. Or do a data export for my account. And it’s not like they were easily accessible in the first place, but buried deep down in some subfolder. So yeah, go ahead and delete that directory (once everything works).

What next?

I still have a problem that sometimes avatars are not loading. My hunch, based on some logfile entries I saw, is that some process is running out of memory when doing this. So I’m considering upgrading to a more beefy vServer. Since the extra cost for the second volume is now gone and the cost for the cloud storage is lower (hopefully), this is something I will look into next.

#MastodonAdventures #RunningMastodon

So a week after my Mastodon instance ran out of disk space, and after moving the cache to a separate volume, that volume also filled up completely. Oh well. At least the instance kept on running this time. It just couldn’t, well, cache things any more.

I haven’t had a chance to look into moving the cache to a cloud provider. And by the rate things are going, I’m not sure if I really want to, since I have to wonder if this could potentially ruin me with excessive additional costs.

As a short-term measure, I increased the available disk space some more; and then some more again. Good thing that these days all of these things are virtual and can be scaled easily with a few clicks. And then I started looking into understanding what was actually taking up so much space and how to possibly keep it at bay.

But first, a mysterious error message

But first, I had to figure out another problem that had been popping up with that additional volume. Mastodon’s “bundle” task would complain about the new volume being a read-only file system, even though it wasn’t and the Mastodon user could create and delete files and directories without a problem when logged in directly. I got it working somehow two or three times, but the problem would always reappear whenever I fiddled with some other configuration option.

Eventually I found an unrelated, yet helpful forum post that pointed me in the right direction: Mastodon’s config files in /etc/systemd/system/mastodon-* all have an entry ReadWritePaths which by default only points to the live directory in the Mastodon user’s home directory. Adding the new volume as an additional item to these lists finally got rid of the problem.

Taming the cache

Back to the cache. What, exactly, is taking up so much space? There are four sub directories in the cache directory and the culprit can easily be identified: It’s the media_attachments.

In my case, after a week, that directory alone took up 19 GB and therefore the majority of the 25 GB I had allocated solely for the cache. Forget about the preview_cards (the other directory that is regularly cleaned up by a cronjob), they are only a few hundred MB. That’s nothing in comparison.

My solution, for now at least, is to only keep the media attachments for 2 days (and make sure the cronjob actually runs):

RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --days 2

Disk usage of the cache directory has now been stable over the Christmas weekend and I hope it will stay this way while I consider my next steps.

#MastodonAdventures #RunningMastodon

Logging into it, the Mastodon logfile only showed some nondescriptive error message regarding Postgres. But I had missed the more obvious reason: The server had run out of disk space. I quickly concluded that this was due to me not having activated one of the recommended cleanup tasks (mostly because I misunderstood what it actually does). So I deleted and compressed some logfiles until there was enough disk space available again for the instance to come back to life. Then I ran both cleanup tasks. That brought back about 1.5 GB. Phew? To my horror, I had to watch them quickly fill up again. Something else was going on.

Poking around, I noticed it was the cache directory, live/public/system/cache, that kept on growing. Eventually, it hit me: It was the relay I had subscribed to.

What’s a relay?

Quick excursion: When you are running a small instance with only a few users, you will notice that your federated timeline won’t have many posts you haven’t already seen, that your list of hashtags is not very extensive, and that search just doesn’t seem to return what you are looking for. This is because your Mastodon instance isn’t pulling in a lot of what is out there. Effectively, it is only following the accounts that your local users are following. That will only bring in a small portion of the #Fediverse and you won’t see much beyond that.

One way to fix this is to have more users. The other is to subscribe to a relay. That will effectively pull in posts from other instances, thus enriching your federated timeline, your hashtags, and your search.

But, and that was my problem here, it also means that your instance will have to cache all those posts, including their media attachments and their preview cards. And those will take up disk space.

Solution?

Since my disk space was filling up rapidly, I unsubscribed from the relay and ran the cleanup tasks again. That fixed the immediate problem but was asking for a long-term solution. I considered my options:

• live without a relay
• get more users
• get more disk space

Running a somewhat disconnected instance isn’t fun. It’s not what I joined the Fediverse for in the first place. And while I’m not opposed to getting a few more users on this instance eventually, this was not a short-term solution. So, I had to get back to the relay and, therefore, get more disk space.

There are two ways to do that: Get more “local” disk space or get “disk” space in the cloud. I’m a bit reluctant hooking up with some cloud provider, as this is all new to me and I need to do some research first. So I chose the other option. Since the instance is running on a highly configurable vServer anyway, getting a scalable volume of additional disk space was only a few clicks away. I moved the cache directory over to the newly attached volume, restarted Mastodon and it’s been running off of that happily since.

Current status

The immediate danger of the instance grinding to a halt again is over. I will have to watch the disk space on the new volume, making adjustments as needed. And I will research the cloud option, to see which is more cost effective.

#MastodonAdventures #RunningMastodon

It has been a while since I administered a webserver. That was back in the Apache 1.3 days, long before Docker and all this new-fangled stuff (which I still don’t understand to this day) came along. So I got myself the smallest vServer I could find and gave the instructions a try. What can I say – it worked (almost) without a flaw.

Now, those instructions are not for the faint of heart. But if you know where to look (and what to do) when, at the end of the day, nginx reports a permission issue trying to serve a CSS file, then you can do it.

Setting up a Mastodon instance is one thing. Running an instance is another, which we’ll explore in another post …

#MastodonAdventures #InstallingMastodon