It's already September and I haven't yet blogged about July... That's partly because of holidays and some personal stuff. But mostly also because of the amount of work Lemmy has brought...

In July, user count has grown from 58.000 to 118.000, so that doubled! Mind, this server was only started June 1st... This sort of growth has brought a lot of challenges, but the technical challenges are the fun ones. What has been less fun is the continuous attacks we faced, probably because we're the largest Lemmy server and we've pissed off some folks by moderation actions (defederating from some instances, banning users etc). Luckily we've built a very good and capable team, for both the sysadmin side as well as the moderation side.

There also has been good cooperation with admins of other servers, especially those of and And a lot of app and frontend developers are involved.

We've put the site behind Cloudflare. This is an emergency measure we needed to take, and although CF isn't the popular option, we decided to use it because we knew other Lemmy servers used it successfully. We had no time to look for and test other options. We will do that when stuff settles down.

We also started using Digital Ocean to develop terraform scripts for re-deploying the server for scaling purposes.

Actually, for things have been quiet in July. Which is good.

The Calckey software has been re-branded to Firefish. We've upgraded to the latest, so now it's running Firefish. Unfortunately, the domain is already taken, so we're thinking about a new name where we'll create a new Firefish instance. We'll keep running for those who don't want to migrate.


Like June, July also saw many one-time donations because of the Lemmy user influx, and a steady growth in recurring donations. The one-time donations are good for building a reserve, the recurring ones are for the continuity of the platforms. (In August you'll see the one-time donations going down significantly, but the recurring still more than cover our costs).

Because of the attacks on our Lemmy, we have enabled Cloudflare as an emergency measure. We've paid for 1 year, and hope to have found an alternative before that's over.


  • Hetzner: EUR 741,35 (masto & lemmy)
  • DigitalOcean: EUR 0,59 (lemmy)
  • Cloudflare: EUR 224,82 (lemmy)
  • Wasabi: EUR 101,75 (masto)
  • Mailgun: EUR 87,24 (masto & lemmy)
  • Donations: EUR 99,22 (mastodon & lemmy Patreon)

Total costs: EUR 1254,97

The Cloudflare is a one-time (yearly) cost.\ Mailgun will be much cheaper from now on, because the largest user influx of Lemmy seems over.


These were the donations received:

  • OpenCollective: EUR 2218,-
  • Patreon: EUR 1509,33

Total income: EUR 3727,33

What's next

August has also finished already, we'll make the blog post as soon as all the invoices are in. Costs were less, donations were less. You'll see.

What's also planned, is the creation of a non-profit Foundation which will own all Fediverse platforms run by us (and maybe by others). Main purpose of it will be to not have 1 person be liable in case any bad stuff happening on those instances leads to lawsuits. Also good for continuity to have all things fedi separated.

This has been a hectic and busy month. There have been all kinds of challenges, but also a lot of good stuff!


As you might have read in my previous blog, we have setup a Lemmy server 6 weeks ago. In those 6 weeks, it has grown to be the largest Lemmy server, with over 100k users.

This of course also brought many challenges. From a spam wave, to being hacked and also a lot of moderation challenges.

But all in all I'm very pleased with how it's running now, and the team we have assembled. We have around 25 people in our Discord, who help with sysadmin / troubleshooting / moderation etc. Even admins from other instances, and app developers.


For, there hasn't been too much news. There has been a security fix which we installed, and we're waiting for 4.2.0 to be released so we can get back to stable releases. (We're running a nightly build at the moment, because we needed captcha in order to fight off spam waves)


Due to all the attention that Lemmy needed, has been a bit 'forgotten' last month. Performance hasn't been good, we've upgraded the server specs twice.

July 19th there will be a re-branding of Calckey to a new name, together with a release of the new version (which will be v1.0 instead of v14.0). We will upgrade to the new version, and decide what to do with it once the name of Calckey has changed. The new name isn't available as a .world domain, so either we think of a new cool domain name, or we keep using

Meta / Threads

The whole threads commotion has been my biggest headache during the past months. I've seen so much negativity, hate, threats etcetera about the topic, that I've not communicated about it a lot.

First of all, I'm a bit neutral here. I want what's safe for our users. In order to have that, we first need to know what it is exactly that's coming. I see a lot of people panicking without knowing or understanding what's to be expected.

Secondly, I prefer to have the moderation teams deciding on this. They have to deal with it more than me, I just keep the server running. After lengthy discussions, the team made this statement on Lemmy.

What I would like to add, is that it's OK to post your opinion, it's OK to block threads for your account if you like (although on Lemmy that doesn't work yet), and of course it's also OK to migrate to another server if you disagree with our decisions. But please, keep it polite. Understand that Fediverse servers are indeed run by real people, not the big corporations. Volunteers, doing their best to provide services for you in their spare time. I've seen admins/mods get threatened over this...


And now for the fun stuff. The finances.

Running Lemmy brings extra costs, but more so it brings extra donations. And, like with Mastodon in November, this new user influx brought a spike in donations, which will probably also fade over time. But this spike gives us a nice buffer, so we can guarantee the continuity of the servers for at least the next year.


The expenses for June were:

  • Mailgun (e-mail): EUR 107,93
  • Hetzner (servers, backup storage etc): EUR 760,81
  • Wasabi (media storage): EUR 84,37
  • Other*: 264,65

(*Other was Netdata, which we use for monitoring all servers (EUR 165,43) and Patreon donations to Mastodon and Lemmy (EUR 99,22))

Total costs: EUR 1217,76


As said, donations spiked this month, and seem to get even higher in July.

  • Patreon: EUR 958,74
  • OpenCollective: EUR 1588,89

Total income: EUR 2547,63

Extending the .world

As some may know, I run several other Fediverse instances next to

When I decided it was time to also run a Calckey server, I chose the domain, and started thinking about expanding the .world.


So when I wanted to start a Lemmy server, first thing I did was check if was available. And it was!

So I started installing Lemmy on it, and while I was doing this, I read a post about Reddit charging app developers millions for their API use. Speaking about timing... By the time the server was finished and operational (June 1st), it was announced that Reddit would see some subreddits go dark on June 12th.

So, with this announcement, people started fleeing Reddit, and looking for Lemmy servers to join. I started using a VPS (virtual cloud server) for Lemmy, with 2 CPU and 4GB RAM. Soon after opening, that appeared not to be enough, so I doubled it. And after a few thousands of users, I had to double it again, to 8 CPU and 16 GB RAM.

This really remembered me of Mastodon in November. The larger servers were struggling with the load, causing people to choose my server to create an account. Also, there aren't real 'generic' large servers. The largest server states it's for 'privacy and FOSS enthusiasts'. So the .world grew fast..

I had put the links to the Opencollective and Patreon on the Lemmy site as well, because I don't want to have multiple of those. Also I started paying for the Lemmy cloud servers from the funds. Soon, new subscribers to these donation platforms started coming in. And even so much, that I decided to do the same for Lemmy as I did for buy a ridiculously large server.

The current VPS couldn't be resized that much anymore, and load was going up with all the new users. So I bought the same server at Hetzner: a 32-core/64 thread 128GB RAM dedicated server. (For Mastodon, I doubled the RAM. For Lemmy I don't think it's needed yet.) I migrated the Lemmy software and database there, and moved over. This took 4 minutes of downtime.

After that, the server is performing very smoothly.We're over 33k users now, having the second largest Lemmy server there is. (See here for stats).

Biggest challenge now is the moderation; I see that there's more unwanted content than on Mastodon, and the moderation tools are very basic, or even lacking. But we're working on getting this organised!

It's been a while, but here's the update about what's been happening to

What happened

The most important thing that happened was the wave of spam accounts. This wave started at the largest Mastodon server, But as soon as they installed a Captcha to prevent spam signups, the spammers chose other servers, among which ours.

We got hit by a wave of many spam accounts, sending DMs to users with crypto-spam. We then installed the latest build of Mastodon, which had the emergency-captcha-fix in it, and enabled captcha. This will prevent most of the automated spam signups. It has been quiet since.

Financial update

An update on the finances:



  • Mail: EUR 31,80
  • Hetzner (server etc): EUR 382,70
  • Storage: EUR 66,13

Total: EUR 480,63


  • Patreon $557,39 = EUR 517,87
  • OpenCollective: EUR 444,- Total: EUR 961,87



  • Mail: EUR 32,52
  • Hetzner (server etc): EUR 424,34
  • Storage: EUR 75,69

Total: EUR 532,55


  • Patreon $535,19 = EUR 492,37
  • OpenCollective: EUR 351,- Total: EUR 843,37

In May, you see that costs have gone up a bit. That's because we've also started a server and we needed some extra space for the backup server for

You'll also notice that donations are going down every month. That's to be expected, number of active users has also gone down, until end of may, when it stabilized. We expext donations to stabilize as well, and if they won't we'll post a message asking for people to donate if they can/want. (We don't do that very often). But also we now have around EUR 8000 reserve, so even if donations might fall below expenses, we're good for quite some time.

But in June, we've started, which drove costs up, but also generated a lot of new donations. I'll write about that in a next blog post.


This is the financial overview for March 2023:


We've had these donations in March:

  • Opencollective: EUR 473,73
  • Patreon: $585,15


  • Hetzner : EUR 382,16
  • Wasabi : EUR 62,28
  • Mailgun : EUR 32,35

Also we're Silver sponsor to the Mastodon project, which costs around 50 EUR per month


Last few weeks we've seen a lot of spam accounts. The famous 'thegx' account which was created at dozens of instances, but also a flood of Vietnamese casino accounts. If you see one of these, please do report them.

I'm sorry, this will be a brief blog post. If you have questions, contact

What happened

In February, no extraordinary things happened.



In February, we received these donations:

  • Patreon: $ 580,97
  • OpenCollective: EUR 663,73


  • Hetzner (servers): EUR 349,51
  • Wasabi (storage) : EUR 64,78
  • Mailgun (E-mail): EUR 33,02 Total : EUR 447,31

As said in last month's blog post, we also donated EUR 130 to which was in financial troubles. We promised to do that again this month. Also, starting this month, we will be donating to the Mastodon development team (in their 'silver' tier), just over EUR 50 per month, to support further development of the Mastodon software.

February is almost over, which made me realize I hadn't posted about January yet. I'll be brief.

What happened

The month of January was quite slow and stable. We had under 6000 new registrations this month, while at some moments back in November we had the same amount in under 2 hours..

January Statistics January Statistics2



We had many donations again in January:

  • Patreon donations: $600,82
  • OpenCollective donations: EUR 549,73

This is awesome. It guarantees we can keep the lights on for months ahead, and plan for scaling when needed.


Expenses were down, as expected. Due to decreased usage, the e-mail costs were down by over 80%.

  • Server costs: EUR 338,94
  • E-mail costs: EUR 32,10
  • Media storage costs: EUR 55,08

Total running cost for January 2023: EUR 426,12

In February, we made a EUR 130,– donation to a Mastodon instance which had some financial troubles ( I promised to make another donation to them next month, to give them time to downsize to reduce some costs.


We had posted about this, but let's also mention it on the blog. Some Mastodon instances have announced they would shut down, leaving tens of thousands of users needing to migrate their account. At, we have more than 1 admin. If 1 of them wants out, or gets hit by a bus, others have access to take over. And as stated above, finances allow this service to run for months ahead. I hope this takes away any worries for our users they might have gotten due to the other shutdowns.

Enjoy Mastodon!

What happened

So after the first blog post, we saw another spike in registrations mid-december. After that, growth has slowed and stabilised. Only a handful of new users every hour. The server is running smoothly.


There have been some challenges with moderation. Many reports come in, which are handled promptly and thoroughly by the moderation team (currently 7 people). Some decisions are difficult. They are all made with the rules in mind. There's always the option to appeal a decision (please engage in polite discussion, instead of name-calling and aggressive reactions). Appeals will be handled by a different moderator than the one that made the decision. All in all I think we're doing a good job in keeping a friendly and welcoming community with almost 150.000 users. I am really thankful to the moderation team!



We saw many donations come in in December. Thank you all for that!!

  • OpenCollective : 1985,53 EUR was raised in December
  • Patreon: $673,85 was raised in December


Expenses were lower in December, and will be even lower this month, because we have scaled down the e-mail subscription with Mailgun.

  • Mail hosting (Mailgun): 201,36 EUR
  • Hetzner server costs (server, backup, test server, monitoring etc): 332,29 EUR
  • Wasabi media storage: 27,06 EUR

Total costs for December were 560,71 EUR.


Plans for include:

  • Disaster recovery: Next to backups, I would like to setup replication to another data center, just in case..
  • Scalability: We're looking into using Kubernetes to scale in case of load-peaks and future growth
  • Location: When migrating to Kubernetes, we plan to move the server(s) to The Netherlands, this has to do with jurisdiction. We need to sort out legal stuff, but having the server running in another country than the owner, isn't making it easier..
  • Media cache: Media is now served straight from the block storage. This needs some cache in between, and an URL masking that.

Other instances

Next to, I have setup some instances for my own interests:

(I already have a Pixelfed, Peertube and Misskey instance, and of course this WriteFreely blog).


If you have any questions or remarks, you can contact the admin team via, and the admins and moderation team can be reached by e-mail:

Since the big inflow of users, the running cost for has increased significantly. Before November, it ran on a 100EUR/month server which I use for dozens of services.

That no longer was enough. I am very happy that the users of have donated generously, making the server upgrade and other enhancements possible.

Because the server is now funded by donations, it’s important we’re clear on the expenses we cover from them. That’s why I’ll post the expenses here, hopefully monthly.

Expenses for November

The expenses in November have been quite high, due to the ‘emergency fixing’ and server upgrade. This will be less in following months.

Expenses (in Euro’s):

  • Hetzner – old server plus backup server: 105,90
  • Mailgun
    • Plan 100k Nov: 28,91
    • Plan 250k Nov: 70,29
    • Plan 250k Dec plus overage Nov: 235,25
  • Hetzner:
    • Backup server: 20,80
    • New server: 360,00
    • Cloud servers (monitoring & testserver) : 15,13
  • Wasabi storage: 7,38

Total running costs: 843,66

Donations for November

Donations, after deduction of platform cost:

  • OpenCollective: EUR 3406,08
  • Patreon: $568.67


Costs were quite high in November, but the donations more than covered it. Seeing that part of the donations are recurring, I think we're financially safe for at least the coming months.

There will be less expenses in December (The one-time setup costs are only for this month, the e-mail solution will be changed), but maybe some extra costs will be needed for adding high-availability.

When everything looks stable financially, my goal is to have 6 months in reserve. Might we have money left after that, I will donate that to open-source projects.

After the insane month of November 2022, it's now time to write down what happened. A lot happened.

Let's try this Mastodon thing

Back in 2017, I stumbled upon the Mastodon project and thought, this looks nice! Let's try it. So I registered a domain name ( and installed Mastodon, back then using Yunohost. Started using it, but soon lost interest, so a few months later I shut it down, and in 2018 the domain name expired.

In 2021, I re-gained interest in Mastodon. Twitter became more and more un-useful for me, and I remembered how I liked the Mastodon interface. So, I started looking for a new domain name, and found '' still available. I was running a dedicated server at Hetzner, on which I ran ProxMox. In ProxMox, I ran a virtual machine on which I ran Docker. Running Mastodon in Docker is easy.

As with most of my self-hosted servers, I opened registrations for Mastodon. In my experience, not many people would signup, which was true for the next months. Around 10 accounts per month were created, but only a few remained active.


Early November, Elon Musk's Twitter-takeover became final. That, and some poor decisions he took, hurting people's confidence in Twitter, caused people to leave Twitter and search for alternatives, like Mastodon. I saw a slight increase in user signups, 'my' 100 users became 140 users. I figured, my server could handle some more, so I thought I would apply for the Serverlist, the list maintained by the Mastodon developers, which is also used for the Mastodon mobile app when selecting a server to join. November 3rd, at 19:02 my time, I mailed them requesting my server to be added. At 21:58, I received a reply bij Eugen that my server was added. But I didn't see any signups coming in. After an hour, I went to bed.

The next morning, around 8AM, I woke up, seeing 700 users in my server! Nice, people started finding my server. I welcomed and helped some newcomers before going to work.

While at work, the registrations started rising even faster. The 1000th user was reached at 10:15, the 5000th at 21:00. The next 45 minutes gave 1200 new registrations, by then I started seeing issues on the server. I closed the registrations and started troubleshooting.


One of the important components of Mastodon is Sidekiq. These are queues of tasks that do all sorts of stuff, like pushing the posts to other servers, getting posts from other servers, sending e-mail, checking links etc. There are different queues (default, pull, push, ingress, mailer, scheduler). The default install in Docker starts 1 process for that, with 5 threads, handling all queues. This means, only 5 of these tasks can be handled simultaneously. When investigating the issues I encountered, I saw over 350000 tasks were queued. Sidekiq couldn't keep up.

Without looking into it very much, I changed the number of threads from 5 to 200. Queues were going down quickly.

I re-opened registrations, the biggest inflow of users was over for now, growth was more low and steady.

Growing pains and making plans

November 5th. User count was around 10k. By now, I couldn't consider as just some hobby project. Many people were relying on this service. So, first of all, I was keeping a close look on the performance all day (luckily it was weekend). Closing registrations now and then, to give the server room. Then, I started thinking about continuity. It was just me running this. I decided I would need at least 2 other admins, not only to help out, but also to take over in case something happens to me. Asking around, I found 2 people willing to help: @jeroen and @spaceriker.

I also noticed that moderation was slowly taking up more and more of my time. (At the moment, moderation is the most time-consuming element of running Mastodon). So I decided to look for moderators. I found 3, and a day later a fourth. In different timezones, which is helpful:

Because of the sudden surge in users, I also noticed that other, also some larger, instances had closed registrations. So, whenever I opened registrations, big parts of the network's new users landed on my server.

November 6th, I saw a spike in registrations again. Sidekiq queues went up again, so I started tweaking those. I started more and more processes, with around 800 threads in total. But, that triggered the next issue. The database's max connections setting was reached, so sidekiq could't create more threads. I raised the max connections to 800 (as a database admin, I know that's not very good when using Postgres, but all my actions were emergency fixes...) and restarted the database. Pfew. Got that fixed.

But now, many sidekiq tasks started failing. Investigating, I found that it was caused by the e-mail server, which I also hosted myself. All the accounts send out verification e-mails, sign in e-mails, password reset e-mails, and notification e-mails. The mailserver choked on the number of mails. (I had no idea how many, at that moment). After some failed attempts at getting this to work again, I switched to Mailgun, a service provider for sending e-mails. I knew this was going to be costly, but again, it was an emergency fix. This helped. Everything was working fine again. (Later, I would see at the peak day it would send out 75000 e-mails!)

November 7th, I started created donation pages, so that users could make donations if they wish. I had had some request from users wanting to donate, and also I was concerned about the projected cost for that month, with Mailgun. And I wanted options for scaling out, because I was worried the current setup couldn't handle much more.

I created an OpenCollective and Patreon account. Donations started coming in quickly, which creates possibilities for planning scaling.

Also that day, one of the moderators @stevo, who works in 'Business Ethics and Anti-Corruption Compliance', created a new set of rules for the server. Up till then, I had 4 rules, which I made up when there were a handful of users. By now, we needed more, also for being used by the moderators to moderate by.

That evening, I was interviewed by Joanna Stern, a senior Tech columnist for the Wall Street Journal. She happened to have created her Mastodon account on my server. A few days later, this article was published.

Mostly stable.. let's keep it that way!

Growth was kind of stable. With the e-mail service and the tweaked Sidekiq queues, the server was running well. People complimented me on the 'snappiness' of the server. (Some other big servers were still struggling). I was making plans to get better hardware. First, I setup a new instance ( purely for testing. Same setup as (but smaller). On November 11th, I used this testserver to test migrating the media storage to Wasabi. That was necessary because I noticed this taking up more and more diskspace, and had calculated I could only last a few more days with my current disks. After I got this working on the testserver, I migrated the media for to Wasabi as well. (It was 200GB back then, but grew rapidly. Today, Dec 12th, it takes up 2.2 TB)

November 12th, I ordered a bigger server from Hetzner, the AX161 with a 32-core, 64-thread AMD EPIC CPU and 256GB RAM. It was 'delivered' 2 days later.

On November 13th, @jeroen added uptime monitoring ( so we can see (and show) how reliable the server is for the users.

Around that same time, Wasabi started having issues. Users couldn't see or upload images intermittently. I reported this to Wasabi support, it would take a day before they got that fixed.

Also the 13th, @stevo wrote a Code of Conduct, to further guide the moderators and the users in how we want the community to be a happy place for all.

Server migration and upgrade to v4

On November 14th, it was time to move to the new server. This was actually quite easy: * Install docker on the new server * Copy the mastodon software and database to the new server using rsync * Stop Mastodon * Do a final rsync to get the latest changes * Start Mastodon on the new server * Change the DNS entry to point to the new server

(Left out some smaller steps, but these are the main ones)

This resulted in around 20 minutes downtime, the most of that being the result of a typo I made in the Nginx configuration.

On the new server, Mastodon ran mega fast. Sidekiq tasks are executed very quickly because of the fast CPU, so queues stay low.

The next day, I upgraded to Mastodon v4.0.2, after first testing it on The upgrade took less than a minute of downtime. Version 4 was long awaited with some key features (Editing of posts, following hashtags etc).

On the 16th, I did some tweaking on OS-parameters (open files) because Nginx started to give errors.

Elon again

November 18th started with 30.000 users. But then Elon shook up the Twitter community again. Apparently he fired some staff (a lot actually), and at a certain point even closed Twitter offices so staff couldn't get in anymore. This caused an even larger exodus of Twitter users who started to be concerned that Twitter would fall down. (Which I don't think it will), Thousands of new users registered every hour. Within 12 hours, user count doubled, to 62.000. I then closed registrations because I started seeing some database connection issues again.

So, finally I started to look into the database configuration (I am a database admin, so perhaps should/could have done that sooner :–) ). What I also use at work when dealing with too many connections to a Postgres database, is pgbouncer. So, I installed and configured that on the test server, and after some configuring, got it working. The next day I installed that on Now that bottleneck was solved, we were ready to accept more registrations.

Because of the growth, we started getting more and more reports on content and users that violates our rules. So, between 18 and 22 November, we added 3 more moderators to the team: * @KTR (for helping moderate Spanish posts) * @Sodfabasha (for helping moderate Arabic posts) * @Paul


On November 21st, we reached 100.000 users. Growth was slowly getting less, system was stable, moderators doing an excellent job!

I was getting more and more e-mails on, so I installed a ticketing system, enabling me to have some of the moderators help me reply to the mails.

November 22nd, I was interviewed by The Verge, resulting in this article.


November was over, but on December 3rd, something happened which I would like to add to this blog. Some users reported seeing 'weird accounts' spamming the Federated timeline of our server. They were accounts with names like @<random string>@<random string> The only thing they did, was post messages with 2 accounts of the same trolls. When such post arrives at your server (because someone boosts it or follows it), the server will check the post and follow the linked accounts. On each of those linked accounts, again they will find a post with 2 accounts. And so on. On our server, already 20.000 of such accounts were known. Luckily, the server could handle that extra load. Other, large servers couldn't, and were unreachable. When I found out what was happening, I blocked the domain (which took some time because of the number of subdomains known to our server). That solved the issue. I then posted this information to mastodon, mentioning #mastoadmin, so that other admins could take action. I also notified other admins via Matrix channels. My post was boosted 3700 times, it reached many admins.


This has been a very stressful, but fun, and educating month. Apart from the few bad comments you get, I got many friendly messages and compliments on running this server. I met (mostly online) some new people. I am really blown away by how the moderators are doing their job, like professionals (although this is all volunteer-work!). I have had 3 interviews (last week also for the Süddeutsche Zeitung) and an interview with a student from the University of Alberta.

I did get some remarks from people that think a Mastodon server shouldn't host this many accounts. Although I agree, there was no alternative place for these new users to go to, during the peak moments. Large servers closed registrations or were down because of the influx. Many smaller servers couldn't be found because they are not on the Joinmastodon serverlist (mind you, many new users sign up via the Mastodon app, which uses this serverlist when signing up). This needs attention for the future growth of Mastodon.

There are also some follow-up tasks and questions. How do I want to scale for possible future growth? Do I want growth at all? How do I make sure there is high-availability, without performance being degraded? And so on.

In my next blog, I will talk about the cost of running this server.