Blog

While upgrading the blog to Writefreely version 0.15, I noticed the last blog was dated January 2024... It seems I've been neglecting the blog and with it you, the users. Sorry!

So let's see what has happened the past 8 months:

Fedihosting Foundation

As I wrote in the last blog, we (@stux@mstdn.social and I @ruud@mastodon.world ) started a non-profit foundation (https://fedihosting.foundation) for hosting our Fediverse instances. A great part of our stuff has been transferred to the foundation.

(To be clear, this blog covers the instances hosted by Ruud, not those hosted by Stux)

Meanwhile, we're still looking for volunteers / people to join us, in any of these roles: – New members of the foundation, possibly instance owners/admins, preferably Dutch. – People wanting to help hosting and think about the future of the instances – People wanting to help out moderating 1 or more of our instances

If you're interested, DM @ruud@mastodon.world or mail to info@mastodon.world

Mastodon

The biggest news on the Mastodon side has been that we've taken ownership of https://toot.community! This instance had been setup and run by Jorijn, but he now lacks the time to run it as well as he would like, so asked us to take over and we did. Toot has around 2500 monthly active users out of 31000 total users. (Compared to mastodon.world: 13000 monthly active out of 183000 total users)

We did notice the running costs of toot no longer were covered by their donations, and the infrastructure was overscaled, so we're planning to downgrade soon.

The other Mastodon instances have been running quite stable and growing very slightly.

Lemmy.world

https://lemmy.world has been steadily growing and been performing quite well the last months. We had a major upgrade from 0.18.5 to 0.19.3 in March which went well, thanks to the preparation and hard work of the team.

Bookwyrm

Our Bookwyrm server https://bookwyrm.world has grown from 37 users in January to 206 users currently. The server has gotten some nice logo's and banners designed by @vsp@mastodon.world, and @sigvie@bookwyrm.world has been helping out moderating the instance.

Pixelfed

I had a Pixelfed instance https://fotofed.nl but wanted a '.world' version, so in February we installed a Pixelfed instance at https://photofed.world. It hasn't got too many users yet, so if you want to try a Federated alternative to Instagram, just try it!

Sharkey / Firefish (Calckey)

Our 'Calckey gone Firefish' instance https://calckey.world has grown to over 700 users. But there has been (and still is) some uncertainty about the future of Firefish, which prompted us to install another Misskey fork in January: Sharkey. Of course it's at https://sharkey.world I personally like it better than Firefish or Misskey, I even re-installed my personal instance (https://ruud.social) using Sharkey.

Finances

We haven't updated you on the finances since January. Let's sum things up: – Costs have slightly gone up, and from July we've started some cost-cutting – Donations have been going down every month (together with number of active users going down) – We've taken over toot.community, and also the donation platforms for it. Their donations don't cover the total running costs, plans for scaling down are in testing phase now. – In August, we saw a yearly subscription for Cloudflare renewing (which we'll replace with Fastly hopefully before the next renewal..) – Also in August, we discovered that the Fastly payments foor toot.community failed in the last 3 months, so we had to pay that at once. (When downscaling toot.community, we'll also migrate it to our foundation's free Fastly account)

Due to the last 2 points, August costs were about 400,– higher then they should normally be. Apart from that, the cost cuttings have worked and will be even better after t.c migration.

Here's the totals for the past 3 months:

Costs June: EUR 2424,80 Costs July: EUR 1953,45 Costs August: EUR 2368,11

Donations June: EUR 2214,16 Donations July: EUR 1971,84 Donations August: EUR 2061,51

And for those interested, these were the costs in August: – Foundation 50,- – Wasabi toot.community 94,50 – Wasabi others 226,74 – Hetzner toot.community 284,79 – Hetzner others 1116,83 – Fastly toot.community (3 months): 190,07 – Cloudflare (yearly): 232,85 – Namecheap: 2,41 – DigitalOcean: 18,30 – Donations to Mastodon and Lemmy: 99,22 – Backblaze: 2,84 – Borgbase: 16,92 – Mailgun: 32,64

As a reminder, might you want to support our instances, you can do so at any of these platforms: KoFi (https://ko-fi.com/fhfworld) Liberapay (https://liberapay.com/fhf/) OpenCollective (https://opencollective.com/mastodonworld) Patreon (https://www.patreon.com/mastodonworld)

It's been a while since the last blog post. Which is actually a good sign, things have been going well. But let's look back at the last quarter of 2023.

Fedihosting Foundation

First of all: we've started a non-profit Foundation, which will own our Fediverse instances. For now, it's started together with Paul (from the Mastodon.world team) and @stux@mstdn.social who also runs several large Fediverse instances. In the future, other (Dutch) fediverse admins can join.

The primary goals for this non-profit are:

• Improve continuity of these instances
• Share knowledge and resources
• Move liability from the admins to the foundation

We're still in the process of setting things up, but the Foundation is now registered! A website will be created (at https://fedihosting.foundation).

Mastodon.world

Except for some minor issues, like a disk getting full, the Mastodon server has been running fine.

Spam

We did however have a spam issue. Despite the fact that we have a captcha enabled, which does prevent automated spam signups, we kept on seeing a lot of manual spam signups. Mostly Vietnamese casino spam. But these didn't really post a lot so we just removed them in batches when we had some time. But, last month, we started seeing troubling spam accounts, which posted links to child pornography. That's obviously something we'll never tolerate, and we tried several things to prevent these spam accounts from signing up. Unfortunately they kept coming back, so we decided to temporarily limit the signups to 'approval required', meaning anyone signing up should specify a reason for signup and we need to approve (or reject) these. Maybe we'll open registrations up again soon, but on the other hand we catch (almost) all other spam accounts too, which is nice.

Lemmy.world

Our Lemmy instance is running fine. There's a lot of work been done behind the scenes, but the number of incidents has gone down a lot.

We have migrated the image storage from local storage to block storage at Wasabi.

Calckey.world (Firefish) and other instances

The other instances have been running fine, Calckey.world grew to almost 600 users.

Only https://fotofed.nl, our Pixelfed instance, has had some issues. We'll be looking into installing a '.world' Pixelfed instance soon.

Finances

This blog post covers 3 months, so we'll not go into too many details. Let's have some numbers first, and then explain what's happened:

Month	Expenses	Income (donations)
Oct 2023	€ 1276,77	€ 1986,37
Nov 2023	€ 1554,84	€ 2026,28
Dec 2023	€ 1465,02	€ 2051,65

Remarks:

• In september, next to the expenses mentioned in the previous blog, we also had €907,56 expenses for getting the non-profit setup.
• Normal expenses are the infrastructure costs for Hetzner (servers, storage etc), Wasabi (block storage for media), Mailgun (for mail sending), Cloudflare (for CDN) and DigitalOcean (for tooling)
• Those normal expenses are quite stable, Hetzner increased somewhat because we've launched some testservers and smaller new instances (like bookwyrm.world).
• Wasabi costs almost doubled, because next to mastodon, we now also host media for lemmy and calckey.world there
• In Oct + Nov, we had another €287 expenses for the foundation setup.

As you can see, donations still cover the expenses. We will see expenses go up somewhat when we have setup the Foundation completely, because we need some extra services like insurance etc.
We might even ask for donations in a post soon, which we don't normally do. For the people who have donated: a big thank you!

For those wondering: donations can be made here: – OpenCollective (preferred) – Patreon

A good day to all! It's time for the monthly update about the .world.

Here's what happened last month.

Mastodon.world

It's all nice and steady with our Mastodon server. The user count has increased from 176.000 to 179.000 this month.

We have been looking for new moderators for a while now, and since last month we've found 1 more (welcome to the team, Sharon!).

But still we are looking to extend the moderator team. So, anyone that wants to help out, please mail to info@mastodon.world. As a moderator, you would need a bit of free time, and of course find yourself in our site rules. We would very much appreciate if we could find volunteers from the black community, as well as from the LGBTQ+ community to assist us with the very difficult topics we sometimes see in moderation.

Lemmy.world

The Lemmy.world server is more and more stable. The infra and moderation teams have done a wonderful job making the server run smoothly and fighting off unwanted content. It's still a lot of work keeping it all running and keeping it a safe place, for which I want to thank the entire team! We have almost 140.000 users!

Lemmy.world is setting up its own blog, so we'll put the details there once it's finished.

Calckey.world

The Calckey.world server (now running Firefish) has been steadily growing to over 500 users.

Bookwyrm.world

Last month we launched the Bookwyrm instance, which now has 30 users. It's a platform to keep track of your book collection and reading progress. It's federated so you can follow Bookwyrm accounts from Mastodon and others.

Finances

Income

We received these donations last month:

• Patreon: EUR 1317,33
• OpenCollective: EUR 718,60 Total: EUR 2035,93

Expenses

These were our expenses last month:

• Hetzner: EUR 959,98 (masto + lemmy)
• Wasabi: EUR 104,28 (masto)
• Mailgun: EUR 33,40 (masto + lemmy)
• DigitalOcean: EUR 4,74 (lemmy)
• Cloudflare: EUR 9,45 (lemmy)
• Donations: EUR 99,22 (Monthly donations to the Mastodon and Lemmy devs)

Total costs: EUR 1211,07

This month we also started setting up a non-profit foundation for owning our instances. This is still in progress, costs will be added up and listed in November (hopefully it's finished by then).

So this month we still have more donations than expenses. We do see an expected decrease in donations, but still very healthy finances overall. As soon as donations fall below expenses, we'll create some posts to raise awareness for the donation platforms.

For those wondering: donations can be made here: – OpenCollective (preferred) – Patreon

A good day to all! It's time for the monthly update about the .world.

Here's what happened last month.

Mastodon.world

It's all nice and steady with our Mastodon server. The user count has increased from 173.000 to 176.000 this month.

We have been looking for new moderators, and so far we've only found 1 (welcome to the team, Branka!). In the meantime also 1 of the moderators left due to lack of time.

So, anyone that wants to help out, please DM @mwadmin@mastodon.world or mail to info@mastodon.world. As a moderator, you would need a bit of free time, and of course find yourself in our site rules. We would very much appreciate if we could find volunteers from the black community, as well as from the LGBTQ+ community to assist us with the very difficult topics we sometimes see in moderation.

Lemmy.world

August was the 3rd month since we started lemmy.world. It has been a crazy run from the start, but at the end of the month slowly we see things settle down. Usercount increased from 118.000 to 130.000 this month.

The team has been working hard on keeping the server running despite all the attacks we saw. The worst one was a CSAM attack, which also federated to other servers so all Lemmy admins were impacted by this. We have taken measures to prevent these as much as we can, and are looking for tools to detect and remove such content.

At this moment, we see the server being more and more stable, so we can focus on other stuff, like improvements (and setting up a Minecraft server ...)

Calckey.world

I haven't had time to setup a new server for Firefish yet. But calckey.world (running Firefish 1.0) works nicely, and we'e passed the 500 users!

Bookwyrm.world

Well this was actually just setup last week, but I want to mention it here: we've setup a BookWyrm instance! In Bookwyrm, you can keep track of the books you own, have read and are reading. You can rate the books and write reviews. And of course, it's federated, so you can follow others (and get inspired on which books to read next) and be followed by other Bookwyrm users, or Mastodon/Calckey etc...

Finances

Income

We received these donations last month:

• Patreon: EUR 1472,15
• OpenCollective: EUR 1177,10 Total: EUR 2649,25

Expenses

These were our expenses last month:

• Hetzner: EUR 800,50 (masto + lemmy)
• Wasabi: EUR 127,70 (masto)
• Mailgun: EUR 32,44 (masto + lemmy)
• DigitalOcean: EUR 6,78 (lemmy)
• Cloudflare: EUR 31,24 (lemmy)

Other: 107,66 (spent on Lemmy.world stickers for our volunteers)

Total costs: EUR 1205,54

So this month we have more donations than expenses. We do see an expected decrease in donations, but still very healthy finances overall.

It's already September and I haven't yet blogged about July... That's partly because of holidays and some personal stuff. But mostly also because of the amount of work Lemmy has brought...

Lemmy.world

In July, user count has grown from 58.000 to 118.000, so that doubled! Mind, this server was only started June 1st... This sort of growth has brought a lot of challenges, but the technical challenges are the fun ones. What has been less fun is the continuous attacks we faced, probably because we're the largest Lemmy server and we've pissed off some folks by moderation actions (defederating from some instances, banning users etc). Luckily we've built a very good and capable team, for both the sysadmin side as well as the moderation side.

There also has been good cooperation with admins of other servers, especially those of lemm.ee and sh.itjust.works. And a lot of app and frontend developers are involved.

We've put the site behind Cloudflare. This is an emergency measure we needed to take, and although CF isn't the popular option, we decided to use it because we knew other Lemmy servers used it successfully. We had no time to look for and test other options. We will do that when stuff settles down.

We also started using Digital Ocean to develop terraform scripts for re-deploying the server for scaling purposes.

Mastodon.world

Actually, for Mastodon.world things have been quiet in July. Which is good.

Calckey.world

The Calckey software has been re-branded to Firefish. We've upgraded calckey.world to the latest, so now it's running Firefish. Unfortunately, the firefish.world domain is already taken, so we're thinking about a new name where we'll create a new Firefish instance. We'll keep calckey.world running for those who don't want to migrate.

Finances

Like June, July also saw many one-time donations because of the Lemmy user influx, and a steady growth in recurring donations. The one-time donations are good for building a reserve, the recurring ones are for the continuity of the platforms. (In August you'll see the one-time donations going down significantly, but the recurring still more than cover our costs).

Because of the attacks on our Lemmy, we have enabled Cloudflare as an emergency measure. We've paid for 1 year, and hope to have found an alternative before that's over.

Expenses

• Hetzner: EUR 741,35 (masto & lemmy)
• DigitalOcean: EUR 0,59 (lemmy)
• Cloudflare: EUR 224,82 (lemmy)
• Wasabi: EUR 101,75 (masto)
• Mailgun: EUR 87,24 (masto & lemmy)
• Donations: EUR 99,22 (mastodon & lemmy Patreon)

Total costs: EUR 1254,97

The Cloudflare is a one-time (yearly) cost.\ Mailgun will be much cheaper from now on, because the largest user influx of Lemmy seems over.

Income

These were the donations received:

• OpenCollective: EUR 2218,-
• Patreon: EUR 1509,33

Total income: EUR 3727,33

What's next

August has also finished already, we'll make the blog post as soon as all the invoices are in. Costs were less, donations were less. You'll see.

What's also planned, is the creation of a non-profit Foundation which will own all Fediverse platforms run by us (and maybe by others). Main purpose of it will be to not have 1 person be liable in case any bad stuff happening on those instances leads to lawsuits. Also good for continuity to have all things fedi separated.

This has been a hectic and busy month. There have been all kinds of challenges, but also a lot of good stuff!

Lemmy

As you might have read in my previous blog, we have setup a Lemmy server 6 weeks ago. In those 6 weeks, it has grown to be the largest Lemmy server, with over 100k users.

This of course also brought many challenges. From a spam wave, to being hacked and also a lot of moderation challenges.

But all in all I'm very pleased with how it's running now, and the team we have assembled. We have around 25 people in our Discord, who help with sysadmin / troubleshooting / moderation etc. Even admins from other instances, and app developers.

Mastodon

For Mastodon.world, there hasn't been too much news. There has been a security fix which we installed, and we're waiting for 4.2.0 to be released so we can get back to stable releases. (We're running a nightly build at the moment, because we needed captcha in order to fight off spam waves)

Calckey

Due to all the attention that Lemmy needed, Calckey.world has been a bit 'forgotten' last month. Performance hasn't been good, we've upgraded the server specs twice.

July 19th there will be a re-branding of Calckey to a new name, together with a release of the new version (which will be v1.0 instead of v14.0). We will upgrade calckey.world to the new version, and decide what to do with it once the name of Calckey has changed. The new name isn't available as a .world domain, so either we think of a new cool domain name, or we keep using calckey.world.

Meta / Threads

The whole threads commotion has been my biggest headache during the past months. I've seen so much negativity, hate, threats etcetera about the topic, that I've not communicated about it a lot.

First of all, I'm a bit neutral here. I want what's safe for our users. In order to have that, we first need to know what it is exactly that's coming. I see a lot of people panicking without knowing or understanding what's to be expected.

Secondly, I prefer to have the moderation teams deciding on this. They have to deal with it more than me, I just keep the server running. After lengthy discussions, the team made this statement on Lemmy.

What I would like to add, is that it's OK to post your opinion, it's OK to block threads for your account if you like (although on Lemmy that doesn't work yet), and of course it's also OK to migrate to another server if you disagree with our decisions. But please, keep it polite. Understand that Fediverse servers are indeed run by real people, not the big corporations. Volunteers, doing their best to provide services for you in their spare time. I've seen admins/mods get threatened over this...

Finances

And now for the fun stuff. The finances.

Running Lemmy brings extra costs, but more so it brings extra donations. And, like with Mastodon in November, this new user influx brought a spike in donations, which will probably also fade over time. But this spike gives us a nice buffer, so we can guarantee the continuity of the servers for at least the next year.

Expenses

The expenses for June were:

• Mailgun (e-mail): EUR 107,93
• Hetzner (servers, backup storage etc): EUR 760,81
• Wasabi (media storage): EUR 84,37
• Other*: 264,65

(*Other was Netdata, which we use for monitoring all servers (EUR 165,43) and Patreon donations to Mastodon and Lemmy (EUR 99,22))

Total costs: EUR 1217,76

Income

As said, donations spiked this month, and seem to get even higher in July.

• Patreon: EUR 958,74
• OpenCollective: EUR 1588,89

Total income: EUR 2547,63

Extending the .world

As some may know, I run several other Fediverse instances next to mastodon.world.

When I decided it was time to also run a Calckey server, I chose the domain calckey.world, and started thinking about expanding the .world.

Lemmy

So when I wanted to start a Lemmy server, first thing I did was check if lemmy.world was available. And it was!

So I started installing Lemmy on it, and while I was doing this, I read a post about Reddit charging app developers millions for their API use. Speaking about timing... By the time the server was finished and operational (June 1st), it was announced that Reddit would see some subreddits go dark on June 12th.

So, with this announcement, people started fleeing Reddit, and looking for Lemmy servers to join. I started using a VPS (virtual cloud server) for Lemmy, with 2 CPU and 4GB RAM. Soon after opening, that appeared not to be enough, so I doubled it. And after a few thousands of users, I had to double it again, to 8 CPU and 16 GB RAM.

This really remembered me of Mastodon in November. The larger servers were struggling with the load, causing people to choose my server to create an account. Also, there aren't real 'generic' large servers. The largest server lemmy.ml states it's for 'privacy and FOSS enthusiasts'. So the .world grew fast..

I had put the links to the mastodon.world Opencollective and mastodon.world Patreon on the Lemmy site as well, because I don't want to have multiple of those. Also I started paying for the Lemmy cloud servers from the mastodon.world funds. Soon, new subscribers to these donation platforms started coming in. And even so much, that I decided to do the same for Lemmy as I did for mastodon.world: buy a ridiculously large server.

The current VPS couldn't be resized that much anymore, and load was going up with all the new users. So I bought the same server at Hetzner: a 32-core/64 thread 128GB RAM dedicated server. (For Mastodon, I doubled the RAM. For Lemmy I don't think it's needed yet.) I migrated the Lemmy software and database there, and moved over. This took 4 minutes of downtime.

After that, the server is performing very smoothly.We're over 33k users now, having the second largest Lemmy server there is. (See here for stats).

Biggest challenge now is the moderation; I see that there's more unwanted content than on Mastodon, and the moderation tools are very basic, or even lacking. But we're working on getting this organised!

It's been a while, but here's the update about what's been happening to Mastodon.world

What happened

The most important thing that happened was the wave of spam accounts. This wave started at the largest Mastodon server, mastodon.social. But as soon as they installed a Captcha to prevent spam signups, the spammers chose other servers, among which ours.

We got hit by a wave of many spam accounts, sending DMs to users with crypto-spam. We then installed the latest build of Mastodon, which had the emergency-captcha-fix in it, and enabled captcha. This will prevent most of the automated spam signups. It has been quiet since.

Financial update

An update on the finances:

April

Expenses

• Mail: EUR 31,80
• Hetzner (server etc): EUR 382,70
• Storage: EUR 66,13

Total: EUR 480,63

Income

• Patreon $557,39 = EUR 517,87
• OpenCollective: EUR 444,- Total: EUR 961,87

May

Expenses

• Mail: EUR 32,52
• Hetzner (server etc): EUR 424,34
• Storage: EUR 75,69

Total: EUR 532,55

Income

• Patreon $535,19 = EUR 492,37
• OpenCollective: EUR 351,- Total: EUR 843,37

In May, you see that costs have gone up a bit. That's because we've also started a calckey.world server and we needed some extra space for the backup server for mastodon.world.

You'll also notice that donations are going down every month. That's to be expected, number of active users has also gone down, until end of may, when it stabilized. We expext donations to stabilize as well, and if they won't we'll post a message asking for people to donate if they can/want. (We don't do that very often). But also we now have around EUR 8000 reserve, so even if donations might fall below expenses, we're good for quite some time.

But in June, we've started Lemmy.world, which drove costs up, but also generated a lot of new donations. I'll write about that in a next blog post.

Finances

This is the financial overview for March 2023:

Income

We've had these donations in March:

• Opencollective: EUR 473,73
• Patreon: $585,15

Expenses

• Hetzner : EUR 382,16
• Wasabi : EUR 62,28
• Mailgun : EUR 32,35

Also we're Silver sponsor to the Mastodon project, which costs around 50 EUR per month

Moderation

Last few weeks we've seen a lot of spam accounts. The famous 'thegx' account which was created at dozens of instances, but also a flood of Vietnamese casino accounts. If you see one of these, please do report them.

I'm sorry, this will be a brief blog post. If you have questions, contact @mwadmin@mastodon.world

What happened

In February, no extraordinary things happened.

Finances

Donations

In February, we received these donations:

• Patreon: $ 580,97
• OpenCollective: EUR 663,73

Expenses

• Hetzner (servers): EUR 349,51
• Wasabi (storage) : EUR 64,78
• Mailgun (E-mail): EUR 33,02 Total : EUR 447,31

As said in last month's blog post, we also donated EUR 130 to toot.lgbt which was in financial troubles. We promised to do that again this month. Also, starting this month, we will be donating to the Mastodon development team (in their 'silver' tier), just over EUR 50 per month, to support further development of the Mastodon software.