Technology

The local communities are such a great mix of uplifting and informative posts that I am sure to return.

I agree. I replied to Kris elsewhere saying this, but I am super glad to have been a part of this instance because it feels like a nice balance of being large enough to be robust and diverse, but small enough to have a distinctive culture. I don't often interact with the communities that were on the instance, but I always enjoy seeing my peers crop up in the comments of various posts — it's one of my favourite parts of Lemmy being federated (db0 is another example of an instance that has such a distinctive vibe that seeing it as someone's instance is often useful metadata that affects how I parse their comment)

Count me in! Slrpnks all the way! 💚

Itself embedded in an auto extractable zip executable.

yes, that's why it's been down for this long...

it's on archive.org

https://web.archive.org/web/20240223204706/https://wiki.f-hub.org/books/slrpnknet/page/server-hosting

Server hosting

This is a work in progress
Slrpnk.net is hosted by F-hub.org, a volunteer driven and non-commercial effort to host federated community services in a resource efficient and ethical way. F-hub.org grew out of a community of open-source game developers (freegamedev.net) that has existed since the early 2000s.

All the servers are operated and maintained (as a hobby) by one of the founding members and are currently located on the Azores in Portugal. Connectivity is provided through a dedicated high-speed fibreglass connection.
Hardware

The F-hub.org servers are based on second-hand consumer PC and data-centre hardware, but optimized for low energy consumption. Battery backup power is provided and data is stored with triple redundancy (off-site backups are still a work in progress). Electricity is currently provided by the utility grid (about 60% green-energy, mostly from a geothermal power-plant) but it is a work in progress to upgrade to a on-site solar PV system for near 100% renewable power.

Slrpnk.net itself currently runs on a dedicated 6th gen Intel CPU server with 8 threads, 16GB RAM and SATA SSD storage in raid configuration. Image uploads are stored on a large HDD raid array. All data is snapshotted and transferred for backup to a second shared server on daily basis.
External services

Given how difficult it is to have outgoing email accepted by the large email providers, all emails are currently routed through an external SMTP server hosted by OVH in France (same as the domain registration and DNS routing).

There are backup plans to move the servers to a co-location data-center owned by Altice, should this be required for scaling. However, monthly rent of rack-space is quite costly (starts at around 300€/month), thus this will require a substantial regular donation base to be possible.

I think they're located in the Azores, and yes, running their own servers on solar (of course) and other renewable energy. Coolest instance out there.

(speaking as a slrpnk user): Another backup communication strategy (once things are backup) might be to designate somewhere on a non-slrpnk instance as a place where people can check for updates if things go down; when I first discovered the outage, I wasn't sure where to go to check for info/updates.

Unrelatedly, I hope that this unexpected outage isn't causing you or other admins too much stress. Whilst the extended nature of this outage is unfortunate, I respect that you're using this as an opportunity to migrate to a more robust solution. This kind of resilience focussed response is a key part of the solarpunk ethos, in my view.

Some people have said that such a long outage seems likely to kill an instance, but for my part, this community is worth waiting for — I have enjoyed having an account on this instance because it feels like the perfect blend of small enough to have a distinct culture and ethos, but is large enough to be robust and diverse.

Slrpnk.net admin here.

The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.

There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time...

An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.

I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I'll know more when I have physical access again.