this post was submitted on 07 Jun 2025
530 points (100.0% liked)

Technology

71088 readers
3559 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
530
Slrpnk instance is down till mid July; they might relaunch their server on piefed. (lemmynsfw.com)
submitted 2 days ago* (last edited 2 days ago) by [email protected] to c/[email protected]
88 comments fedilink hide all child comments
 

Source. Thread.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 12 points 2 days ago* (last edited 2 days ago) (2 children)

Is it in a data centre or someone's house? If the latter, would they let a stranger in?

Surely they would need a backup and replicate db to so in case of hardware failure they switch over.

Sounds like they could improve their setup.

Too much of a single point of failure.

[–] [email protected] 38 points 1 day ago* (last edited 1 day ago) (3 children)

Slrpnk.net admin here.

The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.

There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time...

An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.

I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I'll know more when I have physical access again.

[–] [email protected] 2 points 1 day ago (1 children)

Is it run out of a private residence? How could it happen if it’s in a real data center…?

[–] [email protected] 3 points 1 day ago (1 children)

It is run from a private residence in the DIY punk spririt (and this also allows us to run of a local solar PV system), but more or less the same would happen if you rent rack-space in a "real" data-center. Only if you rent a managed server or VPS someone else will be responsible to fix such issue and this comes at a significantly higher cost at the scale we operate at (slrpnk is part of a bigger project that also hosts other services).

[–] [email protected] 0 points 1 day ago (1 children)

If you rent rack space, you can at least call the DC to send someone to reboot it or smt.

[–] [email protected] 3 points 1 day ago

Sure, but they will not replace hardware and set up remote access for you, which seems like this would have been required in this case.

[–] [email protected] 6 points 1 day ago

I've done a lot of SysAdmin and DCOps stuff in the past so, thought I'd give you some plausible suggestions (haven't dug deep into Lemmy DB stuff and DNS/Federation of the stack, so not sure all is practical).

Scenario 1 - Preserve and merge when access is restored

Setup

  • Spin up two VMs/VPS (or one that has enough grunt for two Lemmy servers). Call them robak.slrpnk.net and slrpnk.net and point DNS appropriately.
  • Pull federated content from other instances and place it on robak, set as read-only.
  • Sync important comms to (new) slrpnk.net without content.
  • Allow users to sign up, vetting as possible (all mods). Keep a list of those that are vetted (call it vetted.list). Inform all users that any non-vetted users will have their content dropped when access is restored.

Merge!

  • Once access is restored, ensure that (old) slrpnk.net is set to read-only.
  • Schedule a maintenance window (announce more time than you are likely to need).
  • During the maintenance window, put (new) slrpnk.net into R/O, or just block external access.
  • Query the db on (old) slrpnk.net for all users.
  • Subtract the vetted users from vetted.list from the list.
  • Drop all records from the resulting list of non-vetted users from (new) slrpnk.net.
  • Insert the records from vetted and new users (those without conflicts) into the DB on (old) slrpnk.net.
  • Validate that everything is working
  • Cut over DNS and spin down the new VMs/VPS.

Scenario 2 - Server is in DC or Admin able to facilitate access

  • Get a db dump/backup.
  • Spin up temporary slrpnk.net on a VM/VPS.
  • Use backup of temporary server to restore data to original, when possible.
[–] [email protected] 2 points 1 day ago

Appreciate the answer and the detail. Good luck getting it all resolved.

[–] [email protected] 3 points 1 day ago

Probably quite expensive, and when doing something as a hobby it's often hard to get the funds.