this post was submitted on 21 Jun 2023
24 points (100.0% liked)

Selfhosted

45450 readers
317 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
24
Spawned my own instance of lemmy: now I've got a lot of questions about federation (lemmy.g97.top)
submitted 2 years ago by [email protected] to c/[email protected]
32 comments fedilink hide all child comments
 

cross-posted from: https://lemmy.g97.top/post/761

cross-posted from: https://lemmy.g97.top/post/723

Hi! I spawned my own instance of lemmy on my server and I discovered new things about how lemmy and federation works, and I have a lot of doubt. I don't know exactly if those doubts are problems of my implementation of if they are normal, so!

  1. My main account is on lemmy.world and I see that new posts from communities I follow show up before on lemmy.world and then on my instance. Is it normal?
  2. With comments happens the same thing and they are slower to "sync". Why?
  3. If a community has been never discovered from the search form with the full format !community@instance, it will never appear on my instance. This means that is not possible to search for an argument (i.e. steam deck) and finding all the posts and communities about it. Is this normal or a feature that we/you would like to see in future/is adaptable to the concept of the fediverse? Because if I am on a big instance with a lot of users maybe I found that specific community or post, but on smaller instances like mine it will never appear If I don't know the exact name.
  4. I created a community on my instance and subscribed it from lemmy.world but I don't see any post nor are they in sync. Why? https://lemmy.world/c/[email protected] vs https://lemmy.g97.top/c/announcements.
  5. From my instance I am unable to follow lemmy.ml communities (they are pending, usually on lemmy.world the pending status is faster)
  6. I am unable to search for communities on Kbin.social, and when I try I see this log message of type "couldnt_find_object: error decoding response body: missing field properties at line 1 column 206" from my docker instance:

2023-06-20T22:02:16.056226139Z 2023-06-20T22:02:16.055937Z ERROR HTTP request{http.method=GET http.scheme="https" http.host=lemmy.g97.top http.target=/api/v3/ws otel.kind="server" request_id=8211e6a4-2b30-4f8c-98b3-d93843a0e293 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: error decoding response body: missing fieldpropertiesat line 1 column 206 2023-06-20T22:02:16.056276976Z 0: lemmy_apub::fetcher::search::search_query_to_object_id 2023-06-20T22:02:16.056286500Z at crates/apub/src/fetcher/search.rs:17 2023-06-20T22:02:16.056293804Z 1: lemmy_apub::api::resolve_object::perform 2023-06-20T22:02:16.056300316Z with self=ResolveObject { q: "[[email protected]](/c/[email protected])", auth: Some(Sensitive) } 2023-06-20T22:02:16.056307712Z at crates/apub/src/api/resolve_object.rs:21 2023-06-20T22:02:16.056314152Z 2: lemmy_server::root_span_builder::HTTP request 2023-06-20T22:02:16.056320693Z with http.method=GET http.scheme="https" http.host=lemmy.g97.top http.target=/api/v3/ws otel.kind="server" request_id=8211e6a4-2b30-4f8c-98b3-d93843a0e293 http.status_code=101 otel.status_code="OK" 2023-06-20T22:02:16.056351870Z at src/root_span_builder.rs:16

  1. I have a lot of warnings in the lemmy log of type "Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Header is expired" such as:

2023-06-20T21:58:12.484449111Z 2023-06-20T21:58:12.484275Z WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Header is expired 2023-06-20T21:58:12.484510012Z 0: lemmy_server::root_span_builder::HTTP request 2023-06-20T21:58:12.484517559Z with http.method=POST http.scheme="https" http.host=lemmy.g97.top http.target=/inbox otel.kind="server" request_id=caf194c5-cac3-4c37-a29c-577d65deb050 http.status_code=400 otel.status_code="OK" 2023-06-20T21:58:12.484525578Z at src/root_span_builder.rs:16 2023-06-20T21:58:12.484530286Z LemmyError { message: None, inner: Header is expired, context: "SpanTrace" }

I have more questions/doubt but for now this is enough I think! Thank you!

top 32 comments
sorted by: hot top controversial new old
[–] [email protected] 6 points 2 years ago* (last edited 2 years ago) (3 children)
  1. Yes, because posts/comments are created there and then federated to subscribed instances, which happens on a queue. Depending how popular that community is, there's probably a lot of other instances ahead of you in the queue, but lemmy.world is also a huge instance in general so they might be under al ot of load.
  2. Same reason as above, federation is asynchronous and happens on a queue.
  3. In theory you could probably write a bot that just auto-subscribes to communities in other instances, but you're going to have a LOT of traffic going through to your instance and you'll basically need it to be as powerful as those instances. Sending you only the data you've subscribed to receive is how all federated services work, an instance just can't reasonably get everything. However I've put some thought into this and I think a way to browse other communities without subscribing to them would be a game changer.
  4. This could be a lot of things unfortunately. Have you tried making new posts?
  5. Same on my end, I think lemmy.ml is waaay overloaded at this point and I think it might also be behind CloudFlare which complicates the federation process.
  6. Seems like a problem on the kbin end. You might want to create an issue on the kbin's codeberg.
  7. You'll probably want to post that in one of the lemmy communities or on its issue list (make sure to search first).
[–] [email protected] 5 points 2 years ago

Same on my end, I think lemmy.ml is waaay overloaded at this point and I think it might also be behind CloudFlare which complicates the federation process.

The IP addresses (both IPv4 and IPv6) I'm seeing for lemmy.ml trace to OVH, not Cloudflare.

[–] [email protected] 2 points 2 years ago

Thank you for the detailed response! I'll open the issues for 6. And 7.

[–] [email protected] 4 points 2 years ago (1 children)

I can only answer a couple.

  1. This is by design, due to how federation works. Federation is literally just:
  • Instance A requests information from Instance B
  • Instance B responds to the request and sends it back
  • Instance A follows Instance B
  • Instance B now federates all future posts to Instance A

There's nothing more complicated to it, but it does mean that instances cannot know about other instances without being told, as there is no central location that instances connect to in order to find out about all other instances.

  1. Only posts after subscribing are federated to an instance, it doesn't backfill. An option for admins of an instance to request a backfill would not be a bad option though, but as time goes on, backfilling an entire community could take too much data on instances.

  2. Issue with Lemmy.ml, although when you see Subscribe Pending, you tend to still see things in your feed.

[–] [email protected] 1 points 2 years ago (1 children)

So if you're using a new instance all old posts are not visible? Really? Can it not load them on demand? Or am I misunderstanding?

[–] [email protected] 1 points 2 years ago

Yes, this is correct.

It's possible to fetch older posts by requesting them directly but there's no automated way to grab old posts on demand.

[–] [email protected] 2 points 2 years ago* (last edited 2 years ago) (2 children)

I can't even see this post on my selfhosted instance, so that's fun.

With regard to #6, you apparently have to search for the URL of the Kbin magazine (e.g. https://kbin.social/m/RedditMigration) and then you can subscribe as normal from that point on.

And yeah, the discovery and comments thing is super annoying. There will be completely different conversations based on where you're viewing the post, which kind of kills the idea of all these instances federating in the first place.

That being said, it appears to be feeds coming from lemmy.world and lemmy.ml that have the majority of the problems with comment/post sync. Unfortunately that's the selfhosted community.

[–] [email protected] 1 points 2 years ago

The suggestioni for kbin works flawless, thanks! I don't get why Lemmy and kbin instances have a different way to do this.

Anyway, seems like my instance is more in sync (by order of magnitude, I still have to verify it better) with kbin then with Lemmy.world or .ml.

[–] [email protected] 1 points 2 years ago (2 children)

Until 5 minutes ago I was seeing 6 comments. Now Lemmy says there are 11 comments (as I was saying on lemmy.world until minutes ago) but I only count 6 comments by hand 😂

Yep the problem is that we all see different things so we are not having a consistent conversation between us.

I still don't know and I would like to, how the activityPub and federation protocol works and where the fault is (in the activityPub or in the Lemmy's implementation). As I understood everything that happens on Lemmy.ml or lemmy.world is sent to all the connected instances with HTTP. This means that if there are 10k connected instances, a like produces 10k HTTP request and this is multiplied for each actions done on these servers. Is this right?

[–] [email protected] 2 points 2 years ago

It sounds like you know more about ActivityPub than I do, but that seems right. My guess is that .ml and .world are so overloaded that there's just a massive delay coming from them. It is kind of funny that, like you mentioned above, with my Lemmy instance I have almost perfect sync with Kbin servers but problems with Lemmy.

I'm hopeful that this is like Mastodon, where things calm down in a few weeks and there's a smaller but more sustainable userbase.

[–] [email protected] 1 points 2 years ago (1 children)

Can confirm that all of the above applies to my instance as well

[–] [email protected] 1 points 2 years ago (1 children)

Well, this is good. At least we are sure almost everything is good on our side

[–] [email protected] 2 points 2 years ago

Hoping someone found a solution for at least the slow comment updating. Especially older posts (48 hours) dont seem to sync properly

[–] [email protected] 1 points 2 years ago (1 children)

I'm only just now learning more about federation, so I'm no expert.

Is your instance setup to use NTP? federation requires fairly accurate time from my understanding, and shared hosting normally drifts quite a bit, which might be where your "Header is expired" issue comes from. It could also have something to do with your instance not processing incoming federation stuffs fast enough I think?

There is a chat space at https://matrix.to/#/#lemmy-space:matrix.org with people that run their own instances, which might help answer these questions

[–] [email protected] 1 points 2 years ago

thanks for the hint regarding ntp. I configured my VM to use NTP now but the http error remained

[–] [email protected] 1 points 2 years ago

Number 3 was the most important learning I got from setting up my own self-hosted Pleroma instance. Finding new content is super hard in the fediverse if you self-host, in the case of Mastodon/Pleroma I tried federating with some Relays, but still...

[–] [email protected] 1 points 2 years ago

Don't use lemmy.world for now if you want to experience federation(atleast for now). I have a separate account for this instance because of the popularity and use my personal instance for other instances except lemmy.ml