I recently discovered an interesting (and somewhat disappointing, as we’ll find later) fact. It may surprise you to hear that the two most upvoted comments on any Lemmy instance (that I could find at least) are both on Feddit.dk and are quite significantly higher than the next top comments.
The comments in question are:
- This one from @bstix@feddit.dk with a whopping 3661 upvotes.
- This one from @TDCN@feddit.dk with 1481 upvotes.
These upvote counts seems strange when you view them in relation to the post - both of the comments appear in posts that do not even have 300 upvotes.
Furthermore, if you go on any instance other than Feddit.dk and sort for the highest upvoted comments of all time, you will not find these comments (you’ll likely instead find this one from @Plume@lemmy.blahaj.zone).
Indeed, if you view the comments from another instance (here and here), you will see a much more “normal” upvote count: A modest 132 upvotes and a mere 17 upvotes, respectively.
What’s going on?
Well, the answer is Mastodon. Both of these comments somehow did very well in the Mastodon microblogging sphere. I checked my database and indeed, the first one has 3467 upvotes from Mastodon instances and the second one has 1442 upvotes from Mastodon instances.
Notice how both comments, despite being comments on another post, sound quite okay as posts in their own right. A Mastodon user stumbling upon one of these comments could easily assume that it is just another fully independent “toot” (Mastodon’s equivalent of tweet).
Someone from Mastodon must have “boosted” (retweeted) the comments and from there the ball started rolling - more and more people boosted, sharing the comments with their followers and more and more people favorited it. The favorites are Mastodon’s upvote equivalent and this is understood by Lemmy, so the upvote count on Lemmy also goes up.
Okay, so these comments got hugely popular on Mastodon (actually I don’t know if 3.4k upvotes is unusual on Mastodon with their scale but whatever), but why is there this discrepancy between the Lemmy instances then? Why is it only on Feddit.dk that the extra upvotes appear and they don’t appear on other instances?
The reason is the way that Mastodon federates Like objects (upvotes). Like objects are unfortunately only federated to the instance of the user receiving the Like, and that’s where the discrepancy comes from. All the Mastodon instances that upvoted the comments only sent those upvotes directly to Feddit.dk, so no other instances are aware of those upvotes.
This feels disappointing, as it highlights how Lemmy and Mastodon still don’t really function that well together. The idea of a Lemmy post getting big on Mastodon and therefore bigger on Lemmy and thus spreading all over the Fediverse, is unfortunately mostly a fantasy right now. It simply can’t really happen due to the technical way Mastodon and Lemmy function. I’m not sure if there is a way to address this on either side (or if the developers would be willing to do so even if there was).
I personally find Mastodon’s Like sharing mechanism weird - only sharing with the receiving instance means that big instances like mastodon.social have an advantage in “gathering Likes”. When sorting toots based on favorites, bigger instances are able to provide a much better feed for users than smaller instances ever could, simply because they see more of the Likes being given. This feels like something that encourages centralization, which is quite unfortunate I think.
TL;DR: The comments got hugely popular on Mastodon. Mastodon only federates upvotes to the receiving instance so only Feddit.dk has seen the Mastodon upvotes, and other instances are completely unaware.
Unless lemmy devs have changed something since last year, this shouldn’t be the case, there’s a bug in there.
All interactions are recived by the instance hosting the community, and that instance is responsible for broadcasting that interaction to each instance where a user subscribed to it is hosted.
So, mastodon is only responsible for sending the upvote to feddit.dk and then feddit.dk to all other instances.All interactions are recived by the instance hosting the community
Exactly - but Mastodon doesn’t do it like that. Mastodon sends the upvote directly to the instance with the user receiving the Like. So the community never sees the Like at all. So this is Mastodon not supporting groups, it is not a bug in Lemmy.
deleted by creator
But that’s how ActivityPub was designed? Are groups going to be implemented in Mastodon by replicating the crazy “pretend there’s an account boosting everything that’s happening on the server” behaviour?
But that’s how ActivityPub was designed?
This design is incredibly badly suited for something like a forum or content aggregator, such as Lemmy. Only sending Likes to the direct receiver means that other instances are unable to sort content based on votes accurately. Mastodon unfortunately doesn’t care much about this since they prefer chronological timelines, and then Like counts and such don’t matter as much.
It’s really sad that ActivityPub, a supposedly very flexible protocol, seemingly is made mostly for microblogging and doesn’t support other use cases very well at all.
Are groups going to be implemented in Mastodon by replicating the crazy “pretend there’s an account boosting everything that’s happening on the server” behaviour?
Call it crazy but what other options are there? And honestly it is entirely within the spec so it’s not that crazy.
The crazy thing, if you ask me, is that ActivityPub does such a poor job of modelling something like a subreddit. Modelling a subreddit as a Group of users who are subscribed to that subreddit seems unnatural. Most people would say a subreddit is more like a category for posts or a collection of posts.
ActivityPub was designed for social networks, not for forums. I’m not surprised it doesn’t work well for Lemmy.
The alternative, in my opinion, would be shared inboxes combined with adding the “public” audience to outbound messages.
I find it disappointing that everyone says how the Fediverse will allow all kinds of social media, personal blogs and other things to be interconnected, but in the end it kinda sorta works for Twitter clones and barely works for anything else.
It can work for other use cases Lemmy is proof of that. However, it’s not the main objective to be a federated Reddit.
Lemmy, Mastodon, Pixelfed, WordPress, Friendica, and Mobilzon all have different design considerations that are all equally valid. Some of the edge cases necessary to make Lemmy work well are unnecessary or even unwelcome for other designs (see: following Lemmy communities turning your timeline into an endless stream of boosted replies).
ActivityPub isn’t magic. It’s very deliberate in its push-based one-on-one entity relations (for performance reasons mostly). For social media, that works fine. The problem comes in when you get massive communities all posting in single threads, or people with thousands of followers that don’t know about each other rather than a friend group interacting with each other. Mastodon absolutely sucks for people with a large following; this isn’t something that’s unfixable, but fixes do need work and deliberation to work well.
I’d say ActivityPub works great for most social media. The biggest exception may be chat (DMs are footguns that work almost accidentally) and ranked-score forums like Reddit, which require a lot more post-processing than other social media protocols.
It should also be said that both Lemmy and Mastodon use the ActivityPub spec mostly correctly (neither is fully spec compliant as far as I know), despite using different mechanisms to achieve the same goal that aren’t always compatible. We can barely get chat apps to interoperate, it’s impressive how well ActivityPub has proven to be.
I don’t think the situation would be very different had Lemmy been based on ATProto or Nostr. Reddit is a bit of an odd duck in the social media landscape, and very few other websites share its core patterns and algorithms.
ActivityPub was designed for social networks, not for forums.
If you ask me, forums are social media. I think it’s very prescriptive to say that forums should somehow not be supported by a social protocol. I’m not really sure how ActivityPub was designed, but in some ways it feels like they tried to make the protocol too flexible and somehow they managed to make it not flexible enough in other areas, or at least somehow didn’t think to support other use cases very well. It’s unfortunate.
The alternative, in my opinion, would be shared inboxes combined with adding the “public” audience to outbound messages.
So with this model, when I post to, say !technology@lemmy.world, my own instance would send the post to all the instances that I know of? Or would it send to only those instances following that community (how does my instance know that?)? I think there’s also the problem of how moderation is handled - I mean, how does the community in question enforce bans for example? With the current model, the community is kind of “in control” of everything happening, because it is the one sending out the activities. But if everyone sent them themselves, that seems less clear. What if the community defederates an instance but my instance doesn’t defederate that one - will my instance send the post to the instance that is defederated by the community? It’s all very complicated. I’m not sure what a good solution is.
ActivityPub has been designed as a push-only protocol as push/pull would involve too many web requests. It was designed for designs like Facebook and Twitter. This flexibility made Lemmy possible.
As for sending the post, it’s not that hard to keep track of all servers that follow a group or person or hashtag. Following people is not passive, you send a follow request which usually gets approved automatically.
Bans are already questionable. Servers choose not to forward every message (personal, group, public) to every account. There is no “themselves” in a shared server. Either the entire server gets banned, or the server is trusted to enforce bans. Lemmy federates those bans (though Lemmy and Kbin don’t always communicate right).
What if the community defederates an instance but my instance doesn’t defederate that one - will my instance send the post to the instance that is defederated by the community?
I would say “the server publishes a banlist containing *@bad-server.tld”. The defederation list on Lemmy is public anyway, but you could also argue for making such behaviour a setting, and there are probably other/better alternatives.
The way you describe this, it sounds like it would need to work on trust a lot more than it already does. What if there’s a malicious instance actively circumventing bans, ignoring any pulbished banlist?
As for sending the post, it’s not that hard to keep track of all servers that follow a group or person or hashtag
I was talking about the scenario where you are instance A and you don’t know the followers of a user of instance B. That is not easy to keep track of, since you obviously don’t get any of the follow requests for a user on another instance.
Ohhh! Now I understand!
Yeah, then that’s an issue on mastodon.
I mentioned some time ago, the fact that mastodon and Lemmy use the same protocol is annoying, because the experiences are different, so it causes a lot of issues :/the fact that mastodon and Lemmy use the same protocol is annoying
Well I think this is still better than the alternative, which is no interaction between them at all 😅. The protocol is what binds different Lemmy instances together too.
Interesting. This explains why posts never seem to have more than a few likes on the small Mastodon instance I use. I have to say I’m not a fan.
Yep exactly, it also leads to Mastodon instances only seeing local likes for remote posts. You’ll never see remote likes on remote posts as they wouldn’t be sent to your instance. I honestly don’t understand how this hasn’t been a bigger problem for Mastodon, but I guess Mastodon is more about boosts and chronological timelines and less about sorting stuff based on likes.
It causes people to wander off as they think theres not enough interactions compared to other social media. The first comment you linked shows only 141 points to me.
I don’t understand why feddit.dk doesn’t display upvotes received from Mastodon users. Why is this dependent on my instance?
Then why, if I view the post on feddit.dk, does it not show me those likes/votes? What is dependent on my instance?
I don’t understand what you mean, how does it now show you the likes? If you see the two comments here and here as I linked above, you can see the high upvote count. Almost all the upvotes are from Mastodon instances.
The upvotes do not appear if you view the comments from another instance, like here and here, because those instances did not receive the Like.
Sorry, I mean when I view the comment via my instance. I don’t understand why my instance needs to receive the votes/likes directly, instead of my instance fetching them from feddit.dk when I request the comment.
Your instance doesn’t pull the upvotes from other instances. That would not be scalable. How would it know when to pull again, to see new upvotes? When would it stop pulling periodically? Never? And you’d have to do this for every single post and comment everywhere.
No, instead ActivityPub uses a push mechanism here. So any new activity is pushed out to the ones that are deemed relevant to know about the activity. Any other instances are unaware.
Pulling the data when a user requests a post/comment (with a cooldown/cache for popular posts) isn’t any more or less scalable than feddit.de pushing the same data whether it’s been requested or not. If anything, I’d think pushing data when it’s not necessarily needed would be less scalable.
But if it has to be a push model, why doesn’t feddit.dk push the votes it knows about along with the rest of the data?
Pulling the data when a user requests a post/comment (with a cooldown/cache for popular posts) isn’t any more or less scalable
That would definitely be less scalable. That would entail pulling every single time a user views a post or comment. That’s simply not feasible. There are far, far more views of content than there are posts, comments and votes.
Also what about stuff that isn’t seen? What if nobody is logged in or nobody looks at the New sort? You need the votes before you even show the user anything, otherwise you can’t sort the votes.
But if it has to be a push model, why doesn’t feddit.dk push the votes it knows about along with the rest of the data?
This has been explained elsewhere in the thread, see https://feddit.dk/post/7628338/10255563
(mentioning people mentioned in post because post mentions apparently do not work)
Its not really a “not playing well” with each other, it is just the mastodon works. That is the reason why most toots in my mastodon timeline have 0 favourites (upvotes) and only a few boosts… I don’t know why they do it, because at the minimum the followers of that user should be notified about that like…
at the minimum the followers of that user should be notified about that like…
I agree - the problem is that the instance that sends the Like (on instance A) doesn’t know the followers of the user receiving the Like (on instance B), because followers are not (necessarily) public. So it doesn’t know which instances to send the Like to. And instance B can’t forward the Like to the followers itself, because the signatures in ActivityPub are not made for that, as I explained elsewhere in the thread.
AP has a tool for that called inbox forwarding and mastodon uses it for sharing the comments under posts. It works like this: you send a reply to a user with their follower collection as the recipient. You of course cannot know who is following that user, however they than just forward this reply to the follower collection, because the server knows that it has authority over that collection. https://www.w3.org/TR/activitypub/#inbox-forwarding
however they than just forward this reply to the follower collection
How do the receivers of this indirect activity verify that the activity was indeed produced from the original instance?
16 & 4 upvotes here on Fedia (mbin), with 0 boosts on each.
Is this a bug? Sounds like a feature of decentralization.
Mastodon doesn’t support groups so it’s maybe not a “bug” per se, but it is at least a missing feature.
Consider also that if Lemmy shared upvotes the same way, you would only see the upvotes on posts from your own instance, i.e. upvotes would only appear on the local feed. The all feed would be pointless and in general it would be pointless to try to sort posts across the whole fediverse, as you only receive upvotes for your local posts.
Lemmy simply would not function if it shared votes like that. So in that sense, it’s a bug kind of. And as mentioned above, I think it’s a bad way of doing it, as it encourages centralization.
Perhaps you should report this as a bug to the devs. Otherwise, this was an interesting read.
The Mastodon devs are aware of how their Like federation works and considers it a feature, not a bug.
I was referring to the lemmy devs, but ok, I guess
I don’t think there is anything the Lemmy devs can do to fix this. The ball is in Mastodon’s court, so to speak.
Can’t even see these posts, I clicked and got:
400 {“error”:“couldnt_find_post”}
Weird, what client are you using?
Browser. I’m on feddit.nl and use Photon as UI on it (ph.feddit.nl).
Interesting. Seems like Photon attempts to do some kind of clever (and entirely non-standard) hack and tries to load the comment from your own instance? But obviously you need to see it from Feddit.dk to see the votes. This is a bug with Photon I would say.
I found the bug, and i fixed it. (btw, ph.feddit.nl is out of date)
PS: I just tried it without Photon and got another error.
https://feddit.nl/post/feddit.nl/20706530Anyway, all good and not something I can influence so I let it go :-)
I don’t understand why your instance seems to be rewriting the link somehow, but it shouldn’t. This happens with lemmy-ui?
Ok, sounds plausible. Thanks for the explanation!
It simply can’t really happen due to the technical way Mastodon and Lemmy function. I’m not sure if there is a way to address this on either side (or if the developers would be willing to do so even if there was).
Mastodon needs to implement group support, you can follow the issue here (don’t get your hopes up though).
Group support would fix it for Lemmy, but it doesn’t fully fix the problem as I see it with this way of sharing the Like objects. For toots outside of any group (in Lemmy terms: comments/posts outside a community), presumably it would continue to function like this, i.e. only the receiving instance is aware of the Like. This still encourages centralization if you ask me.
Yeah, I’m not going to defend Mastodon’s frankly bizarre Like system. It’s not even a privacy thing as favourites are fully public.
Can we all at least agree that “toots” is a terrible sounding term? It sounds like what happens when one of the powerpuff girls farts. She toots.
A toot is literally the sound of a birb. It’s got more comfy than “tweet” does.
I’m not a native speaker so I don’t hear the fart association so much. But isn’t tooting also just what a trumpet or elephant does? In that way it makes sense. But I do think the terminology is a bit silly. Why not just “post” instead of toot? Why not just “repost” or “share” instead of “boost”? It feels a bit too much like corporate social media where every feature needs a “wacky and fun” name.
Does the receiver instance federate that like object to other instances? If not, it is shit for sure.
No, but how could it? Let’s say Feddit.dk receives a Like from mastodon.social. Then Feddit.dk would have to tell the other instances that mastodon.social sent that Like. But how can Feddit.dk prove that the Like actually did come from mastodon.social, i.e. it is not just a fabricated Like that Feddit.dk made up and hid by pretending it came from mastodon.social. That’s not easy.
You’re right, that’s worse.
The like is an activity. Any activity has an actor. Every actor has a public key. If the activity is sent with a cryptographic signature (like LD signatures, which Mastodon does implement) then any one can verify that the activity is legit.
Mastodon explicitly discourages support of LD signatures. It seems it has kind of landed in what I would call “specification hell”.
Discouraged, but still supported. There is also another FEP (forgot the code now) being worked on and implemented by Mitra.
The point is that it is possible for an instance to federate an activity which is not originated by them.
I mean it could be proven by having every account create a cryptographic key and adding a public key to the vote. Memory might be an issue though.
This is in fact how Feddit.dk knows that the Like came from mastodon.social at first. The problem is that the signature is a HTTP Signature which is only associated with the HTTP request that mastodon.social makes to Feddit.dk. It is not on the Like object itself. Thus that signature can’t be transferred to the Like object if Feddit.dk wanted to share it further.
So we need to convince the mastodon devs to change it so the signature is on the like object itself.
Unfortunately it is not that easy. It’s not Mastodon that places the signature like that, it is the ActivityPub protocol. Lemmy, Mastodon and all other ActivityPub instances do it this way. You’d need to extend or change the protocol to somehow fix this. That is not easy and not something that will be done overnight.
Of fucking course they did it that way. If i where to hack sonthing together u could also just quote the entire request signature and all and forward it.
So there’s no way for feddit.dk to translate into a Lemmy style upvote?
There’s not such a thing as a “Lemmy style” upvote. It’s all Like objects under the hood shared via the ActivityPub protocol. But ActivityPub has no mechanism for sharing an activity further than the original receiver (i.e. forwarding from A to B to C and so on). It’s really only made for direct sharing from A to B.
I’m not sure if that’s true. From the spec:
Additionally, if an object is addressed to the Public special collection, a server MAY deliver that object to all known sharedInbox endpoints on the network.
This requires implementing sharedInbox support, but I believe this should permit federating any content of choice to any server.
That would still be directly from one server to another server. I.e. from A to B and from A to C. But forwarding is a different matter, i.e. A sends something to B which sends it further to C. There’s complications with signatures and verification in that case and it’s less clear how to handle that.
Ohh so the object doesn’t change at all through the process. I see the difficulties
But surely the receiving server could validate that signature by verifying the existence of the received activity (by asking the origin server for the object referenced and validating the signature).
If like objects are distributed in URL form, this is already how it works. The extra load wouldn’t be fun, for sure, but the lack of an embedded signature makes it very easy to falsify anything on the Fediverse.
Yes, fetching the URL directly would be a way to verify it. I don’t know if Lemmy currently does that. In any case, it seems very round-about.
Lemmy doesn’t do it currently. It blindly trusts communities to not lie to people. I just found out about this myself.
In theory the JSON body could include all the necessary information to validate a signature and the signature itself. Then, a simple HEAD request could validate the contents without having to re-download everything, and users’ public keys could be cached to minimise HTTP requests necessary.
If you have a signature you can also sign the contents, so you wouldn’t need to download the content. But AFAIK ActivityPub has no mechanism for including signatures in objects as it is right now. There’s only HTTP signatures, which aren’t on the object itself.
I seriously doubt Lemmy currently does any validation whatsoever. There were communities using this blatant security issue for non-malicious purposes (see https://endlesstalk.org/c/tails@lemmon.website, which re-wrote posts from people (which is only possible if the posts weren’t validated, or at least re-fetched from their origins)).
There is a way to re-share and validate remote activities, either through LD signatures (ew, JSON-LD processing :vomit:) (which only Mastodon and Misskey implement) or the newfangled FEP-8b32 Object Integrity Proofs (which nobody relevant on the microblogging space implements).
There were communities using this blatant security issue for non-malicious purposes (see https://endlesstalk.org/c/tails@lemmon.website, which re-wrote posts from people (which is only possible if the posts weren’t validated, or at least re-fetched from their origins)).
The reason this is possible is because of the way Lemmy federates activities.
When you on instance A post, comment or upvote something in a community on instance B, your instance sends the activity to instance B, regardless of the instance of who you’re replying to or upvoting. It is sent to the community, and the community then shares it out to all other instances. AFAIK, lemmy does nothing to verify that received content from a community actually comes from the original instance. See here for one of the main Lemmy devs commenting on this..
Is this secure or reasonable? I’m honestly not sure but it doesn’t feel great. Signatures on objects could fix this I think.
Instead of sending the entire object embedded in the activity the secure way would be to only the URI instead. This is permitted by JSON-LD.
In the receiving side, if the object is untrusted (i.e. if it isn’t signed or if it’s from a separate authority from the parent object containing it) it should be thrown away and the id should be fetched from the remote instance directly (same as it would happen if it was a URI instead of an inline object). This is completely an oversight on Lemmy’s implementation and not a protocol problem.
That would be a way to do it, but it seems needlessly wasteful as it requires an additional HTTP request. But yea, that could be a way.
Yeah, that is a shortcoming of the protocol. But it’s necessary in order to be secure until things improve (and given this is AP, that’s gonna be a while. People seem to love bikeshedding in circles instead of doing actual work)
and given this is AP, that’s gonna be a while. People seem to love bikeshedding in circles instead of doing actual work
Out of curiosity, what do you mean by this? Any examples? I’ve not followed the development of AP very much at all honestly so I don’t know the history.
great post, thanks
A Mastodon user stumbling upon one of these comments could easily assume that it is just another fully independent “toot” (Mastodon’s equivalent of tweet).
Wait, back up… Mastodon calls these “toots”? So, everybody is posting farts?
Do people seriously not know what a freaking mammoth is?!
Tooting the way you’d toot a horn/trumpet, or in this case an elephant trunk.
That was a good investigation and explanation about a weird number of up votes. Thanks for explaining it.
Interesting observation and analysis, and illustrates the potential of more lemmy-mastodon interaction.
Indeed mdon like-federation seems weird but I presume it was setup this way for efficiency, to reduce the number of small communications? Although Lemmy has a backend in rust - more efficient than mdon’s ruby - still I wonder whether the lemmy system of federating all upvotes would scale well if the number of users grows to that of mastodon and beyond ? Could there be some intermediate compromise solution (e.g. federate batches of 100 likes)?still I wonder whether the lemmy system of federating all upvotes would scale well if the number of users grows to that of mastodon and beyond ?
It’s a good question and really we just don’t know yet I think. It’s very hard to predict performance of complex systems. The only way to know, is basically by measuring, and the only way to do that is if we actually had that amount of users.
Could there be some intermediate compromise solution (e.g. federate batches of 100 likes)?
Unfortunately ActivityPub has no way to “batch” activities like this.
