I just realized that none of the comments or posts I made in the last week from my instance are getting to lemmy.world.
I went to see if I my instance was defederated. No, still showing as connected.
I then went to see if I got blocked or banned. Nope, my username is not showing up in the modlog anywhere.
Is it because my instance is small? I guess not, because I can interact with people and communities from anywhere else just fine.
At the moment, the only plausible explanation I have is that lemmy.world is overwhelmed and dropping messages from smaller instances. They do however everything in their power to keep more users coming up.
Yeah, I get that they were being attacked. I can only imagine that getting DDOS’d is not fun, and worrying about the Schmoes on the smaller instances is not a top concern.
But even in the middle of these constant outages and attacks, the lemmy.world admins are still keeping registrations open? Why? Wouldn’t it be better if they encouraged the users to move out of the instance to reduce the load? Isn’t the whole point of decentralized technologies to be, you know, decentralized?
I shouldn’t have to come here, create an account and make things even more centralized just so that I can tell people that this attitude is hurting the fediverse.
I wouldn’t be so pissed at this if it weren’t for the fact that some many communities were created here and is making this particular instance a crucial part of the fediverse, but the admins seems to be more worried about getting their user count up than the health of the overall system.
Please, admins, the more you go with this unstable federation and open registrations, the more of an incentive you are creating to centralize this further here. Help the fediverse and help yourselves. Close down registrations and focus on ensuring that everyone can access the communities that are being formed here.
deleted by creator
Hey, this happened to us recently. In your database check the table called 'instance ’ and make sure the value for ‘updated’ is less than three days old for lemmy.world
There are false positives regarding the detection of “dead instances” in the latest version of Lemmy and it’s actually your instance that stops sending out messages to lemmy.world
Ok, you might be on to something. The
updatedvalue is set to2023-06-11. Should I force this value to be recent?Yes, try it and see if it works. Lemmy really does skip federating with an instance if hasn’t been marked as updated for more than three days: https://github.com/LemmyNet/lemmy/blob/da031a4ce2f556deae1b3dd1a542db97b3c01d0e/src/scheduled_tasks.rs#L374
Yes. That should fix it. There is instances that are genuinely down. Later today I’ll try to share a script to detect which ones are down and which aren’t via curl. In our case we had 350+ false positives.
Thank you, unfortunately I don’t think it worked. My posts from the last week or so are still not visible here.
It only works for new posts. Try creating something new in a community that you know lemmy.world knows about.
Older posts may appear progressively, but there’s no guarantees.
Nice! That did it! Thank you so much!
Do you know if you had any cronjob running close to 0:00 (server time, possibly UTC) that could have interfered with the validation of dead instances that lemmy now does?
I’m trying to figure out what could have interfered with these checks in the first place.
@jon@lemmy.tf dunno if this is helpful. I’m seeing everything, I think. But just in case.
This sounds serious. I’m going to check my own instance too just in case.
Edit: Phew, looks like there is nothing wrong in my instance. Here is a comment from Nutomic describing how the dead instances check works: https://github.com/LemmyNet/lemmy/pull/3427#issuecomment-1618405012 . Basically, if your instance is unlucky enough to fail to connect to, say, lemmy.world when it runs the daily check three times in a row (because lemmy.world happen to be down at that time), then it might decided to skip federating with lemmy.world. Three days might be too short for this check, should’ve been a week imo.
A general thought about centralisation on the fediverse …
Across various platforms, relatively large instances appear to be a common, arguably “natural” phenomenon.
mastodon.social,firefish.social,kbin.social,lemmy.world,pixelfed.socialetc (see the fedidb page for platforms for data).So, without judging it as inherently good or bad for the fediverse, it might be worth understanding why/how this comes to be. In that vein, my immediate thoughts:
- Decentralisation isn’t naturally attractive and has obvious flaws. It’s off-putting if you don’t understand what’s going on, raises questions that hard to get answers to such as “how are the admins of that instance” and “what problems will I face that I’m not being told about”, and generally introduces decisions and information that get in the way of starting an account and jumping in to the platform.
- Network effects are real and would naturally lead to run away centralisation.
- People seek “trust-worthiness” (related to the first point), and amidst uncertainty about which instance to pick, the obvious, and maybe only clear signal of trustworthiness is the size of an instance.
- In the case of lemmy, the location of communities adds an extra dimension, which, given that you need an account on an instance to create a community there, contributes an additional centralising factor.
If excessive centralisation is to be avoided, whatever the threshold is, I’d bet accounting for these factors (and whatever others are in play) would be necessary. Some random thoughts along these lines:
- Large central instances could do more to promote instance diversity. I’ve seen ruud say that lemmy.world will be doing something along those lines shortly (stated on mastodon).
- Smaller instances could do more to clearly state why and how their instance is attractive. How invested are the admins into maintaining the instance long term, what’s their moderation/(de-)federation policy, do they have a team of some sort, who are they generally on some sort of personal level etc.
- join-lemmy and bigger instances could do more to surface the above information about other instances.
- The lemmy community (or any other fediverse community) could do more to establish norms about what is expected of instances, admins and communicating where instances and admins are in terms of these norms.
- I see open source licences as a good model. Basically, various “pacts” get written up over time, which are statements of values and commitments that admins and users enter into when they run or join an instance. Various instances adopt particular (or various) pacts which made clear to all new members.
- A bit more adventurous … I’m wondering if “community only” and “user only” instances might make sense at all?
- No idea if the division of load here would actually help anything, but I’m curious.
- One issue would be how do users create communities on a “community-only” instance, and I figure the easiest way through that is allow users to sign in to the instance with their credentials from “user only” instances. A bit of software work would be required, but it’s been done on the fediverse before.
Beyond all of that, it might be worth considering the benefits of a relatively big “central” instance. Namely, AFAIU, that they test the limits of the software, which is useful for future growth, they can probably muster larger moderation teams, though such often has scaling issues, and, to be fair, provide the easy landing spot for newcomers who don’t know how (or why) to pick an instance.
A bit more adventurous … I’m wondering if “community only” and “user only” instances might make sense at all?
I suggested this in another thread. However I’ve come to see an issue with that idea. The users.
It’s very hard to change the URL of an instance for federation purposes. All the existing posts and comments that exist everywhere will reference the original URLs.
My suggestion was to move the user side to a new URL. But, the problem there is that lemmy.world is one of the main instances that are suggested to potential fediverse denizens. I suppose the user based URLs could redirect to a new lemmy user only instance keeping the original federating.
It would make some difference in terms of load because the one central point both federation and user activity is going to hit is the SQL server. The user side would be just one more instance pulling data from the federating site. And all user activity would hit the new dedicated DB. There’s still going to be an upper limit without moving toward clustered SQL servers and the like.
I think there’s a real problem with advertising the fediverse on places like reddit. Because, in one post you’ll never properly explain the fediverse. It’s a bit like the Matrix. You need to see it for yourself. As such, what would seem like the sensible thing to do (point people to fedidb or the observer one) will likely confuse people initially. Pointing them to lemmy.world or kbin.social makes sense for this reason, get engaged and understand later. So, I’m not sure how the long term solution to this goes.
Just spitballing an idea I haven’t fully thought out: One interesting way to avoid excessive centralization may be to display a “Join lemmy” button to logged-out users, more prominent than the “Register” button, which redirects to join-lemmy.org/instances instead of the instance registration page. Though I agree with your point about join-lemmy in its current form being somewhat underwhelming as an onboarder. In addition to what you mentioned it could do with some filters (For language, at a minimum) and sorting (For instance age, size). Even adding a button that basically says “I understand the instance I pick isn’t that important, just send me to a random reliable one please” would help a lot.
In the long term I also hope Lemmy evolves to actually make the instance you’re on matter less - for example, by changing defaults on the homepage and search page to “All” instead of “Local”, showing global subscriber count instead of instance subscriber count when searching communities, adding ways to migrate accounts and communities between instances, and perhaps even adding a way to merge one instance into another if one admin no longer wants to maintain their instance, and another admin is willing to absorb them.
That is now a viable option since join-lemmy has finally updated with more instances. Before it was the same 6…
I feel like picking your first instance is not as important as you make it out to be here.
It’s similar to registering on a web site, and all these decisions you talk about sound like you are choosing a partner for life. Is she responsible? Will she take care of the children?
It really isn’t the end of the world if you pick a small instance and you don’t like it, I promise.
So I’m going to push back against your response here pretty hard.
First, I’m not talking about myself, I’m trying to understand general user behaviour … so going ad hominem here or presuming I’m projecting my own problems isn’t productive or useful.
Second … “it isn’t that hard” is, IMO, the mantra of someone choosing not to understand the users, which can become a pretty toxic behaviour or perspective.
You state …
picking your first instance is not as important as you make it out to be here
(emphasis mine)
The pertinent questions here are:
- How is a new user supposed to know that?
- Where has such information been provided?
- How clearly and easily discoverable is that information?
- How convincing and comprehensible is that information for a newcomer?
- How much does digesting this information ultimately contribute to the load of signing up for lemmy such that it ultimately doesn’t alter the fact that picking an instance, or learning that it doesn’t matter which instance you pick, is friction that is easy to bypass by simply picking the big central instance?
If you were trying to help me … thanks … but I wasn’t talking about me … rather the generic “new user”.
And none of what you say about it being
not as importantdoesn’t really alter the reality that the friction of decentralisation makes (re-)centralisation around a big instance the path of least resistance and therefore the common choice for many newcomers.I joined vlemmy and subscribed to a bunch of communities. What communities was I subscribed to? I’ll never know because it’s down forever
Doesn’t sound like a huge personal loss to me but sure… You would have to resub to stuff that interests you. :)
There are tools out to help you (at least partly) migrate your account. It won’t migrate your posts and comments but it will migrate your subscriptions.
https://github.com/wescode/lemmy_migrate
Also !syncforlemmy@lemmy.world is working on implementing this feature.
Yeah i know, I think it’s very good that it’s coming in the clients soon. Us technical folks can run command line utilities for this but probably a few users would run into issues with those.
Yeah it’s command line and python so definitely not suited for everyone. It’s why I point out that the Sync dev is working in it. All of it is just growing pains…
lol your instance is considered “Compromised” from totalvirus and malwarebytes.
Can you send me a link? That sounds weird. :)
Your post made me curious so I scanned it with a number of tools:
https://www.urlvoid.com/scan/lemmy.today/
https://transparencyreport.google.com/safe-browsing/search?url=lemmy.today
https://sitecheck.sucuri.net/results/lemmy.today
Looks fine?
The first url says directly Found: 1 from fortinet. Could be false positive. But still weird.
Hmm it didn’t a minute ago.
That’s weird. Probably a false positive but will ask the instance owner.
Honestly, I moved away from that instance after days of issues, downtimes and timeouts. Been a smooth experience since then. It’s unfortunate that they are more focussed on grabbing as many new users as possible instead of giving the existing users a stable experience.
A number of client apps, at least on the android side, set the site as a default when asking users to pick an instance. It’s not surprising given their status at the moment for the app devs to do that but it does lend itself to some of the centralization problem/risk that’s developing. Scaling a site for a reletively new platform at a very rapid pace is going to be a big challenge regardless even if you had a full scale IT team behind it just because of all the unknowns involved. Throw in a pile of new apps all pushing users to that instnace first, and there’s no question that most people trying something on a whim won’t bother to search out anything beyond that first ‘default’ option, and it just snowballs. Suffering from success as it where. I suspect if more of the app devs did some of that lifting by pointing new users to pick from a list without a fixed order it would help a lot.
You have no idea how much work is being put in to get things stable. That’s an insult to everyone working in the background together even with other instance admins. Other admins help because they know we all benefit from these growing pains.
A lot of you who come with these remarks seem to forget that the fediverse is still very small. If you want to compare it to reddit the entire fediverse isn’t even the size of a medium subreddit.
I actually want to make it super clear that I think the admins of Lemmy.world are amazing in the way they take care of their instance. They are absolutely doing everything they can to make it a good place for everyone.
But at the same time, I feel that having all users on a few large instances is going back to being centralized, and I think they would see this if they took a step back and looked at what’s happening now. They clearly wants all users to be on their own or the other already large instances. This is their egos making decisions, not their hearts.
But still I have respect for what they are doing. It’s a lot of work and they are under fire all the time.
Again, the fediverse is still very small. We hope that the fediverse will continue to grow and what is wrong about actively working towards, helping the fediverse as a whole? We work together with other instance admins… But for some reason it is bad to be successful and we don’t have our hearts in this?
Damned if you do, damned if you don’t…
Yeah I kind of sympathize with that feeling. You are trying to do good, that’s very obvious. Hope you have a good day.
Tangential comment about sorting algorithms.
As of writing, this post has a score of
3, with36up and33downvotes. There’s a “Controversial” sorting algorithm in the works for lemmy, where controversity is high “engagement” (ie, lots of votes) but diverging opinions (ie low total scores) … and this is precisely what that feed would be for … kinda interesting!We are not ‘dropping’ messages from smaller instances. @ruud@lemmy.world and others did tighten up some things to fend off some attacks but that should not interfere with the federation.
If you could have at least mentioned your instance name we could ask our backend administrators to have a look and we could get back to you with an informed answer.
We will not close registrations, what we will do is show people information on how the fediverse works and give them the option to register on other instances. That will be implemented in the next couple of days.
Why is it so vital for you to keep registrations open? Can you please help me understand the logic behind it?
Personal anecdote: I probably wouldn’t have joined Lemmy if it wasn’t for lemmy.world and I think I’m far from alone.
Same, I started with Lemmy.world to begin with because I didn’t really understand the fediverse and didn’t want to go through choosing. After I understood federation more I set up my own instance. If I didn’t start with Lemmy.world I probably wouldn’t have tried Lemmy at all.
People hear from others that Lemmy is the place you go and when searched for you end up here. Having registrations closed doesn’t sound like a great idea when most people have no idea about the fediverse
I concur. I am one of those people as well.
Instead of mentioning your instance so we can help you you seem to have other motives.
Edit: but if you want the logic. We had discussions internally as well as with other instance admins about this and the conclusion was - even from other instance admins - that closing registrations would be a bad idea. Currently a lot of people are sent to lemmy.world from reddit and 3rd party apps for example. If people land here and are not able to register they might not look into it further and give up because it is too complicated or overwhelming. So that would actually be worse for the fediverse. That is why we want to make the sign up page more informative and provide options.
Perfectly fair. People coming from Reddit specifically need to be eased into the fediverse and saying “whoops sorry we’re full / closed” isn’t great.
Ideally if we want things balanced people should be on multiple instances but I’d rather several slower, bigger instances in the meantime until the community expands.
Cheers for the work you guys do!
If people land here and are not able to register they might not look into it further and give up because
Sorry, that’s a really bad excuse. Lemmy allows to have custom messages/taglines on the registration page, why not just add a quick paragraph saying “registration is closed because this server is overwhelmed, but here is a list of servers that you can use”.
Make it a rotating list, change it every 2-3 days if you want.
So are you here to make us close registrations or do you want help with your problem?
I’m also here to expose bad excuses.
As for help, @Wander@yiffit.net has pointed me to the right direction, and he didn’t need to know my instance to do so…
I’m also here to expose bad excuses.
Not being able to help someone who is refusing to provide technical detail is a pretty damn good excuse in this industry.
If your goal is to expose the bad excuses of others, step one is to put in as much effort as you’re expecting from others. :P
Edit for good measure: (links fixed, forgot about direct linking comments from outside of a lemmy instance)
- Your instance was not federating with lemmy.world. [1]
- You assumed that the blame had to rest on lemmy.world because you had “eliminate[d] all the possibilities [you] had at hand”. [2]
- You made this post to vent about a bunch of unrelated nonsense and refused to provide technical detail that would assist the admins in troubleshooting. It’s a given fact that your privacy is your choice, but it’s also a given that you shouldn’t be a dick about it if you choose to withhold details, even from PM. For the record, the information being requested was the bare minimum for an instance administrator to troubleshoot network interactions with a remote instance.
- A random (but cool) third party identified the issue with your instance not federating. [3]
- Instead of apologizing, you proceeded to act like you were entitled to that solution from the admins you wrongly accused. [4] You are not god’s gift to the internet and they are not technical support for your instance.
There’s no room for niceties here, you are either an asshole in denial or some brat who is too young to know any better. Sleep on it. Come to terms with that fact and make good on it, or don’t. You aren’t worth anyone’s energy, and I’m only bothering with this summary for everyone else’s sake. Your problem is fixed, it was never on lemmy.world’s side to begin with, and somehow you are still acting like the failure of the admins to figure out what was busted with your shit is some Sherlock gotcha moment.
I am unaffiliated with lemmy.world and my toxicity does not represent the opinions of the admins. (but they’re probably thinking it)
Well he didn’t implement IP bans on his instance I bet. The reason I asked for this information is so I could check if your IP ended up on our banlist or check if it was marked as inactive.
Is the large amount of users actually the reason for the downtime? I thought it was coordinated attacks.
It’s the attacks. The amount of users or communities is not the problem but there are people who want us to close down (registrations) for a variety of reasons.
I get the feeling that you’ve never been in charge of UX before. What you’re suggesting would be an absolutely terrible user experience. You have to realize that not everybody is as interested in new things as you or I might be, so even the smallest of hurdles will be prohibitive for most users.
So he just wants us to close and leave them in the dark, right?
This is being handled by a team that knows what they are doing. Together with someone that contributes to Lemmy.
Apes together strong.
“If you could have at least mentioned your instance name we could ask our backend administrators to have a look and we could get back to you with an informed answer.”
Try answering their request. Stop going on about registrations, at least until the above is resolved.
I think it is already answered in this thread.
- let us know which instance (even by pm) do we can have an informed reply as to what is going on. There have been changes made in the backend to fend of attacks but this should not affect federation.
- There is a bug in Lemmy where instances are too early marked as ‘inactive’ and will stop federating.
Btw the title of this thread is ‘stop registrations’ not ‘help me find out what is going on’.
As for why I am not naming my instance: this post went to -1 as soon as it got posted. My comments are also being downvoted quite quickly. I don’t believe that I am being rude or uncivil, yet it feels like mods and admins are more worried about fighting to keep the status quo than helping the community at large, and I’d worry about retaliation.
I mean, you are demanding something in a very unrespectful way, without considering the negative impact that could have in the grow of the fedi as a whole, and say that any reason why you could be wrong is an excuse that you need to adress. But it must be the admins who need to mantain the status quo if you get downvoted. I think you are not being very self aware.
I mean, you named your account “you are hurting the fediverse” to come and ask them to close registrations then you ignore their questions. Seems kinda rude to me.
The reason you’re being downvoted is because you experienced a problem (Posts from your instance won’t show up in this instance), came up with a pet theory for why that problem might be happening (This instance must be dropping posts from small instances because it’s overloaded from all the users), assumed it was correct (Based on what, exactly? Because it’s definitely not correct), then came here to post about it in a very confrontational, demanding, and accusatory tone, with a seeming lack of desire or ability to consider that you may be the mistaken one. Moreso, the change you’re suggesting would have dramatic and perhaps negative repercussions for both this instance and Lemmy as a whole.
Maybe the reason is that you behave like a first-class asshole idiot? You wanted help, but evaded the reasonable question for your instance name. Several times. While making unreasonable demands. I’ve seen a lot of idiots in thirty years of being on the internet, and seriously, you are one of the bigger ones.
I don’t believe that I am being rude or uncivil
Well, herein lies the problem, because the rest of us do. You’re assuming that you know better than the people who have been doing this for years, and dismissing their attempts to help as “excuses”. I really don’t know what other outcome you could have possibly foreseen with this combative attitude.
I’m not seeing dropped messages. I can see that a comment in this thread made it through just fine. However, I am seeing since around 09:30 yesterday that get requests for comments (and maybe other activities) are being responded to with 400 “not_logged_in”. I’ve quite a few messages in error queue stacking up because of this.
Edited to clarify. The instance is running kbin.
That’s possible because those requests are being blocked to non-logged-in users.
It’s a federation message. For example I receive a federated like message, and so my instance fetches the comment (and all parent comments) if not already present in my DB. It has the correct headers to be activitypub. So it should be allowed.
Would be nice if you could mention more instances than the top 5. The way things look right now, Lemmy is becoming centralized with all communities and users on a handful of servers.
Another thing to keep in mind is that when an instance goes down you lose the content. Instances should be reliable too. So just listing any random instance and overloading them would not help either. Imagine running a private instance for yourself and some friends and getting randomly listed on the signup page here…
I don’t think small instances have a reliability problem. They actually lack users right now. And of course you wouldn’t sign every new person up to the same instance, because that is against the entire point of the technology here.
I seriously believe it would be much better for the Lemmy network to spread people out on the existing instances.
I think it’s more a case of long-term reliability than day-to-day reliability. Anyone can create a lemmy instance - including people who may lack the resources or long-term motivation to keep their instance on the internet for basically forever. That’s definitely going to be a much smaller risk with the more established instances - though I’m sure we’ll eventually have a lot of drama over some instance or another in the top 10 shutting down.
Yeah but if they don’t get users, they will shut down guaranteed.
If they get users, they will likely enjoy having an instance more and take care of it.
So it’s a bit like chicken and the egg here. You have to give smaller instances a chance to grow or they will shut down of course.
I don’t disagree with what you said, but I am more risk-averse than you are about it. I think it would be best in the long term to list servers with a proven dedication to the long-term existence and health of the community. This doesn’t necessarily mean the top 5, or even the top X, but it does mean excluding a server hosted on a laptop in someone’s closet with 8 users, or instances from completely unknown in the community admins that have existed for a month.
Which is why we will adjust our sign-up page. But not every small instance wants new users because that would increase operating costs which they might not be willing to cover.
No that is caused by instance owners not closing registrations when their instance is full.
It’s a lot of common sense actually in how Lemmy software is built. You can choose to upgrade your instance or you can close registrations if the load becomes too big.
It’s up to the instance owner how to deal with that, given the chance.
This was brought up a few days ago on ml
@ruud@lemmy.world when you have time
Do you have any proof of any of this? Lol this just reads like a rant that your server isn’t as popular and it isn’t fair so the popular host should just stop taking on new users because reasons
What’s the point of your post? You’re just critiquing for the sake thereof.
My instance is not meant to be “popular”. It’s just for me and some friends, it’s federating just fine, I can receive all the posts but whatever I write does not get here.
I understand. Send it to me in PM sonwe can investigate. This will be kept private.
You don’t need my instance to help me here. I already asked: can I just take the comment from https://lemmy.world/u/Wander@yiffit.net as an indication that I can bump the
updatedvalue in the instance record? If I run something likeUPDATE instance SET updated = 'NOW()' WHERE domain = 'lemmy.world', will the communication be re-established?Technically that should work.
And the reason I asked for your instance name is so we could check this in our db table or check if your instance IP didn’t end up on our banlist.
Isn’t the banlist public, available in the modlog?
No this is done on the webserver not in Lemmy. And these are not “instances” specifically these are IP’s who are overloading the API’s with expensive requests.
Aside from version updates, I haven’t changed my server setup since I set it up months ago, and my instance has only a handful of active users. Only two of them subscribed to !fediverse.
Also, keep in mind that I am receiving updates just fine (maybe with some delay) the problem is that nothing I post on the lemmy.world communities end up visible here.
