An update:
- fmhy.ml is gone, due to the ongoing fiasco with mali government taking all their .ml domains back
- As such, lemmy.fmhy.ml is also gone, we are currently exploring ways to refederate (or somehow restart federation entirely) without breaking anything substantial
- We have backups, so don’t worry about data loss (you can view them on other instances anyway)
Currently, we have fmhy.net and are exploring options to somehow migrate, thank you for your patience.
What happens when it goes down, will all of the posts be gone?
Re-federation is probably possible. BUT! You’re going to always have problems with older content. Case in point my federation error messages is at 2300. About half are failed requests on fmhy.ml.
So for re-federation what’s needed:
1: Remote instances should unsubscribe all users from any fmhy groups. They’re dead now. They can only announce that and hope they do. I reckon when their errors start ramping up (as I saw yesterday) they will be looking into why. Probably to help de-federate from the old URL
2: The fmhy instance should unsubscribe all users from all remote groups but keep a note of the groups while identifying as fmhy.ml. Then once on a configuration for the new domain re-subscribe to each one. The first step should hopefully stop them trying (and failing) to federate new events to the old URL. The second step should trigger federation with the new one.
3: They could be able to keep the DB. But I am not sure in what places the old domain might be stored in the DB and what would need fixing there. Also not sure if they’d need to regenerate keys. Not sure if they’ll see the key was attached to the old domain and refuse to talk to the instance.Now what’s going to be a problem? Well ALL the existing content out there has references to users on the old domain. It’s VERY hard to fix that. Like every instance would need to fix their database. Not worth it. But, whenever someone likes/unlikes or comments or whatever a post made from fmhy.ml then there’s a good chance a remote instance will queue up a retrieval of:
1: User info about the poster/commentor/liker
2: Missing comments/posts for a like/comment eventAnd those will fail and error log. I don’t think there’s a way around that aside from editing the whole database on every instance. Again, IMO not worth it.
Would be a nice federation feature if, provided you could identify with the correct private key, announce a domain change which would automatically trigger the above in federated instances, or at the very least some kind of internal redirect for outgoing messages.
Afaik mastodon has a way for instances to migrate to a new domain, but the old domain must be up during the migration process. Lemmy on the other hand don’t even have any domain migration procedure yet. People will probably go nuts about this on their GitHub issues portal.
Possibly. I think mastadon has been around a bit longer though? Not sure why the old domain must be up. Unless they don’t store public keys of known instances and they rely on DNS for the security.
e.g. Instance A signs a request, Instance B queries Instance A via DNS lookup (as is normal) and checks public key confirms signature and allows it.
I got curious so I start digging into how mastodon do it. It’s more like a hack, really. Mastodon uses WebFinger to resolve user account, so when you change domain, you can leave the old domain up so your federated servers can still resolve your users and realized the domain has been changed and update their federation data. But it turns out you can’t exactly retire the old domain either because it’s still tied to user account internally. So if you lose control of your old domain, you’re probably as screwed as fmhy.ml.
Yeah, which is why I think storing remote user and instance public keys might be better. Then that can be used to authenticate the migration request (it’d probably need to be an extension to the activitypub standard).
The biggest problem I see is that an instance doesn’t know about all the instances that have data pointing to them. So how does it communicate the changes to everyone? The mastadon way is probably the sensible way to do it, despite not supporting the loss of control of domain scenario.
If I’m running lemmy.world, I wouldn’t unsubscribe my people. I’d wait for that instance to move to a new domain and just find/replace in the database.
Not every instance needs to migrate fmhy. Some can just leave that stuff broken. If the biggest half dozen instances migrate manually, fmhy would be able to keep most of their subscribers.
I do wonder how often instances will keep looking for fmhy without intervention. Seems like tooling to migrate or discontinue an instance wouldn’t be too difficult to build. At least it wouldn’t if they didn’t have a million other things on their plate.
We could use a few less third party clients and more work on Lemmy itself. Unless you’re going to bring over your userbase like RiF and Apollo can.
Yes, although you might need to fudge keys if they’re properly enforced. Looking at kbin I can see requests are at least signed with the private key. Not sure if the public key is stored somewhere in database, or is pulled from the instance using DNS as a security guarantor (I guess) every time.
I don’t have any subscriptions to them, but I have those 1000+ errors just from posts their users were involved in.
What about lemmy.ml? Can govt just take their domain?
I think in theory yes, since the .ml tld is now managed by the Mali government instead of some guy that had an agreement with them.
Lemmy has had such a crazy month and a half. Insane growth, XSS injections, DDOS attacks, admin takeover, domain name seizures. What a wild ride
and still better than using official Reddit app (and still better than Reddit overall) lol
Facts. And such drama does make Lemmy more appealing in a way
What’s this about an admin takeover?
I think they are referring to this
oh, the XSS injection? I thought it would be some admins fighting over who owned the instance or something like that lol
And we’re still here, going strong. Long live the Fediverse.
Welcome to the real world
My boy Lemmy is growing up!
That’s how every service starts. Let’s keep going
I’ve been seeing posts from users on lemmy.ml though? How’s that possible
They are probably paying for the domain
have fmhy not paying for theirs?
No idea… Sorry
I’d like to think FMHY was true to their name and didn’t pay for the domain.
No, that is the crux of the matter. They used freedom, which offered .ml for free.
This is why we host our instance on a .org. Honestly another huge blow for Lemmy. It doesn’t really inspire confidence in the platform. Hopefully after enough time passes smaller instances like us and the bigger ones left will have help up a good track record to inspire confidence again.
This isn’t really that huge of a blow, it’s a learning curve sure but just because some people made dumb decisions on what TLD to use based on something they decided it means (and backtracked to say they chose it because it was free, I know) doesn’t mean federated platforms don’t work. Actually imo it points to the strength of federation that we can still be here using lemmy on our instances while they switch.
All this really did was teach instance owners (who this might be their first experience hosting things too btw) that you have to use a TLD that is more stable like a .org, .com, .net, etc over a “free” one, and this is afaik the first instance of something like this happening, so honestly they didn’t have precedent to base this on before.
By no means did i mean that federated platform’s wont work, far from it. Im more considering the reputational damage of the platform. For me all the .ml instances unresolvable. And it doesn’t look good when the “official” instance is lost. If i were an outsider considering moving to lemmy and i saw vlemmy disappear, lemmy world get hacked, and .ml loosing their domains id be pretty hesitant on making the switch. Thats where i sit for the moment. Lets all hope we can get a good track record going now.
Exactly, IMO this has nothing to do with lemmy itself and everything to do with dumbass domain names.
Only one went down - most everyone else are still up.
This is the opposite of a blow. It directly shows the resilience of the fediverse.
Where I live i cant access any .ml instances so for me even the lemmy.ml is down
This just shows how putting all eggs into one basket is a bad idea.
Only the fewest of few instances should have used .ml - they should have spread out. This is the beauty of the fediverse when done as intended.
The only drawback I see, is that people want to make instances bigger, scaling tall, when they should have been scaling wide + wanting to use the same odd TLD. We reap what we sow.
Hello! I’m new to Lemmy, could someone break this down like I’m 5 and explain what it means for the people who were already on there?
Only instances with a “.ml” at the end of the name may or may not be affected. Lemmy is a collection of instances so the loss of a few will not cripple the whole thing. Content over the whole is not greatly affected.
If your home log-in instance is one that’s affected, you’ll have to find a new one. You’ll know right away because the instance will be unreachable. Not a big deal, last time I looked there was over 1200 instances to chose from.
Another consideration is any communities living on an affected instance may have issues. All communities are common to Lemmy, but each originates from a particular instance. We’ve not yet seen a major instance go down so I don’t know how Lemmy deals with communities getting orphaned like that.
It means anybody who will want to go to site lemmy.fmhy.ml will not load site and would think its down, maybe some will find out on google about it, some are already on multiple instances…
Also all links to lemmy.fmhy.ml are dead/gone now.
Btw the domain *.ml was free as i read, at least they could get some 1-5 USD domain name extension.
Is there some news coverage on the mali governments actions? I couldn’t find anything on my quick google search
I hope FMHY comes back. Didn’t realise how much I liked it until it was gone.
FYI discussion on lemmy.ml about it
I migrated, but I would happily rejoin under a different tld.
WIll this also affect all other .ml domains? Or is this some anti-piracy thing? (I don’t know fmhy, but from the name I guess it’s about piracy.)
It seems to be Mali just wanting their domains back, in which case it’s uncertain times for all .ml domains.
rip lemmy.ml
lemmy.ml is still up as of right now. Possibly they contracted a subscription to the domain name to keep it up. They had to do something to retain it otherwise the site would be unreachable. If lemmy.ml does have to change names it will be a hassle since I’ve got a good number of community subscriptions there.
This wouldn’t happen to an instance with a regularly subscribed domain name. Problem is the .ml domains were free and the associated country decided to claim them back. The risk of using a free top level domain is something that should have been considered. I don’t think it’s worth the risk versus the cost savings considering how difficult it is to migrate a Lemmy instance.
It’s just the domain, though. That’s not a big deal to change.
The instance is known by its domain name in the federation network. If that domain name changes, it’s like starting a new instance from scratch.
Sounds like a complicated project to migrate communities and posts and users to a new instance without breaking something.
Currently, activitypub identity is tied to domain name. Mastodon support migration as long as the old domain is still up during the migration process, but AFAIK Lemmy doesn’t even have a process to migrate an instance to a new domain yet.
Someone should tell Lemmy devs and send them a crate of coffee because it’ll be a race to implement domain migration before all .ml domains got shut down.
Federation connections are by domain name, so … it is a big deal
Right. This will basically make nearly every /c live in .world as all of the .ml /c’s go defunct. That, or Beehaw, which is walled off from everyone else.
(Side note… my work’s firewalls block everything *.ml – and that’s the only thing that saved me from creating my account there)
From that point of view, yes. That’d mess things up, you’re right. But from my understanding, they won’t lose any data, accounts will remain, as well as subscriptions that lemmy.ml users have. Or am I wrong?
The problem is, if they don’t have access to their original
.mldomain, their accounts are still tied to it. That means if they try to interact, such as subscribing to a community, when the data for that action tries to be sent back (such as updates) it’ll go to the.mldomain, which they wouldn’t receive.Lemmy doesn’t have a built in way to just change the domain name, or really any of the ActivityPub services AFAIK. You’d have to either really do some hacky stuff to get around it (which could result in unknown issues down the line) or reset everything.
Most of the hacky ways around it involve retaining ownership of the old domain and leaving it up indefinitely as a pointer to the new location. If your domain is taken from you though there is not much you can do.
Seriously dumb to have used this TLD considering there are a ton of choices these days.
Oh, it’s more complex than I expected. Thanks for explaining. I was wrong.
deploying the fediverse instances-instance communication on top of a mesh-net like yggdrasil, using their addresses as domain names, may be a quick fix without having to change the paradigm
Think about all the links to lemmy.ml
Good thing join-lemmy is safely tucked away in a .org domain.
This is extremely bad timing for Lemmy (if it ends up happening), but also a good example of how federation makes the entire social media landscape more robust. Had this happened to a centralized service it would be devastating.
If it was always going to happen, now isn’t really a bad time. Sure, a month ago would have been better, but people still haven’t been here that long. If I wind up needing to migrate, and lose my current account, oh well. No big loss. I imagine others feel similar.
I was frustrated with the outage yesterday and created a new account on a different instance so I could still browse. Couple hours later I had all my subscriptions filled out and the experience is almost identical to my first account.
Not really. Most centralized services are accessible via multiple domains, e.g. for different countries. This would just disable one of them, but users could still use another to log into their accounts. For the Fediverse it “disables” an entire instance, cuts it off from federation and locks out users.
Lets not put a positive spin on a situation that exposes a weakness of the current system. The federation protocol needs to be able to handle these things gracefully, like propagating domain changes and migrating accounts between instances!
I’m now wondering what happens if the Mali government (or someone else) begins using those domains with their own lemmy instance, potentially with malicious content.
Would the instances they’ve federated with begin ingesting and serving that content automatically? Or would that be blocked due to key mismatch?
Afaik it is all connected to the domain name, so they could definitely start to impersonate any .ml instance. Other instances could detect that the signing key for federation messages changed, but that’s about it. Their admins would probably have to block/defederate them manually.
I think they need the private key for the https certificate to do that
Shall I make an account in another instance?
Never hurts. Could be a good opportunity to look around the threadiverse and see if you find anything interesting.
However, as it only affects the domain, I expect the Lemmy developers will manage to migrate user data to the new domain should lemmy.ml go down. So your account won’t just disappear, but it might go down for a while. It might also affect communities hosted on .ml domains, as followers from other instances will not have the correct path any more.
Thanks for the info.
Yeah, they are actively working on functionality to migrate user accounts and other data between instances, so that they can use that functionality to migrate everything on an instance to another instance.
Since migrating data affects all the replicated data on other instances as well, I guess when they migrate lemmy.ml somewhere else, all of Lemmy will be down for a day or two, being just overloaded with all the migration stuff.
I’ve migrated from fmhy to feddit.uk, luckily my subscriptions were on a cached web page soon was able to manually re-subscribe.
Nope. Domains don’t store data. They can change domain and keep all the data.
Unfortunately, no.
Currently, activitypub identity is tied to domain name. While mastodon support migration as long as the old domain is still up during the migration process, AFAIK Lemmy doesn’t even have a process to migrate an instance to a new domain yet.
So basically, if you switch your instance domain, you’ll mess up all your federation network, unless Lemmy devs implement a solution soon.
Calckey.social will be transferring all data to new firefish.social, first in the Fediverse.
I understand it as the Mali government is taking back all the domains after a subletting contract ran out. A lot of sensitive emails that should go to .mil (US military) has been typo-sent to .ml-addresses instead. Here’s some more reading.
(I am very tired here and might have misunderstood everything, please correct me if I am wrong)
Perhaps the military should have a system in place to not allow emails to be sent outside of very specific TLDs if it’s that sensitive? And perhaps have an automated contact book, instead of relying on someone typing out the to: address manually to be able to make that mistake in the first place?
Seems like some very basic security measures for something so serious.
Internally they do block that but the problem are people outside the network sending something to a .mil address and mistyping.
This says that they block outgoing mail to .ml domains from its network.
https://domainincite.com/28897-freenom-is-losing-another-cctld-after-collecting-military-emails
Edit: wrong link
For most situations, there is a global address list that members can use. There are instances where emails need to be sent outside of the .mil domain though, such as to other government agencies that use a .gov, or to contractors on commercial domains, as well as to partner nations that will be on their own countries’ domains.
Does this have anything to do with the whole email thing from the American military? According to the financial times, there are about 117.000 emails send to .ML addresses instead of .MIL…
Nah, it’s more that this is about to get worse when those mails all go to a state closely associated with Russia.
Somewhat related. Basically, the management of the .ml TLD are being handed back to Mali government, and they seem to revoking.ml domains left and right.
I suspect they’re revoking registration for .ml domains that was registered for free. the company that originally managed .ml domains had a free domain offers where you could register any .ml domain for free, the caveat is you don’t have the ownership right to that free domain. Maybe Mali government doesn’t honor such free domain registration and wish to revoke them all.
Posting here for visibility as I guess most people on Lemmy are not on Firefish/Mastodon
If this was a planned takeover by the government, why was there no notification sent in time? Why is lemmy.ml not shut down in parallel?
There was a report in the beginning of June that things started looking weird. The registration of new .ml domains shut down in the beginning of the year.
In short, the Mali government just gave some random 3rd party a ten-year contract to hand out the domains for free, which the third party did without too much care or attention. It would have been up to the third party to notify domain owners, but as they’re not paying and probably don’t even have contracts themselves, there was little incentive to do so.
As far as I can understand, it relates to the US military scandal only indirectly: As the .ml domains are now returning to the government of Mali, it becomes a lot more problematic that the US keep directing their emails there, and the person in charge of managing the domain went public about the security threat.
