I woke up this morning to a text from my ISP, “There is an outage in your area, we are working to resolve the issue”
I laugh, this is what I live for! Almost all of my services are self hosted, I’m barely going to notice the difference!
Wrong.
When the internet went out, the power also went out for a few seconds. Four small computers host all of my services. Of those, one shutdown, and three rebooted. Of the three that ugly rebooted some services came back online, some didn’t.
30 minutes later, ISP sends out the text that service is back online.
2 hours later I’m still finding down services on my network.
Moral of the story: A UPS has moved to the top of the shopping list! Any suggestions??
I’m a big fan of running home stuff on old laptops for this reason. Most UPSs give you a few minutes to shut down, laptops (depending on what you run) could give you plenty of extra run time and plenty of margin for a shutdown contingency.
Small, good value, quiet, power efficient, built in battery backup and server terminal. Laptops are dope for home labs!
When you are bored, backup a VM then hard kill it and see if it manage to restart properly.
Software should be able to recover from that.
If it doesn’t, troubleshoot.That reminds me of Netflix’s Chaos Monkey (basically in office hours this tool will randomly kill stuff).
While I appreciate the sentiment, most traditional VMs do not like to have their power killed (especially non-journaling file systems).
Even crash consistent applications can be impacted if the underlying host fs is affected by power loss.
I do think that backup are a valid suggestion here, provided that the backup is an interrupted by a power surge or loss.
most traditional VMs do not like to have their power killed (especially non-journaling file systems).
Why are you using a non-journaling file system in 2024 when those were common 10+ years ago?
Or even better use something like ZFS with CoW that can’t corrupt on power loss
I would still consider that generation of filesystem to be effort to use while regular journaling filesystems have been so ubiquitous that you need to invest effort to avoid using one.
It was supported and the default out of the box when I installed my OS
Maybe on some distros that is the case if you install a recent version but to get a non-journaling filesystem you literally have to partition manually to avoid using one on any distro that is still supported today and meant for full sized PCs (as opposed to embedded devices).
Are you talking about Linux distros? What manual partitioning has to occur?
and don’t fuck with sync writes
It’s been a while since a power cut affected my services, is this why?
I remember having to troubleshoot mysql corruption following abrupt power loss, is this no longer a thing?
Databases shouldn’t even need a journaling filesystem, they usually pay attention to when to use fsync and fdatasync.
In fact journaling filesystems basically use the same mechanisms as databases only for filesystem metadata.
Your system should be fine after a hard kill. If its not stop using it as that’s going to be a problem down the road.
When I built my home server this is what I did with all VMs. Learned how to change the start up delay time in esxi and ensured everything came back online with no issues from a cold built.
Rip VMware.
Did the services fail to come back due to the bad reboot, or would they have failed to come back on a clean reboot? I ugly reboot my stuff all the time, and unless the hardware fails, i can be pretty sure its all going to come back. Getting your stuff to survive reboot is probably a better spend of effort.
I didn’t mean to imply that Services actually broke. Only that they didn’t come back after a reboot. A clean reboot may have caused some of the same issues because, I’m learning as I go. Some services are restarted by systemctl, some by cron, some…manual. This is certainly a wake up call that I need standardize and simplify the way the services are started.
I reboot every box monthly to flush out such issues. It’s not perfect, since it won’t catch things like circular dependencies or clusters failing to start if every member is down, but it gets lots of stuff.
We’ve all.committed that sin before. Its better to rely on it surviving the reboot than to try prevent the reboot.
Also worth looking into some form of uptime monitoring software. When something goes down, you want to know about it asap.
And documenting your setup never hurts :D
On the uptime monitoring I’ve been quite happy with uptime kuma, but… If you put it on the same host that’s down… Well, that’s not going to work :p (I nearly made that mistake)
It’s not the most detailed thing, but I just use a free account on cron-job.org to send a head request every two minutes to a few services that are reachable from the internet (either just their homepage or some ping endpoint in the API) and then used the status page functionality to have a simple second status page on a third party server.
You can do a bit more on their paid tier, but so far I didn’t need that.
On the other hand, you could try if a free tier/cheap small vps on one of the many cloud providers is sufficient for an uptime Kuma installation. Just don’t use the same cloud provider as all other of your services run in.
Oh, I’m fine with my setup, I have a couple of external servers that can monitor all my web accessible stuff with kuma and then I’ve got another local one to monitor my non-web accessible stuff.
Thanks for those tips though, definitely useful to consider other options
Same, Uptime Kuma is fantastic. I put it on my most critical server, if Kuma is down, everything is down :D
Yeah an unclean reboot shouldn’t break anything as long as it wasn’t doing anything when it went down. I’ve never had any issues when I have to crash a computer unless it was stuck doing an update.
My suggestion just changes your threat model, so may not be a good one based on your wants.
Perhaps consolidate systems? Managing less devices = less points of failure. But adds the risk of any given failure being more severe.
This thought came to me this morning. I have 4 machines both because the BEAST grows organically, and because we’re always trying to avoid that single point of failure. Then a scenario comes along that makes you question your whole way of thinking, diversifying may actually create more problems
UPS with usb allows you to configure a script to properly shutdown your server when a power outage happens and the UPS battery is about to run out.
IMHO you’re optimizing for the wrong thing. 100% availability is not something that’s attainable for a self-hoster without driving yourself crazy.
Like the other comment suggested, I’d rather invest time into having machines and services come back up smoothly after reboots.
That being said, an UPS may be relevant to your setup in other ways. For example it can allow a parity RAID array to shut down cleanly and reduce the risk of write holes. But that’s just one example, and an UPS is just one solution for that (others being ZFS, or non-parity RAID, or SAS/SATA controller cards with built-in battery and/or hardware RAID support etc.)
I agree that 99.999% uptime is a pipedream for most home labs, but I personally think a UPS is worth it, if only to give yourself the option to gracefully shut down systems in the event of a power outage.
Eventually, I’ll get a working script that checks the battery backup for mains power loss and handle the graceful shutdown for me, but right now that extra 10-15 minutes of battery backup is enough for a manual effort.
I run nut on a pi.
The problem is that for most self-hosters, they would be working and unavailable to do a graceful shutdown in any case even if they had a UPS unless they work fully from home with 0 meetings. If they are sleeping or at work, (>70% of the day for many or most) then it is useless without graceful shutdown scripts.
I just don’t worry about it and go through the 10 minute startup and verification process if anything happens. Easier to use an uptime monitor like uptimekuma and log checker like dozzle for all of your services available locally and remotely and see if anything failed to come back up.
Some of the nicer models of UPS have little servers built in for remote management, and also communicate to their tenants via USB or Serial or Emergency Power Off (EPO) Port.
You shouldn’t have to write a script that polls battery status, the UPS should tell you. Be told, don’t ask.
Two pitfalls I had that you can avoid:
- look at efficiency. It’s not always neglible, was like 40% of my energy usage because I oversized the UPS. The efficiency is calculated from top power the UPS can supply. 96% efficient 3kW UPS eats 4% of 3kW, 120 watts, even if the load you connected is much smaller than 3kW
- look at noise level. Mine was loud almost like a rack server, because of all the fans.
I replaced that noisy, power hungry beast with a small quiet 900W APC and I couldn’t be happier
This is why I have about five of these bad boys: CyberPower CP1500PFCLCD.
One is in my utility room for my cable modem and our chest freezer, three back up my homelab and wifi AP, and one is for my office.
They’ve been bulletproof through storms, and when we’ve lost power, but not Internet I can’t keep on working.
The big thing to look for is number of battery+surge outlets vs just surge outlets. Typically they top out at 1500VA - the more overhead for what you’re powering, the longer you can go without mains power.
A screen/display is helpful for at-a-glance information like expected runtime, current output, etc.
Never heard of someone using a UPS on a Fridge/Freezer. Does it make a difference? Seems like the UPS would just died after 10-20 minutes and not really make much difference to your freezer.
I didn’t intend to use it on the chest freezer - it was mostly for the modem, but since I had spare battery capacity and outlets I thought what the heck.
The power load is practically nothing until it cycles, and even then it’s fairly efficient - my current runtime is estimated to be about 18 hours, more than enough to come up with an alternative if we lose power in a storm.
Yeah if you self host, a UPS is very important.
when you say some services on your network are you talking about machines or softwares?
for machines yes ups makes sense for softwares writing some scripts to run on start up should be enough another alternative can be setting up wake on Lan that way you can bring all up again wherever you are
A general tip on buying UPSes: look for second hand ones - people often don’t realise you can just replace the battery in them (or can’t be bothered) so you can get fancier/larger ones very cheap.
Also, a larger capacity one is better, and it’s likely you’ll find a secondhand one with more capacity/features for a similar price.
Why? If the power has gone out there are very few situations (I can’t actually think of any except brownouts or other transient power loss) where it would be useful to power my server for much longer than it takes to shut down safely.
Longer means you’re more likely to be able to ride out a power cut, and gives you more options if you want/need to complete something more involved than saving and shutting down.
Could also be a good opportunity to add a service monitor like Uptime Kuma. That way you know what services are still down once things come back online with less manual discovery on your part.
I use a laptop and external jbod covered with a low power ups. As other said, the point is to bridge powergaps now long term working powerless. I live in the countrisied, so small powergaps happens specially when my photovoltaic don’t produce (no, i have no battery accumulators, too expensive)
Figure out how much power your servers use on average with the help of a wattage meter, then enter that number and how many minutes battery backup you want in Eatons UPS Power Calculator to find a suitable unit. I’m sure other vendors have similar tools too.
A UPS should always be your first or second purchase if only for power conditioning and brown-out protection.
They will do power conditioning? My modem is such a sensitive baby I cannot plug anything else in next to it or it starts dropping packets. Would a UPS help with that? Unfortunately I cannot replace the modem, that’s the only one the ISP will give me.
Yes. An online/double-conversion UPS will be the most effective, because it actually runs off the battery the whole time, so it’s disconnected from any line quality issues.
A line-interactive UPS is cheaper, but doesn’t do full power conditioning.
An offline UPS doesn’t do any at all, only comes online when power drops.
https://community.fs.com/article/line-interactive-vs-online-vs-offline-ups.html
FYI Few downside of an online/double conversion UPS will use extra power if that is something your trying to avoid.
Also some of them will have a 24/7 fan so there will be extra noise.
You should buy a UPS if those things are concerns for you. If not, then don’t.
