I’m in the process of wiring a home before moving in and getting excited about running 10g from my server to the computer. Then I see 25g gear isn’t that much more expensive so I might was well run at least one fiber line. But what kind of three node ceph monster will it take to make use of any of this bandwidth (plus run all my Proxmox VMs and LXCs in HA) and how much heat will I have to deal with. What’s your experience with high speed homelab NAS builds and the electric bill shock that comes later? Epyc 7002 series looks perfect but seems to idle high.
About 30 watts for a old Lenovo Thinkcentre with a i5-6500T and 8 GB RAM in combination with a DAS and 2x2TB HDD’s. I’m currently waiting for parts for my new server I’m building, a small N100 Mini-ITX board with 4x4TB HDD’s that hopefully has a similar power consumption.
Around 100 Watts for
- NAS with 4x3.5" HDD,
- Minisforum HM90 for Proxmox with 2x2.5" HDDs,
- 16 Port TP Link PoE Switch,
- TP Link router
- 2x Raspberry Pi 4b
But everything with gigabit speed. Doesnt need more at home
Minisforum HM90 for Proxmox with 2x2.5" HDDs
How did you connect HDDs to that thing?
Inside the bottom tray you have two cutouts in order to put two 2.5" HDDs/SSDs in
Well look at that, it sure does. Too bad they don’t make those anymore.
I have four Raspberry Pi 4 running, so that’s 15W max each or 60W max total. Usually they consume much much less.
Let’s see…
My servers (tiny/mini/micros) in total are about… 600W or so. Two NASs, about 15-20W a piece.
I spend a out $150/mo in electricity, but my hot water/HVAC/etc are the big power draw. I’d say about $40-50/mo is what I’m spending on powering the servers in my office.
Definitely puts off some heat, but that’s partially because it’s all in one rack, and I’ve got a bunch of other work hardware in there. It’s about 2 degrees warmer in my office than the rest of my home, but I also have air cycling all the time since it’s a single unit HVAC and I need to keep the air moving to keep it all the right temp in the other rooms anyway (AC will come on more often otherwise, even without my rack).
125W (Less than $15/month) or so for
- Ryzen 9 3900X
- 64GB RAM
- 2x4TB NVMe (ZFS Mirror)
- 5x14TB HDD (ZFS RAID-Z2)
- 2.5GBe Network Card
- 5-port 2.5GBe Network Switch
- 5-port 1GBe POE Network Switch w/ one Reolink Camera attached
I generally leave
powerManagement.cpuFreqGovernor = "powersave"in my Nix config as well, which saves about 40W ($4/mo or so) for my typical load as best as I can tell, and I disable it if I’m doing bulk data processing on a time crunch.5950x in an matx board with 15 x 3.5in drives 1 x sata sad 1 x optane u.2 drive (pulls like 10watts) 1 x Nvidia A2000 1 x Lsi 9305 16i 1 x 2.5gbe intel nic 3 x 140 mm fans at full tilt
Runs at like 120 watts at idle, like 220 watts with a good amount of work and peaks at like 320 watts if I make it do a lot of work
What in the world…
Dafuq you doing over there?
My real server (Nextcloud/NAS/several more vm’s) uses 28 Watts on average. In addition, there is one Pi 4B running, and I don’t even know it’s wattage.
I’m planning on replacing the real server with a new one, with lots of cores and approx. 50 Watts then.
Pi4 tend to stick around 5w
For my main server only… If HP iLO is to be believed, averaging around 130W.
Running: deluge, homarr, jellyfin, lidarr, navidrome, nextcloud, prowlarr, sonarr, whoogle and a minecraft server (VM) on TrueNAS Scale.
As for everything else (my router, switch and DNS/DHCP server, which is a separate machine, you can add another maybe 50W on top of it…
- Fujitsu motherboard
- Intel pentium G5600
- 6 HDD (4 x 4 TB 2 x 8 TB) spinned down
- 2 SSD for proxmox
- 6 CT and no VM for now
it runs at 16W mostly idle
Actually if anyone has advice I’d love to hear. My server is a b450 mobo with an athlon 320ge. Even with no hard drives spinning it uses 65watts. I don’t understand how it is possible. 6 hard drives bring it to 85-90. Running truenas if it matters
I keep it off most of the time to safe power, I was expecting especially with such a cou much lower wattage, like under 20 idle. My assumption is the power supply can’t get low enough.
What CPU governor are you using? I saved about 40W idle powerdraw switching to powersave vs the default on a Ryzen 9 3900X.
Is that a bios or truenas setting? The CPU is only 35 watt max
Operating system so TrueNAS in your case
The last time I checked, mine runs at about 5-10 watts usually.
- Intel i7-3770
- 16gb DDR3
- 2 1TB SSDs
Are you sure. I was thinking those specs you would be more in the 50-80 watts range.
Yep, my homeserver spends most of it’s time idling, so power management kicks in.
Now when one of my build VMs are running, it’ll get up to that range, but that’s why I said it runs at 10 watts usually
deleted by creator
About 120W total for:
- 2 Proxmox hosts with 4 spinning disks between them
- Opnsense firewall
- 24 port GbE switch
- Fiber ONT
- Unifi AP
5 node proxmox cluster (each node on 40gbps networking[yes ceph…], ~80TB of SSD storage, 180cores, ~630GB of ram total)
1 slow storage node (~400TB)
2x opnsense servers in HA
2x icx7750s
2x icx7450sPoE to all the things… and 8gbps internet.
Usually run ~15-17amps. So about 2000 watts. It’s my baby datacenter.
Sometime this month I’ll be installing 25000kwh solar system on my roof and batteries.
As far as heat goes… It’s in the garage with an insulated door, heat pump water heater, and there’s a tripplite ac unit in the bottom of the rack. The waste air(from the a/c) exhausts outside through a direct vent in the wall. The garage is downright tolerable to me for extended periods of time. The servers don’t complain at all.
Reading about all you guys being under 200w or whatever makes me wonder if it’s worth it. Then I realize that the cost to do even a 1/4 of what I do in the cloud is more expensive than buying my solar.
Power costs for the rack is about $100-120 a month. If it wasn’t for solar.
Edit: 75 LXC containers, 22VMs.
Edit: 75 LXC containers, 22VMs.
That’s a lot of power draw for so few VMs and containers. Any particular applications running that justify such a setup?
That’s total draw of the whole rack. No indicative of power per vm/lxc container. If I pop onto management on a particular box it’s only running at an average of 164 watts. So for all 5 processing nodes it’s actually 953 watts (average over the past 7 days). So if you’re wanting to quantify it that way, it’s about 10W per container.
Truenas is using 420 watts (30 spinning disks, 400+TiB raw storage…closer to 350 usable. Assuming 7 watts per spinning drive were at 210Watts in disks alone, spec sheet says 5 at idle and 10 at full speed). About 70 watts per firewall. Or 1515 for all the compute itself.
The other 1000-ish watts is spent on switches, PoE (8 cameras, 2 HDHR units, time server and clock module,whatever happens to be plugged in around the house using PoE). Some power would also be lost to the UPS as well because conversions aren’t perfect. Oh and the network KVM and pullout monitor/keyboard.
I think the difference here is that I’m taking my whole rack into account. Not looking at the power cost of just a server in isolation but also all the supporting stuff like networking. Max power draw on an icx7750 is 586Watts, Typical is 274 according to spec sheet. I have 2 of them trunked. Similar story with my icx7450s, 2 trunked and max power load is 935W each, but in this case specifically for PoE. Considering that I’m using a little shy of 1k on networking I have a lot of power overhead here that I’m not using. But I do have the 6x40gbps modules on the 7750.
With this setup I’m using ~50% of the memory I have available. I’m 2 node redundant, and if I was down 2 nodes I’d be at 80% capacity. Enough to add about 60GB more of services before I have to worry about shedding load if I were to see critical failures.
Damn that’s a setup alright!
If you’re making use of the hardware it’s well worth it over anything cloud based for sure.
Just out of curiosity, what do you use all that storage for?
On the Sata SSD ceph storage. That’s just live stuff on the containers/vms. I’m at 20% usage of the 70TiB usable at the moment. I don’t use it all that heavily. Because of the way ceph works it’s really ~23 TiB of usable space and ~4.5 TiB written since it writes 3 copies in my cluster.
On the slow storage node it’s running Truenas with 28 spinning disks at 16TB each. 2 hot spares, and 2 ssds each for cache, log, and metadata (eating up total of 36 bays). That’s 342.8TiB usable after raidz nonsense. And I’m 56% usage. I have literally everything I’ve done that I cared to save from like 2005 or 2006 or so. Backups for the ceph storage (PBS). Backups for computers I’ve had over the years. Lots of linux ISOs(105TiB) archived, including complete sets of gaming (37TiB) variants. Oh and my full steam library as well which currently sits at 14TiB. Flashpoint takes up a few TiB as well…
I have a small setup for some self hosted apps and media.
- Beelink Mini S.
- 2 external 5TB drives.
- A USB fan used as an exhaust because the SSD inside gets a bit warm.
I think total power is about 30W.
