pfSense on an old PC with two NICs should do well. You could buy dedicated hw like a protectoli. Ive had one for 6 years now no issues.

Fritzboxes are rock stable, and support Wireguard from FritzOS 7.5 onwards, see https://avm.de/service/vpn/wireguard-vpn-zur-fritzbox-am-computer-einrichten/

(Apparently NOT the cable versions!)

What nags me most with them is that you have no separate Firewall controll over their WiFi, and the WiFi range is not really great. So probably consider going with dedicated APs instead.

I have been using opnsense on a very cheap celeron nuc for a few years, very happy with it

I’m very happy with my FritzBox (7590), it handles de ADSL connection to the ISP, supports various DDNS providers, Wireguard VPN, 4 port gigabit switch (5 of you don’t need the WAN port), guest WiFi with client isolation.

It also has basic media server and NAS functionality (with USB3 external hard drives).

Of course you can change the DNS server and other network controls like QOS, wake on LAN, port forwarding, different profiles with parental controls, filters, connection times, etc.

They also seem to take security seriously.

I personally would flick through the OpenWRT supported devices and pick the best supported device with 802.11ax.

You haven’t mentioned what sort of access link or speed you have, that seems very relevant here.

For my 1Gbit/s fiber connection the Edgerouter 6P has been pretty good. It has an SFP port and can route 1 Gbit/s of traffic without issue and my dual-stack setup works well too.

The only significant downside is that its switching is slow, it has no hw support. So I put my NAS on a separate subnet instead so that the traffic to it can be routed instead.

Adding another Mikrotik recommendation with the standard warnings – a bit of a learning curve, although it has a default configuration that “just works”. If you mess something up you can just apply the default config to get back online.

Don’t buy from Amazon. For whatever reason people have problems with those units. Fakes maybe? Who knows. If you’re in the US buy from streakwave, roc-noc, ISP supplies, Double Radius, or Getic (international shipping).

The RB5009 series is very good if you want something beefier with more ports.

Hi-end Xiaomi router, they have WiFi ax and enough ram and support open wrt so you can host your things on it, better yet, do DIY router on orange pi board, there’s tutorials you can follow

I can kind of recommend Firewalla. They run all open source software under the hood, but their UI is their own. I’m not super impressed with some of the decisions they’ve made, but it works and has almost every feature a firewall/router device needs.

Things I like

• VPN client support with selective VPN routing. Beats having to manually maintain a routing table for a VPN interface.

• SSH access with sudo to root

• comes with an Ad blocker, but can run pihole in a docker container. I find the onboard ad blocker paired with NextDNS via TLS is good enough.

Things I don’t like:

• UI is a phone app. WebUI is neutered. You will require all three (SSH included) to set up any advanced configs

• SSH access is a pain to use.

• Firewall rule creation is kind of a nightmare. I can see what they were going for, but they missed.

• You can’t easily configure the onboard IDS or Adblocker. You can dive into the filesystem if you want, but I don’t wanna.

I have a mesh system made up of Asus Zenwifi ET8s, and I have been very happy with them. They have a lot of cool features, such as having a VPN server and VPN client, with the VPN client allowing me to apply the VPN to only selected devices. It has tons of customization options for those that are knowledgeable about that sort of thing. For example, I can tweak at what signal strength AP steering happens. It has WiFi 6E and 2.5 Gbps wired backhaul.

When I first got it, it was very buggy, and some features straight up didn’t work. But they eventually got all the bugs that I found fixed. It’s in a really good state right now.

To address your desired features, it does have wireguard. I don’t know about DDNS, but it does not have pihole built in. It has adguard built in, but it doesn’t really seem to do much, tbh. Then again, pihole didn’t really do anything for me either. I ended up shutting off my pihole because I didn’t even notice a difference.

If you want to start small, I’d go with one supported by Asuswrt-Merlin, “a third party alternative firmware for Asus routers, with a special emphasis on tweaks and fixes rather than radical changes or collecting as many features as possible.” Keeps it close to stock with minor upgrades, and a faster release cycle for fixes. The RT-AX88U_PRO is one of the higher end routers that is supported by Merlin.

When I reached your situation, I started rackmounting which has saved me a lot of time.

I got a 1u dell poweredge r210 and slapped in a 10Gb network card. Loaded up OPNsense onto it. OPN sense was not easy to learn how to use, for me at least. Struggled to get everything running smoothly. But I am very happy I went with rack mounting instead of adding to the rat’s nest.

A raspberry pi with pihole running as dhcp server. In the ISP router turn off DHCP, DNS problem solved as pi will advertise its DNS to all connected devices on the network.

I guess maybe too mainline for everyone here but I use an Asus router flashed with the Merlin OS (a painless easy process) and it works excellently. No issues setting up all the things you mentioned.

How much wifi and open-source do you really want?

If you are willing to go with commercial hardware + open source firmware (OpenWrt) you might want to check the table of hardware of OpenWrt at https://openwrt.org/toh/views/toh_available_16128_ax-wifi and https://openwrt.org/toh/views/toh_available_864_ac-wifi. One solid pick for the future might be the Netgear WAX2* line or the GL.iNet GL-MT6000. One of those models is now fully supported the others are on the way. If you don’t mind having older wifi a Netgear R7800 is solid.

For a full open-source hardware and software experience you need a more exotic brand like this https://www.banana-pi.org/en/bananapi-router/. The BananaPi BPi R3 and here is a very good option with a 4 core CPU, 2GB of RAM Wifi6 and two 2.5G SFP ports besides the 4 ethernet ports. There’s also an upcoming board the BPI-R4 with optional Wifi 7 and 10G SPF.

Both solutions will lead to OpenWRT when it comes to software, it is better than any commercial firmware but be aware that it only support wifi hardware with open-source drives such as MediaTek. While MediaTek is good and performs very well we can’t forget that the best performing wifi chips are Broadcom and they use hacks that go behind the published wifi standards and get it go a few megabytes/second faster and/or improve the range a bit.

DD-WRT is another “open-source” firmware that has a specific agreement with Broadcom to allow them to use their proprietary drivers and distribute them as blob with their firmware. While it works don’t expect compatibility with newer hardware nor a bug free solution like OpenWRT is.

There are also alternatives like OPNsense and pfSense that may make sense in some cases you most likely don’t require that. You’ve a small network and OpenWRT will provide you with a much cleaner open-source experience and also allow for all the customization you would like. Another great advantage of OpenWRT is that you’ve the ability to install 3rd party stuff in your router, you may even use qemu to virtualize stuff like your Pi-Hole on it or simply run docker containers.