I recently set up a personal Owncast instance on my home server, it should do what you’re looking for. I use OBS Studio to stream random stuff to friends, if your webcam can send RTMP streams it should be able to stream to Owncast without OBS in the middle - else, you just need to set up OBS to capture from the camera and stream to Owncast over RTMP.

the communication itself should be encrypted

I suggest having the camera/OBS and Owncast on the same local network as RTMP is unencrypted and could possibly be intercepted between the source and the Owncast server, so make sure it happens over a reasonably “trusted” network. From there, my reverse proxy (apache) serves the owncast instance to the Internet over HTTPS (using let’s encrypt or self-signed certs), so it is encrypted between the server and clients. You can watch the stream from any web browser, or use another player such as VLC pointing to the correct stream address [1]

it seems that I might need to self-host a VPN to achieve this

Owncast itself offers no authentication mechanism to watch the stream, so if you expose this to the internet directly and don’t want it public, you’d have to implement authentication at the reverse proxy level (HTTP Basic auth), or as you said you may set up a VPN server (I use wireguard) on the same machine as the Owncast instance and only expose the instance to the VPN network range (with the VPN providing the authentication layer). If you go for a VPN between your phone and owncast server, there’s also no real need to setup HTTPS at the reverseproxy level (as the VPN already provides encryption)

Of course you should also forward the correct ports (VPN or HTTPS) from your home/ISP router to the server on your LAN.

There are also dedicated video surveillance solutions.