Many of the posts I read here are about Docker. Is anybody using Kubernetes to manage their self hosted stuff? For those who’ve tried it and went back to Docker, why?

I’m doing my 3rd rebuild of a K8s cluster after learning things that I’ve done wrong and wanted to start fresh, but when enhancing my Docker setup and deciding between K8s and Docker Swarm, I decided on K8s for the learning opportunities and how it could help me at work.

What’s your story?

  • You999@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    I’ve spent the last two weeks on getting a k3s cluster working and I’ve had nothing but problems but it has been a great catalysts for learning new tools like ansible and load balancers. I finally got the cluster working last night. If anyone else is having wierd issues with the cluster timing out ETCD needs fast storage. Moving my VMs from my spinning rust to a cheap SSD fixed all my problems.

  • thekernel@kbin.social
    link
    fedilink
    arrow-up
    6
    arrow-down
    1
    ·
    1 year ago

    I like the concept, but hate the configuration schema and tooling which is all needlessly obtuse (eg. helm)

    • Anonymouse@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      Helm is one of the reasons I became interested in Kubernetes. I really like the idea of a package where all I have to do is provide my preferences in a values file. Before swarm was mature, I was managing my containers with complicated shell scripts to bring stuff up in the right order and it became fragile and unmaintainable.

  • pixxelkick@lemmy.world
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    I have a K3OS cluster built out of a bunch of raspberry pis, it works well.

    The big reason I like kubernetes is that once it is up and running with git ops style management, adding another service becomes trivial.

    I just copy paste one if my e is ting services, tweak the names/namespaces, and then change the specific for the pods to match what their docker configuration needs, ie what folders need mounting and any other secrets or configs.

    I then just commit the changes to github and apply them to the cluster.

    The process of being able to roll back changes via git is awesome

  • orizuru@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    13
    ·
    1 year ago

    Seems a bit overkill for a personal use selfhosting set-up.

    Personally, I don’t need anything that requires multiple replicas and loadbalencers.

    Do people who have homelabs actually need them? Or is it just for learning?

    • egonallanon@lemm.ee
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 year ago

      I find mine useful as both a learning process and as a thing need. I don’t like using cloud services where possible so I can set things up to replace having to rely on those such as next loud for storage, plex and some *arr servers for media etc. And I think once you put the hardware and power costs vs what I’d pay for all the subs (particularly cloud storage costs) it comes out cheaper at least with hardware I’m using.

      • orizuru@lemmy.sdf.org
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 year ago

        Yes, those are all great uses of it. But could all still be achieved with docker containers running on some machines at home, right?

        Have you ever had a situation where features provided by kubernetes (like replicas, load balancers, etc) came in handy?

        I’m not criticizing, I’m genuinely curious if there’s a use-case for kubernetes for personal self-hosting (besides learning).

        • MigratingtoLemmy@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          I was a big proponent of k3s in the homelab, but I’m starting to think otherwise these days. I still expel choice words towards Docker’s networking, but it starts becoming more of a philosophical issue with what the company is doing and whoever decided this kind of networking is nice.

          Is the networking on Podman any better? I understand using k8s at home to learn, but what if you don’t care about learning? I have never seen a point to k8s in homelabs other than in home-datacentres, and I’m starting to veer away from k3s too, since I don’t need extreme HA over 3 machines for my services (I would have used Proxmox if I wanted that).

          Yeah, could someone give me a primer on how Podman is better than Docker? I’m adamant that I don’t want to use anything with the name “Docker” in my lab.

    • Anonymouse@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 year ago

      For me, I find that I learn more effectively when I have a goal. Sure, it’s great to follow somebody’s “Hello World” web site tutorial, but the real learning comes when I start to extend it to include CI/CD for example.

      As far as a use case, I’d say that learning IS the use case.

    • kelvie@lemmy.ca
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 year ago

      A lot of people thought this was the case for VMs and docker as well, and now it seems to be the norm.

      • orizuru@lemmy.sdf.org
        link
        fedilink
        English
        arrow-up
        4
        ·
        1 year ago

        A lot of people thought this was the case for VMs and docker as well, and now it seems to be the norm.

        Yes, but docker does provide features that are useful at the level of a hobbyist self-hosting a few services for personal use (e.g. reproducibility). I like using docker and ansible to set up my systems, as I can painlessly reproduce everything or migrate to a different VPS in a few minutes.

        But kubernetes seems overkill. None of my services have enough traffic to justify replicas, I’m the only user.

        Besides learning (which is a valid reason), I don’t see why one would bother setting it up at home. Unless there’s a very specific use-case I’m missing.

  • alphafalcon@feddit.de
    link
    fedilink
    English
    arrow-up
    7
    ·
    1 year ago

    I run a 2 node k3s cluster. There are a few small advantages over docker swarm, built-in network policies to lock down my VPN/Torrent pod being the main one.

    Other than that writing kubernetes yaml files is a lot more verbose than docker-compose. Helm does make it bearable, though.

    Due to real-life my migration to the cluster is real slow, but the goal is to move all my services over.

    It’s not “better” than compose but I like it and it’s nice to have worked with it.

  • ahto@feddit.de
    link
    fedilink
    English
    arrow-up
    6
    ·
    1 year ago

    I love kubernetes. At the start of the year I installed k3s von my VPS and moved over all my services. It was a great learning opportunity that also helped immensely for my job.

    It works just as well as my old docker compose setup, and I love how everything is contained in one place in the manifests. I don’t need to log in to the server and issue docker commands anymore (or write scripts / CI stages that do so for me).

    • sour@feddit.de
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Are most of your services just a single pod? Or do you actually have them scaled? How do you then handle non-cloud-native software?

  • mlg@lemmy.world
    link
    fedilink
    English
    arrow-up
    6
    ·
    1 year ago

    I am insane and use bare bone LXC.

    Stupid ramblings you can probably ignore:

    spoiler

    Usually though it’s because I run most stuff bare metal anyway so LXC is for temporary or random cases where I need a weird dependency or I want to run a niche service.

    Only use docker for when I actually want faster setup like docker-osx which does all the vm stuff for running a virtual Mac for you.

    I don’t really mind docker, but for homelab I just find myself rewriting dockerfile anytime I want to change something which I don’t really need to do if I’m not publishing it or even reusing it.

    Kubernates is really more effective for actual load services, which you never need in homelab lol. It’s great to use to learn k8s cluster, but the resources get eaten fast.


  • RegalPotoo@lemmy.world
    link
    fedilink
    English
    arrow-up
    12
    arrow-down
    1
    ·
    1 year ago

    Kubernetes is awesome for self hosting, but tbh is superpower isn’t multi-node/scalability/clustering shenanigans, it’s that because every bit of configuration is just an object in the API, you can really easily version control everything - charts and config in git, tools like Helm make applying changes super easy, use Renovate to do automatic updates, use your CI tool of choice to deploy on commit, leverage your hobby into a DevOps role, profit

  • Oliver Lowe@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Here’s a slightly different story: I run OpenBSD on 2 bare-metal machines in 2 different physical locations. I used k8s at work for a bit until I steered my career more towards programming. Having k8s knowledge handy doesn’t really help me so much now.

    On OpenBSD there is no Kubernetes. Because I’ve got just two hosts, I’ve managed them with plain SSH and the default init system for 5+ years without any problems.

  • tungtungss@lemmy.wanderia.online
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    Used k3s to manage my single instance. Lots of gotcha moments to learn! Will add Flux for CD after I decide on how to self-host the Git server

  • dap@lemmy.onlylans.io
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Running an RKE cluster as VMs on my ceph+proxmox cluster. Using Rook and external ceph as my storage backend and loving it. I haven’t fully migrated all of my services, but thus far it’s working well enough for me!

  • Avid Amoeba@lemmy.ca
    link
    fedilink
    English
    arrow-up
    6
    ·
    1 year ago

    Docker with or without Compose and systemd is good enough for most of my use cases. SaltStack is good enough for config-as-code.

  • PriorProject@lemmy.world
    link
    fedilink
    English
    arrow-up
    15
    ·
    edit-2
    1 year ago

    I use k8s at work and have built a k8s cluster in my homelab… but I did not like it. I tore it down, and currently using podman, and don’t think I would go back to k8s (though I would definitely use docker as an alternative to podman and would probably even recommend it over podman for beginners even though I’ve settled on podman for myself).

    1. K8s itself is quite resource-consuming, especially on ram. My homelab is built on old/junk hardware from retired workstations. I don’t want the kubelet itself sucking up half my ram. Things like k3s help with this considerably, but that’s not quite precisely k8s either. If I’m going to start trimming off the parts of k8s I don’t need, I end up going all the way to single-node podman/docker… not the halfway point that is k3s.
    2. If you don’t use hostNetworking, the k8s model of traffic routes only with the cluster except for egress is all pure overhead. It’s totally necessary with you have a thousand engineers slinging services around your cluster, but there’s no benefit to this level fo rigor in service management in a homelab. Here again, the networking in podman/docker is more straightforward and maps better to the stuff I want to do in my homelab.
    3. Podman accepts a subset of k8s resource-yaml as a docker-compose-like config interface. This lets me use my familiarity with k8s configs iny podman setup.

    Overall, the simplicity and lightweight resource consumption of podman/docker are are what I value at home. The extra layers of abstraction and constraints k8s employs are valuable at work, where we have a lot of machines and alot of people that must coordinate effectively… but I don’t have those problems at home and the overhead (compute overhead, conceptual overhead, and config-overhesd) of k8s’ solutions to them is annoying there.

    • whyrat@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      arrow-down
      1
      ·
      1 year ago

      I’d suggest Podman over docker if someone is starting fresh. I like Podman running as rootless, but moving an existing docker to Podman was a pain. Since the initial docker setup was also a pain, I’d rather have only done it once :/

      For me the use case of K8s only makes sense with large use cases (in terms of volume of traffic and users). Docker / Podman is sufficient to self-host something small.