Mixing Self-Hosted/Cloud-Hosted: Thoughts on cost effectiveness, availability, reliability? (i.e. External Proxy/Ingress Controller or ControlPlane on AWS w/ Backend Nodes on my own hardware) - eviltoast

I am completely new to Kubernetes, (getting ready to take the CKA exam in December)

I want to host a cluster to get some practice maintaining production-hardened infrastructure

I also don’t want to pay out the nose for cloud services before I really know what I’m doing.

I’ve installed Kubernetes from scratch a few dozen times now using Kubeadm on virtual machines and various hardware that I have lying around at home.

Here is the setup that I want to build:

A control plane node on AWS that also hosts an ingress controller with a public static IP address (needed for TLS certificates and a domain name)

A few RaspberryPi’s and an old Linux Laptop connected to a route to act as worker nodes

Applications:

  • Heimdall
  • Gitea
  • Wordpress for Blogging and Elementor
  • Kubernetes Dashboard
  • Prometheus
  • JupyterLab
  • A NAS to host PersistentVolumes
  • Postgres with WAL enabled for hot standby

Can anyone comment on this setup? Does it make sense from a cost-savings perspective? Are there performance considerations that I should be aware of? I would likely initialize Kubelet myself on an Ubuntu EC2 instance rather than using EKS because I plan to have my cluster mixed between cloud and self-hosted, but is there a better way to handle this?

Thanks in advance for your input!

​

  • stefantigro@alien.topBEnglish
    1·
    1 year ago

    What exactly do you wish to get input on? Reliability, cost effectiveness, availability? You also mention maintaining production hardened hardware?

    Let’s start somewhere with one of these.

    Availability of services in cluster is reliant on many things:

    • Load Balancing
    • Rolling updates
    • Replication
    • Scaling
    • etc

    Reliability is ensuring efficient monitoring, good recovery and fallback mechanisms, self healing, etc. Realistically you need to make sure you have n amount of replicas of a service in multiple physical locations, all backed up.

    Cost effectiveness involves optimizing resource utilization, scaling efficiently, and managing infrastructure in a way that maximizes value for the resources consumed.

    My suggestion:

    1. Automate creation of the cluster. If you want to learn industry standards, you can take a look at terraform or ansible and one of the public cloud providers. K8s was designed to run on a cloud provider ( even tho it can also be ran locally ofc ).
    2. Deploy a CSI
    3. Ingress
    4. Certificates and ssl
    5. Deploy a monitoring solution and make sure that it survives you unplugging one of the servers from the plug.
    6. Repeat step 5 for other apps and stumble through the unique challenges that you’ll face with scaling most of them

    Side Note: take a look at k3s