K3s Setup

Pre-reqs

  1. Install OS on machines, Debian iso’s
  2. We need to configure static ip’s for these machines and for metallb
  3. SSH access

I gave you debian as an example because most instructions are written for ubuntu, and ubuntu is based on debian, just with extra garbage preinstalled. If it turns out we need any of that we’ll get it when we need it. A lot of people, probably including your ai, are going to recommend Talos, Because it’s written for kubernetes. But there’s less examples of using it on the internet. So more security but harder to work with. Do not use anything Arch based, it’s funner but it breaks alot and I don’t think it’s going to be a good fit for you.

When you do the install on these machines. We probably want a desktop for 1 machine. That machine will be our server node, and we can use it to control your cluster, run your database, look at metrics and all that. Any additional machines we use for the cluster can be headless to save on resources. If that doesn’t make sense, then just ask your ai while you do this.

When you run this through your AI, couple things to add to your prompt. You are going to want to verify the hash on the iso, ask your ai how to do that. Pretty common vulnerability in linux is downloading OS’s with additional stuff in them. We verify the sha256 hash to ensure we aren’t grabbing extra stuff like virus’s or spyware

When you run the ssh commands, you’ll get a public key and a private key. If your private key makes it on the internet in any way, then we want to delete it, make a new pair, and then replace public keys anywhere we might have put them.

Do not give the private key to claude or your buddy. If you want to set remote access to the system, then we can do that later with tailscale

Also if you want me to do all this with/for you then we can do that too. Just not gonna give you copy and paste commands because I have all this set up already.

  • Commands for ssh ssh-keygen just take defaults.
  • copy the public key from your host machine to your remote ssh <username>@<local ip> first time you need a password
  • paste the public key at ~/.ssh/authoized_keys

K3s install

Server node

  • curl -sfL https://get.k3s.io | sh -s - server --write-kubeconfig-mode 644 --disable servicelb <- for the control/server node
  • sudo cat /etc/rancher/k3s/k3s.yaml > ~/.kube/kubeconfig <- for putting the kubeconfig in the defualt location
  • sudo cat /var/lib/rancher/k3s/server/node-token <- for grabbing the token to add worker nodes

Worker node

  • curl -sfL https://get.k3s.io | K3S_URL=https://<ENTER YOUR SERVER IP>:6443 K3S_TOKEN=<ENTER YOUR NODE-TOKEN> sh -s -

Test from server node

Metallb

  • kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.16.1/config/manifests/metallb-native.yaml <- install loadbalancers
  • kubectl get pods -A <- ensure you can see the loadbalancers
  • create an ipaddresspool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: adress-pool
  namespace: metallb-system
spec:
  addresses:
  - <change to your cidr range>
  • kubectl apply -f ipaddresspool.yaml <- set the cidr range for your loadbalancers
  • creat l2config.yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: adress-pool
  namespace: metallb-system
  • kubectl apply -f l2config.yaml <- apply l2config

Testing it

  • mkdir test
  • cd test
  • create deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
  • kubectl apply -f deploy.yaml <- apply deployment
  • create service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: my-nginx
  • kubectl apply -f service.yaml <- apply service
  • kubectl get svc <- check if you have a service and that it’s running on an ip
  • kubectl get pods -w <- wait for pods to come up, ctrl+c to exit this watch screen

Clean up test

  • kubectl delete -f deploy.yaml
  • kubectl delete -f service.yaml

A note about my notes

I’m running into some issues with my OS of choice here. I’m going to switch over to debian, this will mean i won’t have screen recording until I get around to setting it up.

istio

I didn’t take real good notes for this. We want istio for our ingress/egress in and out of our deployed applications. Later we’ll make k8 resources for it but for now I ran these commands:


curl -L https://istio.io/downloadIstio | sh -
sudo mv bin/istioctl /usr/bin/
istioctl
istioctl install --set profile=default -y

After it all installs let’s make sure we have all the correct resources. I’m gonna paste my output below:

# kubectl get svc -A

NAMESPACE        NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                      AGE
default          kubernetes                    ClusterIP      10.43.0.1       <none>          443/TCP                                      4h41m
istio-system     istio-ingressgateway          LoadBalancer   10.43.92.106    192.168.1.201   15021:30629/TCP,80:32725/TCP,443:30697/TCP   4h8m
istio-system     istiod                        ClusterIP      10.43.6.143     <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP        4h9m
istio-system     istiod-revision-tag-default   ClusterIP      10.43.56.157    <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP        4h8m
kube-system      kube-dns                      ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP                       4h41m
kube-system      metrics-server                ClusterIP      10.43.120.98    <none>          443/TCP                                      4h41m
kube-system      traefik                       LoadBalancer   10.43.3.87      192.168.1.200   80:30685/TCP,443:31688/TCP                   4h39m
metallb-system   metallb-webhook-service       ClusterIP      10.43.195.185   <none>          443/TCP                                      4h19m

Up there there in the services, the one we are looking for has the type Load Balancer, it’s in the istio-system namespace and it’s name isistio-ingressgateway. You see the External-IP there, that’s where we redirect our router network to go and we’ll just use port 443:80.

# kubectl get pods -A

NAMESPACE        NAME                                      READY   STATUS      RESTARTS        AGE
istio-system     istio-ingressgateway-5fdd5ff474-8s7pc     1/1     Running     0               38m
istio-system     istiod-5688cf5cb9-9c6tf                   1/1     Running     1 (38m ago)     4h10m
kube-system      coredns-5f5694d56b-vzvws                  1/1     Running     1 (38m ago)     4h42m
kube-system      csi-nfs-controller-fccdc96d7-gc7qc        5/5     Running     9 (23m ago)     4h6m
kube-system      csi-nfs-node-xhs8x                        3/3     Running     3 (38m ago)     4h6m
kube-system      csi-nfs-node-xncrf                        3/3     Running     3 (38m ago)     4h6m
kube-system      helm-install-traefik-crd-fcrgg            0/1     Completed   0               4h42m
kube-system      helm-install-traefik-w9vdg                0/1     Completed   1 (4h41m ago)   4h42m
kube-system      local-path-provisioner-58d557dc48-xm6zk   1/1     Running     1 (38m ago)     4h42m
kube-system      metrics-server-7c86f97b8d-fsgxk           1/1     Running     1 (38m ago)     4h42m
kube-system      traefik-6cd8c7cd89-mz7zb                  1/1     Running     1 (38m ago)     4h41m
metallb-system   controller-658745d67-mjq6r                1/1     Running     1 (38m ago)     4h20m
metallb-system   speaker-tlllg                             1/1     Running     1 (38m ago)     4h20m
metallb-system   speaker-xgt7m                             1/1     Running     1 (38m ago)     4h20m

NFS csi server

I also didn’t take good notes for setting up the NFS server, but it’s probably going to be way different for you anyways.

  • Helm Charts, we’ll use these to install it but not yet, first we need to prep the nas.
  • NFS Setup <- this has instructions for tting up our file sharing system on a linux distro. Your NAS you bought is probably going to be different. The steps for some of this will be the same though:
  1. We need to make a partition or share with open permissions available to everyone, or atleast the users we designate.
  2. We need to expose that share/partition to a cidr range with some permission sets.
  3. Then we need to install the K8 resources with helm.
  4. And finally we build a storage class object for it.
  5. Test it

Helm 

helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --version 4.12.0

For the storga Class object, it’s gonna be better we just provide your AI with the information about the share/partition you made and let it give make it, on that note we’ll have your ai make a pvc resource to test it as well.

Database

This might end up being the hard part for you. I’m going to deploy a database on the cluster. We might consider hosting this differently on your setup if anything goes wrong here. This is a part where we’d start setting stuff up for your application as well. I’m not deploying my own app right now, so I’m going to be deploying a photo server instead. So this is an example, not a step by step copy and paste this time.

Helm install of cnpg

From helm docs

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update

helm upgrade --install cnpg \
  --namespace cnpg-system \
  --create-namespace \
  cnpg/cloudnative-pg

K3s Setup

Comments

Loading comments...

Post a Comment

Youtube Transcripts

Static IP example for john

And I want you >> Hey, John. This is my how I set up my static IP. So, this is my router's typically you probably are at the same address. There's a couple other addresses your your router might be at, but 192.168.1.1 is pretty common place for the routers to be. >> [clears throat] >> Anyways, what we have here, what I want to show you is I found a spot on here that handles my dynamic leases. That's what DHCP is. This is for the DNS on my local machines. And right here, I said you can use anything for dynamic between 168.1.10 and .100. So, that allows me 90 devices on my local network, which is fine for me. Maybe not for you. May And you can see on my home cluster, right? So, I don't really have a whole lot that I needed. So, I have a static IP for my main machine. And then I have a This is the machine that I'm recording from here. This is my would be my control or server node. And then for my worker node, we have another one right here, right? So, that's This is a Debian machine right here. And this is running workloads for the cluster, which I'm going to tear down and rebuild. And then this one right here is for my load balancer. And so, >> [clears throat] >> we'll get into that later, but address. So, this isn't a physical machine, but it does still have a MAC address. So, and the way these work is every machine in your that you connect can connect to the internet with or you maybe even more than that has a MAC address built into it. And so, anytime the router senses this MAC address, it will assign it this machine. So, if the power's out for a long period of time, it won't come up with a new IP, which is important when you are running your servers that everything routes to the same places all the time. >> [clears throat] >> I crashed.

ssh stuff for jonh

Hey, John. Okay, so what we need to do is we need to set up the SSH key on our uh remote from our host machine. So the way you do that is you're going to run SSH key gen and that's going to create a new file for you in the correct location. So this is uh just the default name and it's uh this is the type of encryption that's going to use which is the just going to do the defaults to give you an example of what it gives here. We get I'm going to rename it to this local directory just so we can see it here real quick. Okay. So we have two two things here. So if we cat out if we look at the contents of what's in test.pub hub. This is the This is what you need to copy and paste. So, you're going to copy and paste this copy this line here, right? And we're going to put that on the remote machine. So, um and again, don't copy what I'm doing from the test here. We're going to remove that remove that. Um what where you want to do it from is when it created when you run SSH key gen you you're just going to take the defaults and this is where it's going to set it. Right? So when I go and look at that at that location. So we go to home.sh we now you can see my two keys here. Right? So this is the one I'm going to open up and copy the contents to. So, put this on your clipboard. Okay. And then we're going to go ahead and SSH into our remote. So, if you looked at my other one when I set up the um the router and I was showing you how I where my router routes were, I that's where you can find your IP address. We could find it from the other machine as well, but I have that one already. So, I'm going to put the username for um that particular machine. And we're going to put the IP address. And um I already set the set set up the SSH login. So I don't have a password, but it should prompt you for a password here. Your AI might tell you to remove the password. But if you want to run all this headless all the time, it's probably good to keep um a password so that you can SSH in without going over there and plugging in a monitor to wherever you have this hooked up at. And so when we look here, we're going to look at the in the home. SSH file and we have this authorize keys and that is where you're going to paste. So, you're going to open up this file and paste the key in there. And then that'll allow us to connect into the remote host. And that's all we're doing there.

k3s install

Hey, John. Uh, I was gonna we were going to do K0, but I couldn't get it to work. So, we're going to go with what I do know. So, to install, we're over here at K3S on the documentation page. And, uh, we can't go with the quick start guide because we need to disable the load balancers. So, um, that comes from in here. We can do this afterwards if it's still on there, but uh I don't remember how to do it. So, we're just going to pass this in to the command. So, the command I'll uh copy and paste it into uh the web page so that we have it. But we're going to run the same configuration command. We're going to pass in the uh configuration with binary right here. So, we have that write-ash config mode 644. This is just uh for the cube config, I think, um to make it so that we don't have to run it in pseudo every time. And then we're going to disable the service load balancers. And we'll let that script run real quick. And it should only take a minute. And then we have to run an a code to we have to run a different script to get the agent of which is what they call it in K3S which would be also the worker note. So yeah, I guess I should have said that this is the host machine. This is our server machine that we're running the command in first. Down here is the is the remote. This is going to be our worker node that we're going to run it in. So, I already have this. I tested this earlier. Um, however, we're going to have a new token and that comes from these commands here. So, you can see you throw in the URL. Um, in this case, I'm just throwing in my IP address and then the port which is that runs on port 6443. And then the password is the K3S token that we're going to get from here. Um, uh, we can that actually it's easiest to find from the quick start guide. So, if we look at the quick start guide, it does tell us where it's at. Says it's at this location. So, I'm going to go ahead and copy this guy and we'll run pseudo cat to get what's in there. And don't worry about expos I'll undo this afterwards so you can see my token. No big deal. But typically probably shouldn't show this in a YouTube video. And we'll run it like this. And this should download and attach it to the cluster. We'll give it a pause so that this can download without you watching it. I'm sorry, Denali. You You cannot lick your your skin off. I know. I know, girl. Made me feel so bad. I feel like she Dad, I'm sorry. I won't do it no more. I'm so sorry, Dad. Cool. Now that that's done, we can grab our cube config file, which you probably are going to that's going to exist right here. So we want to create it so that it's not attached to root because everything in root in this f under Etsy folders is belongs to root. So we're going to cut it instead of copying it and we're going to go ahead and just put it into a cube config file that we can find. You'll notice that I made a folder. That's what this is. So um that's where we're at. Uh, that's not a file. Yes. Okay. So, we're going to export cube config with to this file which we can talk about later and we can set it up so it runs default but cubectl get nodes and here's our server and here's our worker. So says control plane that means it's a server node and says none means it's an agent node. So that's all there is to doing this. The next thing we need to do is install load balancers.

metallb install on k3s

All right. Let's do the load balancers. So, we we installed K3S without load balancers installed, so now we need to install it. So, we're going to go to the MetalLB. This is for installing met- bare metal load balancers onto your Kubernetes cluster. Um and I'm going to go ahead and just install by manifest here. Um if you wanted to be able to see what is in the manifest, we could look at it here. It some There's a lot a ton of Kubernetes stuff in here that is it is what it is. We don't really need to know most of this stuff. I'm being honest with you, so we'll do the configuration afterwards. So, I'm going to go ahead and apply these with the command that's given and we'll do that. Go into my cluster into my cluster's area. I'm going to MetalLB and if we for the sake of keeping track of all of this stuff, um go ahead and curl this. And it gives me the look all the information in there and I going to so I'm going to curl that again, but I'm going to put it into a manifest so we can see it later. Just for record keeping purposes. And we'll go ahead and run a apply on that. And you could just copy this command. In fact, that's what we'll do for this one. Uh-oh. Connection refused. >> Oh, man. What's going on? Okay. No problem. Turned out it was uh when I ran cube CTL, it wasn't doing what it wanted, so I went back and made sure that I am grabbing my cube config so that when we run our cube CTL commands, it goes to the right place. So, I made sure that we are doing that there. Now that we have it, now we should be able to go back to MetalLB and run this command. There it goes. So, now we have the custom resource definitions. And I'd use this one, the lightweight one. So, let's go ahead and hop over into our L2 configuration. Um >> [groaning] >> is this correct? So, now I run cube CTL get pods and we're going to do {dash} capital A to get them in all the name spaces. We see that we have our metal load balancers running. That's what we want. And we need to um if we are looking at this, the next thing we need to do is assign a set of IPs. And when I look at my old uh notes that I had, I did do the same thing, so uh first pool is the name. Um I don't think it matters what we call it. Let's go ahead and just set this up, so we're going to set up our IP address pool .yaml. We'll save that in there, and this is where we need to make sure we get our static IPs in there. >> [clears throat] >> So, I don't want this huge cider range. So, what I'm going to do is we'll go to chat GPT. And remember, I have a small selection of ciders that I want. So, I'm going to go ahead and I think it's installed on 111, but we're going to go ahead and give it a uh 115 to 120. Um give me a cider range for 192.168.1. 115 to 111 192.168.1. 1 125. Let's see what that says. So, it's telling me to use So, we'll go ahead and put this in there. Oops. Delete that one. Um I don't think the name matters, so we're just going to call this calico. And we'll go ahead and exit out of that. And we're going to apply kubectl calico IP address pool. >> [clears throat] >> And then we need this L2 configuration thing, which this is the one I have here, so um L2 pool config.yaml. Um I think yeah. That's what we use. So So now we can apply that. Cool. And I don't know why I had this in here, but let's go ahead and pause the video and see what this is about. Yeah, okay. So I'm looking at it and it's it was just how I was doing test. So I'm going to go ahead and do the kubernetes.io and we'll go ahead and run uh test to make sure that it's coming up on the load balancer correctly. So Let's see here. Let's make a test. I'm not going to keep any of this, so nginx. Uh deploy.yaml. So we'll make that and then we're going to create the service. And we're going to go ahead and change some of the service on here instead of for the end we want to We're going to go ahead and select type load balancer. And that should be all we have to change there, so kubectl apply check deploy kubectl my dash app. service Um So, this command is checking and waiting for this thing to pop up. And so, what we did is running Engine X as the image. It's going to bring up two replicas. We really only care about the one. Um And we have my Engine X is going to be showing up here at this IP address. Um All right. So, they're up. And now we're going to go ahead and check out this IP address. If this works, then we have a working load balancer. Welcome to Engine X. Look at that. So, that's it for this. We now have a load balancer. The next thing we can do is install We'll be needing to get our uh network routing set up. So, that's going to be Istio. And then the cert managers, and then after that, all you have to do is deploy your apps. So, we're coming close to the end.