WEBVTT

00:00:01.010 --> 00:00:10.010
yes and so the session records itself as well off of my VM I believe that they

00:00:10.010 --> 00:00:18.830
stay up for a week or two weeks inside the DA desktop so tonight I'll push up

00:00:18.830 --> 00:00:26.770
the instructions for building your own mini cube on a VM so that you can

00:00:26.770 --> 00:00:32.090
practice on your own and we'll go through why we use minikube for training

00:00:32.090 --> 00:00:36.630
on kubernetes and then you can go through this course as well for it's

00:00:36.630 --> 00:00:46.470
either one or two weeks it'll stay up recorded so and get started so I've been

00:00:46.470 --> 00:00:52.990
a kubernetes instructor since 2016 I built this kubernetes cluster as part of

00:00:52.990 --> 00:01:00.030
a government project years ago Google had I don't know who funded it one of

00:01:00.030 --> 00:01:07.390
agencies funded it for Google to build their own container orchestrator and part of that funding

00:01:07.390 --> 00:01:17.390
they spun out a binary which became Kubernetes eventually and it was out for I think among

00:01:17.390 --> 00:01:25.470
Google engineers in 2014 and early 2015 and then they released it to the public sometime in the

00:01:25.470 --> 00:01:31.950
summer of 2015 that's when i was asked to figure out how to build a cluster so all we had was a

00:01:31.950 --> 00:01:42.590
binary um and a make file for i believe it was red hat um like version i don't know five or something

00:01:44.670 --> 00:01:49.390
and that was it no instructions no nothing um you had to figure out how to build your own control

00:01:49.390 --> 00:01:56.110
plane. And you had to figure out how to build your own worker node. And no one could figure

00:01:56.110 --> 00:02:05.130
out how to build a highly available control plane. So it was a lot of fun. We did competitions among

00:02:05.130 --> 00:02:09.770
government contractors to see who could build the first Kubernetes cluster.

00:02:11.690 --> 00:02:22.250
So that was the early days and it's progressed a lot. So I design, engineer and build production

00:02:22.250 --> 00:02:29.210
great clusters and I train clients and instruct them on using infrastructure as code and declarative

00:02:29.210 --> 00:02:36.970
GitOps. So the course this week, we'll have our introduction lessons. There will be nine lessons

00:02:36.970 --> 00:02:43.370
and we'll have course feedback at the end. Lesson one will be on container architecture. It's a basic

00:02:43.370 --> 00:02:51.530
overview of containerization and how it then works within the Kubernetes cluster. We'll have an

00:02:51.530 --> 00:02:59.440
overview of Kubernetes architecture, the Kubernetes command line tools, and the

00:02:59.440 --> 00:03:05.580
Kubernetes core concepts. Lesson 5 will get into the more detailed networking,

00:03:05.580 --> 00:03:14.790
config maps, and secrets. And then Lesson 6 will discuss how scheduling interacts

00:03:14.790 --> 00:03:22.770
with the cluster. Lesson 7 will discuss Helm charts, operators, and storage within

00:03:22.770 --> 00:03:28.470
the cluster. Lesson eight, Kubernetes security and automation, which is where

00:03:28.470 --> 00:03:35.530
it'll begin to tie in for your own cluster that I built in RKE2. And then

00:03:35.530 --> 00:03:40.550
lesson nine, logging and monitoring. And then we'll have our course feedback and final

00:03:40.550 --> 00:03:47.450
Q&A. This will be four days of interactive training and about 70% of the

00:03:47.450 --> 00:03:52.470
course will be labs, so it'll be hands-on. And the DMA desktop works really well

00:03:52.470 --> 00:04:01.240
well for that it allows me to see your terminal assuming that your pixel size is you know large

00:04:01.240 --> 00:04:06.740
enough on your terminal i can kind of see what you're typing in and and help you catch any typos

00:04:06.740 --> 00:04:15.400
or mistakes so it'll be four hours of instruction per day with a 30 minute break so we'll do two

00:04:15.400 --> 00:04:20.920
hour session a 30 minute break and then another two hour session and start at 11 a.m each day

00:04:20.920 --> 00:04:28.520
and finish up around 3.30. So we'll be using the Minikube Kubernetes environment. It enables many

00:04:28.520 --> 00:04:34.920
of the production feature sets without requiring us to spin up multiple VMs.

00:04:37.640 --> 00:04:44.360
It allows us to demonstrate HA mode as well as multiple nodes at the same time. We can also

00:04:44.360 --> 00:04:53.160
just spin up a single node and so to start up if you could search your menu for your terminal

00:04:53.160 --> 00:05:02.840
i believe it's going to be a mate terminal so up in the top left corner you click on menu and just

00:05:02.840 --> 00:05:10.760
type in terminal um it should pop up and you may want to extend that across the um your entire

00:05:10.760 --> 00:05:19.880
screen. Just as we type in commands and read the results, it'll take up most of the width of the

00:05:19.880 --> 00:05:26.840
desktop. And it shouldn't require a username or password, but if it does, it's minicoop minicoop.

00:05:26.840 --> 00:05:33.510
Right, first thing, we're going to check the versions, make sure everything is working correctly.

00:05:35.030 --> 00:05:42.070
And you should just need to type in minicoop space version. Yeah, sometimes my screen delays here.

00:05:44.150 --> 00:05:51.270
there you go uh so the terminal if you just type in the word terminal i think on the right it'll

00:05:51.270 --> 00:05:56.710
pop up mate terminals that's what's on mine yeah there you go and you might want to extend that

00:05:56.710 --> 00:06:04.470
across the width of your screen just as we read the output that way it doesn't truncate okay 1.37

00:06:04.470 --> 00:06:13.130
is correct and we're going to check our coop control version and it should see one dot yep

00:06:13.130 --> 00:06:21.130
That's correct. And that is because MiniKube is on 1.3.4, and so we want to keep our kube control

00:06:21.130 --> 00:06:36.150
version within one version of that. So the next is a Helm version. Let's see 3.19.3. Now there

00:06:36.150 --> 00:06:43.270
is a Helm version 4 out, but it is not really tested with a lot of upstream Helm charts.

00:06:43.270 --> 00:06:51.670
So you could, if you download, for example, a Snap with Ubuntu, I think it installs Helm 4 by default.

00:06:52.610 --> 00:06:53.610
You just want to be careful.

00:06:54.150 --> 00:07:00.410
It's 4.0.0, which in Kubernetes means you're basically the experiment.

00:07:02.030 --> 00:07:08.770
So I'll wait until they get a little further along on Helm before I jump to 4.0.0.

00:07:08.770 --> 00:07:17.010
but 3.19 is fairly mature. It should work with pretty much every home chart. And then we'll

00:07:17.010 --> 00:07:22.730
check our Docker version. We should see 29, I believe. And last, I believe we're going to check

00:07:22.730 --> 00:07:31.750
the Cilium version. The cluster that I built for you uses Cilium, and it provides its own CLI.

00:07:31.750 --> 00:07:44.150
and we should see 0.18.9 and it pulls its own default image as part of the CLI so it's on

00:07:44.150 --> 00:07:51.670
1.18.3 and I believe cluster wise around 18.4 the latest is 18.5 so

00:07:53.110 --> 00:07:59.110
yes and that's because we're not connected to a cluster and so when we run cilium version it

00:07:59.110 --> 00:08:05.350
expects us to be connected to the cluster and to provide information inside the cluster since it's

00:08:05.350 --> 00:08:12.070
not connected it can only tell us the cli version and what the latest stable version is and what

00:08:12.070 --> 00:08:18.470
it's using for its own image so if we were running psyllium inside the cluster it would provide us

00:08:18.470 --> 00:08:30.070
with that last one as well so now we've verified our environment uh we are going to start up

00:08:30.070 --> 00:08:38.150
a Minikube cluster. Has anyone worked with Minikube before? Okay. I assume it's probably...

00:08:39.910 --> 00:08:49.190
Okay. Okay. Yeah. So Minikube is a great way to learn Kubernetes. And again, this evening,

00:08:49.190 --> 00:08:53.510
I'll push up how to spin up your own Minikube environment in a Linux environment. I believe

00:08:53.510 --> 00:09:01.270
you can even install it on macOS and Windows. Just be advised if you're a Windows user,

00:09:01.830 --> 00:09:06.870
there's a lot of tooling with Kubernetes that does not work in a Windows environment. So Linux

00:09:07.590 --> 00:09:15.750
by far has the best environment for your Kubernetes tooling. MacOS is okay and Windows is

00:09:15.750 --> 00:09:22.310
probably the least favorable for managing the Kubernetes cluster. And so yeah, so I think that

00:09:22.310 --> 00:09:27.690
And also because the cloud-native foundation and the way it started, it started with Red Hat.

00:09:28.210 --> 00:09:35.410
So back in 2016, we installed it on Enterprise Red Hat and then CentOS, which was the free open-source version.

00:09:35.870 --> 00:09:38.930
So everything was Linux when the tooling was being developed.

00:09:41.490 --> 00:09:50.350
And so I believe you can run Kubernetes on Windows now, but clients have told me that the tooling just does not work correctly.

00:09:50.350 --> 00:09:57.230
you can you you know not all of the feature sets of the tooling are available so i don't have a

00:09:57.230 --> 00:10:03.810
windows environment so i don't have any way to test that but how i work with clusters because

00:10:03.810 --> 00:10:13.530
i work with quite a few with clients is i build a vm specifically with ubuntu and i access that

00:10:13.530 --> 00:10:20.090
cluster wherever it is whether it's a cloud or bare metal i access it through my vm and then

00:10:20.090 --> 00:10:24.950
And then when I'm done, you know, with whatever maintenance work I'm doing or building a cluster,

00:10:24.950 --> 00:10:27.190
I shut my VM back down.

00:10:27.190 --> 00:10:34.330
So that VM is always a Linux-based Ubuntu VM, which allows all of the tooling to work.

00:10:34.330 --> 00:10:46.210
So once you type this command in, let's see, I see, and we're going to verify the creation.

00:10:46.210 --> 00:10:48.710
Yeah, you can also abbreviate it.

00:10:48.710 --> 00:10:55.030
So in this case, we did double hyphen driver equals Docker and double hyphen container

00:10:55.030 --> 00:11:02.470
runtime equals container D. We can also abbreviate some of the flags with Minikube, hyphen D

00:11:02.470 --> 00:11:04.710
for driver and hyphen C for container runtime.

00:11:04.710 --> 00:11:07.010
We're going to verify that it was successful.

00:11:07.010 --> 00:11:12.790
So the kubectl get nodes, Minikube works automatically with kubectl.

00:11:12.790 --> 00:11:19.030
automatically places our kube config in the proper area so that kube control can

00:11:19.030 --> 00:11:24.010
work directly with it. So that's the nice thing about learning Kubernetes with

00:11:24.010 --> 00:11:31.990
Minikube. This makes a lot of the processes. And you should see a single node and it

00:11:31.990 --> 00:11:37.690
should be named minikube when you run kube control get nodes. And it is a control

00:11:37.690 --> 00:11:47.130
plane let's see yeah okay and uh yeah so what is your say three days 21. okay you know we're

00:11:47.130 --> 00:11:51.930
gonna blow that out here in a minute and i'll show you how to do that uh yeah okay so

00:11:54.650 --> 00:11:59.450
now we're going to check our pods so coop control get pods and we're going to use a hyphen capital

00:11:59.450 --> 00:12:05.930
a and the hyphen capital a denotes all namespaces now we can of course type it out with a double

00:12:05.930 --> 00:12:12.330
hyphen and then all namespaces but um hyphen capital a to show and we can see we have a core

00:12:12.330 --> 00:12:25.370
dns an etcd um we have a kind net which is minicoob's networking overlay very very basic

00:12:25.370 --> 00:12:33.450
overlay we have our api server we have our controller we have the older kube proxy which

00:12:33.450 --> 00:12:38.250
we'll talk about later on why modern clusters generally don't use kube proxy

00:12:40.170 --> 00:12:46.890
you can see kube scheduler and then we have storage provision as well all right now we are

00:12:46.890 --> 00:12:55.050
going to blow this out so we're going to use minikube stops and once that has stopped then

00:12:55.050 --> 00:13:01.290
we'll use minikube delete hyphen hyphen all and what this does is this deletes the minikube profile

00:13:02.170 --> 00:13:09.290
so that it doesn't spin up this profile again. We'll be using different Minicoob profiles

00:13:09.290 --> 00:13:13.930
throughout the course and so we want to make sure that we always, after we stop the Minicoob,

00:13:14.810 --> 00:13:22.250
that we delete that profile. So if you were to type Minicoob start, I believe it would start

00:13:22.250 --> 00:13:28.650
back up with that same profile that had been running. Now it depends on what, what's that?

00:13:28.650 --> 00:13:35.530
If you didn't rename it, so I believe that minicube is the standard name, and then you can

00:13:35.530 --> 00:13:41.050
also add your own name for the profile. So you could technically have a profile listing of

00:13:41.050 --> 00:13:47.550
multiple profiles. So you could create a profile for a single node. You could create a profile for

00:13:47.550 --> 00:13:55.550
an HA. You could create a profile for multiple worker nodes. And then you could just spin those

00:13:55.550 --> 00:14:00.150
but you would have to tell it the name of the profile to spin that up.

00:14:00.350 --> 00:14:02.730
So it does give you the ability to have multiple profiles,

00:14:03.050 --> 00:14:07.750
but in order to make sure that we don't run into issues

00:14:07.750 --> 00:14:11.070
with spinning up the wrong profile for the course,

00:14:11.150 --> 00:14:14.330
we just generally delete everything afterwards.

00:14:14.990 --> 00:14:18.630
So any questions on the Minicoop environment?

00:14:34.980 --> 00:14:38.320
Yeah, so the upstream Kubernetes SIG team,

00:14:38.320 --> 00:14:44.680
uh actually it's an amazing engineering feat what they did it's a kubernetes and docker

00:14:44.680 --> 00:14:51.460
and getting the network to work correctly uh took a lot of engineering on their part

00:14:51.460 --> 00:14:57.760
so i think they did a great job with minikube it's essentially you're running it on docker

00:14:57.760 --> 00:15:08.040
um and you're spinning up a minikube container and then each of those minikube containers

00:15:08.040 --> 00:15:15.080
is named so the first the first container is a node so it's minicube and then the next one is

00:15:15.080 --> 00:15:21.900
o2 so it's hyphen o2 which we'll see in a minute and so each container is its own node which has

00:15:21.900 --> 00:15:30.920
docker inside of it and runs its own kubernetes node so and when you think about it if you've

00:15:30.920 --> 00:15:36.460
ever tried to do this with docker on your own machine the networking does not cooperate with

00:15:36.460 --> 00:15:43.100
docker network and so they did a lot of what i just referred to as magic you know under the hood

00:15:43.100 --> 00:15:49.420
in order to enable the networking to work yes it is a kubernetes cluster if you spin it up and

00:15:49.420 --> 00:16:02.810
we'll we'll spin up one um as a availability here in a few minutes and also multiple worker

00:16:02.810 --> 00:16:08.330
nodes as well. Okay, so now lesson one. Let's start with container architecture and for some

00:16:08.330 --> 00:16:13.050
of you this may be fairly basic and for others it may be the first time you're seeing this.

00:16:14.810 --> 00:16:20.730
So a container is a standard unit of software that packages up code and all of its dependencies

00:16:21.530 --> 00:16:25.610
and enables it to run quickly and reliably across computing environments.

00:16:27.820 --> 00:16:34.380
So container images become containers at runtime. And the Docker container technology launched in

00:16:34.380 --> 00:16:42.220
2013. Kubernetes technically launched in 2015. Containers and VMs have similar resource isolation.

00:16:43.580 --> 00:16:49.660
VMs virtualize hardware and networking. There's an abstraction of physical hardware turning one

00:16:49.660 --> 00:16:54.940
server into many. The hypervisor allows multiple VMs to run on a single machine

00:16:55.740 --> 00:17:04.220
and each VM includes a full copy of an operating system. VMs can take up to tens of gigabytes

00:17:04.220 --> 00:17:09.900
of space and VMs are slow to boot. Containers virtualize the operating system. There's an

00:17:09.900 --> 00:17:15.500
abstraction at the aft layer that allows you to package code and dependencies together.

00:17:16.620 --> 00:17:24.140
Multiple containers can run on the same machine, so the same VM, and they share the same OS kernel

00:17:24.940 --> 00:17:27.320
as isolated processes.

00:17:27.320 --> 00:17:29.040
They also take up less space

00:17:29.040 --> 00:17:31.840
than the more efficient resource utilization.

00:17:31.840 --> 00:17:33.540
And that's why the industry began

00:17:34.880 --> 00:17:39.460
turning towards containerization instead of virtualization

00:17:40.480 --> 00:17:45.420
exclusively back in that 2013 to 2015 time.

00:17:45.420 --> 00:17:49.460
The container images are lightweight standalone packages.

00:17:49.460 --> 00:17:52.000
The container engine is responsible

00:17:52.000 --> 00:17:54.200
for maintaining the container lifecycle.

00:17:54.200 --> 00:17:58.140
Container runtime handles the execution of containers

00:17:58.140 --> 00:18:00.440
and the interaction with the host OS.

00:18:00.440 --> 00:18:03.100
And then we have a container management system

00:18:03.100 --> 00:18:06.960
that automates the deployment, scaling, and management.

00:18:06.960 --> 00:18:10.340
The container management system is where we get into.

00:18:10.340 --> 00:18:14.380
Container management systems optimize resource utilization

00:18:14.380 --> 00:18:16.580
and streamline our workflows.

00:18:17.660 --> 00:18:20.380
They enable faster development cycles

00:18:20.380 --> 00:18:22.020
and quicker deployments.

00:18:22.020 --> 00:18:28.180
They allow for easier scaling of applications on demand, they can be used across multiple

00:18:28.180 --> 00:18:32.380
environments such as bare metal or in the cloud, and they enable automation of both

00:18:32.380 --> 00:18:34.180
routine and complex tasks.

00:18:34.180 --> 00:18:40.860
And if you have questions, feel free to throw them out or feel free to discuss.

00:18:40.860 --> 00:18:46.920
That's why we have the teams meeting up so we can communicate back and forth.

00:18:46.920 --> 00:18:50.040
So OAST, go ahead.

00:18:50.040 --> 00:18:55.040
So Docker provides its own runtime and engine.

00:18:55.040 --> 00:19:04.400
So when you run, for example, let's say you're using Docker Compose on your own laptop, Docker

00:19:04.400 --> 00:19:07.860
would provide the engine and the runtime.

00:19:07.860 --> 00:19:14.800
But with Kubernetes, the Docker runtime had its own issues, and so Kubernetes went to

00:19:14.800 --> 00:19:17.020
container D.

00:19:17.020 --> 00:19:23.580
they started splitting out the engine and the runtime and we'll see that in

00:19:23.580 --> 00:19:30.260
one of our next few slides where we can actually change the container runtime

00:19:30.260 --> 00:19:34.320
and the engine although I don't believe we're gonna change the engine on any of

00:19:34.320 --> 00:19:41.740
our tasks. Yes and Kubernetes switched to container D from the Docker container

00:19:41.740 --> 00:19:45.720
runtime. In container D, Docker switched

00:19:45.720 --> 00:19:49.760
to that as their underlying. Keep in mind, Docker had their

00:19:49.760 --> 00:19:53.820
own Swarm product as a container

00:19:53.820 --> 00:19:57.920
management. And that is pretty much deprecated now, I think.

00:19:58.600 --> 00:20:02.100
It's kind of hard to find an engineer

00:20:02.100 --> 00:20:05.960
who can build something with Docker Swarm, since some of the features

00:20:05.960 --> 00:20:09.820
just aren't as well supported today. Kubernetes has

00:20:09.820 --> 00:20:17.000
pretty much, I think, taken that over. Sure, I can discuss that. So I use Docker Compose.

00:20:17.320 --> 00:20:23.420
And so I develop, when I want to develop containers for Kubernetes, I'll write them

00:20:23.420 --> 00:20:29.720
with a Docker file. And then I'll use Docker Compose to spin that up. So I'll have a Docker

00:20:29.720 --> 00:20:35.660
Compose for development and a Docker Compose for production. Now, technically, that's cheating.

00:20:35.660 --> 00:20:42.980
and the reason why is docker-compose was written for docker swarm i don't use docker swarm it's

00:20:42.980 --> 00:20:50.820
deprecated for the most part but i do use the docker-compose for development and it allows me

00:20:50.820 --> 00:20:59.580
to feed in variables with an emd file and spin it up similarly to the way that it is going to run

00:20:59.580 --> 00:21:07.000
within kubernetes and then i can build that image and and store that image and then i can build a

00:21:07.000 --> 00:21:15.420
helm chart and i can pull that image in from my image repo so docker compose is very useful

00:21:15.420 --> 00:21:22.760
from a development standpoint and if you were running the old days docker swarm you would use

00:21:22.760 --> 00:21:31.580
Docker Compose to run that particular, as you say, cars, cars project, you would you would have used

00:21:31.580 --> 00:21:37.480
Docker Compose to run that inside a Docker swarm. But we kind of cheat today, we just use Docker

00:21:37.480 --> 00:21:44.260
Compose to spin it up for development purposes and tests. Docker Compose in production, I have

00:21:44.260 --> 00:21:51.400
actually done that on a single VM on a VM. Yeah. And it provides an excellent way to do that. And

00:21:51.400 --> 00:21:55.340
And like I said, I use Docker Compose for development for Kubernetes.

00:21:56.220 --> 00:22:00.020
But what you will find when you run this in production with Docker Compose

00:22:00.020 --> 00:22:05.300
is the Docker network overlay is very limited.

00:22:05.880 --> 00:22:13.020
It's limited in throughput, and it's also limited from a firewall standpoint

00:22:13.020 --> 00:22:17.340
because it literally takes control over your IP tables

00:22:17.340 --> 00:22:19.180
and your networking inside the VM.

00:22:19.180 --> 00:22:28.440
and so if you're exposing that on a production cluster to the public you'll

00:22:28.440 --> 00:22:33.000
want to put a web application firewall in front of Docker so that's one of the

00:22:33.000 --> 00:22:40.320
downsides of running Docker Compose in production on a VM is the way that the

00:22:40.320 --> 00:22:47.520
firewall interacts the firewall doesn't actually work with Ubuntu or any of the

00:22:47.520 --> 00:22:53.220
other distros. So just kind of as a heads up, I don't know if you've run into that

00:22:53.220 --> 00:22:59.040
yet, but if you put a web application firewall in there as well, and if you

00:22:59.040 --> 00:23:07.200
could use an NGINX WAF, so they actually build that into NGINX, then it

00:23:07.200 --> 00:23:13.920
will automatically block the malicious requests, which is what the firewall

00:23:13.920 --> 00:23:20.000
should do but it doesn't and the reason for that is because docker is designed for development

00:23:20.000 --> 00:23:28.500
and if you can imagine you know what hundreds of thousands or millions of developers out there

00:23:28.500 --> 00:23:35.800
when they're first starting out trying to install docker on their machine and then deal with the

00:23:35.800 --> 00:23:40.580
networking issues so that's why when you install docker you have to have root level access

00:23:40.580 --> 00:23:43.500
is it automatically takes control over the network.

00:23:44.320 --> 00:23:48.020
It has root-level access, and it defeats the firewall.

00:23:48.800 --> 00:23:51.140
And that eliminated a lot of complaints from developers.

00:23:51.320 --> 00:23:53.560
Hey, Docker doesn't work. I installed it.

00:23:53.660 --> 00:23:55.120
I can't reach the outside network.

00:23:55.620 --> 00:24:00.920
That solved the problem, but it was designed for development purposes

00:24:00.920 --> 00:24:02.940
only and not for production.

00:24:03.480 --> 00:24:06.760
You can run it in production as long as you have a WAP in front of it

00:24:06.760 --> 00:24:08.140
to protect your application.

00:24:08.140 --> 00:24:15.080
so I don't know if you've run into that yet but yeah and you can just swap out yeah and there's an

00:24:15.080 --> 00:24:22.840
OWASP version of NGINX where they they bring in the OWASP best practices into the NGINX layer

00:24:22.840 --> 00:24:29.240
and it has a WAF built in and then you can set that up and you want to really test it out in

00:24:29.240 --> 00:24:34.640
development first because you will trigger a lot of rules with your application and you need to

00:24:34.640 --> 00:24:37.820
determine whether that rule should be triggered or whether you should bypass

00:24:37.820 --> 00:24:43.890
that rule. And then that way you can use Docker Compose in

00:24:43.890 --> 00:24:47.790
production. Just keep in mind you'll be limited on the number of containers that

00:24:47.790 --> 00:24:52.210
can effectively run because they can overload the actual network overlay.

00:24:52.210 --> 00:25:04.220
Alright, let's go to OCI and CRI. OCI is the Open Container Initiative and it's a

00:25:04.220 --> 00:25:13.160
So Linux Foundation Project provides low-level specs, it standardizes container runtimes,

00:25:13.160 --> 00:25:19.620
images and registries, it defines image specs, runtime specs and distribution specs.

00:25:19.620 --> 00:25:28.660
It is not specific to Kubernetes and it provides fundamental building blocks and standardization.

00:25:28.660 --> 00:25:34.780
So this is what you would see in your Docker underlying running Docker Compose on.

00:25:34.780 --> 00:25:41.560
And then we have a CRI, which is our Container Runtime Interface.

00:25:41.560 --> 00:25:50.640
This is a high level spec, unlike the OCI, it is specific to Kubernetes and it enables

00:25:50.640 --> 00:25:58.560
kubelet to use container runtimes that support v1 of the CRI API, enables communication with

00:25:58.560 --> 00:26:06.480
container runtime on each node and it builds on the OCI standards to provide orchestration

00:26:06.480 --> 00:26:14.320
and management for containerized applications at scale. Kubelet uses a gRPC connection to the

00:26:14.320 --> 00:26:21.440
CRI implementation. This allows for a runtime service which gives you version, run pod sandbox,

00:26:21.440 --> 00:26:25.840
you're probably not familiar with any of these commands because we don't run these commands

00:26:25.840 --> 00:26:35.220
with KubeControl or with Docker. So version, run pod, sandbox, stop pod, sandbox. It's a

00:26:35.220 --> 00:26:41.180
different concept. These are treated as sandboxes within Kubernetes. Remove pod

00:26:41.180 --> 00:26:45.900
sandbox, create container, start container, stop container, remove container. For

00:26:45.900 --> 00:26:52.340
image service, pull image, list images, image status, and remove image. And

00:26:52.340 --> 00:26:58.640
And probably no one has run these commands before inside of Kubernetes, but we're going to here in a second.

00:26:59.380 --> 00:27:20.000
So how this works, the kubelet using gRPC interacts with a container runtime interface, or CRI, which uses a container D SOC to connect to the container runtime, which is container D for Kubernetes, which then controls the pod.

00:27:20.120 --> 00:27:23.200
And all of this takes place within the individual nodes.

00:27:23.200 --> 00:27:25.960
And so the kubelet is key here.

00:27:25.960 --> 00:27:28.960
And how do we ensure a fresh Minikube environment?

00:27:28.960 --> 00:27:38.520
Yeah, so we minikube stop, and then we minikube delete, hyphen, hyphen, double hyphen, all,

00:27:38.520 --> 00:27:40.240
and then minikube start.

00:27:40.240 --> 00:27:42.240
Correct, it should not be.

00:27:42.240 --> 00:27:45.200
So you should just need to run minikube start.

00:27:45.200 --> 00:27:53.000
And once it's up, you're going to get the nodes and determine what container runtime

00:27:53.000 --> 00:28:02.940
used and I'll be right back in just a few seconds guys all right so what what

00:28:02.940 --> 00:28:08.640
container runtime do we see we see docker yeah so we see docker for the

00:28:08.640 --> 00:28:17.070
container runtime right yeah all right so how do we ensure a fresh piniku

00:28:17.070 --> 00:28:21.890
environment so we'll be doing this a lot through the class yep stop and delete

00:28:21.890 --> 00:28:30.130
all and in fact you can hit the up button up arrow on your terminal and it'll probably pull up the

00:28:30.130 --> 00:28:35.730
stop one and then the delete all it'll save you some time typing in what we're going to do this

00:28:35.730 --> 00:28:41.810
time is so to ensure a fresh minicoop environment we just stop and delete all and then we're going

00:28:41.810 --> 00:28:48.530
to be spinning up different types of environments throughout the course and so to ensure a fresh

00:28:48.530 --> 00:28:53.090
environment, we just stop and delete all. And then on the next slide, I'll usually give you the

00:28:53.090 --> 00:28:58.850
command for what type of cluster we'll be starting. Yes, if you did minicoop start, it would be the

00:28:58.850 --> 00:29:08.540
wrong one for this one. So this one's going to be minicoop start, and then we'll hyphen c for

00:29:08.540 --> 00:29:16.940
container runtime. And we're going to use container d, which is what Kubernetes. Yeah, so

00:29:16.940 --> 00:29:26.720
so dash o stands for options so we give it a dash o and we tell it we're going to give it an option

00:29:26.720 --> 00:29:34.480
and the option is wide and if you notice the difference between get nodes and and then adding

00:29:34.480 --> 00:29:42.460
to the next command you can try that um you can do kubectl get nodes without the option

00:29:42.460 --> 00:29:48.920
so just kubectl get nodes and then type it in with the option you see that when

00:29:48.920 --> 00:29:56.500
we get yeah so it more information in this case there's a bug with minikube

00:29:56.500 --> 00:30:02.920
it with this particular version of Ubuntu it reports the wrong version but

00:30:02.920 --> 00:30:09.700
um generally minikube reports the correct version and it gives us the

00:30:09.700 --> 00:30:18.140
kernel that we're running with Linux and the container runtime. Occasionally you

00:30:18.140 --> 00:30:25.080
will run into issues with major upgrades of Kubernetes where it will provide a

00:30:25.080 --> 00:30:33.380
specific kernel version that you need to be using. For example, Cilium uses a lot

00:30:33.380 --> 00:30:39.220
of cutting-edge technology that runs at the kernel level and that's why Cilium

00:30:39.220 --> 00:30:48.100
can get away with doing a lot of things that weren't possible before. So one of

00:30:48.100 --> 00:30:53.440
the things, if you're experiencing issues, you can run the option of wide and

00:30:53.440 --> 00:30:57.280
check your kernel version. It's a quick and easy way to do that while you're

00:30:57.280 --> 00:31:05.670
using your kubectl. It also gives you your internal IP for each node. So if

00:31:05.670 --> 00:31:09.690
you're kind of used to naming your nodes based on IP address as well,

00:31:10.330 --> 00:31:16.890
that allows you to to see what that ip is okay yes so if we were using like a node port

00:31:17.930 --> 00:31:25.050
um i believe it would populate that okay we already have container d running and we're

00:31:25.050 --> 00:31:30.250
going to use cry control to inspect our cluster this time so since we already have this running

00:31:31.130 --> 00:31:37.850
we've already verified we are now going to ssh into our node so our node name is minicoob

00:31:37.850 --> 00:31:45.230
and Minikube provides a very convenient way to SSH into it. Minikube SSH. So we'll

00:31:45.230 --> 00:31:51.350
Minikube SSH in and we're going to sudo cry control images. And if you've ever

00:31:51.350 --> 00:31:57.890
run Docker images you'll know that this is very similar to running Docker images.

00:31:57.890 --> 00:32:04.530
And let's view the running containers. sudo cry control ps. Again very similar

00:32:04.530 --> 00:32:09.630
running Docker PS on your development environment. And so how would we view the

00:32:09.630 --> 00:32:15.690
pods that are running using cry control? Well we would do sudo cry control pods.

00:32:15.690 --> 00:32:21.350
Very similar to the result that we would see if we ran kubectl get pods. You

00:32:21.350 --> 00:32:25.890
can also start up a container and run it with cry control. Now we're not going to

00:32:25.890 --> 00:32:30.870
do that but this is the command that you would use. Cry control run would give it

00:32:30.870 --> 00:32:39.930
your option container config and then our pod config and we could run that

00:32:39.930 --> 00:32:47.550
internally. I'm going to check the runtime endpoint for CRY control. So we'll

00:32:47.550 --> 00:32:56.360
cat etsy CRY control. And our runtime endpoint is, remember how we talked about

00:32:56.360 --> 00:33:01.220
we were connecting with the container dsoc. So CRY control connects with the

00:33:01.220 --> 00:33:08.060
container D SOC to container D. So that is the runtime endpoint that it is using.

00:33:08.060 --> 00:33:16.390
So, quick overview. Kubelet uses gRPC to connect to the container runtime

00:33:16.390 --> 00:33:23.050
interface within Kubernetes. This connects to the container D SOC, which

00:33:23.050 --> 00:33:28.630
connects to the container runtime, which is container D, which then allows it to control

00:33:28.630 --> 00:33:34.950
the pod and all this takes place within each individual node so does everyone understand how

00:33:36.150 --> 00:33:40.630
the pods are controlled within the node does anyone have any questions about how this works

00:33:40.630 --> 00:33:46.630
within kubernetes it's the magic of kubernetes and how it works under the hood kind of takes

00:33:46.630 --> 00:33:53.910
the mystery out of it i think where you see it's very similar to running you know docker commands

00:33:53.910 --> 00:34:02.390
on your own development machine. And Kubernetes works very similarly under the hood.

00:34:02.390 --> 00:34:09.750
All right, so for our review, we went through container architecture, how images become

00:34:09.750 --> 00:34:17.270
containers at runtime, discuss containers versus VMs with respect to resource utilization,

00:34:17.270 --> 00:34:24.230
how container management systems work the oci versus cri and what does oci designate

00:34:24.230 --> 00:34:29.670
initiative close very close open container initiative and it is a linux foundation project

00:34:30.550 --> 00:34:38.310
and then what does cri designate correct all right you learned how the kubelet uses a cri to

00:34:38.310 --> 00:34:44.390
communicate with the runtime how to use a cry control tool to communicate with the container

00:34:44.390 --> 00:34:52.710
runtime in Kubernetes, how to check the CRY control runtime endpoint, that tells us what

00:34:52.710 --> 00:35:00.030
SOC we're connecting to, and how to use CRY control to debug Kubernetes runtime issues

00:35:00.030 --> 00:35:01.030
from inside.

00:35:01.030 --> 00:35:11.910
Yeah, so I can provide the slides, I'll push up the code.

00:35:11.910 --> 00:35:19.410
So today we'll have a good portion of labs but the code we'll be running will just be

00:35:19.410 --> 00:35:30.290
a few lines of yaml as we study syntax and then tonight i'll push up the code for the rest of the

00:35:30.290 --> 00:35:36.370
course the yaml files will get pretty lengthy 20 30 lines of yaml so you don't have to type those

00:35:36.370 --> 00:35:44.770
out you'll be able to just access that through gitlab and you can download it to your desktop

00:35:44.770 --> 00:35:50.610
or however you want to bring it in tomorrow morning and then i'll make the slides available

00:35:51.490 --> 00:35:56.450
after the course is over i'll do a pdf that way if i have to make any corrections to it

00:35:56.450 --> 00:36:02.290
and those current corrections will be in there and you'll have pdfs for the um for the actual

00:36:02.290 --> 00:36:10.610
slides and lessons so there'll be nine lessons um and then that way you can kind of go through

00:36:10.610 --> 00:36:24.030
the pdfs with your own minikube instance if you want and if you see any typos please let me know

00:36:24.030 --> 00:36:30.990
i'm using libreoffice which works really well with ubuntu but it does occasionally crash and

00:36:30.990 --> 00:36:42.240
i found that whatever slide it crashes on it can introduce problems with that slide

00:36:42.240 --> 00:36:58.730
typos or now that we're done with containers and bms let's talk about kubernetes that's why we're

00:36:58.730 --> 00:37:05.950
here, right? So what is Kubernetes? Kubernetes is a system that enables the automated deployment

00:37:05.950 --> 00:37:13.810
and management of containerized applications. K8 derives its name from the first letter K,

00:37:14.450 --> 00:37:20.890
last letter S, and eight letters in between. And Kubernetes was designed to be a cost-effective

00:37:20.890 --> 00:37:26.350
alternative for managing containerized workloads. When I was first asked to build a Kubernetes

00:37:26.350 --> 00:37:31.510
Cluster is part of a government project. My PI, which are called primary

00:37:31.510 --> 00:37:38.350
investigators, couldn't even pronounce it. I think we ended up laughing trying to

00:37:38.350 --> 00:37:43.230
pronounce Kubernetes. Apparently we weren't the only ones and someone came

00:37:43.230 --> 00:37:48.850
up with calling it K8s. So that's a lot better than trying to pronounce Kubernetes.

00:37:48.850 --> 00:37:53.590
But you can use either one. Kubernetes versus the cloud. Conventional cloud

00:37:53.590 --> 00:37:59.910
providers utilize a team of infrastructure engineers in the background who maintain APIs

00:37:59.910 --> 00:38:04.870
and UIs for cloud engineering tasks. Cloud engineers may then provision cloud infrastructure

00:38:04.870 --> 00:38:13.830
using infrastructure as code or IAC for use by DevOps personnel. In contrast, K8's providers

00:38:13.830 --> 00:38:19.590
utilize a Kubernetes infrastructure engineer to design and automate the creation of a cluster

00:38:19.590 --> 00:38:25.270
involving server and agent nodes using infrastructure as code. And this relates more to

00:38:25.270 --> 00:38:33.310
bare metal. DevOps personnel may then provision new clusters using that IAC and also workloads

00:38:33.310 --> 00:38:40.150
using manifest files or Helm chart. Conventional cloud may use state management tools such as

00:38:40.150 --> 00:38:47.750
Terraform to provision and maintain resources in the cloud. Terraform will enforce the state,

00:38:47.750 --> 00:38:54.950
It will store the state, and when you Terraform apply, it will enforce that state in the cloud.

00:38:55.250 --> 00:38:57.050
This is different from Kubernetes.

00:38:57.590 --> 00:39:02.270
Kubernetes generally maintains its own state within the cluster itself.

00:39:02.370 --> 00:39:06.730
So Terraform may be used to provision virtual machines with the OS layer.

00:39:08.670 --> 00:39:14.390
And then Ansible may be used to install and configure Kubernetes on the OS layer.

00:39:14.390 --> 00:39:24.090
The declarative GitOps may then be used to provision and maintain the state for workloads and databases that are running within the cluster, so our applications.

00:39:26.230 --> 00:39:32.450
The Kubernetes cluster design process involves determining the workload needs along with the available budget,

00:39:34.130 --> 00:39:37.590
engineering a solution that can be automatically provisioned,

00:39:37.590 --> 00:39:48.680
And then coding a solution using infrastructure as code tools, such as Terraform and Ansel's training DevOps personnel on maintaining the cluster.

00:39:49.820 --> 00:39:56.940
And then training DevOps personnel on provisioning workloads using GitOps or declarative GitOps.

00:39:57.080 --> 00:39:59.620
Kubernetes can be provisioned to be resource-friendly.

00:39:59.900 --> 00:40:07.300
It was designed from day one to be maintainable by DevOps personnel and not require an engineer to actually maintain it.

00:40:07.300 --> 00:40:11.100
and it can provide savings versus the cloud of up to 90%

00:40:11.100 --> 00:40:12.640
if you're running on bare metal.

00:40:13.300 --> 00:40:16.840
So the largest savings I've ever seen for a client,

00:40:18.260 --> 00:40:21.600
they were a VC, they had eight tech companies,

00:40:22.260 --> 00:40:25.560
and I think their AWS cloud bill for Kubernetes

00:40:25.560 --> 00:40:28.860
was a little over half a million a year,

00:40:29.860 --> 00:40:34.680
and the VC was able to get that down below 50,000 a year

00:40:34.680 --> 00:40:35.880
running on bare metal.

00:40:35.880 --> 00:40:44.760
So I did not verify, but the client says that they saved about 93% over their AWS

00:40:44.760 --> 00:40:50.520
cloud bill by switching to bare metal. So quite a save. Cloud-native Kubernetes.

00:40:50.520 --> 00:40:57.940
Kubernetes version 1 was released in July of 2015. Rapid growth took place

00:40:57.940 --> 00:41:03.300
three years later with the addition of cloud engineers. So Kubernetes went from just a

00:41:03.300 --> 00:41:08.880
basic binary to cloud-native Kubernetes and that brought in a lot of the cloud

00:41:08.880 --> 00:41:14.540
engineers, particularly the AWS, Rackspace, and IBM cloud engineers at the time.

00:41:14.540 --> 00:41:19.020
The introduction of cloud-native also brought in a host of cloud tools and

00:41:19.020 --> 00:41:24.820
enabled our early adoption and the cluster was designed and engineered by a

00:41:24.820 --> 00:41:29.340
Kubernetes engineer. So our cloud engineers became Kubernetes engineers,

00:41:29.340 --> 00:41:38.950
However, with all of the cloud tooling, this led to a lot of bloat in our Kubernetes clusters.

00:41:39.810 --> 00:41:50.410
So on the one hand, the government sponsored Kubernetes early on, and then private industry had to take it over and basically run the Kubernetes ecosystem.

00:41:50.410 --> 00:42:04.630
So they brought in the cloud engineers and all of the tooling, but Kubernetes then quickly got away from being a simple resource-friendly container orchestrator into a very bloated ecosystem.

00:42:04.630 --> 00:42:30.850
So from 2018 to about 2020, 2021, it was to the point where I was seeing clusters spun up that consumed, you know, six, seven nodes, 20 CPUs, 30 CPUs, vCPUs, and a lot of RAM just for overhead without ever running a single workload.

00:42:30.850 --> 00:42:33.890
And that's not what Kubernetes was designed for.

00:42:34.630 --> 00:42:44.390
And so in 2022, the industry finally, after losing several companies who had switched to Kubernetes and then switched away due to the overhead,

00:42:45.250 --> 00:42:49.850
the industry made the switch from cloud native to Kubernetes native,

00:42:50.510 --> 00:42:58.070
which is basically removing a lot of that cloud tooling that wasn't necessary with Kubernetes and streamlining the clusters,

00:42:58.070 --> 00:43:03.650
making them more efficient and more resource friendly so that you could spin

00:43:03.650 --> 00:43:08.330
them up with smaller BI. Kubernetes release schedule. Kubernetes maintains a

00:43:08.330 --> 00:43:15.130
minimum of three releases at a time. It has one new release every four months.

00:43:15.130 --> 00:43:21.410
The initial .0 and .1 versions are generally for development testing.

00:43:21.410 --> 00:43:26.290
So I best practice, I wait until dot two for production.

00:43:27.430 --> 00:43:33.410
So your cluster is 1.3.4.2 that I built for you with RKE2.

00:43:33.890 --> 00:43:37.410
And so I generally wait for dot two for production.

00:43:38.950 --> 00:43:44.290
Each release receives one year of updates before reaching EOL or end of life.

00:43:45.590 --> 00:43:50.130
And the release schedule generally runs from a dot zero to a dot 13.

00:43:50.130 --> 00:43:59.230
yes yes so the dot zero introduces new concepts um and if you just want to play around with it

00:43:59.230 --> 00:44:02.970
and see if it'll break your cluster there's a chance that there's something in there that

00:44:02.970 --> 00:44:08.790
will break your cluster and so it's designed for you let's say you're a kubernetes engineer and

00:44:08.790 --> 00:44:13.950
you're all caught up and you're saying hey let's go ahead and take a look at the latest version

00:44:13.950 --> 00:44:18.290
if you're the type of engineer that wants to debug and troubleshoot for everybody else

00:44:18.290 --> 00:44:25.090
then that's for you that dot zero is for you right um i've done that i spent years doing that

00:44:25.090 --> 00:44:33.630
and so i don't like to do that um for the dot zeros and um although i did make an exception

00:44:33.630 --> 00:44:39.530
for gateway api but as a general rule my clients don't want to pay me to do that either

00:44:39.530 --> 00:44:46.010
so i wait for dot two just because over the years i know that dot one fixes it's a bug fix

00:44:46.010 --> 00:44:50.490
and .2 generally has the final bug fixes for production.

00:44:51.310 --> 00:44:55.370
So for Kubernetes, I have found that .2 is safe to run in production.

00:44:56.570 --> 00:45:03.630
.0 and .1, you can oftentimes fix it yourself with a lot of work

00:45:03.630 --> 00:45:05.430
or just wait for .2.

00:45:07.290 --> 00:45:12.030
And everything beyond that is a lot of times a backport.

00:45:12.590 --> 00:45:15.530
So for example, when they're working on 1.35,

00:45:16.010 --> 00:45:23.290
look up with something new or add in a security feature and then they'll backport that to the

00:45:23.290 --> 00:45:31.530
older versions. So when you see a .3, .4, a lot of times that will have a backport from something

00:45:31.530 --> 00:45:41.130
and 1.35 for example. So you can see from our release history that looks like they had a 1.35.0

00:45:41.130 --> 00:45:45.450
right before Christmas, so they went ahead and kicked that off

00:45:45.450 --> 00:45:49.790
on schedule. We're on 1.34.2

00:45:49.790 --> 00:45:52.670
for our RKE2 cluster, so 1.34

00:45:52.670 --> 00:45:57.170
is now ready for production. And then

00:45:57.170 --> 00:46:01.390
1.33 is, looks like they're on a

00:46:01.390 --> 00:46:04.830
1.36, so pretty stable, pretty mature.

00:46:06.330 --> 00:46:10.920
That will EOL out June

00:46:10.920 --> 00:46:24.880
28th of 2026 and 1.34 will EOL out on October 27th of 2026. Yeah, and so what I do, and I was

00:46:24.880 --> 00:46:29.400
asked to do this several years ago, so I had kind of figured out something with automation

00:46:29.400 --> 00:46:37.620
and so I use infrastructure as code and that's how I train teams to install Kubernetes. So how

00:46:37.620 --> 00:46:43.140
would do this with yours and we'll do a day of training on infrastructure as code not this week

00:46:43.140 --> 00:46:49.700
but we'll do a day of training on that and we'll do a day of training on declarative git ops but

00:46:49.700 --> 00:46:54.340
on the infrastructure as code we would build a development cluster so we just spin up a dev

00:46:54.340 --> 00:47:01.780
we'd give it the environment variable of dev and we would update the version if we were on 1.33.3

00:47:01.780 --> 00:47:10.560
we would change that to 1.3.4.2 and we would spin it up and we would address any issues that are in

00:47:10.560 --> 00:47:15.660
the release notes. So there may not be any issues at all in the release notes or there may be an

00:47:15.660 --> 00:47:22.900
upgrade issue that we need to take into account. We'd spin it up in dev and we would test that in

00:47:22.900 --> 00:47:30.000
dev. So we do our smoke test initially and then we test it along running with our applications

00:47:30.000 --> 00:47:33.260
And once we feel comfortable with that, it would go to staging.

00:47:33.460 --> 00:47:39.140
If you have a staging environment, not all teams do, but it would go to staging and then to production.

00:47:39.660 --> 00:47:44.620
And that's how we use our infrastructure as code rather than upgrading in place.

00:47:45.140 --> 00:47:51.640
So if you have a full-time Kubernetes engineer, and we'll get into this a little bit later today,

00:47:53.140 --> 00:47:57.180
you can upgrade in place and it will work about 90% of the time.

00:47:57.180 --> 00:47:59.860
And then 10% of the time, the cluster crashes.

00:48:00.000 --> 00:48:15.060
So back in, you know, 2018, 2019, we knew there had to be a better way than just hiring a full-time Kubernetes engineer who sits around in front of the command line all day to rescue your cluster.

00:48:15.060 --> 00:48:29.960
And infrastructure as code has really reduced the operating expense of Kubernetes clusters because it allows us to spin up and test versions without doing live upgrades in place on production clusters.

00:48:30.000 --> 00:48:33.840
Just keep in mind, upgrades in place, as well as adding and removing nodes,

00:48:34.360 --> 00:48:39.520
may have roughly a 90% success rate on a cluster just because of the complexity.

00:48:39.820 --> 00:48:43.460
But with infrastructure as code, we can eliminate that from the equation.

00:48:43.560 --> 00:48:46.820
So yes, you could do an upgrade in place as well.

00:48:47.160 --> 00:48:50.000
There's a procedure involved in that.

00:48:50.000 --> 00:48:54.340
And you could, if you were running 1.33.3 on a long-running cluster,

00:48:54.940 --> 00:48:57.720
you could do an upgrade in place to 1.34.2.

00:48:57.720 --> 00:49:03.560
but 10% of the time that will fail on a long-running cluster.

00:49:03.600 --> 00:49:04.480
Something to keep in mind.

00:49:04.560 --> 00:49:09.200
Yeah, so my job is to try to eliminate those issues for teams

00:49:09.200 --> 00:49:12.720
and show them the easiest path to run Kubernetes

00:49:12.720 --> 00:49:18.180
and try to avoid the long weekends and working all night

00:49:18.180 --> 00:49:20.940
trying to resurrect the cluster, because that's no fun.

00:49:21.020 --> 00:49:22.260
I've done it over the years.

00:49:22.300 --> 00:49:23.960
Did Stanley or Ralph want that?

00:49:23.960 --> 00:49:32.660
So with the RKE2 cluster that I'm building for AWN, it uses infrastructure as code.

00:49:33.280 --> 00:49:38.680
And so you'll be able to run everything with infrastructure as code once you're ready.

00:49:38.860 --> 00:49:41.560
And then it also will utilize declarative GitOps.

00:49:41.560 --> 00:49:46.720
So the idea is it'll make your life as developers a lot easier and as engineers.

00:49:47.080 --> 00:49:49.740
Okay, let's go into Kubernetes components.

00:49:50.960 --> 00:49:53.880
So we have a lot of components within Kubernetes.

00:49:53.960 --> 00:50:05.120
We have nodes and pods, we have etcd, we have our API server, our kube-scheduler,

00:50:05.120 --> 00:50:16.040
our kube-controller-manager, and then we have a cloud-controller-manager, not to be confused

00:50:16.040 --> 00:50:22.040
with the controller-manager. We have a kube-proxy and older clusters, and we have our kubelet.

00:50:22.040 --> 00:50:28.520
This can be represented as the chart provided by the upstream Kubernetes SIG team.

00:50:29.880 --> 00:50:35.800
You can see we have a control plane here with the blue dash outline. Within that we have our

00:50:36.440 --> 00:50:46.440
controller manager, scheduler, ncd, api server, and optionally the cloud controller

00:50:46.440 --> 00:50:53.940
manager manager as you can see here connects to a cloud provider api so if you were running

00:50:55.140 --> 00:50:58.820
bare metal kubernetes and you wanted to also run that in the cloud

00:51:00.020 --> 00:51:06.260
you could run your bare metal kubernetes in the cloud um and then you would need to build out a

00:51:06.260 --> 00:51:11.860
piece to work with the cloud ap i'm seeing this example we have three worker nodes as well and

00:51:11.860 --> 00:51:18.500
each worker node has kproxy installed in it and all right for our nodes there are two

00:51:18.500 --> 00:51:24.500
primary node types in a cluster the control plane or server node

00:51:25.700 --> 00:51:30.740
and the worker or agent node the names have changed over the years we started out

00:51:32.660 --> 00:51:36.420
in 2015 they were called master slave and then they went to

00:51:36.420 --> 00:51:48.580
to master worker, and then they went to calling it a control plane in worker, and today the

00:51:48.580 --> 00:51:54.680
naming convention is actually server node, which designates your control plane typically,

00:51:54.680 --> 00:51:57.660
or agent node.

00:51:57.660 --> 00:52:06.380
And the reason they went to agent is because sometimes we might have a storage node, right?

00:52:06.380 --> 00:52:10.380
the storage node technically is not a worker node it's not running workloads it's running

00:52:11.180 --> 00:52:19.180
um our database application so we went to agent go ahead yes and so your storage node would be an

00:52:19.180 --> 00:52:27.020
agent and storage and your worker node would be just an agent and your control plane would be a

00:52:27.020 --> 00:52:33.420
server so once you you add a storage node they're either going to be a control plane or a worker

00:52:33.420 --> 00:52:37.680
They're always going to be a worker. You wouldn't run your storage on a control

00:52:37.680 --> 00:52:43.360
plane unless you wanted to risk losing your control plane. So the storage node

00:52:43.360 --> 00:52:47.780
will generally have two roles and we'll get into roles a little bit later. So

00:52:47.780 --> 00:52:53.420
it'll have a storage role and an agent role. Did that make sense to you Stanley?

00:52:53.420 --> 00:52:59.180
Okay, correct. Yeah, so you'll have agent nodes that are just workers and you will

00:52:59.180 --> 00:53:07.900
have storage nodes that are an agent and a storage. Okay, so we have a max of 5,000 nodes in a cluster.

00:53:08.700 --> 00:53:15.820
I don't think anyone on this will probably run into that in their Kubernetes career. However,

00:53:17.260 --> 00:53:23.180
that is the way we used to build clusters, and there are agencies that had multi-thousand node

00:53:23.180 --> 00:53:28.460
clusters within the government, and they had teams of Kubernetes engineers to manage them.

00:53:29.180 --> 00:53:30.920
Very resource inefficient.

00:53:32.800 --> 00:53:38.480
Also, just imagine running, say, a 4,000-node Kubernetes cluster

00:53:38.480 --> 00:53:42.280
with three control plane nodes, right?

00:53:42.360 --> 00:53:46.240
Imagine the traffic that's coming into those three control plane nodes.

00:53:48.080 --> 00:53:53.360
You're going to have a lot of delays with your etcd and your Kube API server.

00:53:53.800 --> 00:53:56.820
But that is the way that it used to be done years ago.

00:53:56.820 --> 00:54:00.240
And today, we use infrastructure as code.

00:54:00.460 --> 00:54:04.900
And we would spin up probably no more than 100 nodes in a cluster.

00:54:05.060 --> 00:54:14.100
So although we can have a max of 5,000, today with infrastructure as code, we would generally limit that to about 100 nodes per cluster.

00:54:14.320 --> 00:54:15.860
And nodes may also be late.

00:54:16.000 --> 00:54:19.520
Oh, so with infrastructure as code, we can build duplicate clusters.

00:54:20.280 --> 00:54:26.580
So instead of building, let's say that your operational requirements, you needed 3,000 nodes.

00:54:26.580 --> 00:54:29.800
But because of the complexity, you would build it for 4,000 nodes.

00:54:30.460 --> 00:54:35.820
That would be a single cluster, and it would be run manually, so you wouldn't use infrastructure as code.

00:54:35.940 --> 00:54:38.420
That's too many nodes to spin that up.

00:54:39.960 --> 00:54:47.400
But with infrastructure as code, we can duplicate, so we can build multiple clusters that are no more than 100 nodes.

00:54:48.300 --> 00:54:55.940
And by maximizing the efficiency of that, we would not need 4,000 nodes in total anymore.

00:54:55.940 --> 00:55:08.420
An older 4,000-node cluster, just because of the failure points, could probably be duplicated with infrastructure as code with five or six, you know, 100-node clusters.

00:55:09.460 --> 00:55:20.560
So the older clusters, and there are still Kubernetes clusters that are multiple thousand nodes, and that's because they were created years ago.

00:55:20.560 --> 00:55:28.640
And they're still running, and they're still being upgraded in place, and they still require teams of Kubernetes engineers to manage them.

00:55:29.560 --> 00:55:33.600
But it's very resource inefficient to do it that way.

00:55:33.680 --> 00:55:36.480
So what you would do is you would segment your workloads.

00:55:36.820 --> 00:55:39.060
So, for example, you have automation in cars.

00:55:39.720 --> 00:55:45.300
So let's say the automation in cars were to be very large applications.

00:55:45.300 --> 00:55:52.880
then you you could have a cluster for cars and a cluster for automation if that makes sense

00:55:52.880 --> 00:55:59.220
so you would split your workloads and that's kind of where that again you can split them even

00:55:59.220 --> 00:56:06.020
further in the form of microservices so i just had to do um back when we had very complex monoliths

00:56:06.880 --> 00:56:13.480
think of the the old monolithic and the databases that were associated with that all of the sharding

00:56:13.480 --> 00:56:19.960
that took place if you're running mongo etc it's a different way of doing things it was basically

00:56:19.960 --> 00:56:28.280
a bridge between the cloud and large cloud um you know engineering setups uh bringing porting

00:56:28.280 --> 00:56:36.440
them over to kubernetes and requiring thousands of nodes to run that all right pods each node

00:56:36.440 --> 00:56:42.600
so we can have up to 5 000 nodes each pod can manage up to 110 pods you want to guess why the

00:56:42.600 --> 00:56:49.880
node is limited to 110 pods, so the pod limitation is imposed by a 24-sider block in each node.

00:56:50.600 --> 00:57:00.040
It's given a 24-sider block of IP addresses to manage, and based on the containers that are

00:57:00.040 --> 00:57:07.000
spinning up, they're terminating, that are being deleted, they're stuck because containers do

00:57:07.000 --> 00:57:15.160
stick, pods do stick and have to be manually deleted when you're troubleshooting or testing.

00:57:16.120 --> 00:57:21.480
They've limited it to 110 pods to allow for those additional IP addresses that are used.

00:57:21.480 --> 00:57:28.600
So pods may contain multiple containers and the pod containers may also contain sidebars.

00:57:28.600 --> 00:57:32.200
So think of it as a pod can have an init container

00:57:32.200 --> 00:57:39.260
which handles a task a pre-startup task and then it can have its primary container

00:57:39.260 --> 00:57:47.440
and then let's say that we needed to send metrics somewhere so we can use a side card to collect

00:57:47.440 --> 00:57:54.180
those metrics and then ship those out to you know prometheus for example so in that case we would

00:57:54.180 --> 00:58:00.640
actually have three containers we'd have an init and then we would have a primary container and

00:58:00.640 --> 00:58:07.440
we'd have a third container that runs as a sidecar and is attached to the primary container. So it

00:58:07.440 --> 00:58:15.280
would be a total of three containers running within that pod. After startup, only two would be

00:58:15.280 --> 00:58:25.360
continuously running. Right, so yeah, and if you were, it depends on how you work that out.

00:58:25.360 --> 00:58:33.120
you can actually run a separate chart for your migration so if your migration um you could run

00:58:33.120 --> 00:58:39.280
a separate chart for that migration um or you can run it as an init container so there's you

00:58:39.280 --> 00:58:44.320
actually have a couple of options there and maybe even more depending on how your migration works

00:58:44.320 --> 00:58:51.430
but you know a couple of different ways so that's how you're doing it now you're running the

00:58:51.430 --> 00:58:57.590
migration first with a container um so you're spinning up your primary container and running

00:58:57.590 --> 00:59:01.750
the migration alongside it or do you run the migration first and then spin up your product?

00:59:01.750 --> 00:59:06.310
Okay. Yep. Right. Okay. Yep. Yep. So we could keep it the same way.

00:59:06.870 --> 00:59:13.430
All right. Because etcd, etcd is a key value store. It's a Kubernetes backing store.

00:59:15.030 --> 00:59:22.310
It's designed for high availability. It needs to be run in odd numbers. So you can have one etcd,

00:59:22.310 --> 00:59:29.990
three SEDs or five SEDs, and that's why you see control planes run as one, three, or five as well.

00:59:29.990 --> 00:59:38.870
It's a leader-based distribution system. It's an Alexa leader. The writes may run behind on slow

00:59:38.870 --> 00:59:47.620
SSD storage, and encryption may also be innate. All right, we'll go ahead and take like a two

00:59:47.620 --> 00:59:55.300
minute break um so generally we do just a very quick break after an hour like one or two minutes

00:59:55.940 --> 01:00:02.660
so one can get up and stretch but um go ahead and take a quick two minute break and then we'll come

01:00:02.660 --> 01:00:11.860
back here and uh i have eastern time 12 24 so we can do 12 26. all right see if we get everybody

01:00:11.860 --> 01:00:27.400
back here in about two minutes. Okay, so in further discussing etcd, you know in the early days

01:00:29.480 --> 01:00:35.640
we had primarily spinning drives and etcd did not do well at all on spinning drives.

01:00:35.640 --> 01:00:47.400
Yes, SSDs were a lot faster, but you'll find that it will run behind with etcd.

01:00:49.080 --> 01:00:55.720
If you have a lot of requests to the etcd server to write, reads are generally find on a single SSD.

01:00:56.440 --> 01:01:04.200
So striping generally takes care of that issue, either a RAID 0 or RAID 10,

01:01:04.200 --> 01:01:13.020
which is a RAID 1 plus 0. And then NVMe, what's that? Oh yeah, so striping is set up either at

01:01:13.020 --> 01:01:21.320
the software layer or the hardware layer. So for example, your Proxmox box that you have Proxmox on,

01:01:22.540 --> 01:01:31.220
I think Chris said that that was a Supermicro board. And so depending on which board you have,

01:01:31.220 --> 01:01:38.560
you either have the ability to set up two SSDs or four SSDs. If you can only do two, you can do

01:01:38.560 --> 01:01:49.840
striping zero. Or if you can do four, you can do a striping and a duplication. So it would be arrayed

01:01:49.840 --> 01:01:59.160
one plus zero. And striping splits the data into two pieces. So if you have data piece one and data

01:01:59.160 --> 01:02:05.320
piece two it will send data piece one to the first drive and data piece two to the second drive and

01:02:05.320 --> 01:02:13.100
it writes them at the same time so it almost doubles your write speed and then your read

01:02:13.100 --> 01:02:21.360
speeds are a lot faster as well and then the RAID keeps track of which which drive has which data

01:02:21.360 --> 01:02:26.060
the only problem with the RAID zero is there's no redundancy if you lose one you lose everything

01:02:26.940 --> 01:02:35.500
so RAID 10 is better off for that if you can add four drives and generally with a Supermicro

01:02:37.900 --> 01:02:45.020
motherboard you can add four or eight generally you can do four to eight drives so RAID 10 is

01:02:45.020 --> 01:02:51.900
generally not an issue for the underlying and a lot of Supermicro boards today have at least one

01:02:51.900 --> 01:03:00.060
nvme slot and then your nvme is going to be four times faster than your ssd than a single ssd

01:03:00.060 --> 01:03:05.340
several several options there i don't know if you guys are utilizing all of the capabilities of your

01:03:05.340 --> 01:03:12.140
proxmox box or not but if you do run into problems as you start running multiple kubernetes clusters

01:03:12.140 --> 01:03:17.980
on there you might find that there are options available within your hardware that can

01:03:17.980 --> 01:03:29.540
significantly speed up your data rights. Correct. Correct. And with today's technology on the

01:03:29.540 --> 01:03:37.060
consumer SSDs, the high quality consumer SSDs oftentimes use the same underlying tech as the

01:03:37.060 --> 01:03:43.800
enterprise. It's just easier to run it all off the same die. And so one's branded as a consumer

01:03:43.800 --> 01:03:49.060
because it has a lower warranty and the other is branded as an enterprise. So if you're using

01:03:49.060 --> 01:03:54.980
something like a Samsung, for example, you can actually run the consumer drives on your Super

01:03:54.980 --> 01:04:00.560
Micro hardware and they're a fraction of the cost of enterprise. Okay, so Kube API server

01:04:00.560 --> 01:04:10.100
is located in the control plane node or our server node. It validates and configures

01:04:10.100 --> 01:04:17.020
API objects such as pods and services. It uses traditional HTTP verbs post, put,

01:04:17.020 --> 01:04:23.220
patch, delete, and get. It interacts directly with the etcd key value store.

01:04:23.220 --> 01:04:28.860
If resource constrained, it will timeout on API calls. And if all API servers are

01:04:28.860 --> 01:04:34.960
unavailable, that cluster may not be recoverable. So that is how we interact

01:04:34.960 --> 01:04:39.720
with the clusters to the API server. Poob's scheduler is located in the

01:04:39.720 --> 01:04:45.660
control plane node as well. It assigns pods to nodes. It determines which nodes

01:04:45.660 --> 01:04:50.720
are available. It assigns the pods based on constraints and available resources

01:04:50.720 --> 01:04:57.660
and it communicates with it Kubernetes APIs. The controller manager

01:04:57.660 --> 01:05:04.080
located in the control plane node. It embeds cloud-specific logic. It maintains

01:05:04.080 --> 01:05:09.100
the state of the cluster using control loops. Control loops are non-terminating

01:05:09.100 --> 01:05:13.540
and regulate the state of the system. Some of the controller examples are

01:05:13.540 --> 01:05:22.820
replication, namespace, endpoints, and service accounts. And it communicates

01:05:22.820 --> 01:05:30.220
with the Kubernetes APIs. And one that we won't run into with RKE2, but you may

01:05:30.220 --> 01:05:35.800
see a reference to this in a Kubernetes cluster, is a Cloud Controller Manager.

01:05:35.800 --> 01:05:42.340
and this is located in the control plane node and this embeds cloud-specific

01:05:42.340 --> 01:05:50.780
logic that decouples a logic between Kubernetes and the cloud provider. The

01:05:50.780 --> 01:05:57.680
cloud controller manager includes node, route, and service controller and it

01:05:57.680 --> 01:06:02.600
communicates with the Kubernetes API. And then kubeproxy which has been around

01:06:02.600 --> 01:06:08.180
for a long time within the Kubernetes cluster and is now being replaced.

01:06:08.180 --> 01:06:15.260
It's located in every node. TCP, UDP, and SCP stream forwarding takes place through

01:06:15.260 --> 01:06:20.680
KubeProxy. Pretty much standard on a base cluster today. Modern CNIs like Cilium

01:06:20.680 --> 01:06:25.520
contain a KubeProxy replacement and replacing the KubeProxy improves

01:06:25.520 --> 01:06:29.860
reliability and can also boost your network performance. If you notice in our

01:06:29.860 --> 01:06:35.380
MiniKube, we do have Kubeproxy. Kubla, which we talked about earlier with the CRI control,

01:06:36.340 --> 01:06:45.940
is located in every node. It runs as a daemon, so it'll restart itself. It manages the lifecycle

01:06:45.940 --> 01:06:52.980
of containers within pods. It reports the status of pods to the API server.

01:06:52.980 --> 01:07:01.200
It monitors the API server for new pod assignments and you can view status by

01:07:01.200 --> 01:07:06.420
viewing the node details. You can see the status of the kubelet. All right

01:07:06.420 --> 01:07:12.120
Kubernetes flavors or distributions. This is very opinionated from having worked

01:07:12.120 --> 01:07:19.380
with Kubernetes since 2016. There is only one flavor in 2015. It was called plain

01:07:19.380 --> 01:07:26.340
vanilla and that's where the term flavor came from with kubernetes so many flavors to choose

01:07:26.340 --> 01:07:33.460
from today you can choose your flavor based on security or opinionated options you can choose

01:07:33.460 --> 01:07:40.580
between toy demonstration or production flavors and the flavor chosen can have an impact on

01:07:40.580 --> 01:07:47.780
cluster long-term viability so we have enterprise production flavors or distributions

01:07:49.380 --> 01:07:59.240
And those are Vanilla, Rancher RKE2, and then Red Hat has their MicroShift OpenShift,

01:07:59.240 --> 01:08:05.920
although they're technically container orchestrators which use Kubernetes under the hood

01:08:05.920 --> 01:08:11.060
because they're very opinionated on how they run containers,

01:08:11.640 --> 01:08:16.320
and they do generally require an enterprise service agreement.

01:08:16.320 --> 01:08:26.400
I see OpenShift used a lot with Department of Defense contracting. It

01:08:26.400 --> 01:08:34.080
allows you to check those boxes if you are a DOD contractor. So for development and

01:08:34.080 --> 01:08:40.740
demonstration we have RKE1 which is Docker in Docker. It's their first

01:08:40.740 --> 01:08:49.440
version with K3S which is designed for Raspberry Pis. So that's your IOT version

01:08:49.440 --> 01:09:00.900
if you will. We have Micro K8s and then Minikube that we're using, Docker K8s,

01:09:00.900 --> 01:09:08.200
Rancher which is their ClickOps version of Kubernetes. Not all features work in

01:09:08.200 --> 01:09:17.140
that. It's basically a way to try RKE2 in a demonstration version if you

01:09:17.140 --> 01:09:23.980
don't know how to use the command line. It's a ClickOps interface. Rancher

01:09:23.980 --> 01:09:32.920
Desktop, which was Rancher's attempt to duplicate Docker K8s, did not work

01:09:32.920 --> 01:09:40.040
very well. I've actually destroyed a laptop battery on a MacBook Pro trying

01:09:40.040 --> 01:09:45.960
to run Rancher Desktop on it. It has a lot of system processes that run

01:09:45.960 --> 01:09:54.220
continuously. And then Kaniykl has their own version. Kaniykl Kubernetes.

01:09:54.220 --> 01:09:58.980
I don't know if anyone's still installing that one today. It would be considered a

01:09:58.980 --> 01:10:05.900
development demonstration that's not actively made. Yeah, so Rancher is Ranchers. So you have

01:10:05.900 --> 01:10:13.720
basically three enterprise paid solutions out there. You don't have to pay Rancher to use

01:10:13.720 --> 01:10:20.560
Rancher. But if you want the features turned on inside the Rancher UI, you have to pay them

01:10:20.560 --> 01:10:28.340
an enterprise fee. And it's the same way with Red Hat's OpenShift and Talos. Talos is

01:10:28.340 --> 01:10:33.720
quickly becoming the same way where if you want advanced features inside of

01:10:33.720 --> 01:10:37.960
Talos you're going to have to pay them an enterprise fee. So all three of them

01:10:37.960 --> 01:10:43.420
to get all of the features either require an enterprise fee or will

01:10:43.420 --> 01:10:50.520
require an enterprise fee and basically Rancher and Talos are competing with the

01:10:50.520 --> 01:10:55.400
one pool of enterprise fees that's collected today which is collected by

01:10:55.400 --> 01:11:01.220
Red Hat. Red Hat is the undisputed champion of enterprise fees when it

01:11:01.220 --> 01:11:08.000
comes to Kubernetes and they have, you know, earned that and it's, you know, if

01:11:08.000 --> 01:11:15.280
you apply for funding or you apply for an award as a contractor with the

01:11:15.280 --> 01:11:19.860
Department of Defense, basically you're going to have to run an OpenShift if you

01:11:19.860 --> 01:11:26.680
want to receive that award or that contract and so if you think about it

01:11:26.680 --> 01:11:32.220
I've had clients tell me that they paid $200,000 to Red Hat to run OpenShift

01:11:32.220 --> 01:11:38.080
for their DoD contract that seems like a lot of money to me when if you don't

01:11:38.080 --> 01:11:45.520
have that requirement RKE2 will do everything and works well with

01:11:45.520 --> 01:11:54.600
infrastructure as code for free so so there are different routes that you can go there but ranchers

01:11:54.600 --> 01:12:00.680
ui if you want all of the features you generally have to pay them an enterprise fee for them to

01:12:00.680 --> 01:12:06.520
and that answer your question about rancher yeah most of my clients are small and medium-sized

01:12:06.520 --> 01:12:14.040
businesses and they just do not want to pay enterprise fees at all and so i try to find

01:12:14.040 --> 01:12:15.940
solutions that avoid that.

01:12:15.940 --> 01:12:19.480
But if you're a DoD contractor, you really today have no choice.

01:12:19.480 --> 01:12:20.160
You have to go the route.

01:12:20.160 --> 01:12:20.520
Yeah.

01:12:20.520 --> 01:12:24.360
And with Rancher, if you pay them the fees, you can probably meet the DoD

01:12:24.360 --> 01:12:30.300
requirements because they'll pair an engineer with you, but it's not going to be cheap.

01:12:31.140 --> 01:12:36.900
So day one versus day two clusters, a day one cluster designed for a single day or task.

01:12:37.740 --> 01:12:41.280
So it's designed for a test for demonstration purposes.

01:12:41.280 --> 01:12:45.040
It can be deleted, modified, and rebuilt in minutes.

01:12:45.040 --> 01:12:49.320
So think of our mini-kube, and we spin it up and spin it down in a matter of minutes.

01:12:49.320 --> 01:12:52.780
It's typically used for development and staging clusters.

01:12:52.780 --> 01:12:58.320
A day two cluster is designed to survive beyond day one.

01:12:58.320 --> 01:13:01.900
It is not designed for testing.

01:13:01.900 --> 01:13:07.600
It can be modified with prior tested configurations, and that's where infrastructure as code comes

01:13:07.600 --> 01:13:08.600
in.

01:13:08.600 --> 01:13:14.420
a configuration to test, and then we can move that to our production day 2 cluster.

01:13:14.420 --> 01:13:18.980
It can be modified with prior tested configurations typically used for

01:13:18.980 --> 01:13:25.160
production. By availability relating to our control plane nodes, on bare metal we

01:13:25.160 --> 01:13:31.840
use a leader election to assign a VIP to the leader. A VIP is a virtual IP address.

01:13:31.840 --> 01:13:36.140
So on bare metal we need to have something in front of our control plane

01:13:36.140 --> 01:13:44.420
nodes where we can access and always control plane node in the event that

01:13:44.420 --> 01:13:51.440
there is a loss of leadership or the VM with the control plane leader fails

01:13:51.440 --> 01:13:57.700
loses network connectivity we need a process in place outside of that that

01:13:57.700 --> 01:14:02.100
enables a new control plane to be elected as a leader and that's done

01:14:02.100 --> 01:14:07.860
through a VIP process or virtual IP. So if the leader changes the new leader can be queried

01:14:07.860 --> 01:14:14.820
after the failover using the same virtual IP address. So everything queries that virtual IP

01:14:14.820 --> 01:14:21.220
address and there's a leader election in the background which makes sure that you always

01:14:21.220 --> 01:14:27.300
have a running control plane node assuming you have at least one run. In HA pod anti-infinity

01:14:27.300 --> 01:14:34.980
settings are used to evenly distribute pods and that is it's a repelling meaning that

01:14:34.980 --> 01:14:41.700
if you already have a pod on a node running an application and you have the same application

01:14:41.700 --> 01:14:47.540
and you want two instances if you have an anti-infinity setting it repels that second pod

01:14:48.180 --> 01:14:53.220
and it forces that second pod to find another node to launch and allows you to avoid resource

01:14:53.220 --> 01:14:58.180
concentration on a single node with a control plane because in the event that that control

01:14:58.180 --> 01:15:03.860
plane goes down now instead of just electing a new leader with a control plane you have to wait for

01:15:03.860 --> 01:15:11.460
all of those pods to then be scheduled and then spin up something we would want to avoid at all

01:15:11.460 --> 01:15:19.780
costs with a control plane node high availability with a workload node enable multiple instances

01:15:19.780 --> 01:15:27.060
of stateless workloads across the nodes so we can actually load balance in front of the ha pods

01:15:28.580 --> 01:15:34.820
if they're truly stateless in ha pod anti-infinity settings are used to evenly distribute pods again

01:15:34.820 --> 01:15:42.020
used to repel a duplicate pod so that it settles onto another node made resource concentration

01:15:42.020 --> 01:15:49.140
on a single node one of the benefits of that and it gives us the ability to auto scale pods across

01:15:49.140 --> 01:15:57.060
HA nodes. So we can autoscale based on CPU consumption. If you have an API back-end,

01:15:57.060 --> 01:16:02.820
for example, that's truly stateless and you're running three instances and you have a fourth

01:16:02.820 --> 01:16:09.940
node available, you can actually set your horizontal pot autoscaler to spin up a fourth

01:16:10.500 --> 01:16:16.260
instance on another node. Or if you have additional capacity available on the node

01:16:17.140 --> 01:16:27.220
that you're running and then you can spin up one on a node that maybe is not utilized as

01:16:27.220 --> 01:16:34.100
heavily. All right let's talk about Kubernetes self-healing. So KH is designed with self-healing

01:16:34.100 --> 01:16:41.380
capabilities, enables you to maintain availability of workloads, automatically reschedules workloads

01:16:41.380 --> 01:16:48.980
during node failure so you have a node that goes down and you have an application that needs to

01:16:51.060 --> 01:16:57.780
schedule a pod a workload it will schedule it on the other automatically replace failed containers

01:16:58.740 --> 01:17:04.580
so we'll get into later on discussing probes and when a probe says that the container has failed

01:17:04.580 --> 01:17:10.340
it will replace it automatically it ensures the desired state of the cluster is maintained it

01:17:10.340 --> 01:17:17.780
does this through container level restarts. Load balancing for services. So when you're load

01:17:17.780 --> 01:17:25.860
balancing a service and you have high availability and it has, say you have three pods across three

01:17:25.860 --> 01:17:31.780
nodes, NetService will show three endpoints. If one of those is no longer available, it will drop it

01:17:32.340 --> 01:17:37.940
and that way it will only load balance between the two remaining endpoints. It enables persistent

01:17:37.940 --> 01:17:44.340
storage recovery because you can run your volumes across multiple nodes replica replacement when

01:17:44.340 --> 01:17:51.780
you're running a replica set it will replace a replica if it fails and self-healing is a key

01:17:51.780 --> 01:17:57.860
feature for devops personnel you can have a failure take place within a cluster and to a

01:17:57.860 --> 01:18:04.420
certain extent it will self-heal or restart to eliminate the issue all right let's talk about

01:18:04.420 --> 01:18:13.460
the causality dilemma. So as the picture represents the chicken versus the egg which came first.

01:18:14.020 --> 01:18:19.700
So the causality dilemma is often experienced when building Kubernetes clusters or building

01:18:19.700 --> 01:18:26.180
Helm charts. We need one resource to be installed first before we can run a second,

01:18:27.700 --> 01:18:32.820
but vice versa that first resource needs a second to be installed. So how do we solve that?

01:18:34.420 --> 01:18:39.160
And it's often solved by splitting our components

01:18:39.160 --> 01:18:44.860
into, let's say, a Helm chart into smaller components,

01:18:44.860 --> 01:18:47.320
and then sequencing it a little differently than the way

01:18:47.320 --> 01:18:51.460
that we might otherwise want to sequence it.

01:18:51.460 --> 01:18:55.060
And that allows us to solve the causality dilemma.

01:18:55.060 --> 01:19:02.320
So Kubernetes is very strict on the resources.

01:19:02.320 --> 01:19:06.480
For example, when you're installing a Helm chart,

01:19:06.480 --> 01:19:10.220
you may want to install the CRDs with that Helm chart.

01:19:10.220 --> 01:19:12.500
You may have something that you're installing

01:19:12.500 --> 01:19:14.440
with the Helm chart before that,

01:19:15.360 --> 01:19:17.820
that requires the CRDs to be in place

01:19:17.820 --> 01:19:19.820
from the second Helm chart.

01:19:19.820 --> 01:19:22.540
And in that case, we would split it

01:19:22.540 --> 01:19:26.380
and we would install the CRDs first at the very beginning

01:19:26.380 --> 01:19:30.880
before we installed the other Helm charts afterwards.

01:19:30.880 --> 01:19:34.540
and that would enable that particular resource to work.

01:19:34.540 --> 01:19:37.720
And one of those would be Prometheus, for example.

01:19:38.760 --> 01:19:45.650
In order to build the service objects for Prometheus,

01:19:45.650 --> 01:19:49.430
we need the CRDs to be in place for our service monitors.

01:19:49.430 --> 01:19:51.950
That's one example of the causality dilemma

01:19:51.950 --> 01:19:54.010
that we run into when we're building a cluster

01:19:54.010 --> 01:19:55.910
with infrastructure is great.

01:19:55.910 --> 01:19:58.130
Kubernetes node scaling.

01:19:58.130 --> 01:20:00.930
Kubernetes worker nodes can be scaled up or down

01:20:00.930 --> 01:20:02.530
as plus or minus N.

01:20:03.050 --> 01:20:10.990
Kubernetes control plane nodes must be scaled in odd numbers, so remember 1, 3, or 5.

01:20:10.990 --> 01:20:16.370
Minikube provides the ability to add or delete nodes, so Kubernetes nodes scaling downward.

01:20:16.370 --> 01:20:21.930
Kubernetes scaling downward involves coordinating the node.

01:20:21.930 --> 01:20:26.890
Coordinating the node prevents future pods from being scheduled.

01:20:26.890 --> 01:20:32.410
Next we drain the node of all of its pods, and then our pod disruption budgets may need

01:20:32.410 --> 01:20:40.490
be set to avoid interruption if you are going to follow this procedure. Draining a node forces pods

01:20:40.490 --> 01:20:48.410
to be scheduled on other nodes. The final step is to remove the node, and deleting a node removes

01:20:48.410 --> 01:20:54.730
it from the Kubernetes cluster. Kubernetes node scaling upward, the opposite of downward.

01:20:55.530 --> 01:21:02.170
Kubernetes staying upward involves creating a node, and we join the node to the cluster.

01:21:03.610 --> 01:21:07.610
We then uncoordinate a node if it was coordinated during the startup.

01:21:09.290 --> 01:21:15.370
This notifies the pod scheduler that it can allow pods to be added to this, and then upgrading our

01:21:15.370 --> 01:21:20.410
Kubernetes nodes when we upgrade in place. And again, this is a 90-10 proposition.

01:21:20.410 --> 01:21:27.970
version. Upgrading the Kubernetes version requires coordinating the node. Next, we drain

01:21:27.970 --> 01:21:38.390
the node. We upgrade the node. Then we restart the node. And finally, uncoordinate the node.

01:21:38.390 --> 01:21:45.920
And typically, this is performed with KubeAdmin by a Kubernetes administrator. Because when

01:21:45.920 --> 01:21:51.020
it fails, you need someone who can rescue the cluster. Cluster scaling and upgrading

01:21:51.020 --> 01:21:57.820
using infrastructure as code. Cluster scan or upgrading involving IAC with stateless workloads

01:21:59.180 --> 01:22:06.620
can create a new cluster with the latest tested version. The workload state

01:22:06.620 --> 01:22:14.460
can be mirrored using declarative GitOps onto the new cluster. After reaching the desired state,

01:22:14.460 --> 01:22:20.700
the DNS can then be transferred over to the new IP. So we have our existing cluster.

01:22:22.940 --> 01:22:27.980
Let's see, we want to upgrade it as we build another cluster alongside. We test the new

01:22:27.980 --> 01:22:33.820
versions and then when we're ready, we then using infrastructure as code, we will spin up a

01:22:33.820 --> 01:22:38.860
production version of that cluster. So we'll have our current production version and we'll have a

01:22:38.860 --> 01:22:45.420
second production version running alongside that, and then all we do with the stateless workloads

01:22:46.060 --> 01:22:53.180
is we transfer the DNS over to the new cluster. This is the easiest way to avoid the issues

01:22:53.740 --> 01:23:00.780
relating to upgrading in place or adding or removing nodes, for example, and it does not

01:23:00.780 --> 01:23:06.300
require having a full-time Kubernetes engineer. Any DevOps personnel can handle this.

01:23:06.300 --> 01:23:18.460
And in this case, in the event that we have an issue with the second cluster, the replacement cluster, we can point the DNS back to the existing workload cluster.

01:23:18.740 --> 01:23:21.120
Does anyone have any questions about how this would work?

01:23:21.220 --> 01:23:25.180
Stanley, you had a few questions the other day when we discussed it.

01:23:25.180 --> 01:23:33.300
Yeah, so when we upgrade and scale with infrastructure as code,

01:23:33.300 --> 01:23:38.700
we're actually building a new cluster alongside the existing cluster.

01:23:39.540 --> 01:23:42.820
And so, yeah, so when you can actually see that in place,

01:23:43.560 --> 01:23:45.740
and we won't demonstrate that with MidiCube.

01:23:46.120 --> 01:23:48.540
Obviously, we can only run one at a time.

01:23:48.800 --> 01:23:53.220
But essentially, it's just a matter of pointing your DNS over.

01:23:53.220 --> 01:24:00.600
and you keep the old cluster running just in case and generally after two

01:24:00.600 --> 01:24:06.400
weeks I think that's enough time that if nothing has failed and you feel

01:24:06.400 --> 01:24:11.480
confident in the cluster then you can go ahead and delete that old cluster out.

01:24:11.480 --> 01:24:16.800
Alright yeah so generally the only thing that's going to change is let's say that

01:24:16.800 --> 01:24:22.440
we were on 1.3 or 4.2 and let's say it's May of this year and you say hey we

01:24:22.440 --> 01:24:28.080
should be on 1.35.2 now, right? It runs four months behind. You could set up a four-month

01:24:28.080 --> 01:24:33.520
replacement schedule if you wanted to mirror Kubernetes, or you could do three months.

01:24:33.720 --> 01:24:38.060
It's all based on your team and what you feel comfortable with. You could even do six months,

01:24:38.060 --> 01:24:47.380
really. But yeah, so you would be upgrading and testing that out, not in production. And then if

01:24:47.380 --> 01:24:52.240
you needed to add additional worker nodes, you could do that as well. So let's say that you were

01:24:52.240 --> 01:24:58.800
adding in your airflow right you said hey we had one that we've been using but we're running up

01:24:58.800 --> 01:25:04.320
at 80 cpu let's go ahead and add a second node or a third one or whatever then you would do that at

01:25:04.320 --> 01:25:11.400
that time and you could also if in the middle of your schedule you said hey you know airflow is

01:25:11.400 --> 01:25:15.960
taking off and we really need five nodes for your phone now for whatever reason then you could go

01:25:15.960 --> 01:25:21.800
ahead and build a cluster with just several extra worker nodes and go ahead and spin that up

01:25:21.800 --> 01:25:26.420
it's going to mirror what's already running in production with the exception that you'll have

01:25:26.420 --> 01:25:32.500
additional nodes, worker nodes, agent nodes, and then you could transfer your DNS over to your

01:25:32.500 --> 01:25:38.560
new cluster. And this eliminates pretty much that 90-10 problem that we run into in Kubernetes,

01:25:39.280 --> 01:25:45.060
which is where we have that 10% failure, meaning one out of 10 times we have a failure in the

01:25:45.060 --> 01:25:49.040
cluster and then we have to try to get that cluster back. So it's two different ways of

01:25:49.040 --> 01:25:56.660
running a kubernetes cluster one with a full-time kubernetes engineer and the other where devops

01:25:56.660 --> 01:26:04.660
personnel can run it all right well we're at our one o'clock time and we're going to work with our

01:26:04.660 --> 01:26:11.360
labs when we come back from depending on where you're at it's one o'clock eastern time so i guess

01:26:11.360 --> 01:26:17.300
this would be lunch break so when we come back from our 30 minute break we'll have practical

01:26:17.300 --> 01:26:25.860
application Q&A, Kubernetes version, container runtime. We're going to work on adding and

01:26:25.860 --> 01:26:31.060
removing a control plane node, so you can see how that actually takes place. We're going to

01:26:31.060 --> 01:26:37.620
add and remove a worker node from a cluster. We'll set up an HA cluster with a CNI, which is

01:26:37.620 --> 01:26:46.380
a container network interface, and I'll show you how to find the config files, and we're going to

01:26:47.340 --> 01:26:53.580
read the etcd key value data right out of the etcd data all right let's see it's

01:26:53.580 --> 01:27:01.580
one eastern so see everyone back here at 1 30. oh no okay yep no worries see one back here at 1 30.