Rkt - Getting Started with RKT

We all know how docker has taken the world of containers but it’s not perfect either. Docker has its own flaws and problems.

Docker architecture is Flawed - The first problem when people talk about docker is about its architecture. It is fundamentally flawed. What exactly does this mean?

in Linux, Init is a abbreviation for initialization. This is a daemon process which starts as soon as the machine starts or boot up. This process runs until the machine is shutdown. This is the first process that starts and also is responsible for starting other process and thus becoming the parent for all other process that starts. This will typically be assigned “pid=1”. The problem with this intialization process is that it starts processes in a serial order. It starts one tasks only after the last task startup was successful and it is loaded into the memory. This can result in longer startup.

There are many works done in enhancing the init process and they came up with the systemd. Though the systemd was not designed for speed but to start things faster ,cleaner and nicer.

Now the docker problem, When ever systemd start a service it creates a series of cgroups. When the same service spawns any child process , they stay in that same cgroup thus blocking them with the restrictions provided by that cgroup. This way systemd will always know what child process are created ,how much resources they are using etc. If that parent service dies all clean up can be done easliy regardless of the parent as systemd know all details about that parent and children.

Docker breaks all this by making a RPC call to the docker daemon when a “docker run” command is triggered. When this call is made, docker spawns a child process that becomes the PID 1 of the container. As soon as the docker client makes the RPC call , systemd will lose track of whats going on as the container is spawned in a different cgroup. this cgroup is different from the docker client cgroup which is being handled by the systemd. Hence systemd cannot manage the docker containers since they are trigged by a docker daemon and running in a different cgroup.

Hence docker is called as flawed. So what is Rkt and how is that different?

rkt is a container system developed by CoreOS as a light weight and secure alternative to docker. it is build based on the Open Container standard known as “App Container” or “appc” specification. The image build on this specification are protable across many container systems that follow the same standard.

How is Rkt different than this?
No Daemon - Rkt from CoreOs handles the setting up of the resources for the containers and then delegates everything to the systemd-nspawn to do the rest.

Protable image format - Since Rkt uses the “appc” specification it is portable to other container technologies that uses the same “appc” specification.

Light weight - Similar to Docker or runC, rkt also minimizes the number of process, resource utilization. It is easier to put all Application depdencies into a small image and ship that every where

Secure by Design - In Docker, if a user break out of a container using a kernel exploit, the attacker can control the whole machine where the container is started. An Additional security is required to handle this situation in docker. Rkt run containers as un priviliged users so that even some thing breaks out , they cannt affect other container or take control the underlying machine.

Rkt uses cryptographic signature checks on downloaded images so that only trustered containers are allowed to start and run.

More to Come,Happy learning :-)

Read More

Resource Management

A Capacity Planning Is used to determine the resources needed in order to meet the future workflow demands. In Kubernetes resource management is very important. We learned that scheduler is the component that identifies best nodes for creating pods.

The Scheduler has to take into account many factors when it is scheduling pods on nodes. If there are overlapping details or conflicting details the scheduler can find problem when allocating pods in nodes.

Let’s say if we nave a daemon set that is running on all nodes which  require 50% of memory, then scheduler can find it hard to allocate pods that would require around same memory.  Problems of this type can easily mitigated by using namespace resource quotas and correct management of the resource like Cpu, memory and storage

Kubernetes out of box supports resource quotas. The Admission control element in the API server makes sure to enforce resource quotas when one is defined. Kubernetes enforces one resource quota per name space.

The Resource Quotas in Kubernetes are 3 types

Compute

Storage

Object

Compute Resource Quota

This Quota talks about compute resources which are memory and Cpu. In this we can either request for a quota or limit a quota. This can be done by using limits.[cpu,memory] and requests.[cpu,memory]

Limits.cpu & limits.memory – This values talk about the sum of the CPU and Memory limits across all pods that cannot be exceeded.

Requests.cpu & requests.memory – This value talks about the sum of CPU and memory requests by all pods that cannot be exceeded

Storage Resource Quota

This storage resource quota talks about resource that can be restricted per namespace. The resource includes the amount of storage and persistent volume claims.

Object Resource Quota

The Object resource quota talks about the limitations on objects that can be created per namespace. The goal of this resource quota is to make sure that Api server is not wasting time performing things that are useless.

Compute Quota

Lets create a simple compute quota and see how things go. In the below manifest file we are limiting the number of pods to 2. When this quota is attached to a namespace, kubernetes blocks creation of the 3 pods in the namespace.

Create a resource quota config file as,
[root@manja17-I13330 kubenetes-config]# cat basic-resourcequota.yml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: pod-demo
spec:
  hard:
    pods: "2"

In our resource quota file we have limited our number of pod creation to 2. 

Create the resource quota as,

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-resourcequota.yml --namespace=sample-testing

resourcequota "pod-demo" created

Once created , add the resource quota to a namespace as,

[root@manja17-I13330 kubenetes-config]# kubectl get resourcequota --namespace=sample-testing

NAME       AGE

pod-demo   55s

When we get the pods, we see we already have 2 pods running in this namespace,

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing

NAME                                             READY STATUS RESTARTS AGE

test-service-deploy-85c8787b6f-sfprn   1/1 Running   0             57s

test-service-deploy-85c8787b6f-wvnfc  1/1 Running   0             14s

Now lets try to create a new pod and see how it goes

[root@manja17-I13330 kubenetes-config]# cat basic-single-container-pod.yml

apiVersion: v1

kind: Pod

metadata:

 name: testing-service

spec:

 containers:

   - name: test-ser

     image: docker.io/jagadesh1982/testing-service

     ports:

     - containerPort: 9876

     resources:

       limits:

         memory: "64Mi"

         cpu: "500m"

[root@manja17-I13330 kubenetes-config]# kubectl create -f  basic-single-container-pod.yml --namespace=sample-testing

Error from server (Forbidden): error when creating "basic-single-container-pod.yml": pods "testing-service" is forbidden: exceeded quota: pod-demo, requested: pods=1, used: pods=2, limited: pods=2

Now when i run the pod creation it gives me an error saying that the resource quota limit for pod is reached and cannot create any more.

Let’s see another example of Compute Quota,

[root@manja17-I13330 kubenetes-config]# cat basic-computequota.yml

apiVersion: v1

kind: ResourceQuota

metadata:

  name: computequota

spec:

  hard:

    pods: "2"

    requests.cpu: "1"

    requests.memory: 20Mi

    limits.cpu: "2"

    limits.memory: 2Gi

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-computequota.yml –namespace=sample-testing

resourcequota "computequota" created

[root@manja17-I13330 kubenetes-config]# kubectl get quota –namespace=sample-testing

NAME                AGE

computequota    14s

Describe the Quota using,

[root@manja17-I13330 kubenetes-config]# kubectl describe quota computequota

Name:               computequota

Namespace:       default

Resource                Used  Hard

--------                    ----  ----

limits.cpu                 0      2

limits.memory          0     2Gi

pods                        7     2

requests.cpu            0     1

requests.memory     0     20Mi

[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --namespace=sample-testing

deployment.apps "nginx" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing

No resources found.

[root@manja17-I13330 kubenetes-config]# kubectl get deploy --namespace=sample-testing

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

nginx       1               0                  0                   0              40s

[root@manja17-I13330 kubenetes-config]# kubectl describe deploy nginx --namespace=sample-testing

Name:                      nginx

Namespace:              sample-testing

CreationTimestamp:   Tue, 14 Aug 2018 02:29:26 -0400

Labels:                      run=nginx

Annotations:              deployment.kubernetes.io/revision=1

Selector:                   run=nginx

Replicas:                   1 desired | 0 updated | 0 total | 0 available | 1 unavailable

StrategyType:            RollingUpdate

MinReadySeconds:        0

RollingUpdateStrategy:  1 max unavailable, 1 max surge

Pod Template:

  Labels:  run=nginx

  Containers:

   nginx:

    Image:        nginx

    Port:         <none>

    Host Port:    <none>

    Environment:  <none>

    Mounts:       <none>

  Volumes:        <none>

Conditions:

  Type             Status  Reason

  ----             ------  ------

  Progressing      True    NewReplicaSetCreated

  Available        True    MinimumReplicasAvailable

  ReplicaFailure   True    FailedCreate

OldReplicaSets:    <none>

NewReplicaSet:     nginx-65899c769f (0/1 replicas created)

Events:

  Type    Reason             Age   From                   Message

  ----    ------             ----  ----                   -------

  Normal  ScalingReplicaSet  51s   deployment-controller  Scaled up replica set nginx-65899c769f to 1

Now if we check the replica set in the namespace sample-testing, we can see

[root@manja17-I13330 kubenetes-config]# kubectl describe rs nginx --namespace=sample-testing

Name:           nginx-65899c769f

Namespace:      sample-testing

Selector:       pod-template-hash=2145573259,run=nginx

Labels:         pod-template-hash=2145573259

                run=nginx

Annotations:    deployment.kubernetes.io/desired-replicas=1

                deployment.kubernetes.io/max-replicas=2

                deployment.kubernetes.io/revision=1

Controlled By:  Deployment/nginx

Replicas:       0 current / 1 desired

Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed

Pod Template:

  Labels:  pod-template-hash=2145573259

           run=nginx

  Containers:

   nginx:

    Image:        nginx

    Port:         <none>

    Host Port:    <none>

    Environment:  <none>

    Mounts:       <none>

  Volumes:        <none>

Conditions:

  Type             Status  Reason

  ----             ------  ------

  ReplicaFailure   True    FailedCreate

Events:

  Type     Reason        Age               From                   Message

  ----     ------        ----              ----                   -------

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-xgwrb" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-mr975" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-whtp9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-g284h" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  32s (x6 over 1m)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-65899c769f-jzsv9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

The nginx container failed to create with the warning

 Warning  FailedCreate  32s (x6 over 1m)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-65899c769f-jzsv9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

This clearly indicates the quota need to be defined for a pod to be created in sample-testing. Since we haven’t defined that the pod creation failed.

We can then  define the quota and then create the pod as,

[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --requests=cpu=100m,memory=4Mi --limits=cpu=200m,memory=8Mi --namespace=sample-testing

deployment.apps "nginx" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing

NAME                    READY     STATUS    RESTARTS   AGE

nginx-97fb87dfc-9lcxb   1/1       Running   0          8s

Limit Quota

Rather then defining pods with quotas while creating, we can define out quota to allow default values if missing. This is what we call as limit Ranges. Lets create a quota with a limit range and see how things work,

[root@manja17-I13330 kubenetes-config]# cat basic-limitrangequota.yml

apiVersion: v1

kind: LimitRange

metadata:

  name: limits

spec:

  limits:

   - default:

       cpu: 200m

       memory: 6Mi

     defaultRequest:

       cpu: 100m

       memory: 5Mi

     type: Container

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-limitrangequota.yml --namespace=sample-testing

limitrange "limits" created

[root@manja17-I13330 kubenetes-config]# kubectl get limits --namespace=sample-testing

NAME      AGE

limits      1m

we can create pods with out providing any values. The default values will be assigned to the pod while creation,

[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --namespace=sample-testing

More to come, Happy Learning :-)

Read More

Kubernetes Deployment Strategy - Blue/Green

In this 3 post we will see how to do a blue/green deployment. A blue/green deployment is different from the rolling update in which a green version of the application is deployed along with the blue version. After testing the green version and once satisfied we will update the service to point to the green version. This way we will route the traffic from blue version to green version. This is done by playing with the labels in the selector fields. Once the traffic is moved to the green version, the blue version pods are terminated.

Lets create the blue deployment first as,

[root@manja17-I13330 kubenetes-config]# cat  blue-deployment.yml

apiVersion: v1

kind: Service

metadata:

  name: testing-service

  labels:

    app: testing-service

spec:

  type: NodePort

  ports:

  - name: http

    port: 80

    targetPort: http

  # Note here that we match both the app and the version

  selector:

    app: testing-service

    version: "1.0"

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: testing-service-v1

  labels:

    app: testing-service

spec:

  replicas: 3

  selector:

    matchLabels:

      app: testing-service

      version: "1.0"

  template:

    metadata:

      labels:

        app: testing-service

        version: "1.0"

    spec:

      containers:

      - name: testing-service

        image: docker.io/jagadesh1982/testing-service

        ports:

        - name: http

          containerPort: 9876

        env:

        - name: VERSION

          value: "1.0"

[root@manja17-I13330 kubenetes-config]# kubectl apply -f blue-deployment.yml

service "testing-service" created

deployment.apps "testing-service-v1" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods

NAME                                             READY      STATUS    RESTARTS   AGE

testing-service-v1-bc6866889-g8zrr    1/1        Running      0              5m

testing-service-v1-bc6866889-lj8f7     1/1        Running      0              5m

testing-service-v1-bc6866889-qnr2r   1/1         Running      0              5m

[root@manja17-I13330 kubenetes-config]# kubectl get svc

NAME                TYPE          CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE

kubernetes        ClusterIP    10.96.0.1           <none>          443/TCP           1d

testing-service   NodePort    10.110.200.110  <none>          80:30009/TCP   5m

[root@manja17-I13330 kubenetes-config]# curl 10.99.143.66/info

{"host": "10.99.143.66", "version": "1.0", "from": "10.32.0.1"}

Lets run the green deployment,with the below manifest file,

[root@manja17-I13330 kubenetes-config]# cat green-deployment.yml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: testing-service-v2

  labels:

    app: testing-service

spec:

  replicas: 3

  selector:

    matchLabels:

      app: testing-service

      version: "2.0"

  template:

    metadata:

      labels:

        app: testing-service

        version: "2.0"

    spec:

      containers:

      - name: testing-service

        image: docker.io/jagadesh1982/testing-service

        ports:

        - name: http

          containerPort: 9876

        env:

        - name: VERSION

          value: "2.0"

[root@manja17-I13330 kubenetes-config]# kubectl apply -f green-deployment.yml

deployment.apps "testing-service-v2" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods

NAME                                              READY     STATUS    RESTARTS   AGE

testing-service-v1-bc6866889-g8zrr    1/1       Running             0          8m

testing-service-v1-bc6866889-lj8f7     1/1       Running             0          8m

testing-service-v1-bc6866889-qnr2r    1/1       Running            0          8m

testing-service-v2-7f966d5d4c-mb2fk  1/1       Running            0          2m

testing-service-v2-7f966d5d4c-qs6fz   1/1       Running             0          2m

testing-service-v2-7f966d5d4c-z4tj4   1/1       Running             0          2m

Side by side, 3 pods are running with version 2 but the service still send traffic to the first deployment. If necessary, you can manually test one of the pod by port-forwarding it to your local environment. Once your are ready, you can switch the traffic to the new version by patching the service to send traffic to all pods with label version=v2.0.0:

[root@manja17-I13330 kubenetes-config]# kubectl patch service testing-service -p '{"spec":{"selector":{"version":"2.0"}}}'

service "testing-service" patched

[root@manja17-I13330 kubenetes-config]# kubectl get svc

NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE

kubernetes        ClusterIP   10.96.0.1        <none>          443/TCP            1d

testing-service   NodePort    10.99.143.66  <none>          80:30126/TCP   3m

[root@manja17-I13330 kubenetes-config]# curl 10.99.143.66/info

{"host": "10.99.143.66", "version": "2.0", "from": "10.32.0.1"}

If you want to rollback to the older version , we can use

[root@manja17-I13330 kubenetes-config]# kubectl patch service testing-service -p '{"spec":{"selector":{"version":"1.0"}}}'

service "testing-service" patched

[root@manja17-I13330 kubenetes-config]# kubectl get svc

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE

kubernetes        ClusterIP   10.96.0.1         <none>          443/TCP           1d

testing-service   NodePort   10.99.143.66   <none>          80:30126/TCP   4m

[root@manja17-I13330 kubenetes-config]# curl 10.99.143.66/info

{"host": "10.99.143.66", "version": "1.0", "from": "10.32.0.1"}

[root@manja17-I13330 kubenetes-config]# kubectl get deploy

NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

testing-service-v1   3               3                 3                 3                5m

testing-service-v2   3               3                 3                 3                2m

Once we are happy with the green version, delete the blue deployment using,

[root@manja17-I13330 kubenetes-config]# kubectl delete deploy testing-service-v1

deployment.extensions "testing-service-v1" deleted

Now check the pods ,

[root@manja17-I13330 kubenetes-config]# kubectl get pods

NAME                                                READY     STATUS        RESTARTS   AGE

testing-service-v1-bc6866889-dthrt      1/1         Terminating      0             5m

testing-service-v1-bc6866889-nk282    1/1         Terminating      0             5m

testing-service-v1-bc6866889-rxdtz      1/1         Terminating      0             5m

testing-service-v2-76dcdbd6bc-g56bh   1/1         Running           0             3m

testing-service-v2-76dcdbd6bc-jbvkw    1/1         Running           0             3m

testing-service-v2-76dcdbd6bc-s4zg2    1/1         Running           0             3m

We can see the blue version of pods are being terminated and green version are up and running. This is how we will be doing a blue/green deployment in kubernetes.

More to Come, Happy learning :-)

Read More