Pages

Friday, August 31, 2018

Rkt - Getting Started with RKT

We all know how docker has taken the world of containers but it’s not perfect either. Docker has its own flaws and problems.

Docker architecture is Flawed - The first problem when people talk about docker is about its architecture. It is fundamentally flawed. What exactly does this mean?

in Linux, Init is a abbreviation for initialization. This is a daemon process which starts as soon as the machine starts or boot up. This process runs until the machine is shutdown. This is the first process that starts and also is responsible for starting other process and thus becoming the parent for all other process that starts. This will typically be assigned “pid=1”. The problem with this intialization process is that it starts processes in a serial order. It starts one tasks only after the last task startup was successful and it is loaded into the memory. This can result in longer startup.

There are many works done in enhancing the init process and they came up with the systemd. Though the systemd was not designed for speed but to start things faster ,cleaner and nicer.
Now the docker problem, When ever systemd start a service it creates a series of cgroups. When the same service spawns any child process , they stay in that same cgroup thus blocking them with the restrictions provided by that cgroup. This way systemd will always know what child process are created ,how much resources they are using etc. If that parent service dies all clean up can be done easliy regardless of the parent as systemd know all details about that parent and children.

Docker breaks all this by making a RPC call to the docker daemon when a “docker run” command is triggered. When this call is made, docker spawns a child process that becomes the PID 1 of the container. As soon as the docker client makes the RPC call , systemd will lose track of whats going on as the container is spawned in a different cgroup. this cgroup is different from the docker client cgroup which is being handled by the systemd. Hence systemd cannot manage the docker containers since they are trigged by a docker daemon and running in a different cgroup.
Hence docker is called as flawed. So what is Rkt and how is that different?

rkt is a container system developed by CoreOS as a light weight and secure alternative to docker. it is build based on the Open Container standard known as “App Container” or “appc” specification. The image build on this specification are protable across many container systems that follow the same standard.

How is Rkt different than this?
No Daemon - Rkt from CoreOs handles the setting up of the resources for the containers and then delegates everything to the systemd-nspawn to do the rest.
Protable image format - Since Rkt uses the “appc” specification it is portable to other container technologies that uses the same “appc” specification.
Light weight - Similar to Docker or runC, rkt also minimizes the number of process, resource utilization. It is easier to put all Application depdencies into a small image and ship that every where
Secure by Design - In Docker, if a user break out of a container using a kernel exploit, the attacker can control the whole machine where the container is started. An Additional security is required to handle this situation in docker. Rkt run containers as un priviliged users so that even some thing breaks out , they cannt affect other container or take control the underlying machine.
Rkt uses cryptographic signature checks on downloaded images so that only trustered containers are allowed to start and run.

More to Come,Happy learning :-)
Read More

Tuesday, August 14, 2018

Resource Management

A Capacity Planning Is used to determine the resources needed in order to meet the future workflow demands. In Kubernetes resource management is very important. We learned that scheduler is the component that identifies best nodes for creating pods.

The Scheduler has to take into account many factors when it is scheduling pods on nodes. If there are overlapping details or conflicting details the scheduler can find problem when allocating pods in nodes.

Let’s say if we nave a daemon set that is running on all nodes which  require 50% of memory, then scheduler can find it hard to allocate pods that would require around same memory.  Problems of this type can easily mitigated by using namespace resource quotas and correct management of the resource like Cpu, memory and storage

Kubernetes out of box supports resource quotas. The Admission control element in the API server makes sure to enforce resource quotas when one is defined. Kubernetes enforces one resource quota per name space.

The Resource Quotas in Kubernetes are 3 types
Compute
Storage
Object

Compute Resource Quota
This Quota talks about compute resources which are memory and Cpu. In this we can either request for a quota or limit a quota. This can be done by using limits.[cpu,memory] and requests.[cpu,memory]

Limits.cpu & limits.memory – This values talk about the sum of the CPU and Memory limits across all pods that cannot be exceeded.

Requests.cpu & requests.memory – This value talks about the sum of CPU and memory requests by all pods that cannot be exceeded

Storage Resource Quota
This storage resource quota talks about resource that can be restricted per namespace. The resource includes the amount of storage and persistent volume claims.

Object Resource Quota
The Object resource quota talks about the limitations on objects that can be created per namespace. The goal of this resource quota is to make sure that Api server is not wasting time performing things that are useless.

Compute Quota
Lets create a simple compute quota and see how things go. In the below manifest file we are limiting the number of pods to 2. When this quota is attached to a namespace, kubernetes blocks creation of the 3 pods in the namespace.

Create a resource quota config file as,
[root@manja17-I13330 kubenetes-config]# cat basic-resourcequota.yml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pod-demo
spec:
  hard:
    pods: "2"

In our resource quota file we have limited our number of pod creation to 2. 

Create the resource quota as,
[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-resourcequota.yml --namespace=sample-testing
resourcequota "pod-demo" created

Once created , add the resource quota to a namespace as,
[root@manja17-I13330 kubenetes-config]# kubectl get resourcequota --namespace=sample-testing
NAME       AGE
pod-demo   55s

When we get the pods, we see we already have 2 pods running in this namespace,
[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing
NAME                                             READY STATUS RESTARTS AGE
test-service-deploy-85c8787b6f-sfprn   1/1 Running   0             57s
test-service-deploy-85c8787b6f-wvnfc  1/1 Running   0             14s

Now lets try to create a new pod and see how it goes
[root@manja17-I13330 kubenetes-config]# cat basic-single-container-pod.yml
apiVersion: v1
kind: Pod
metadata:
 name: testing-service
spec:
 containers:
   - name: test-ser
     image: docker.io/jagadesh1982/testing-service
     ports:
     - containerPort: 9876
     resources:
       limits:
         memory: "64Mi"
         cpu: "500m"

[root@manja17-I13330 kubenetes-config]# kubectl create -f  basic-single-container-pod.yml --namespace=sample-testing

Error from server (Forbidden): error when creating "basic-single-container-pod.yml": pods "testing-service" is forbidden: exceeded quota: pod-demo, requested: pods=1, used: pods=2, limited: pods=2

Now when i run the pod creation it gives me an error saying that the resource quota limit for pod is reached and cannot create any more.

Let’s see another example of Compute Quota,
[root@manja17-I13330 kubenetes-config]# cat basic-computequota.yml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: computequota
spec:
  hard:
    pods: "2"
    requests.cpu: "1"
    requests.memory: 20Mi
    limits.cpu: "2"
    limits.memory: 2Gi

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-computequota.yml –namespace=sample-testing
resourcequota "computequota" created

[root@manja17-I13330 kubenetes-config]# kubectl get quota –namespace=sample-testing
NAME                AGE
computequota    14s

Describe the Quota using,
[root@manja17-I13330 kubenetes-config]# kubectl describe quota computequota
Name:               computequota
Namespace:       default
Resource                Used  Hard
--------                    ----  ----
limits.cpu                 0      2
limits.memory          0     2Gi
pods                        7     2
requests.cpu            0     1
requests.memory     0     20Mi

[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --namespace=sample-testing
deployment.apps "nginx" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing
No resources found.

[root@manja17-I13330 kubenetes-config]# kubectl get deploy --namespace=sample-testing
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx       1               0                  0                   0              40s

[root@manja17-I13330 kubenetes-config]# kubectl describe deploy nginx --namespace=sample-testing
Name:                      nginx
Namespace:              sample-testing
CreationTimestamp:   Tue, 14 Aug 2018 02:29:26 -0400
Labels:                      run=nginx
Annotations:              deployment.kubernetes.io/revision=1
Selector:                   run=nginx
Replicas:                   1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType:            RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:  run=nginx
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  Progressing      True    NewReplicaSetCreated
  Available        True    MinimumReplicasAvailable
  ReplicaFailure   True    FailedCreate
OldReplicaSets:    <none>
NewReplicaSet:     nginx-65899c769f (0/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  51s   deployment-controller  Scaled up replica set nginx-65899c769f to 1

Now if we check the replica set in the namespace sample-testing, we can see
[root@manja17-I13330 kubenetes-config]# kubectl describe rs nginx --namespace=sample-testing
Name:           nginx-65899c769f
Namespace:      sample-testing
Selector:       pod-template-hash=2145573259,run=nginx
Labels:         pod-template-hash=2145573259
                run=nginx
Annotations:    deployment.kubernetes.io/desired-replicas=1
                deployment.kubernetes.io/max-replicas=2
                deployment.kubernetes.io/revision=1
Controlled By:  Deployment/nginx
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  pod-template-hash=2145573259
           run=nginx
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age               From                   Message
  ----     ------        ----              ----                   -------
  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-xgwrb" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-mr975" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-whtp9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-g284h" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  32s (x6 over 1m)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-65899c769f-jzsv9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

The nginx container failed to create with the warning

 Warning  FailedCreate  32s (x6 over 1m)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-65899c769f-jzsv9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

This clearly indicates the quota need to be defined for a pod to be created in sample-testing. Since we haven’t defined that the pod creation failed.

We can then  define the quota and then create the pod as,
[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --requests=cpu=100m,memory=4Mi --limits=cpu=200m,memory=8Mi --namespace=sample-testing
deployment.apps "nginx" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing
NAME                    READY     STATUS    RESTARTS   AGE
nginx-97fb87dfc-9lcxb   1/1       Running   0          8s

Limit Quota
Rather then defining pods with quotas while creating, we can define out quota to allow default values if missing. This is what we call as limit Ranges. Lets create a quota with a limit range and see how things work,

[root@manja17-I13330 kubenetes-config]# cat basic-limitrangequota.yml
apiVersion: v1
kind: LimitRange
metadata:
  name: limits
spec:
  limits:
   - default:
       cpu: 200m
       memory: 6Mi
     defaultRequest:
       cpu: 100m
       memory: 5Mi
     type: Container

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-limitrangequota.yml --namespace=sample-testing
limitrange "limits" created

[root@manja17-I13330 kubenetes-config]# kubectl get limits --namespace=sample-testing
NAME      AGE
limits      1m
we can create pods with out providing any values. The default values will be assigned to the pod while creation,
[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --namespace=sample-testing

More to come, Happy Learning :-)
Read More

Monday, August 13, 2018

Kubernetes Deployment Strategy - Blue/Green

In this 3 post we will see how to do a blue/green deployment. A blue/green deployment is different from the rolling update in which a green version of the application is deployed along with the blue version. After testing the green version and once satisfied we will update the service to point to the green version. This way we will route the traffic from blue version to green version. This is done by playing with the labels in the selector fields. Once the traffic is moved to the green version, the blue version pods are terminated.

Lets create the blue deployment first as,
[root@manja17-I13330 kubenetes-config]# cat  blue-deployment.yml
apiVersion: v1
kind: Service
metadata:
  name: testing-service
  labels:
    app: testing-service
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: http

  # Note here that we match both the app and the version
  selector:
    app: testing-service
    version: "1.0"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testing-service-v1
  labels:
    app: testing-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: testing-service
      version: "1.0"
  template:
    metadata:
      labels:
        app: testing-service
        version: "1.0"
    spec:
      containers:
      - name: testing-service
        image: docker.io/jagadesh1982/testing-service
        ports:
        - name: http
          containerPort: 9876
        env:
        - name: VERSION
          value: "1.0"

[root@manja17-I13330 kubenetes-config]# kubectl apply -f blue-deployment.yml
service "testing-service" created
deployment.apps "testing-service-v1" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods
NAME                                             READY      STATUS    RESTARTS   AGE
testing-service-v1-bc6866889-g8zrr    1/1        Running      0              5m
testing-service-v1-bc6866889-lj8f7     1/1        Running      0              5m
testing-service-v1-bc6866889-qnr2r   1/1         Running      0              5m

[root@manja17-I13330 kubenetes-config]# kubectl get svc
NAME                TYPE          CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
kubernetes        ClusterIP    10.96.0.1           <none>          443/TCP           1d
testing-service   NodePort    10.110.200.110  <none>          80:30009/TCP   5m

[root@manja17-I13330 kubenetes-config]# curl 10.99.143.66/info
{"host": "10.99.143.66", "version": "1.0", "from": "10.32.0.1"}

Lets run the green deployment,with the below manifest file,
[root@manja17-I13330 kubenetes-config]# cat green-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testing-service-v2
  labels:
    app: testing-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: testing-service
      version: "2.0"
  template:
    metadata:
      labels:
        app: testing-service
        version: "2.0"
    spec:
      containers:
      - name: testing-service
        image: docker.io/jagadesh1982/testing-service
        ports:
        - name: http
          containerPort: 9876
        env:
        - name: VERSION
          value: "2.0"

[root@manja17-I13330 kubenetes-config]# kubectl apply -f green-deployment.yml
deployment.apps "testing-service-v2" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods
NAME                                              READY     STATUS    RESTARTS   AGE
testing-service-v1-bc6866889-g8zrr    1/1       Running             0          8m
testing-service-v1-bc6866889-lj8f7     1/1       Running             0          8m
testing-service-v1-bc6866889-qnr2r    1/1       Running            0          8m
testing-service-v2-7f966d5d4c-mb2fk  1/1       Running            0          2m
testing-service-v2-7f966d5d4c-qs6fz   1/1       Running             0          2m
testing-service-v2-7f966d5d4c-z4tj4   1/1       Running             0          2m

Side by side, 3 pods are running with version 2 but the service still send traffic to the first deployment. If necessary, you can manually test one of the pod by port-forwarding it to your local environment. Once your are ready, you can switch the traffic to the new version by patching the service to send traffic to all pods with label version=v2.0.0:

[root@manja17-I13330 kubenetes-config]# kubectl patch service testing-service -p '{"spec":{"selector":{"version":"2.0"}}}'
service "testing-service" patched

[root@manja17-I13330 kubenetes-config]# kubectl get svc
NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes        ClusterIP   10.96.0.1        <none>          443/TCP            1d
testing-service   NodePort    10.99.143.66  <none>          80:30126/TCP   3m

[root@manja17-I13330 kubenetes-config]# curl 10.99.143.66/info
{"host": "10.99.143.66", "version": "2.0", "from": "10.32.0.1"}

If you want to rollback to the older version , we can use
[root@manja17-I13330 kubenetes-config]# kubectl patch service testing-service -p '{"spec":{"selector":{"version":"1.0"}}}'
service "testing-service" patched

[root@manja17-I13330 kubenetes-config]# kubectl get svc
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
kubernetes        ClusterIP   10.96.0.1         <none>          443/TCP           1d
testing-service   NodePort   10.99.143.66   <none>          80:30126/TCP   4m

[root@manja17-I13330 kubenetes-config]# curl 10.99.143.66/info
{"host": "10.99.143.66", "version": "1.0", "from": "10.32.0.1"}

[root@manja17-I13330 kubenetes-config]# kubectl get deploy
NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
testing-service-v1   3               3                 3                 3                5m
testing-service-v2   3               3                 3                 3                2m

Once we are happy with the green version, delete the blue deployment using,
[root@manja17-I13330 kubenetes-config]# kubectl delete deploy testing-service-v1
deployment.extensions "testing-service-v1" deleted

Now check the pods ,
[root@manja17-I13330 kubenetes-config]# kubectl get pods
NAME                                                READY     STATUS        RESTARTS   AGE
testing-service-v1-bc6866889-dthrt      1/1         Terminating      0             5m
testing-service-v1-bc6866889-nk282    1/1         Terminating      0             5m
testing-service-v1-bc6866889-rxdtz      1/1         Terminating      0             5m
testing-service-v2-76dcdbd6bc-g56bh   1/1         Running           0             3m
testing-service-v2-76dcdbd6bc-jbvkw    1/1         Running           0             3m
testing-service-v2-76dcdbd6bc-s4zg2    1/1         Running           0             3m

We can see the blue version of pods are being terminated and green version are up and running. This is how we will be doing a blue/green deployment in kubernetes.

More to Come, Happy learning :-)
Read More