Resource Management

A Capacity Planning Is used to determine the resources needed in order to meet the future workflow demands. In Kubernetes resource management is very important. We learned that scheduler is the component that identifies best nodes for creating pods.

The Scheduler has to take into account many factors when it is scheduling pods on nodes. If there are overlapping details or conflicting details the scheduler can find problem when allocating pods in nodes.

Let’s say if we nave a daemon set that is running on all nodes which  require 50% of memory, then scheduler can find it hard to allocate pods that would require around same memory.  Problems of this type can easily mitigated by using namespace resource quotas and correct management of the resource like Cpu, memory and storage

Kubernetes out of box supports resource quotas. The Admission control element in the API server makes sure to enforce resource quotas when one is defined. Kubernetes enforces one resource quota per name space.

The Resource Quotas in Kubernetes are 3 types

Compute

Storage

Object

Compute Resource Quota

This Quota talks about compute resources which are memory and Cpu. In this we can either request for a quota or limit a quota. This can be done by using limits.[cpu,memory] and requests.[cpu,memory]

Limits.cpu & limits.memory – This values talk about the sum of the CPU and Memory limits across all pods that cannot be exceeded.

Requests.cpu & requests.memory – This value talks about the sum of CPU and memory requests by all pods that cannot be exceeded

Storage Resource Quota

This storage resource quota talks about resource that can be restricted per namespace. The resource includes the amount of storage and persistent volume claims.

Object Resource Quota

The Object resource quota talks about the limitations on objects that can be created per namespace. The goal of this resource quota is to make sure that Api server is not wasting time performing things that are useless.

Compute Quota

Lets create a simple compute quota and see how things go. In the below manifest file we are limiting the number of pods to 2. When this quota is attached to a namespace, kubernetes blocks creation of the 3 pods in the namespace.

Create a resource quota config file as,
[root@manja17-I13330 kubenetes-config]# cat basic-resourcequota.yml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: pod-demo
spec:
  hard:
    pods: "2"

In our resource quota file we have limited our number of pod creation to 2. 

Create the resource quota as,

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-resourcequota.yml --namespace=sample-testing

resourcequota "pod-demo" created

Once created , add the resource quota to a namespace as,

[root@manja17-I13330 kubenetes-config]# kubectl get resourcequota --namespace=sample-testing

NAME       AGE

pod-demo   55s

When we get the pods, we see we already have 2 pods running in this namespace,

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing

NAME                                             READY STATUS RESTARTS AGE

test-service-deploy-85c8787b6f-sfprn   1/1 Running   0             57s

test-service-deploy-85c8787b6f-wvnfc  1/1 Running   0             14s

Now lets try to create a new pod and see how it goes

[root@manja17-I13330 kubenetes-config]# cat basic-single-container-pod.yml

apiVersion: v1

kind: Pod

metadata:

 name: testing-service

spec:

 containers:

   - name: test-ser

     image: docker.io/jagadesh1982/testing-service

     ports:

     - containerPort: 9876

     resources:

       limits:

         memory: "64Mi"

         cpu: "500m"

[root@manja17-I13330 kubenetes-config]# kubectl create -f  basic-single-container-pod.yml --namespace=sample-testing

Error from server (Forbidden): error when creating "basic-single-container-pod.yml": pods "testing-service" is forbidden: exceeded quota: pod-demo, requested: pods=1, used: pods=2, limited: pods=2

Now when i run the pod creation it gives me an error saying that the resource quota limit for pod is reached and cannot create any more.

Let’s see another example of Compute Quota,

[root@manja17-I13330 kubenetes-config]# cat basic-computequota.yml

apiVersion: v1

kind: ResourceQuota

metadata:

  name: computequota

spec:

  hard:

    pods: "2"

    requests.cpu: "1"

    requests.memory: 20Mi

    limits.cpu: "2"

    limits.memory: 2Gi

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-computequota.yml –namespace=sample-testing

resourcequota "computequota" created

[root@manja17-I13330 kubenetes-config]# kubectl get quota –namespace=sample-testing

NAME                AGE

computequota    14s

Describe the Quota using,

[root@manja17-I13330 kubenetes-config]# kubectl describe quota computequota

Name:               computequota

Namespace:       default

Resource                Used  Hard

--------                    ----  ----

limits.cpu                 0      2

limits.memory          0     2Gi

pods                        7     2

requests.cpu            0     1

requests.memory     0     20Mi

[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --namespace=sample-testing

deployment.apps "nginx" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing

No resources found.

[root@manja17-I13330 kubenetes-config]# kubectl get deploy --namespace=sample-testing

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

nginx       1               0                  0                   0              40s

[root@manja17-I13330 kubenetes-config]# kubectl describe deploy nginx --namespace=sample-testing

Name:                      nginx

Namespace:              sample-testing

CreationTimestamp:   Tue, 14 Aug 2018 02:29:26 -0400

Labels:                      run=nginx

Annotations:              deployment.kubernetes.io/revision=1

Selector:                   run=nginx

Replicas:                   1 desired | 0 updated | 0 total | 0 available | 1 unavailable

StrategyType:            RollingUpdate

MinReadySeconds:        0

RollingUpdateStrategy:  1 max unavailable, 1 max surge

Pod Template:

  Labels:  run=nginx

  Containers:

   nginx:

    Image:        nginx

    Port:         <none>

    Host Port:    <none>

    Environment:  <none>

    Mounts:       <none>

  Volumes:        <none>

Conditions:

  Type             Status  Reason

  ----             ------  ------

  Progressing      True    NewReplicaSetCreated

  Available        True    MinimumReplicasAvailable

  ReplicaFailure   True    FailedCreate

OldReplicaSets:    <none>

NewReplicaSet:     nginx-65899c769f (0/1 replicas created)

Events:

  Type    Reason             Age   From                   Message

  ----    ------             ----  ----                   -------

  Normal  ScalingReplicaSet  51s   deployment-controller  Scaled up replica set nginx-65899c769f to 1

Now if we check the replica set in the namespace sample-testing, we can see

[root@manja17-I13330 kubenetes-config]# kubectl describe rs nginx --namespace=sample-testing

Name:           nginx-65899c769f

Namespace:      sample-testing

Selector:       pod-template-hash=2145573259,run=nginx

Labels:         pod-template-hash=2145573259

                run=nginx

Annotations:    deployment.kubernetes.io/desired-replicas=1

                deployment.kubernetes.io/max-replicas=2

                deployment.kubernetes.io/revision=1

Controlled By:  Deployment/nginx

Replicas:       0 current / 1 desired

Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed

Pod Template:

  Labels:  pod-template-hash=2145573259

           run=nginx

  Containers:

   nginx:

    Image:        nginx

    Port:         <none>

    Host Port:    <none>

    Environment:  <none>

    Mounts:       <none>

  Volumes:        <none>

Conditions:

  Type             Status  Reason

  ----             ------  ------

  ReplicaFailure   True    FailedCreate

Events:

  Type     Reason        Age               From                   Message

  ----     ------        ----              ----                   -------

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-xgwrb" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-mr975" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-whtp9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  1m                replicaset-controller  Error creating: pods "nginx-65899c769f-g284h" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

  Warning  FailedCreate  32s (x6 over 1m)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-65899c769f-jzsv9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

The nginx container failed to create with the warning

 Warning  FailedCreate  32s (x6 over 1m)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-65899c769f-jzsv9" is forbidden: failed quota: computequota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

This clearly indicates the quota need to be defined for a pod to be created in sample-testing. Since we haven’t defined that the pod creation failed.

We can then  define the quota and then create the pod as,

[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --requests=cpu=100m,memory=4Mi --limits=cpu=200m,memory=8Mi --namespace=sample-testing

deployment.apps "nginx" created

[root@manja17-I13330 kubenetes-config]# kubectl get pods --namespace=sample-testing

NAME                    READY     STATUS    RESTARTS   AGE

nginx-97fb87dfc-9lcxb   1/1       Running   0          8s

Limit Quota

Rather then defining pods with quotas while creating, we can define out quota to allow default values if missing. This is what we call as limit Ranges. Lets create a quota with a limit range and see how things work,

[root@manja17-I13330 kubenetes-config]# cat basic-limitrangequota.yml

apiVersion: v1

kind: LimitRange

metadata:

  name: limits

spec:

  limits:

   - default:

       cpu: 200m

       memory: 6Mi

     defaultRequest:

       cpu: 100m

       memory: 5Mi

     type: Container

[root@manja17-I13330 kubenetes-config]# kubectl create -f basic-limitrangequota.yml --namespace=sample-testing

limitrange "limits" created

[root@manja17-I13330 kubenetes-config]# kubectl get limits --namespace=sample-testing

NAME      AGE

limits      1m

we can create pods with out providing any values. The default values will be assigned to the pod while creation,

[root@manja17-I13330 kubenetes-config]# kubectl run nginx --image=nginx --replicas=1 --namespace=sample-testing

More to come, Happy Learning :-)