kubernetes - What's inside Etcd?

Etcd is a simple distributed key-value store which uses the Raft algorithm. Kubernetes use Etcd to store information on what's happening in the cluster. Whatever happens in the cluster like creating pods, services, pods details like where they are created etc, all these details will be saved inside the etcd server.

In this article we will see how we can connect to the etcd server , access its datasource and see how things work.

How to get into the Etcd Pod
Every kubernetes cluster has a etcd configured. The api server is the only component in the whole cluster who can talk to the etcd. Whatever we want to do in the cluster, like creating pod, destroying etc will be first sent to the api server and then api server talks to the etcd to delete the requested details and then performs the actions requested.

Api server will  have the certificates to connect to the etcd server. If we go to the /etc/kubernetes/manifests location in the kubernetes master node, we can see the yml files for creating the kube-system pods. We can also see the etcd.yml file which gives us much information about the etcd server. In the manifest file, we can see how etcd is started and how it can be accessed as below,

containers:
  - command:
    - etcd
    - --advertise-client-urls=https://10.0.2.15:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.0.2.15:2380
    - --initial-cluster=logging.machine.vm=https://10.0.2.15:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379
    - --listen-peer-urls=https://10.0.2.15:2380
    - --name=logging.machine.vm
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: k8s.gcr.io/etcd:3.3.10
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - -ec
        -ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
          --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
          get foo
      failureThreshold: 8
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: etcd
    resources: {}
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true

If we see the above manifest file, we can see the client interface is bound to 127.0.0.1 which means we are able to access to etcd only using localhost. The port is 2379 default port for etcd and the -client-cert-auth=true says that the certificate authentication is activated. This means we need to have certificates in order to connect to the etcd.

Also we can see the data is stored in the /var/lib/etcd location from host machine. This means the data is being stored in the host machine /var/lib/etcd location.

First connect to the etcd server and find the location of the certificates,
[root@logging ~]# kubectl exec -it --namespace kube-system kube-apiserver-logging.machine.vm sh

# cd /etc/kubernetes/pki
# ls
apiserver-etcd-client.crt     apiserver-kubelet-client.key  ca.crt  front-proxy-ca.crt        front-proxy-client.key
apiserver-etcd-client.key     apiserver.crt            ca.key  front-proxy-ca.key        sa.key
apiserver-kubelet-client.crt  apiserver.key            etcd    front-proxy-client.crt  sa.pub
# exit

We can see that the certificates are located at /etc/kubernetes/pki location in the kube-apiserver pod.

Lets download the keys that are necessary for connecting to the etcd,
[root@logging ~]# kubectl cp --namespace kube-system kube-apiserver-logging.machine.vm:etc/kubernetes/pki/apiserver-etcd-client.crt apiserver-etcd-client.crt

[root@logging ~]# kubectl cp --namespace kube-system kube-apiserver-logging.machine.vm:etc/kubernetes/pki/apiserver-etcd-client.key apiserver-etcd-client.key

[root@logging ~]# ll
total 28
-rw-r--r--. 1 root root 1090 Apr 12 02:11 apiserver-etcd-client.crt
-rw-r--r--. 1 root root 1675 Apr 12 02:11 apiserver-etcd-client.key

Copy the above downloaded certificates to the etcd server,
[root@logging ~]# kubectl cp --namespace kube-system apiserver-etcd-client.crt etcd-logging.machine.vm:etc/kubernetes/pki/etcd/
[root@logging ~]# kubectl cp --namespace kube-system apiserver-etcd-client.key etcd-logging.machine.vm:etc/kubernetes/pki/etcd/

Connect to the kube-apiserver using,
[root@logging ~]# kubectl exec -it --namespace kube-system kube-apiserver-logging.machine.vm sh
 
We need to set etcdctl tool to v3 API version using following environment variable: export ETCDCTL_API=3

One we set the etcdctl, we can then connect to the local etcd server using,
/etc/kubernetes/pki/etcd # etcdctl --cacert=ca.crt --key=apiserver-etcd-client.key --cert=apiserver-etcd-client.crt endpoint status
127.0.0.1:2379, f074a195de705325, 3.3.10, 1.4 MB, true, 2, 9248

Understand the Data
/etc/kubernetes/pki/etcd # etcdctl --cacert=ca.crt --key=apiserver-etcd-client.key --cert=apiserver-etcd-client.crt get / --prefix
 --keys-only

/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io

The output is too large. We just got a snippet of the content.
/registry/minions/logging.machine.vm : This contains the location of the node details. In this case logging.machine.vm is the node name. This contains many details regarding the node.

To get the details we can use the same above command by passing the location to it as,
/etc/kubernetes/pki/etcd # etcdctl --cacert=ca.crt --key=apiserver-etcd-client.key --cert=apiserver-etcd-client.crt get /registry/minions/logging.machine.vm

/registry/namespaces/ : Namespace details. This list all the available namespaces. We can pass the namespace to the location to get status of the namespace whether active or terminated.

/registry/pods/{namespace}/{pod name} - State of every pod running in cluster. Contains a lot of information like pod IP, mounted volumes, docker image etc.

/registry/ranges/servicenodeports - Ports range for exposing services
/registry/secrets/{namespace}/{pod} - All secrets in cluster stored as plain text in default mode

This way we can connect to the etcd server running in the kubernetes cluster and gather more details about the things running inside a cluster.