Falco - Container Bahavior Analysis

End-to-End protection for containers in production is required to avoid the steep operational costs and also to decrease data breaches. New and Fresh Container attacks and Vulnerabilities continue to increase, a strong runtime security is required for containers.

Runtime container security means vetting all activities within the container application environment from analysis of container, runtime and host activities to monitoring protocols and payloads of network connections. Some of the vulnerabilities can be remediated by Host benchmarking and vulnerability scanning on Host Operating systems. Container images are scanned for vulnerabilities and remediation before running them but there is a need for analysis monitoring of the running containers. We need to understand what is happening within the running containers, what network calls are being made, what directories and drives are being accessed etc. It is very important to understand how the containers are behaving while running. This is where Container Runtime monitoring comes into picture and Falco is one such tool. In this blog we learn the basics of Falco tool and how that can be used.

Introducing Sysdig Falco 

Sysdig Falco is a Powerful behavioral activity monitoring tool to detect abnormal behavior in your applications and containers. Falco is cloud native runtime security systems that works with both containers and raw linux hosts. Developed by Sysdig organization for the Cloud Native computing foundation, it works by looking at the file changes, network activity, the process table and other data for suspicious behavior and then sending alerts through a pluggable backend. It inspects events at the system call level of a host through a kernel module or an extended BPF probe.

Falco works on rules that we can edit for identifying specific abnormal behaviors and it comes with 25 rules installed.

Installation and configuration

On a centos based machine, install the rpm as below,

[root@ip-172-31-32-147]#dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

Enable the repo using,

[root@ip-172-31-32-147]#dnf config-manager --enable epel

[root@ip-172-31-32-147]# rpm --import https://falco.org/repo/falcosecurity-3672BA8F.asc

[root@ip-172-31-32-147]# curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo

[root@ip-172-31-32-147]# yum -y install kernel-devel-$(uname -r)

[root@ip-172-31-32-147]# yum -y install falco

Once the installation is done, a directory /etc/falco is created which contains 2 files

/etc/falco.yml,/etc/falco_rules.yml and /etc/falco/falco_rules.local.yml.

The file /etc/falco.yml controls several logging and high level configurations. 

The file /etc/falco_rules.yml contains the list of rules that can be configured for abnormal behavior checking.It contains a predefined set of rules designed to provide good coverage in a variety of situations.

The file /etc/falco/falco_rules.local.yml is an empty file with some comments. The intent is that additions/modifications/overrides to the main rules file are added to this file. This can be taught of a custom rules file for one organization.

Run the Service as,

[root@ip-172-31-32-147]# service falco restart

Writing Your First Rule

Falco rules are based on Sysdig filter syntax. These filters expose a variety of information about system calls and events that take place in the system. These filters are organized as classes called “field classes”.  These classes can be see running the “sysdig -l”. Some of the filters include 

Fd : file descriptors

Process : processes

Evt : System events

User : Users

Group : groups

Container : container info

K8s : kubernetes events

Falco Rules are written in YAML format with some required and optional keys. A simple syntax of a rule,

Rule        : Name of the rules
Desc        : Description of what the rule is
Condition : the logic that triggers a notification
Output     : message that will be shown in the notification
Priority     : logging level for the notification
Tags        : tags for categorize rules
Enabled   : turn the rule on or off

A simple falco rule for checking if a shell was triggered in a container

Rule 1 : Log if there is shell trigger in a Container

Create a rule as below,

- rule: Terminal shell in container

  desc: A shell was spawned by a program in a container with an attached terminal.

  condition: >

    spawned_process and container

    and shell_procs and proc.tty != 0

  output: "A shell was spawned in a container with an attached terminal (user=%user.name %container.info shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty)"

  priority: NOTICE

  tags: [container, shell]

Now add the rules to the /etc/falco_rules.yml file and restart the Falco service. Now start a container and run bash inside the container as below,

[root@ip-172-31-32-147]#docker run -d -P --name example2 nginx

[root@ip-172-31-32-147]#docker exec -it example2 bash

Come out of the container and see the /var/log/messages file in the host machine. We can see the below type of logs

May 24 13:28:14 ip-172-31-32-147 falco[16586]: 13:28:14.523089250: Notice A shell was spawned in a container with an attached terminal (user=root example2 (id=1546ca8ce5f0) shell=bash parent=runc cmdline=bash terminal=34816)

May 24 13:28:23 ip-172-31-32-147 falco[16586]: 13:28:23.947776826: Warning Shell history had been deleted or renamed (user=root type=openat command=bash fd.name=/root/.bash_history name=/root/.bash_history path= oldpath= example2 (id=1546ca8ce5f0))

The log says that a shell is spawned in the container with an attached terminal. It also gives information about the user log triggered the shell and name of the container with container id. This way we can see what happens inside a container based on the rule that we defined.

Rule 2 : Check if other processes are running.

Docker best practices recommend running just one process per container. It can be a security issue if there are some other processes too running in a container. In our nginx container, we want only the nginx process to run. We want to log whatever process or job that run inside a nginx container other than nginx processes. Our rule would look like,

- rule: Unauthorized process on nginx containers

  desc: There is a process running in the nginx container that is not described in the template

  condition: spawned_process and container and container.image startswith nginx and not proc.name in (nginx)

  output: Unauthorized process (%proc.cmdline) running in (%container.id)

  priority: WARNING

Lets Understand the rule a little,

Spawned_process : a macro to identify that a new process was executed.

Container : the container namespace where it was executed belongs to a container and not the host

Container.image startswith nginx : the image name so you can have an authorized process lists for each one

not proc.name in (nginx) (the list of allowed processes names)

[root@ip-172-31-32-147]# docker run -d -P --name example2 nginx

[root@ip-172-31-32-147]# docker exec -it example2 ls

Run a nginx container and run “ls” inside the nginx container.Now when we check the /var/log/messages file, we can see the below logs as,

May 24 13:34:20 ip-172-31-19-104 falco[17179]: 13:34:20.823028587: Warning Unauthorized process (ls) running in (1546ca8ce5f0)

May 24 13:34:20 ip-172-31-19-104 dockerd[15797]: time="2020-05-24T13:34:20.876166289Z" level=error msg="Handler for POST /v1.39/exec/c4180687a5b2dc2b9d58b7e8ca20f5242e0b3a7657751b7a437645c012f2a67a/resize returned error: cannot resize a stopped container: unknown"

Though falco is a good tool for behavior checking, there are few limitations for the tool. Though it supports integrations with multiple tools , more work needs to be done on enhancing the tool. Hope this helps in starting with the Falco tool.