Docker - Trapping Signals inside a Container

A signal is a message to a process from the kernel to notify that some condition has occurred. When a signal is issued to a process, the process is interrupted and a signal handler is executed. If there is no signal handler, the default handler is called instead.

A Docker container will also receive signals. In docker , we have two commands that we can use to stop it, docker stop and docker kill. When we do a docker stop to a running container, it sends a SIGTERM signal to the main process running inside the container ( pid 1 process ) and after a grace period it issues a SIGKILL to terminate the process. At this moment the process can ignore the signal or let a default action occur or provide a callback function to respond to the signal.

Lets run a container with the sleep command and try to pass a signal to the container as,

jagadishm@[/Volumes/Work/build/trap]: docker run -it centos sleep 100

^C^C

In this case, the container will not exit until the sleep 100 is complete which means the signal that we passed is not received by the container process.

Now let's write a simple bash trap script as below,

jagadishm@[/Volumes/Work/build/trap]: cat signal.py 

import signal

import sys

def signal_handler(sig, frame):

        print('You pressed Ctrl+C!')

        sys.exit(0)

signal.signal(signal.SIGINT, signal_handler)

while True:

    print('Press Ctrl+C')

    signal.pause()

    time.sleep(10) #make function to sleep for 10 seconds

This script goes to infinite sleep mode but it has the trap command which handles the signals that we pass. Lets create a docker image trap with the below contents,

jagadishm@[/Volumes/Work/build/trap]: cat Dockerfile 

FROM centos

COPY signal.py /

WORKDIR /

ENTRYPOINT ["python”,”signal.py”]

Now when we run the container and try to send the signal “ctrl + c “ as below,

jagadishm@[/Volumes/Work/build/trap]: python signal.py 

Press Ctrl+C

^CYou pressed Ctrl+C!

We can see that the signal is sent to the process and even handled by the trap expression by executing the commands defined. One the above container is up and running ,from another terminal window run the “docker kill --signal

Signal by Docker - Similarly Docker allows to send signals to the process running inside them. Signals can be sent by stop,rm and kill commands in docker.

By Stop - Docker stop command allows to send a stop signal to the process running inside it. When we issue a stop command, the process will be asked nicely to stop and if it doesn’t respond in 10 seconds it will forcibly kill it. The docker stop command attempts to stop a running container first by sending a SIGTERM signal to the root process ( ie. process running with pid 1). If the process hasn't exited within the timeout period a SIGKILL signal will be sent.

The only thing that we can control with the stop command is the time to wait until the docker daemon will wait before sending a SIGKILL. A stop command with the time can be defined as “docker stop ----time=30 <Container ID>”

By Kill - Similarly we can send the signal by using the docker kill command. The main process inside the container is sent a SIGKILL signal ( default ) , or a signal that is specified with the --signal option. 

The --stop-signal flag sets the system call signal that will be sent to the container to exit. The signal can be a valid number that matches a position in the kernel syscall table for instance 9, or by a signal name SIGNAME for instance SIGINT.

From the first terminal run the command “docker run -it trap /bin/bash” as below

jagadishm@[/Volumes/Work/build/trap]: docker run -it trap /bin/bash

Press Ctrl+C

Now from another terminal window, run the command “docker kill --signal="SIGINT" <Container ID>” . once we pass the SIGINT signal to the container, we can see the below output,

jagadishm@[/Volumes/Work/build/trap]: docker run -it trap /bin/bash

Press Ctrl+C

You pressed Ctrl+C!

By RM - docker rm command is used to remove already stopped container but when used in conjunction with --force flag a SIGKILL is passed to the pid 1 inside the container. It can be used as “docker rm --force <Container ID>”

Hope this helps in understanding how to pass signals to a container and define a shutdown behavior for the process running inside the container.

Read More

Linux - Trapping Signals

It is a general practice to interrupt a running program by passing a signal to the program or some times there may be situations that you don’t want users of your script to interrupt or exit in between by passing signals. 

A signal is a message to a process from the kernel to notify that some condition has occurred. For example, When you press the “ctrl + c” while running a program we are basically sending a signal to that program to terminate. When a signal is issued to a process, the process is interrupted and signal handler is executed. If no signal handler is available, then the default signal handler is called instead. Just run the “man -k signal” command to see the list of signals available. 

Some of the common signal types are:

SIGINT: user sends an interrupt signal (Ctrl + C). this is sent when user press the ctrl +c .

SIGKILL - The SIGKILLsignal is used to cause immediate program termination. It cannot be handled or ignored, and is therefore always fatal. It is also not possible to block this signal.

SIGTERM : this is the most generic signal used to cause program termination. Unlike SIGKILL, this signal can be blocked, handled, and ignored. It is the normal way to politely ask a program to terminate.

SIGQUIT: user sends a quit signal (Ctrl + C)

Note - the SIGKILL and SIGSTOPsignals cannot be caught, blocked or ignored.

Handling a signal is crucial for programs since they need to perform some cleanup of resources like file deletion etc before the process getting shut completely. Bash provides a simple utility called trap by which we can customize a script behavior when the script receives a signal. This is very useful, for example, to make sure that the system is always in a consistent state

[root@ansible ~]# trap -l

 1) SIGHUP           2) SIGINT     3) SIGQUIT 4) SIGILL 5) SIGTRAP

 6) SIGABRT         7) SIGBUS     8) SIGFPE 9) SIGKILL 10) SIGUSR1

11) SIGSEGV        12) SIGUSR2 13) SIGPIPE 14) SIGALRM

15) SIGTERM         16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT

19) SIGSTOP         20) SIGTSTP 21) SIGTTIN 22) SIGTTOU

23) SIGURG  24) SIGXCPU 25) SIGXFSZ         26) SIGVTALRM

27) SIGPROF 28) SIGWINCH 29) SIGIO         30) SIGPWR

31) SIGSYS         34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2

37) SIGRTMIN+3  38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6

41) SIGRTMIN+7 42) SIGRTMIN+8  43) SIGRTMIN+9 44) SIGRTMIN+10

45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13  48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12

53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8

57) SIGRTMAX-7   58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4

61) SIGRTMAX-3 62) SIGRTMAX-2   63) SIGRTMAX-1 64) SIGRTMAX

Let's write a simple program to understand how we can trap signals,

[root@ansible ~]# cat traptest.sh 

#!/bin/bash

#

# A simple script to demonstrate how trap works

#

set -e

set -u

set -o pipefail

trap 'echo "  signal caught, cleaning..."; rm -irf /tmp/hello' SIGINT SIGTERM

echo "Going to Sleep..."

while true

do 

  sleep 10000

done

In the above script, we used trap to catch the SIGINT and SIGTERM signals. The program goes to an infinite sleep loop and when i press Ctrl + C, the program terminals but it also executes the command that we defined in the trap condition.

[root@ansible ~]# sh traptest.sh 

Going to Sleep...

^C  signal caught, cleaning...

In this case the command are actually two, the first is the echo and second is the removal of the files in /tmp location. Instead of specifying command this way we can define a function and call it with whatever actions we want to do

Hope this short introduction helps in understanding signals and how they can be trapped in bash.

Read More