The haunting of the docker and kubectl exec commands

I have discovered that containers can become haunted with ghosts, this is the story. Ok, it's not that spooky, but it is certainly very irritating and unexpected, and if you use the either the Docker or the Kuberentes version of the exec command, you should know about this.

So, last week, in some scripts that utilized kubectl-exec, I discovered some surprising behaviour. Normally, exec is used for one-off triggers and such operations with running Kubernetes pods.

Is that such a great idea? Is it what you do in production? Sure, it is not.

Shouldn't you be using `jobs` for this?. Maybe. The thing is, there are situations, where exec is the only feasible option.

So, imagine your exec call like this:

kubectl exec pod-name-with-lengthy-uuid -- bash -c "some command that does not end"

Notice the lack of the -i and -t flags. This is intentional; we are running this command from inside a script. Most likely this command will be backgrounded.

What do you think will happen if you Ctrl-C this command or kill it if it's a background process? ps will show that the exec is gone. But has it?

It has not. It has become, what I term, a ghost. Invisible to you on the source machine where you ran the exec, the process inside the container is still running, not dead yet. This exact thing happens with docker exec.

This "feature" has been left hanging in docker since 2014 and discovered in Kubernetes more recently. Neither of those threads indicate any sign of things changing.

So what do you do to dispel the ghosts? Rather, in elegant, I am afraid. You need to run a follow up exec command with pkill to forcibly clean up. But while this takes out the ghosts, be warned, you will be left with zombies inside the container.