Control codes, it flags and the Terminal Raw-mode

When you run watch with a kubectl command with -it flags, sometimes it gets stuck - with none of the Control keys like Ctrl-C, Ctrl-D or Ctrl-Q having any effect. I didn't get to the bottom of this, but learnt some intersting things that I wanted to document here.

First, the source code for watch is housed on Gitlab of all places: https://gitlab.com/procps-ng/procps. Not just that, this is also the home for a number of other familiar unix tools: the likes of ps, free, pkill, top etc!

I found an issue which seemed similar at the outset: https://gitlab.com/procps-ng/procps/-/issues/224. I learnt one trick from that issue, to help debug this sort of things:

strace -o trace.txt --trace=%process -t watch -n1 <cmd>

The idea is to capture the process executions done by watch - obviously. But, this also includes the signal received by the watch command. I guess, you can also focus on the same using %signal directly. Anyway, the point of this command is to check if SIGINT is being received by watch when you press Ctrl-C or not. If the signal is being received, then clearly it is a bug in watch. If not, something else has gone wrong. In this case, I could see that the signal never got to watch.

Though I have not been able to confirm, I had my suspicion: the use of the -it flags in the command which are only needed if you are interactively working with commands like kubectl, docker or podman. In most cases, just omitting these for non-interactive commands should be the way to go anyway. That should solve the issue in the right way.

But, since I was this far, I had a question long un-answered in my mind: what do the -it flags even do internally?

So, this once I decided to go find the source. Specifically for kubectl. The analysis below is obviously valid only as of the date of this post.

The exec subcommand is to be found here: https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/exec/exec.go

In the NewCmdExec function, we see the lines:

cmd.Flags().BoolVarP(&options.Stdin, "stdin", "i", options.Stdin, "Pass stdin to the container")
cmd.Flags().BoolVarP(&options.TTY, "tty", "t", options.TTY, "Stdin is a TTY")

These option flags are stored in a ExecOptions struct that extends StreamOptions. Somewhere down the line, we have the `Run()` method on ExecOptions, which has this line:

t := p.SetupTTY()

The SetupTTY func is to be found as a method of StreamOptions in that very file. After a number of checks using these flags, the following seems to be happening:

1. The terminal is abstracted using another package: https://github.com/kubernetes/kubectl/blob/master/pkg/util/term/term.go
2. If the -i flag is set, uses os.Stdin as the term's In.
3. If the -t flag is set, a bool called Raw is set in the underlying object.

In the end of the above mentioned Run() function, we have a call to the underlying TTY.Safe() method - to be found at the end of the term.go file.

Turns out that the k8s term package is itself a thin layer on top of another: https://github.com/moby/term. So the wrapper calls the term.MakeRaw method from the moby term package when the Raw bool is set. Finally, it just calls the function passed to it with some handling for termination and clean up.

What is this MakeRaw? This leads us to: https://en.wikipedia.org/wiki/Terminal_mode. In Cooked mode, which the default in most cases, data is preprocessed before being sent to the downstream process which takes care of things like signalling. In raw mode, the key presses are directly forwarded to the downstream application.

Using the moby term library, here is an example:

package main

import (
        "github.com/moby/term"
        "os"
        "fmt"
        "time"
)

func main () {
        fd := os.Stdin.Fd()

        if term.IsTerminal(fd) {
                fmt.Println("Running in terminal!")
        }

        var state *term.State
        var err error
        state, err = term.MakeRaw(fd)
        if err != nil {
                fmt.Println("Error in making raw: %v", err)
        }

        var char rune
        for true {
                _, err := fmt.Scanf("%c", &char)
                if err != nil {
                        fmt.Println("Error in reading char: %v", err)
                }
                fmt.Printf("Read: %v\n", char)
         }

        term.RestoreTerminal(fd, state)
}

Careful: when you run this, you will lose that terminal since it is set to RawMode and all signals will simply get absorbed by the program, not as signals, but as raw data.

If you run this and try inputting commands like Ctrl-C or Ctrl-D, you should see output as 3 and 4. Fine, but how are these numbers calculated?

Turns out there is a simple approach to it. As explained in this Stack Overflow answer: https://unix.stackexchange.com/questions/443484/are-ascii-escape-sequences-and-control-characters-pairings-part-of-a-standard, they are a single bit flip from the Ascii equivalent.

It is quite apparent from the man ascii ouput, where data is arranged in the form of 2 columns, the second starting from 64 onwards. The character codes in the first and second columns can be easily mapped.

Now, as far as the original problem is concerned: I am not still not clear. It occured non-deterministically with -it programs, so I think it may also be a timing thing on when the subprocesses called by watch end, when the internal clean up functions for the terminal are called by the called process etc.

So for now, this is just an info dump.