What happened?

Reproduction steps below, but at a high level:

1. I have a pod with 2 containers. I kill the pod. terminationGracePeriodSeconds is 30s
2. Container 1 terminates immediately. Container 2 terminates after 30s.
3. Immediately after the removal, pod is updated with deletionTimestamp
4. Container 1 terminates, but pod status does NOT change at all.
5. 30s later, container 2 terminates and pod dies.

What did you expect to happen?

After container 1 terminates, the containerStatus is updated to indicate the terminated container is no longer running.

How can we reproduce it (as minimally and precisely as possible)?

Apply this pod:

apiVersion: v1
kind: Pod
metadata:
  name: foo
spec:
  containers:
  - name: echo
    image: gcr.io/istio-testing/app:latest
  - name: sleep
    image: howardjohn/shell
    args:
    - /bin/sleep
    - infinity

echo container exits immediately, sleep does not until SIGKILL.

Next, run kubectl delete foo.

Observe that after a few seconds (up to 30s), the container status is not updated:

  containerStatuses:
  - containerID: containerd://399d45c45585232ef561c61267e40793146a6dd1bda93131abc311611dc4bae7
    image: gcr.io/istio-testing/app:latest
    imageID: gcr.io/istio-testing/app@sha256:1c0f400e31d492c2af4a2370943f3697a9a65823a179971ec720e510fba7bf78
    lastState: {}
    name: echo
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-12-09T00:44:44Z"
  - containerID: containerd://55b438c97c5aad6c10256ff90a42e2edd50295697dccef9d3bcae0a5dde96bb8
    image: docker.io/howardjohn/shell:latest
    imageID: docker.io/howardjohn/shell@sha256:9139585d3569e38dffbf035716b7ce1b1c6578e6276b10b82b09e1187f7917ff
    lastState: {}
    name: sleep
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-12-09T00:44:45Z"

Kubelet logs, with -v6:
kubelet.txt

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T21:30:26Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)