Skip to content

Kubelet stops rotating service account tokens when pod is terminating, breaking preStop hooks #116481

@PaulFurtado

Description

@PaulFurtado

What happened?

We have pods that use the kube API during their preStop hooks and we use 12h service account token expirations. Today we had a pod where the token was approaching expiration at the time the pod was deleted. 4h into the preStop hook, the token expired and the preStop hook started failing.

What did you expect to happen?

Service account token rotation should continue working until pods have fully terminated.

How can we reproduce it (as minimally and precisely as possible)?

Run a pod with a 10-minute service account token expiration and long terminationGracePeriodSeconds and a preStop hook that does sleep infinity. Exec into the pod after 10 minutes have elapsed and check the token, it will be expired.

Anything else we need to know?

No response

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"clean", BuildDate:"2023-01-18T16:01:10Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"archive", BuildDate:"2023-03-01T14:28:40Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

AWS, but N/A for this issue

OS version

CentOS 8 stream, but N/A for this issue

Install tools

N/A

Container runtime (CRI) and version (if applicable)

Occurs with both cri-o and cri-dockerd

Related plugins (CNI, CSI, ...) and versions (if applicable)

N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.sig/authCategorizes an issue or PR as relevant to SIG Auth.sig/nodeCategorizes an issue or PR as relevant to SIG Node.sig/storageCategorizes an issue or PR as relevant to SIG Storage.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Projects

    Status

    Pending other SIGs

    Status

    Triaged

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions