Skip to content

Pod deletion timestamp is resetted after preStop hook is executed #132205

Open
@aojea

Description

@aojea

this looks like a bug to me, having deletionTimestamps that fluctuate is confusing at minimum, let me try to understand the timeline:

t0 create pod with finalizer
t1 delete pod -> deletionTimestamp is equal to t1+gracePeriod
at t1 + sleepPeriod -> deletionTimestamp is updated and is equal to t1??

yep, that is confirmed with

apiVersion: v1
kind: Pod
metadata:
  name: prestop-example
  finalizers:
    - test/finalizer
spec:
  terminationGracePeriodSeconds: 30
  containers:
  - name: app-container
    image: registry.k8s.io/e2e-test-images/agnhost:2.48
    lifecycle:
      preStop:
        sleep:
          seconds: 5

after preStop timestamp goes back in time

$ date && kubectl get pods  prestop-example -o yaml | grep -i timestam
Tue Jun 10 11:37:31 AM UTC 2025
  creationTimestamp: "2025-06-10T11:36:20Z"
  deletionTimestamp: "2025-06-10T11:37:56Z"
$ date && kubectl get pods  prestop-example -o yaml | grep -i timestam
Tue Jun 10 11:37:32 AM UTC 2025
  creationTimestamp: "2025-06-10T11:36:20Z"
  deletionTimestamp: "2025-06-10T11:37:32Z"

it seems the kubelet sends a new DELETE after the prestop hook is executed

I0610 11:55:01.644079       1 httplog.go:134] "HTTP" verb="DELETE" URI="/api/v1/namespaces/default/pods/prestop-example" latency="5.239176ms" userAgent="kubectl/v1.33.1 (linux/amd64) kubernetes/8adc0f0" audit-ID="1b48906d-bda7-4599-8cad-7ff983291186" srcIP="192.168.8.1:54692" apf_pl="global-default" apf_fs="global-default" apf_iseats=1 apf_fseats=0 apf_additionalLatency="0s" apf_execution_time="4.622037ms" resp=200




I0610 11:55:06.982183       1 handler.go:152] kube-aggregator: DELETE "/api/v1/namespaces/default/pods/prestop-example" satisfied by nonGoRestful
I0610 11:55:06.982209       1 handler.go:142] kube-apiserver: DELETE "/api/v1/namespaces/default/pods/prestop-example" satisfied by gorestful with webservice /api/v1


Originally posted by @aojea in #128642 (comment)

The logic that sets the deletion timestamp considering the grace period comes from

// CheckGracefulDelete allows a pod to be gracefully deleted. It updates the DeleteOptions to
// reflect the desired grace value.
func (podStrategy) CheckGracefulDelete(ctx context.Context, obj runtime.Object, options *metav1.DeleteOptions) bool {

and is added in

// `CheckGracefulDelete` will be implemented by specific strategy
if !gracefulStrategy.CheckGracefulDelete(ctx, obj, options) {
return false, false, nil
}
if options.GracePeriodSeconds == nil {
return false, false, errors.NewInternalError(fmt.Errorf("options.GracePeriodSeconds should not be nil"))
}
requestedDeletionTimestamp := metav1.NewTime(metav1Now().Add(time.Second * time.Duration(*options.GracePeriodSeconds)))
objectMeta.SetDeletionTimestamp(&requestedDeletionTimestamp)
objectMeta.SetDeletionGracePeriodSeconds(options.GracePeriodSeconds)
// If it's the first graceful deletion we are going to set the DeletionTimestamp to non-nil.
// Controllers of the object that's being deleted shouldn't take any nontrivial actions, hence its behavior changes.
// Thus we need to bump object's Generation (if set). This handles generation bump during graceful deletion.
// The bump for objects that don't support graceful deletion is handled in pkg/registry/generic/registry/store.go.

I didn;t have time to trace the preStop hook logic, but it seems on the experiments that it sends a new Delete request , the logic seems to start here

// executePreStopHook runs the pre-stop lifecycle hooks if applicable and returns the duration it takes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.sig/nodeCategorizes an issue or PR as relevant to SIG Node.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions