The Wayback Machine - https://web.archive.org/web/20211003223322/https://github.com/kubernetes/kubernetes/issues/104432
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

imagePullSecrets should throw error if secret does not exist #104432

Open
neilharris123 opened this issue Aug 18, 2021 · 28 comments · May be fixed by #104535
Open

imagePullSecrets should throw error if secret does not exist #104432

neilharris123 opened this issue Aug 18, 2021 · 28 comments · May be fixed by #104535

Comments

@neilharris123
Copy link

@neilharris123 neilharris123 commented Aug 18, 2021

What happened:

imagePullSecrets can added to a pod manifest to reference a secret containing registry credentials. Currently, if you add the name of a secret that does not exist, there is no warning or error.

What you expected to happen:

I would expect there to be an error and a failed deployment if the secret does not exist/cannot be found in the namespace.
If people are pulling public images, they may still wish to authenticate to a registry, for example dockerhub, before pulling the image, in order to utilise their subscription account rate limits. Currently if they use imagePullSecrets and the referenced secret does not exist, they may assume they are authenticating to their account when they are not, and will still be able to pull public images without any warning that authentication is not taking place.

How to reproduce it (as minimally and precisely as possible):

Add imagePullSecrets: myfakesecret to a pod spec, and pull images from docker.io.

@neolit123
Copy link
Member

@neolit123 neolit123 commented Aug 18, 2021

Add imagePullSecrets: myfakesecret to a pod spec, and pull images from docker.io.

/sig node
(based on the field location)

@neolit123
Copy link
Member

@neolit123 neolit123 commented Aug 18, 2021

(and/or)
/sig auth

@kerthcet
Copy link
Member

@kerthcet kerthcet commented Aug 18, 2021

correct me if not. Pulling private images with nonexists secret, pod will turn to ErrImagePull. However, pulling public image with nonexists secret, no error will display yet.

@pacoxu
Copy link
Member

@pacoxu pacoxu commented Aug 19, 2021

Similar issue with #88142.

Perhaps a warning event should be created?

@neilharris123
Copy link
Author

@neilharris123 neilharris123 commented Aug 19, 2021

correct me if not. Pulling private images with nonexists secret, pod will turn to ErrImagePull. However, pulling public image with nonexists secret, no error will display yet.

Yes that's correct.

@kerthcet
Copy link
Member

@kerthcet kerthcet commented Aug 19, 2021

Similar issue with #88142.

Perhaps a warning event should be created?

I think so. Working on this later. Will also live up to your expectations, right? @neilharris123

@JaneLiuL
Copy link

@JaneLiuL JaneLiuL commented Aug 20, 2021

But I check it already have a warnging events if secret not exist and pullImageFail:
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/images/image_manager.go line:147

@Pingan2017
Copy link
Member

@Pingan2017 Pingan2017 commented Aug 20, 2021

Now, only info message when pullsecret not exist

secret, err := kl.secretManager.GetSecret(pod.Namespace, secretRef.Name)
if err != nil {
klog.InfoS("Unable to retrieve pull secret, the image pull may not succeed.", "pod", klog.KObj(pod), "secret", klog.KObj(secret), "err", err)
continue
}

@Pingan2017
Copy link
Member

@Pingan2017 Pingan2017 commented Aug 20, 2021

maybe direct return and not enter syncPod is better.
return if pullSecret is not existed

// Fetch the pull secrets for the pod
pullSecrets := kl.getPullSecretsForPod(pod)
// Call the container runtime's SyncPod callback
result := kl.containerRuntime.SyncPod(pod, podStatus, pullSecrets, kl.backOff)

@kubernetes/sig-node-bugs

@kerthcet
Copy link
Member

@kerthcet kerthcet commented Aug 20, 2021

maybe direct return and not enter syncPod is better.
return if pullSecret is not existed

// Fetch the pull secrets for the pod
pullSecrets := kl.getPullSecretsForPod(pod)
// Call the container runtime's SyncPod callback
result := kl.containerRuntime.SyncPod(pod, podStatus, pullSecrets, kl.backOff)

@kubernetes/sig-node-bugs

if you pull image from a public repo, the process should not be impacted even no secrets exists. And it's what it is right now.

@adisky
Copy link
Contributor

@adisky adisky commented Aug 20, 2021

I think a warning event FailedToRetrieveImagePullSecret like

kl.recorder.Eventf(pod, v1.EventTypeWarning, "InvalidEnvironmentVariableNames", "Keys [%s] from the EnvFrom configMap %s/%s were skipped since they are considered invalid environment variable names.", strings.Join(invalidKeys, ", "), pod.Namespace, name)

could be added here
secret, err := kl.secretManager.GetSecret(pod.Namespace, secretRef.Name)
if err != nil {
klog.InfoS("Unable to retrieve pull secret, the image pull may not succeed.", "pod", klog.KObj(pod), "secret", klog.KObj(secret), "err", err)
continue
}

/good-first-issue
/triage accepted

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Aug 20, 2021

@adisky:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

I think a warning event FailedToRetrieveImagePullSecret could be added here
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_pods.go#L655
/good-first-issue
/triage accepted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kerthcet
Copy link
Member

@kerthcet kerthcet commented Aug 20, 2021

/assign @EricWvi

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Aug 20, 2021

@kerthcet: GitHub didn't allow me to assign the following users: EricWvi.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @EricWvi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Aug 21, 2021

@calvin0327: GitHub didn't allow me to assign the following users: EricWvi.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

I would like work on this.
/assign @EricWvi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@calvin0327
Copy link

@calvin0327 calvin0327 commented Aug 21, 2021

I would like to work in this.
/assign

@liggitt
Copy link
Member

@liggitt liggitt commented Aug 22, 2021

Thanks for wanting to jump in on this, but it's not clear this is a good first issue… making the proposed change would break deployments/pods that are currently working/running, so it's unlikely this would be acceptable from a compatibility perspective.

@liggitt
Copy link
Member

@liggitt liggitt commented Aug 22, 2021

Requiring image pull secrets to exist would have to be opt-in, which means an API change. I don't think this is worth the addition, but would like to hear sig-node's perspective.

An alternative could be to explore ways to make it more obvious to the user when this situation was occurring without automatically failing the pod

@kerthcet
Copy link
Member

@kerthcet kerthcet commented Aug 22, 2021

Requiring image pull secrets to exist would have to be opt-in, which means an API change. I don't think this is worth the addition, but would like to hear sig-node's perspective.

An alternative could be to explore ways to make it more obvious to the user when this situation was occurring without automatically failing the pod

hi @liggitt , we will not fail the pod, we just want to add some warning to make it obviously, maybe it's the same as you advised. correct me if not. 🧐

@adisky
Copy link
Contributor

@adisky adisky commented Aug 23, 2021

I think a warning event FailedToRetrieveImagePullSecret like

kl.recorder.Eventf(pod, v1.EventTypeWarning, "InvalidEnvironmentVariableNames", "Keys [%s] from the EnvFrom configMap %s/%s were skipped since they are considered invalid environment variable names.", strings.Join(invalidKeys, ", "), pod.Namespace, name)

could be added here

secret, err := kl.secretManager.GetSecret(pod.Namespace, secretRef.Name)
if err != nil {
klog.InfoS("Unable to retrieve pull secret, the image pull may not succeed.", "pod", klog.KObj(pod), "secret", klog.KObj(secret), "err", err)
continue
}

/good-first-issue
/triage accepted

To be clear here, only adding a warning is marked as good first issue here, not changing any pod behaviour.

@liggitt
Copy link
Member

@liggitt liggitt commented Aug 23, 2021

gotcha, that seems more reasonable, thanks for clarifying

as a note, make sure the warning events are generated at most once in that function, rather than once per missing secret object (if there are multiple missing secrets)

@liggitt
Copy link
Member

@liggitt liggitt commented Aug 23, 2021

/good-first-issue

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Aug 23, 2021

@liggitt:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@neilharris123
Copy link
Author

@neilharris123 neilharris123 commented Aug 23, 2021

Similar issue with #88142.

Perhaps a warning event should be created?

I think so. Working on this later. Will also live up to your expectations, right? @neilharris123

Yes, a warning would be good. I feel an error would have be better, but as has already been mentioned above we are too far along now for this change as it could break some users existing deployments, so a warning makes the most sense now.

@calvin0327 calvin0327 linked a pull request that will close this issue Aug 24, 2021
@ehashman ehashman added this to Triaged in SIG Node Bugs Sep 1, 2021
@enj enj added this to Needs Triage in SIG Auth Sep 13, 2021
@ossparvinder
Copy link

@ossparvinder ossparvinder commented Sep 26, 2021

/assign

@ossparvinder
Copy link

@ossparvinder ossparvinder commented Sep 26, 2021

@adisky based on what you have stated above, I see an INFO log already exists! If I'm understanding this correctly, now we need to raise a warning event in the same code block to fulfill this request. Is that right? @liggitt @neilharris123

@kerthcet
Copy link
Member

@kerthcet kerthcet commented Sep 26, 2021

I think @calvin0327 has created a pr above and labeled /lgtm already.

@ossparvinder ossparvinder removed their assignment Sep 26, 2021
@B-AYUSHMAN-PRUSTY
Copy link

@B-AYUSHMAN-PRUSTY B-AYUSHMAN-PRUSTY commented Oct 1, 2021

Hey I don't know anything about how to code but i am interested to contribute on kubernetes
So help me to contribute

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
SIG Auth
Needs Triage
Linked pull requests

Successfully merging a pull request may close this issue.