The Wayback Machine - https://web.archive.org/web/20211003223335/https://github.com/kubernetes/kubernetes/issues/104234
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy should log the payload when iptables-restore fails #104234

Open
cchxn258886 opened this issue Aug 9, 2021 · 13 comments
Open

kube-proxy should log the payload when iptables-restore fails #104234

cchxn258886 opened this issue Aug 9, 2021 · 13 comments

Comments

@cchxn258886
Copy link

@cchxn258886 cchxn258886 commented Aug 9, 2021

my k8s cluster version is 1.17.x 。this cluster has 10 nodes.one master 。
only 2 nodes get problem.

E0809 05:37:21.484590       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 05:37:21.484752       1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:14:37.485082       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 53 failed
)
I0809 06:14:37.485205       1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:45:24.676775       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 53 failed
)
I0809 06:45:24.676868       1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:45:38.141258       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 52 failed
)
I0809 06:45:38.141408       1 proxier.go:779] Sync failed; retrying in 30s
E0809 07:32:42.616867       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 52 failed
)
I0809 07:32:42.617009       1 proxier.go:779] Sync failed; retrying in 30s
E0809 07:42:13.864568       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 07:42:13.864698       1 proxier.go:779] Sync failed; retrying in 30s
E0809 08:00:20.842378       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 08:00:20.842502       1 proxier.go:779] Sync failed; retrying in 30s
E0809 08:06:32.562513       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 08:06:32.562665       1 proxier.go:779] Sync failed; retrying in 30s

it make my svc sometime can use。sometime can timeout。
what can i do can resolve this problem ?
this 2 node's svc is nodeportType 。

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Aug 9, 2021

@cchxn258886: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@cchxn258886
Copy link
Author

@cchxn258886 cchxn258886 commented Aug 9, 2021

@Network

@aojea
Copy link
Member

@aojea aojea commented Aug 9, 2021

you can increase kube-proxy verbosity adding -v7 flag and you should get more information about the cause of the error

Kubernetes 1.17 is EOL, so you should upgrade to 1.19 at least https://kubernetes.io/releases/

@MadhavJivrajani
Copy link
Contributor

@MadhavJivrajani MadhavJivrajani commented Aug 9, 2021

/sig network

@cchxn258886
Copy link
Author

@cchxn258886 cchxn258886 commented Aug 10, 2021

you can increase kube-proxy verbosity adding -v7 flag and you should get more information about the cause of the error

Kubernetes 1.17 is EOL, so you should upgrade to 1.19 at least https://kubernetes.io/releases/

hello bro。my cluster is use kubeadm init .so what should i do .i know this is use configmap 。but in this i can not found how edit this configmap is available?about upgrade this cluster .this time is impossible。

@cchxn258886
Copy link
Author

@cchxn258886 cchxn258886 commented Aug 10, 2021

you can increase kube-proxy verbosity adding -v7 flag and you should get more information about the cause of the error

Kubernetes 1.17 is EOL, so you should upgrade to 1.19 at least https://kubernetes.io/releases/

about stack. i only get this log.

 11:25:38.390292       1 proxier.go:779] Sync failed; retrying in 30s
E0805 11:36:18.240849       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 46 failed
)
I0805 11:36:18.240985       1 proxier.go:779] Sync failed; retrying in 30s
W0805 11:45:14.938674       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
I0805 11:45:14.951384       1 node.go:135] Successfully retrieved node IP: 10.60.44.71
I0805 11:45:14.951437       1 server_others.go:145] Using iptables Proxier.
I0805 11:45:14.951948       1 server.go:571] Version: v1.17.2
I0805 11:45:14.952618       1 conntrack.go:52] Setting nf_conntrack_max to 655360
I0805 11:45:14.953070       1 config.go:131] Starting endpoints config controller
I0805 11:45:14.953105       1 config.go:313] Starting service config controller
I0805 11:45:14.953143       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0805 11:45:14.953144       1 shared_informer.go:197] Waiting for caches to sync for service config
I0805 11:45:15.053346       1 shared_informer.go:204] Caches are synced for endpoints config 
I0805 11:45:15.053355       1 shared_informer.go:204] Caches are synced for service config 
E0805 11:46:53.104982       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 47 failed
)

@aojea
Copy link
Member

@aojea aojea commented Aug 10, 2021

To increase the verbosity on kube-proxy on a kubeadm deployment you can do

 kubectl patch -n kube-system daemonset/kube-proxy \
    --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--v=7" }]'

and wait until pods are restarted.

Despite that, if there is a bug, it will not be backported to 1.17, so you have to go to a supported version first

@thockin
Copy link
Member

@thockin thockin commented Aug 19, 2021

I swear we added code to dump the ruleset when this happens, but I don't see it any more.

@thockin
Copy link
Member

@thockin thockin commented Sep 2, 2021

I'm going to re-purpose this as a feature :)

@thockin thockin changed the title kube-proxy problem Failed to execute iptables-restore: exit status 1 kube-proxy should log the payload when iptables-restore fails Sep 2, 2021
@bhumijgupta
Copy link
Member

@bhumijgupta bhumijgupta commented Sep 3, 2021

Hi, I would like to pick this up. Can someone give me some code pointers on where to start?

@rajibmitra
Copy link
Member

@rajibmitra rajibmitra commented Sep 5, 2021

Let me know if I can help in anyway. @thockin

@rajibmitra
Copy link
Member

@rajibmitra rajibmitra commented Sep 5, 2021

/assign

@thockin
Copy link
Member

@thockin thockin commented Sep 6, 2021

I'd start with pkg/util/iptables/iptables.go - restoreInternal()

It's worth thinking about the cleanest solution here, but maybe something like:

type ParseError interface {
    Line() Int
}

type parseError struct {
    cmd string
    line int
}

func (e parseError) Line() int {
    return line
}

func (e parseError) Error() string {
    return fmt.Sprintf("%s: parse error on line %d", e.cmd, e.line)
} 

When you get an error from exec, parse the error for line %d failed and if it matches, return a parseError. Then let the callers check for a ParseError by interface assertion, extract the Line(), extract their input's data at -3 and +3 from that line and log it. E.g.

 Failed to execute iptables-restore: input error on line 1234:
    1231: ...
    1232: ...
    1233: ...
    1234: ...
    1235: ...
    1236: ...
    1237: ...

You'll proably want a helper func to do that, which could live in that same iptables package. And a test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants