Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

portmap nftables backend invalid rules (dnat from input hook + invalid ipv6 rules) #1115

Open
champtar opened this issue Nov 5, 2024 · 3 comments · May be fixed by #1116
Open

portmap nftables backend invalid rules (dnat from input hook + invalid ipv6 rules) #1115

champtar opened this issue Nov 5, 2024 · 3 comments · May be fixed by #1116

Comments

@champtar
Copy link
Contributor

champtar commented Nov 5, 2024

Trying the new nftables backend, it fails on EL 9.4

# nft --version
nftables v1.0.9 (Old Doc Yak #3)
# uname -r
5.14.0-427.40.1.el9_4.x86_64

conf

{
  "type": "portmap",
  "capabilities": {"portMappings": true},
  "backend": "nftables",
  "conditionsV4": ["ip", "daddr", "!=", "{ 127.0.0.0/8, 198.19.254.254 }"]
},

Error

Nov 05 17:15:34 atsc2 kubelet[4258]: E1105 17:15:34.352838    4258 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err=<
Nov 05 17:15:34 atsc2 kubelet[4258]:         rpc error: code = Unknown desc = failed to setup network for sandbox "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621": plugin type="multus-cni" name="multus-cni-network" failed (add): [traefik/appliance-ingress-traefik-698cd97568-5xg88/3908060c-0419-481c-865d-3230b95c3a84:mgmt]: error adding container to network "mgmt": plugin type="portmap" failed (add): unable to set up nftables rules for port mappings: /dev/stdin:13:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.1.194 . 10081 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:14:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.1.194 . 18444 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:15:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.1.194 . 10082 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:16:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.1.194 . 18445 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:17:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.1.194 . 10083 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:18:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.1.194 . 18446 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:19:1-168: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 198.18.1.194 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:20:1-165: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:  >

Using strace, the rules are:

add table ip cni_hostport { comment "CNI portmap plugin" ; }
add chain ip cni_hostport hostports
add chain ip cni_hostport hostip_hostports
add chain ip cni_hostport input { type nat hook input priority -100 ; }
flush chain ip cni_hostport input
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostports
add chain ip cni_hostport output { type nat hook output priority -100 ; }
flush chain ip cni_hostport output
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } fib daddr type local jump hostports
add chain ip cni_hostport masquerading { type nat hook postrouting priority 100 ; }
add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.0.56 . 10081 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.0.56 . 18444 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.0.56 . 10082 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.0.56 . 18445 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.0.56 . 10083 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.0.56 . 18446 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 198.18.0.56 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"

I think instead of

add chain ip cni_hostport input { type nat hook input priority -100 ; }

we should use

add chain ip cni_hostport input { type nat hook prerouting priority -100 ; }

(and rename input to prerouting)
@champtar champtar linked a pull request Nov 5, 2024 that will close this issue
Open
@champtar
Copy link
Contributor Author

champtar commented Nov 5, 2024

The ip6 rules are also invalid,

kubelet[4258]:         /dev/stdin:18:41-48: Error: syntax error, unexpected string
kubelet[4258]:         add rule ip6 cni_hostport hostports ip6 protocol tcp th dport 8446 dnat ip6 addr . port to fd61:7465:6d65:1000::3cd . 18446 comment "30ba0c9d53d293e8ed7b7dcdc37cd0cb8380f78cbd2b3977bc57f8e9931e04e7"
kubelet[4258]:                                                 ^^^^^^^^

ip6 protocol tcp gives a syntax error, we can use meta l4proto tcp th dport, but I'm curious why aren't we just using tcp dport ? @danwinship ?

@champtar
Copy link
Contributor Author

champtar commented Nov 5, 2024 

nft 'add rule ip cni_hostport hostports meta l4proto tcp th dport 81 dnat ip addr . port to 198.18.3.235 . 10081 comment "1"'
nft 'add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.3.235 . 10081 comment "2"'
nft 'add rule ip cni_hostport hostports tcp dport 81 dnat ip addr . port to 198.18.3.235 . 10081 comment "3"'

# nft list ruleset
		tcp dport 81 dnat to 198.18.3.235:10081 comment "1"
		tcp dport 81 dnat to 198.18.3.235:10081 comment "2"
		tcp dport 81 dnat to 198.18.3.235:10081 comment "3"

@champtar champtar changed the title portmap nftable backend invalid rules (dnat from input hook) portmap nftables backend invalid rules (dnat from input hook + invalid ipv6 rules) Nov 6, 2024
@champtar champtar mentioned this issue Nov 6, 2024
Merged
@danwinship
Copy link
Contributor

hm... I had tested all of this at one point... I must have made a "minor fix" at some point and broken it

I think instead of

need to think about this and compare with iptables portmap and nftables kube-proxy rules...

I'm curious why aren't we just using tcp dport

probably I copied from somewhere else and simplified/rewrote and didn't notice it ended up having unnecessary clauses in the end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

portmap: fix nftables backend champtar/cni-plugins
2 participants
@danwinship @champtar