sfunnel: multi-port/multi-flow session affinity in Kubernetes

sfunnel is an eBPF program designed to funnel multiple traffic flows through a single Kubernetes service port, ensuring under certain conditions consistent sessionAffinity: ClientIP affinity across all ports within the service.

See the original use-case here.

⚠️ sfunnel is still in an early development stage.

⛔ severe performance degradation when funneling TCP over TCP/UDP is being investigated (GSO/TSO issues). Do not use it for real traffic.

At a glance

Example where TCP/8080 traffic is funneled through TCP/80.

Remove ports from the K8s service and e.g. deployment. Add the sfunnel container along with the rules in SFUNNEL_RULESET:

--- a/service.yaml
+++ b/service.yaml
@@ -1,18 +1,12 @@
 apiVersion: v1
 kind: Service
 metadata:
   name: my-loadbalancer-service
 spec:
   type: LoadBalancer
   selector:
     app: my-nginx-app
   ports:
     - protocol: TCP
       port: 80
       targetPort: 80
-    - protocol: TCP
-      port: 8080
-      targetPort: 8080
   sessionAffinity: ClientIP

--- a/nginx.yaml
+++ b/nginx.yaml
@@ -1,21 +1,31 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: my-nginx-deployment
 spec:
   replicas: 4
   selector:
     matchLabels:
       app: my-nginx-app
   template:
     metadata:
       labels:
         app: my-nginx-app
     spec:
       containers:
+        - name: sfunnel-init
+          env:
+            - name: SFUNNEL_RULESET
+              value: ip tcp dport 80 sport 540 actions unfunnel tcp
+          image: ghcr.io/datahangar/sfunnel:0.0.11@sha256:5f130c2bfc95fb0d264ad54c52b1fef26c58e5635f11b8b862efe611b98b1f9a
+          securityContext:
+            privileged: false #Set to true for some public clouds (e.g. GKE standard)
+            capabilities:
+              add: [BPF, NET_ADMIN, SYS_ADMIN]
+          volumeMounts:
+            - name: bpffs
+              mountPath: /sys/fs/bpf
+        - name: sfunnel-init-egress
+          env:
+            - name: SFUNNEL_RULESET
+              value: ip tcp sport 8080 actions funnel tcp dport 540 sport 80
+            - name: DIRECTION
+              value: egress
+          image: ghcr.io/datahangar/sfunnel:0.0.11@sha256:5f130c2bfc95fb0d264ad54c52b1fef26c58e5635f11b8b862efe611b98b1f9a
+          securityContext:
+            privileged: false #Set to true for some public clouds (e.g. GKE standard)
+            capabilities:
+              add: [BPF, NET_ADMIN, SYS_ADMIN]
+          volumeMounts:
+            - name: bpffs
+              mountPath: /sys/fs/bpf
         - name: nginx
           image: nginx:latest
           ports:
             - containerPort: 80
-            - containerPort: 8080
+     volumes:
+       - name: bpffs
+         hostPath:
+           path: /sys/fs/bpf

(funneling HTTPs TCP/443 through TCP/80 would work the same way. Manifest is just too long for this example)

On the other end (e.g. a Linux host, server etc..), deploy it with the matching rules:

IFACES=eth0 LB_IP=1.1.1.1 \
SFUNNEL_RULESET="ip daddr ${LB_IP} tcp dport 8080 actions funnel tcp dport 80 sport 540" \
docker run --privileged --network=host -it -e IFACES -e DIRECTION="egress" -e SFUNNEL_RULESET ghcr.io/datahangar/sfunnel:0.0.11

IFACES=eth0 LB_IP=1.1.1.1 \
SFUNNEL_RULESET="ip saddr ${LB_IP} tcp sport 80 dport 540 actions unfunnel tcp" \
docker run --privileged --network=host -it -e IFACES -e DIRECTION="ingress" -e SFUNNEL_RULESET ghcr.io/datahangar/sfunnel:0.0.11

The sfunnel container will run, load the eBPF code and finish its execution.

Support

Service types

• ClusterIP: supported
• LoadBalancer: supported
• NodePort: untested, but should work

📝 Note

Currently internalTrafficPolicy: Local for ClusterIP and externalTrafficPolicy: Local for NodePort and LoadBalancer services are required.

Environments

• Google Kubernetes Engine(GKE): Standard cluster.
  • Autopilot clusters are not supported due to lack of eBPF support.
• MetalLB with the following CNI plugins:
  • Cilium
  • Flannel
  • Calico
• Dockerd

sfunnel should work on any environments supporting sessionAffinity: ClientIP. If you encounter any issues or have successfully deployed it in other environments, please reach out so that we can update this list.

Requirements

• eBPF-enabled kernel, with support for clsact and direct-action.
• Proper MTU configuration (20 bytes for TCP, 8 for UDP).
• In Kubernetes:
  • Privileged init container (CAP_BPF, CAP_NET_ADMIN, CAP_SYS_ADMIN)
    • In some cloud providers (E.g. Google Cloud) privileged=true is required.
• On the funneling side:
  • Permissions to spawn sfunnel (same caps as before).
  • Route or proxy traffic to be funneled. More on this here

More...

• Use-cases
• Funneling?
• Rule syntax
• sfunnel container
• Deploying it in K8s
• Next steps

Contact

Marc Sune < marcdevel (at) gmail (dot) com>