on
K8s network policies break service encapsulation
Setting the scene
Imagine an organisation with two teams, Team-A and Team-B. Each team owns an application, App-A and App-B respectively. These apps are deployed in kubernetes with each team having their own namespace.
App-B needs to call a HTTP service provided by App-A. Team-A expose their application through the use of a standard ClusterIP service in Kubernetes.
By using a service in this way Team-B don’t need to care how many instances of App-A exist for routing and they do not need to care which port App-A is running on. Using the service Team-A can abstract this implementation detail by having the service forward port 80 to port 8080, then if they change the implementation later such that the App-A runs on a different port, as long as they update the service definition Team-B don’t need to make any changes for their app to keep working.
A simple test setup to demonstrate this could look like the following;
---
# team-a-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: team-a
---
# app-a-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: team-a
name: app-a
spec:
selector:
matchLabels:
app: app-a
template:
metadata:
labels:
app: app-a
spec:
containers:
- name: app-a
image: nginxinc/nginx-unprivileged
---
# app-a-service.yaml
apiVersion: v1
kind: Service
metadata:
namespace: team-a
name: app-a
spec:
selector:
app: app-a
ports:
- protocol: TCP
port: 80
targetPort: 8080
---
# team-b-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: team-b
---
# app-b-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: team-b
name: app-b
spec:
selector:
matchLabels:
app: app-b
template:
metadata:
labels:
app: app-b
spec:
containers:
- name: app-b
image: nginxinc/nginx-unprivileged
Then you can confirm connectivity as follows;
$ kubectl exec -ti -n team-b deploy/app-b -- curl http://app-a.team-a/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Security goals
Now, maybe your team or organisation wants to start being a little more security focussed in the wake of log4j vulnerabilities and think that it could be time to start adding a little more network security to your applications.
"Manage flows out of the network as closely as those allowed in to defeat the persistent cyber adverary" - The Art of Cyber War
— Sun Tzu (@SunTzuCyber) January 8, 2022
While traditional style firewalls can be used to secure traffic flowing to and from a Kubernetes cluster, by default, there are no restrictions on traffic flowing between pods inside the cluster.
There are a couple of tools at your disposal to secure network traffic within the cluster. If you need complete fine-grained control of your traffic then a Service Mesh is probably the right answer, but these come with a high degree of complexity and have a potential performance impact that might not be practical for all workloads.
If you’re looking for something with a lighter touch the next solution to look at would be network policies as long as your CNI supports it.
Using network policies you can control where a pod accepts connections from and where it can create connections to.
A first attempt
So, Team-B have decided to make use of network policies to secure their application. The first step here is to deny all traffic in and out by default;
---
# np-default-deny-all-yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: team-b
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress: []
With this in place, we can see that not even DNS traffic is allowed out of the pod.
$ kubectl exec -ti -n team-b deploy/app-b -- curl http://app-
a.team-a/
curl: (6) Could not resolve host: app-a.team-a
command terminated with exit code 6
So lets enable DNS
---
# np-default-allow-dns.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-allow-dns
namespace: team-b
spec:
podSelector: {}
egress:
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
Testing this configuration we now get a more reasonable error, showing that we cannot connect to App-A;
$ kubectl exec -ti -n team-b deploy/app-b -- curl http://app-a.team-a/
curl: (28) Failed to connect to app-a.team-a port 80: Connection timed out
command terminated with exit code 28
So, to solve this Team-B might think that as they access App-A on port 80 that the following config might work;
---
# np-app-b-allow-app-a-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: app-b-allow-egress-app-a
namespace: team-b
spec:
podSelector:
matchLabels:
app: app-b
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: team-a
podSelector:
matchLabels:
app: app-a
ports:
- port: 80
Unfortunately, applying this policy we get the same error message as before.
$ kubectl exec -ti -n team-b deploy/app-b -- curl http://app-
a.team-a/
curl: (28) Failed to connect to app-a.team-a port 80: Connection timed out
command terminated with exit code 28
The error message even mentions port 80!
However if we update the policy to refer to the real destination port;
---
# np-app-b-allow-app-a-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: app-b-allow-egress-app-a
namespace: team-b
spec:
podSelector:
matchLabels:
app: app-b
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: team-a
podSelector:
matchLabels:
app: app-a
ports:
- port: 8080
All is well again;
$ kubectl exec -ti -n team-b deploy/app-b -- curl http://app-
a.team-a/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
The broken encapsulation
What this means in practice is that Team-B need to know the implementation details of the app from Team-A that they shouldn’t need to know. On one level the implementation makes complete sense, but when trying to use the feature in practice it feels a little confusing. It may be that I’m using it wrong, feel free to contact me if you have solutions for this.
Possible workarounds
There are two possible workarounds that I see for this issue;
- You can ensure that by convention services are always exposed on the same port as they are exposed on the pods. This is easy to implement with tooling and while in theory it gives a little less flexibility to make changes, in practice it’s unlikely that you’ll need to change which port an app is listening on.
- Remove the ports restiction on the policy, allowing traffic to any port on the destination pod. While not ideal, hopefully the other pod has ingress restrictions only allowing traffic to enter to expected ports and blocking traffic to more dangerous ports such as those used for JMX or remote debugging. The combination of an egress policy on one side and ingress policy on the other side would probably provide an acceptable level of protection.
Final notes
The above configuration and behaviour has been tested on an AKS cluster with Azure CNI and a minikube cluster with Cilium CNI, another CNI implementation might behave differently.
The Cilium Network Policy Editor is a great tool for crafting your first network policies.