What a wicked NAT we weave: detangling the cloud

Cloud has 3 levels of address translation. Shocking I know. But… Believe it or not, this is the chain of events for a stream that arrives at your service in the cloud.

The sequence ends up being:

Client->LoadBalancer->Ingress->Sidecar->Service

and, LoadBalancer does a NAT, Ingress and Sidecar are proxies, so, well, Service never sees the IP of client.

Other people have been working at this problem (e.g. RFC7974, HAProxy ‘ Proxy Protocol‘, others.

Today lets look at a practical example, using the HAProxy Proxy Protocol. Specifically, lets look at a tool CloudFlare did that allows adding transparency on the far side. They talk about it more here.

Here’s a recipe for you to try it out at home:

Start a new container (as per first line) and run the following lines

docker run --name mmp --privileged --rm -it -v $PWD:$PWD ubuntu:18.04
apt update && apt install -y iproute2 curl iptables python3 netcat

iptables -t mangle -I PREROUTING -m mark --mark 123 -j CONNMARK --save-mark
iptables -t mangle -I OUTPUT -m connmark --mark 123 -j CONNMARK --restore-mark
ip6tables -t mangle -I PREROUTING -m mark --mark 123 -j CONNMARK --save-mark
ip6tables -t mangle -I OUTPUT -m connmark --mark 123 -j CONNMARK --restore-mark
ip rule add fwmark 123 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
ip -6 rule add fwmark 123 lookup 100
ip -6 route add local ::/0 dev lo table 100
echo 1 | tee /proc/sys/net/ipv4/conf/eth0/route_localnet

python3 -m http.server -b 127.0.0.1 8000

Now run this from host:

docker exec -it mmp $PWD/mmproxy -a $PWD/networks.txt 
  -l 0.0.0.0:80 -4 127.0.0.1:8000 -6 '[::1]:8000'

Now run this from host:

echo -en "PROXY TCP4 1.2.3.4 1.2.3.4 11 11\r\nGET / HTTP/1.1\r\n\r\n" | \
 docker exec -i mmp nc -v 127.0.0.1 80

On the first window, you will see something like:

1.2.3.4 - - [18/Jun/2018 14:32:04] "GET / HTTP/1.1" 200

The 1.2.3.4 indicates the source IP.

What sourcery is this? Is this a tool to undo the magic NAT stuff of the cloud? Or a security nightmare?