Puh…this is a big one…
A client contacted me saying the webserver’s logs on one of our swarm stacks reports the same IP address for any request and it’s not one of any of the clients used:
- 10.255.0.2 [18/Oct/2018:13:38:43 +0000] "GET / HTTP/1.0" 401 113 "HTTP-Monitor/1.1" 1
This is an issue as we require the original caller’s IP address for various reasons:
- Restricting access to admin backends to specific IP addresses
- Log-analyses and tracking user behaviour
- etc. etc.
The IP is the IP of the ingress network gateway. The entrance to the swarm ingress network that each request arrives at when requesting a swarmed service.
The original IP is translated and overwritte, via SNAT (Static Network Address Translation), with the IP of the node it arrived at upon entering the ingress network as origin-ip and the ip of the node where the requested service is located as destination-ip. This is to forward the request to the node containing the service that can actually answer the request and then to be able to determine the node to send the response back to, meaning “the node who knows the origin ip to send the response back to”.
In a fully “web-based” (meaning HTTP based) setting, this issue could be easily overcome by implementing the proxy-protocol and adding a proxy layer to the request before SNATting the ips, just like Nginx or HAproxy do when receiving a request. The X-Forwarded-For header would then contain the IP of the original caller, while the origin-IP would be the IP of the node to return the response to when done servicing.
Sadly, as not all services run on and with Docker are HTTP services, this would limit the usability of the ingress network to “web-” (meaning HTTP) services only. Thus Docker needs to find a solution to this problem on L4 (meaning “layer 4” of the network layers, find descriptions at: The 7 Layers of the OSI Model), which simply doesn’t offer a base for proxy-protocol.
This is where #25526 comes in. It means: https://github.com/moby/moby/issues/25526. It details the struggle since 2016, when this issue was first reported, through a ton of comments and replies without a solution yet.
A workaround is specified there:
- Using a globally deployed Nginx/Traefik container binding to a port with mode=host on each Docker node
- Adding a proxy layer to each request (adding x-forwarded-for Header and others)
- Then forwarding the request to the ingress network where the origin-IP will be overwritten during SNATting but the proxy-headers will persist
- Allowing to read the callers IP from the proxy headers as most applications to automatically
This is super unhappy as it implies multiple problems:
- The proxy service running with port-binding “mode=host” is limited to precisely one container per swarm node
- A service update to this proxy service is not possible without downtime (or you need to manage re-routing traffic of updating proxies to proxy containers that are servicing using an external node-balancer) <- See also next point!
- An external load balancer is effectively required to balance all incoming requests to one of the Docker swarm nodes as this is not done by the ingress network anymore (since rolling with port-binding mode=host lives outside of the swarm networking completely, instead running directly and hard on each node)
- Without the external load balancer a service running running only on a single node is not possible, unless each client requesting this service knows to specifically send their requests to ONLY this Docker node (which pretty much defeats parts of the purpose of having a swarm)
- All OTHER containers must be on a network with the global proxy containers, otherwise the global proxies won’t be able to talk to the containers they are supposed to route to. This is a security issue as services that are NOT supposed to communicate with each other will be able to do just that on the “global-proxy” network
- AND the service cannot be controlled with the Windows Docker client as the Windows client enforces swarm services and non-swarm services (especially including their networks) to be mutually exclusive. Thus: Services cannot be port-binding with mode=host while at the same time taking part in a swarm-overlay-network.
There is a major drawback though:
With swarms dynamic character(*1) it is next to impossible to keep lb-routing (lb = load balancer) in sync with “what container runs where and what is the IP I need to route to?”.
A possible solution: Containous – Traefik load balancer. Traefik is an auto-configuring load balancer that attaches to the orchestrator of your choice (many available, Docker Swarm in our case) and automatically configures itself according to deployed containers and services, whose information it obtains automagically from the orchestrator. If this LB could stay in sync automatically it might solve this problem for web-based services that can leverage an added proxy-layer in requests.
(*1) Any container can be on any swarm node at any time in as many instances as configured and there can be any number of swarm nodes at any time when taking into account that they can be spun up and torn down at any time)