Photon OS 4 R2 - Docker Swarm - Issue with Routing Mesh not routing Published Port on all Nodes

[jmcombs]

Describe the bug

Background

I have a 3 node Docker Swarm running with one Manager node utilizing Overlay networking. The Ingress overlay network configuration is default configuration.

(Manager) Node 1: 192.168.7.236
Node 2: 192.168.7.237
Node 3: 192.168.7.238

I have deployed a Service for a Minecraft server that runs one replica/container on a single node that is publishing TCP Port 25565 to the Ingress overlay network and is also using its own overlay network: minecraft_network.

Overlay networking is working as I can deploy containers on all three nodes in the minecraft_network overlay network and they can ping each other and the external world.

Issue

From my understanding, when I publish a port on a service, that service should be reachable externally from all 3 Docker nodes via Routing Mesh. This is not the case, the service is only reachable externally on the Docker node that is running the Container. Output below from running a test via nc:

➜  ~ nc -z -v 192.168.7.236 25565
Connection to 192.168.7.236 port 25565 [tcp/*] succeeded!
➜  ~ nc -z -v 192.168.7.237 25565
nc: connectx to 192.168.7.237 port 25565 (tcp) failed: Operation timed out
➜  ~ nc -z -v 192.168.7.238 25565
nc: connectx to 192.168.7.238 port 25565 (tcp) failed: Operation timed out

Everything looks correct when I inspect the networks, service and even iptables is showing the port is permitted (by Docker automatically adding it) on each Node.

I have determined that when I disable iptables via systemctl stop iptables the issue is resolved. iptables output is below.

I appreciate any and all help. Thank you!

Reproduction steps

root@docker01 [ ~ ]# more minecraft-server.yml 
version: '3.8'

services:
  server:
    image: itzg/minecraft-server
    ports:
      - target: 25565
        published: 25565
        protocol: tcp
    volumes:
      - "datastore:/data"
    networks:
      - network
    environment:
      EULA: "TRUE"
    restart: unless-stopped
    deploy:
      mode: replicated
      replicas: 1

networks:
  network:
    driver: overlay
    attachable: true

volumes:
  datastore:

root@docker01 [ ~ ]# docker stack deploy -c minecraft-server.yml minecraft
Ignoring unsupported options: restart

Creating network minecraft_network
Creating service minecraft_server

root@docker01 [ ~ ]# docker service ls
ID             NAME               MODE         REPLICAS   IMAGE                          PORTS
128zzthqb5pv   minecraft_server   replicated   1/1        itzg/minecraft-server:latest   *:25565->25565/tcp

root@docker01 [ ~ ]# docker service ps minecraft_server
ID             NAME                 IMAGE                          NODE                      DESIRED STATE   CURRENT STATE            ERROR     PORTS
gddad3afa0xl   minecraft_server.1   itzg/minecraft-server:latest   docker01.<redacted>   Running         Running 13 seconds ago  

Expected behavior

➜  ~ nc -z -v 192.168.7.236 25565
Connection to 192.168.7.236 port 25565 [tcp/*] succeeded!
➜  ~ nc -z -v 192.168.7.237 25565
nc: connectx to 192.168.7.237 port 25565 [tcp/*] succeeded!
➜  ~ nc -z -v 192.168.7.238 25565
nc: connectx to 192.168.7.238 port 25565 [tcp/*] succeeded!

Additional context

Environment:

Application Details:

• Operating System: VMware Photon 4.0 Revision 2
• Docker version 20.10.11, build dea9396.

Cluster:

root@docker01 [ ~ ]# docker node ls
ID                            HOSTNAME                  STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
txjvdmo2s1nj69u6gig01gdeo *   docker01.<redacted>   Ready     Active         Leader           20.10.11
g8uw56gqr7a9eqxhxvwp415ug     docker02.<redacted>   Ready     Active                          20.10.11
5wcwrkjbhz8t4rqxujca4c0xo     docker03.<redacted>   Ready     Active                          20.10.11

IP Tables Output:

Node 1:

root@docker01 [ ~ ]# iptables -L -v -n
Chain INPUT (policy DROP 6489 packets, 2493K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2718  136K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
 130K   80M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    4   256 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2222
    9   540 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2377
 1985  119K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:7946
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:7946
   73  8226 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789
  178  6420 ACCEPT     all  --  eth0   *       0.0.0.0/0            224.0.0.0/8         
    0     0 ACCEPT     112  --  eth0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     icmp --  eth0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  215  162K DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  215  162K DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  209  162K DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
  114  156K ACCEPT     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
   95  5858 ACCEPT     all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  docker_gwbridge docker_gwbridge  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 124K   23M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-INGRESS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    4   220 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25565
    2   112 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:25565
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
   95  5858 DOCKER-ISOLATION-STAGE-2  all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
  209  162K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
   95  5858 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  215  162K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0  

Node 2:

root@docker02 [ ~ ]# iptables -L -v -n
Chain INPUT (policy DROP 6455 packets, 2474K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2670  134K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
61523 7012K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    1    64 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2222
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2377
 2023  121K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:7946
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:7946
   27  2970 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789
20197  807K ACCEPT     all  --  eth0   *       0.0.0.0/0            224.0.0.0/8         
    0     0 ACCEPT     112  --  eth0   *       0.0.0.0/0            0.0.0.0/0           
  811 68124 ACCEPT     icmp --  eth0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           
 6455 2474K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 7 prefix "IPTSUCKS: "

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   22  1360 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   22  1360 DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  docker_gwbridge docker_gwbridge  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
75209 7822K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-INGRESS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   22  1360 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25565
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:25565
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   22  1360 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0 

Node 3:

root@docker03 [ ~ ]# iptables -L -v -n
Chain INPUT (policy DROP 6565 packets, 2518K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2730  137K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
62807 7163K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    1    64 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2222
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2377
 2066  124K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:7946
    2   615 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:7946
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789
20632  825K ACCEPT     all  --  eth0   *       0.0.0.0/0            224.0.0.0/8         
    0     0 ACCEPT     112  --  eth0   *       0.0.0.0/0            0.0.0.0/0           
  818 68712 ACCEPT     icmp --  eth0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   10   640 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   10   640 DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  docker_gwbridge docker_gwbridge  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
76783 7981K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-INGRESS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   640 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25565
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:25565
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   640 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0 

Proof Overlay networking is working:

I added individual Alpine containers to each node to the minecraft_network overlay network and tested pinging for DNS name resolution and communication between the Hosts and external world.

Node 1:

root@docker01 [ ~ ]# docker run -it --name alpine1 --network minecraft_network alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df9b9388f04a: Pull complete 
Digest: sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
Status: Downloaded newer image for alpine:latest
/ # ping server
PING server (10.0.6.2): 56 data bytes
64 bytes from 10.0.6.2: seq=0 ttl=64 time=0.096 ms
64 bytes from 10.0.6.2: seq=1 ttl=64 time=0.048 ms
--- server ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.048/0.072/0.096 ms
/ # ping alpine2
PING alpine2 (10.0.6.6): 56 data bytes
64 bytes from 10.0.6.6: seq=0 ttl=64 time=0.197 ms
64 bytes from 10.0.6.6: seq=1 ttl=64 time=0.215 ms
64 bytes from 10.0.6.6: seq=2 ttl=64 time=0.175 ms
--- alpine2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.175/0.195/0.215 ms
/ # ping alpine3
PING alpine3 (10.0.6.8): 56 data bytes
64 bytes from 10.0.6.8: seq=0 ttl=64 time=0.974 ms
64 bytes from 10.0.6.8: seq=1 ttl=64 time=0.890 ms
64 bytes from 10.0.6.8: seq=2 ttl=64 time=0.854 ms
--- alpine3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.854/0.906/0.974 ms
/ # ping google.com
PING google.com (142.250.64.238): 56 data bytes
64 bytes from 142.250.64.238: seq=0 ttl=116 time=30.473 ms
64 bytes from 142.250.64.238: seq=1 ttl=116 time=64.490 ms
64 bytes from 142.250.64.238: seq=2 ttl=116 time=27.212 ms
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 27.212/40.725/64.490 ms
/ # 

Node 2:

root@docker02 [ ~ ]# docker run -it --name alpine2 --network minecraft_network alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df9b9388f04a: Pull complete 
Digest: sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
Status: Downloaded newer image for alpine:latest
/ # ping server
PING server (10.0.6.2): 56 data bytes
64 bytes from 10.0.6.2: seq=0 ttl=64 time=0.086 ms
64 bytes from 10.0.6.2: seq=1 ttl=64 time=0.048 ms
64 bytes from 10.0.6.2: seq=2 ttl=64 time=0.093 ms
--- server ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.048/0.075/0.093 ms
/ # ping alpine1
PING alpine1 (10.0.6.5): 56 data bytes
64 bytes from 10.0.6.5: seq=0 ttl=64 time=0.134 ms
64 bytes from 10.0.6.5: seq=1 ttl=64 time=0.219 ms
64 bytes from 10.0.6.5: seq=2 ttl=64 time=0.285 ms
--- alpine1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.134/0.212/0.285 ms
/ # ping alpine3
PING alpine3 (10.0.6.8): 56 data bytes
64 bytes from 10.0.6.8: seq=0 ttl=64 time=1.107 ms
64 bytes from 10.0.6.8: seq=1 ttl=64 time=1.282 ms
64 bytes from 10.0.6.8: seq=2 ttl=64 time=1.219 ms
--- alpine3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.107/1.202/1.282 ms
/ # ping google.com
PING google.com (142.250.64.238): 56 data bytes
64 bytes from 142.250.64.238: seq=0 ttl=116 time=35.483 ms
64 bytes from 142.250.64.238: seq=1 ttl=116 time=27.942 ms
64 bytes from 142.250.64.238: seq=2 ttl=116 time=25.849 ms
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 25.849/29.758/35.483 ms
/ # 

Node 3:

root@docker03 [ ~ ]# docker run -it --name alpine3 --network minecraft_network alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df9b9388f04a: Pull complete 
Digest: sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
Status: Downloaded newer image for alpine:latest
/ # ping server
PING server (10.0.6.2): 56 data bytes
64 bytes from 10.0.6.2: seq=0 ttl=64 time=0.071 ms
64 bytes from 10.0.6.2: seq=1 ttl=64 time=0.077 ms
64 bytes from 10.0.6.2: seq=2 ttl=64 time=0.093 ms
--- server ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.071/0.080/0.093 ms
/ # ping alpine1
PING alpine1 (10.0.6.5): 56 data bytes
64 bytes from 10.0.6.5: seq=0 ttl=64 time=1.039 ms
64 bytes from 10.0.6.5: seq=1 ttl=64 time=1.229 ms
64 bytes from 10.0.6.5: seq=2 ttl=64 time=1.228 ms
--- alpine1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.039/1.165/1.229 ms
/ # ping alpine2
PING alpine2 (10.0.6.6): 56 data bytes
64 bytes from 10.0.6.6: seq=0 ttl=64 time=0.613 ms
64 bytes from 10.0.6.6: seq=1 ttl=64 time=1.193 ms
64 bytes from 10.0.6.6: seq=2 ttl=64 time=1.225 ms
--- alpine2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.613/1.010/1.225 ms
/ # ping google.com
PING google.com (142.250.64.238): 56 data bytes
64 bytes from 142.250.64.238: seq=0 ttl=116 time=28.863 ms
64 bytes from 142.250.64.238: seq=1 ttl=116 time=28.646 ms
64 bytes from 142.250.64.238: seq=2 ttl=116 time=27.834 ms
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 27.834/28.447/28.863 ms

Docker Stack/Service Information:

Inspections:

root@docker01 [ ~ ]# docker service inspect minecraft_server
[
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 25565,
                        "PublishedPort": 25565,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 25565,
                    "PublishedPort": 25565,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "s3vwk04tgjgy27dvk2coj72ku",
                    "Addr": "10.0.0.15/24"
                },
                {
                    "NetworkID": "xnlj3dttk4y3kb1nu22d5v9sl",
                    "Addr": "10.0.6.2/24"
                }
            ]
    }
]

root@docker01 [ ~ ]# docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "s3vwk04tgjgy27dvk2coj72ku",
        "Created": "2022-04-17T22:05:48.77504595Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "c2067bdcec04c5f327072200b5bc720ddcb79933bc621e72f248912e4466bbed": {
                "Name": "minecraft_server.1.gddad3afa0xly8zc8ge00ssqb",
                "EndpointID": "6fbd05792c0a9be832e63af57dd5f5a2ce9e6219317e3b707eefc54e895987d5",
                "MacAddress": "02:42:0a:00:00:10",
                "IPv4Address": "10.0.0.16/24",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "5c382c781bb17d93a33789f03f8162ad3c0a28cc170d8607beda2212747835c1",
                "MacAddress": "02:42:0a:00:00:02",
                "IPv4Address": "10.0.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "a8b91c57730c",
                "IP": "192.168.7.236"
            },
            {
                "Name": "c1cb3607cdf7",
                "IP": "192.168.7.237"
            },
            {
                "Name": "048d71f586ca",
                "IP": "192.168.7.238"
            }
        ]
    }
]

[jmcombs]

Update, it looks like iptables is the culprit. But, now I am trying to find out why. When I systemctl stop iptables Routing mesh works.

[dcasota]

Hi,

might be similar to #583 ?

iptables -P FORWARD ACCEPT
iptables -P INPUT ACCEPT

edited:
Are the Linux kernel config the same? There is a docker check-config.sh script which lists all necessary settings, see
check-config_output.txt

[jmcombs]

@dcasota

I tried those rule changes from #583 earlier to no avail. Just re-tried them, again, same results.

iptables -P FORWARD ACCEPT
iptables -P INPUT ACCEPT

I see where it changes the following in an iptables -L:

• Chain INPUT (policy DROP) to Chain INPUT (policy ACCEPT)
• Chain FORWARD (policy DROP) to Chain FORWARD (policy ACCEPT)

Result is still:

➜  ~ nc -z -v 192.168.7.236 25565            
Connection to 192.168.7.236 port 25565 [tcp/*] succeeded!
➜  ~ nc -z -v 192.168.7.237 25565            
nc: connectx to 192.168.7.237 port 25565 (tcp) failed: Operation timed out
➜  ~ nc -z -v 192.168.7.238 25565
nc: connectx to 192.168.7.238 port 25565 (tcp) failed: Operation timed out

Keep in mind, this follows the container, so if the container is running on .237, I will get a success there and timeout everywhere else. Could it be a DNAT issue in iptables since it's not getting refused and, instead, is timing out?

Regarding the Docker check and kernel configurations, they are the same. Output below:

Node 1:

root@docker01 [ ~ ]# curl -L -O https://github.com/moby/moby/raw/master/contrib/check-config.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   140  100   140    0     0    533      0 --:--:-- --:--:-- --:--:--   534
100 12039  100 12039    0     0  32002      0 --:--:-- --:--:-- --:--:-- 32002
root@docker01 [ ~ ]# chmod +x check-config.sh
root@docker01 [ ~ ]# ./check-config.sh 
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-5.10.103-4.ph4-esx ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled, but apparmor_parser missing
    (your best bet is "yum install apparmor-parser")
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: missing

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: enabled
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

root@docker01 [ ~ ]# docker --version
Docker version 20.10.11, build dea9396
root@docker01 [ ~ ]# cat /etc/photon-release
VMware Photon OS 4.0
PHOTON_BUILD_NUMBER=2f5aad892
root@docker01 [ ~ ]# uname -a
Linux docker01.<redacted> 5.10.103-4.ph4-esx #1-photon SMP Thu Apr 7 02:58:03 UTC 2022 x86_64 GNU/Linux

Node 2:

root@docker02 [ ~ ]# curl -L -O https://github.com/moby/moby/raw/master/contrib/check-config.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   140  100   140    0     0    536      0 --:--:-- --:--:-- --:--:--   538
100 12039  100 12039    0     0  27534      0 --:--:-- --:--:-- --:--:-- 27534
root@docker02 [ ~ ]# chmod +x check-config.sh
root@docker02 [ ~ ]# ./check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-5.10.103-4.ph4-esx ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled, but apparmor_parser missing
    (your best bet is "yum install apparmor-parser")
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: missing

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: enabled
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

root@docker02 [ ~ ]# docker --version
Docker version 20.10.11, build dea9396
root@docker02 [ ~ ]# cat /etc/photon-release
VMware Photon OS 4.0
PHOTON_BUILD_NUMBER=2f5aad892
root@docker02 [ ~ ]# uname -a
Linux docker02.<redacted> 5.10.103-4.ph4-esx #1-photon SMP Thu Apr 7 02:58:03 UTC 2022 x86_64 GNU/Linux

Node 3:

root@docker03 [ ~ ]# curl -L -O https://github.com/moby/moby/raw/master/contrib/check-config.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   140  100   140    0     0    965      0 --:--:-- --:--:-- --:--:--   965
100 12039  100 12039    0     0  41649      0 --:--:-- --:--:-- --:--:-- 41649
root@docker03 [ ~ ]# chmod +x check-config.sh
root@docker03 [ ~ ]# ./check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-5.10.103-4.ph4-esx ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled, but apparmor_parser missing
    (your best bet is "yum install apparmor-parser")
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: missing

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: enabled
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

root@docker03 [ ~ ]# docker --version
Docker version 20.10.11, build dea9396
root@docker03 [ ~ ]# uname -a
Linux docker03.<redacted> 5.10.103-4.ph4-esx #1-photon SMP Thu Apr 7 02:58:03 UTC 2022 x86_64 GNU/Linux

[jmcombs]

iptables DNAT config looks fine, and consistent across all three nodes.

iptables -nL -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER-INGRESS  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER-INGRESS  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
MASQUERADE  all  --  172.18.0.0/16        0.0.0.0/0           

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-INGRESS (2 references)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:25565 to:172.18.0.2:25565
RETURN     all  --  0.0.0.0/0            0.0.0.0/0 

[dcasota]

@jmcombs did you use the bridge utils?
brctl show docker_gwbridge

[jmcombs]

@dcasota I apologize, I have not. What would I be looking for?

root@docker01 [ ~ ]# brctl show docker_gwbridgebridge name     bridge id               STP enabled     interfaces
docker_gwbridge         8000.02427a66294c       no              vethb42f4b6
                                                        vetheac5263

root@docker02 [ ~ ]# brctl show docker_gwbridge
bridge name     bridge id               STP enabled     interfaces
docker_gwbridge         8000.02421b51d6b5       no              veth22cfbbe

root@docker03 [ ~ ]# brctl show docker_gwbridge
bridge name     bridge id               STP enabled     interfaces
docker_gwbridge         8000.024209cf94ad       no              vethcb476cd

[dcasota]

Not sure about the root cause.
The virtual bridge that connects to the overlay networks is automatically created, in reference to https://docs.docker.com/engine/swarm/networking/#customize-the-docker_gwbridge.
Forwarding over VXLAN overlay actually works only to one docker host. I’ve ended up reading blog articles, eg. ‘Tunnelling inter-host networking through a Docker Swarm Overlay network’ from Matt Henley, and issue thread moby/moby#41775 about various constellations, one issue was caused by the underlying virtual hardware version (vHW 15 and above fixes it).

[jmcombs]

@dcasota I am assuming you're referring to this: moby/moby#41775 (comment)? I am on ESXi 7.0U3c and running VM Version 19.

I had already tried the recommendations of ethtool -K eth0 tx-checksum-ip-generic off prior to opening this issue. In my case, based on the referenced comment above, to know I was hitting that same issue with vxlan, I could run a tcpdump and would only see traffic in one direction. In my case, I see it in both directions.

Workstation:

➜  ~ nc -z -v 192.168.7.237 25565
^C

Node:

root@docker02 [ ~ ]# tcpdump -i ens192 -n port 4789
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
16:08:55.850821 IP 192.168.7.237.44382 > 192.168.7.236.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.0.0.3.54031 > 10.0.0.5.25565: Flags [S], seq 3792502421, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1548007996 ecr 0,sackOK,eol], length 0
16:08:55.851226 IP 192.168.7.236.50569 > 192.168.7.237.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.0.0.5.25565 > 10.0.0.3.54031: Flags [S.], seq 4257375963, ack 3792502422, win 64308, options [mss 1410,sackOK,TS val 4102421346 ecr 1548007996,nop,wscale 7], length 0
16:08:56.845693 IP 192.168.7.237.44382 > 192.168.7.236.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.0.0.3.54031 > 10.0.0.5.25565: Flags [S], seq 3792502421, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1548008996 ecr 0,sackOK,eol], length 0
16:08:56.845876 IP 192.168.7.236.43540 > 192.168.7.237.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.0.0.5.25565 > 10.0.0.3.54031: Flags [S.], seq 4257375963, ack 3792502422, win 64308, options [mss 1410,sackOK,TS val 4102422341 ecr 1548007996,nop,wscale 7], length 0
16:08:57.845196 IP 192.168.7.237.44382 > 192.168.7.236.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.0.0.3.54031 > 10.0.0.5.25565: Flags [S], seq 3792502421, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1548009997 ecr 0,sackOK,eol], length 0
16:08:57.845482 IP 192.168.7.236.57703 > 192.168.7.237.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.0.0.5.25565 > 10.0.0.3.54031: Flags [S.], seq 4257375963, ack 3792502422, win 64308, options [mss 1410,sackOK,TS val 4102423340 ecr 1548007996,nop,wscale 7], length 0
16:08:58.849163 IP 192.168.7.237.44382 > 192.168.7.236.4789: VXLAN, flags [I] (0x08), vni 4096
IP 10.0.0.3.54031 > 10.0.0.5.25565: Flags [S], seq 3792502421, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1548010998 ecr 0,sackOK,eol], length 0

As soon as I disable iptables, it works.

root@docker02 [ ~ ]# systemctl stop iptables

➜  ~ nc -z -v 192.168.7.237 25565
Connection to 192.168.7.237 port 25565 [tcp/*] succeeded!

I just don't know iptables good enough to figure out how to debug iptables to try and figure out why it's breaking. It's not that it's being refused (dropped), it just seems to not be returning which tells me it's a NAT issue.

[jmcombs]

Just posting this here. I found a way to log PREROUTING, FORWARD and POSTROUTING.

I initiate an nc -z -v 192.168.7.237 25565 from my workstation. .237 is docker02 which forwards the packet to docker01.

docker02:

Apr 24 16:51:01 docker02.<redacted> kernel: JC (PREROUTING): IN=eth0 OUT= MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=192.168.7.237 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=54313 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:01 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54313 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:01 docker02.<redacted> kernel: JC (POSTROUTING): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54313 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:02 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54313 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:03 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54313 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:04 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54313 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:05 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54313 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:33 docker02.<redacted> kernel: JC (PREROUTING): IN=eth0 OUT= MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=192.168.7.237 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=54316 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:33 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54316 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:33 docker02.<redacted> kernel: JC (POSTROUTING): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54316 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:34 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54316 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:35 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54316 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 
Apr 24 16:51:36 docker02.<redacted> kernel: JC (FORWARD): IN=eth0 OUT=docker_gwbridge MAC=00:50:56:8b:8e:ab:bc:d0:74:4b:74:b2:08:00 SRC=192.168.7.223 DST=172.18.0.2 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=54316 DPT=25565 WINDOW=65535 RES=0x00 SYN URGP=0 

docker01:

Apr 24 16:51:25 docker01.<redacted> kernel: JC (POSTROUTING): IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57703 DF PROTO=TCP SPT=49954 DPT=705 WINDOW=65495 RES=0x00 SYN URGP=0 
Apr 24 16:51:32 docker01.<redacted> kernel: JC (POSTROUTING): IN= OUT=eth0 SRC=192.168.7.236 DST=192.168.7.238 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11533 DF PROTO=TCP SPT=52804 DPT=7946 WINDOW=64240 RES=0x00 SYN URGP=0 
Apr 24 16:51:32 docker01.<redacted> kernel: JC (PREROUTING): IN=eth0 OUT= MAC=00:50:56:8b:50:bc:00:50:56:8b:6b:66:08:00 SRC=192.168.7.238 DST=192.168.7.236 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=38159 DF PROTO=TCP SPT=37080 DPT=7946 WINDOW=64240 RES=0x00 SYN URGP=0 
Apr 24 16:51:33 docker01.<redacted> kernel: JC (PREROUTING): IN=eth0 OUT= MAC=00:50:56:8b:50:bc:00:50:56:8b:8e:ab:08:00 SRC=192.168.7.237 DST=192.168.7.236 LEN=114 TOS=0x00 PREC=0x00 TTL=64 ID=56421 PROTO=UDP SPT=43254 DPT=4789 LEN=94 
Apr 24 16:51:33 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=65430 PROTO=UDP SPT=45562 DPT=4789 LEN=90 
Apr 24 16:51:34 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=51 PROTO=UDP SPT=38197 DPT=4789 LEN=90 
Apr 24 16:51:35 docker01.<redacted> kernel: JC (PREROUTING): IN=eth0 OUT= MAC=00:50:56:8b:50:bc:00:50:56:8b:8e:ab:08:00 SRC=192.168.7.237 DST=192.168.7.236 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=10054 DF PROTO=TCP SPT=42650 DPT=7946 WINDOW=64240 RES=0x00 SYN URGP=0 
Apr 24 16:51:35 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=143 PROTO=UDP SPT=44722 DPT=4789 LEN=90 
Apr 24 16:51:36 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=259 PROTO=UDP SPT=33015 DPT=4789 LEN=90 
Apr 24 16:51:37 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=292 PROTO=UDP SPT=57216 DPT=4789 LEN=90 
Apr 24 16:51:37 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=304 PROTO=UDP SPT=52609 DPT=4789 LEN=90 
Apr 24 16:51:38 docker01.<redacted> kernel: JC (PREROUTING): IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:f8:ff:c2:5e:1c:ce:08:00 SRC=192.168.7.108 DST=192.168.7.255 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=12375 PROTO=UDP SPT=61001 DPT=137 LEN=58 
Apr 24 16:51:39 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=442 PROTO=UDP SPT=40390 DPT=4789 LEN=90 
Apr 24 16:51:40 docker01.<redacted> kernel: JC (POSTROUTING): IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58146 DF PROTO=TCP SPT=49962 DPT=705 WINDOW=65495 RES=0x00 SYN URGP=0 
Apr 24 16:51:43 docker01.<redacted> kernel: JC (POSTROUTING): IN=vxlan0 OUT=eth0 MAC=02:42:0a:00:00:03:02:42:0a:00:00:05:08:00 SRC=192.168.7.236 DST=192.168.7.237 LEN=110 TOS=0x00 PREC=0x00 TTL=64 ID=1002 PROTO=UDP SPT=40907 DPT=4789 LEN=90 
Apr 24 16:51:45 docker01.<redacted> kernel: JC (POSTROUTING): IN= OUT=eth0 SRC=192.168.7.236 DST=192.168.7.238 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=56807 DF PROTO=TCP SPT=52812 DPT=7946 WINDOW=64240 RES=0x00 SYN URGP=0 

Trying to figure out how to interpret it.

[dcasota]

In yml file "Ignoring unsupported options: restart" -> 'restart' has been deprecated, 'restart_policy' has been added, see docs. In addition, 'restart' is not supported for docker stack deploy. You could try

      restart_policy:
        condition: on-failure

Docker adds/modifies iptables entries automatically.
The overlay networking works. This means that the normal behavior preserves cluster management communications, communication among nodes and overlay network traffic.

Interesting that the service' published (, non-reserved) port effectively isn't active on all nodes despite the fact of iptables entries. What is the first run behavior with an updated yml? In addition, what happens if you add

    deploy:
      mode: replicated
      replicas: 1
      update_config:
        delay: 10s
        order: stop-first

Hope this helps. Daniel

[jmcombs]

@dcasota

So, oddly, if I remove restart or restart_policy from the YAML, it works...

I've done so much to this cluster for testing, I am going to tear it all down and rebuild it and confirm that was the..problem? Thankfully, Photon OS is quick and easy to deploy.

If it fixes it, I'll feel embarrassed but such is life. Regardless, I'll post back here with my final comment confirming the issue and mark it closed. I really appreciate you dealing with me on your weekend.

EDIT: It is not the restart policy.

[jmcombs]

@dcasota I am editing my previous comment as it's not the restart policy. I believe it is related to ethtool -K [network] tx-checksum-ip-generic off mentioned here VMware and Swarm routing.
It appears that running the ethtool command does not persist after reboots. Two questions:

• Is it possible to make this change without ethtool since that package is not installed, by default, with Photon OS?
• How do I make the change persistent across reboots?