howto/networksettings.md
... ...
@@ -77,8 +77,44 @@ To avoid this issue, use one of these approaches:
77 77
3. Use different IPs for services than for peering
78 78
4. De-couple services from specific nodes
79 79
80
-### Other Non-Trivial Pitfalls
81
-- **MTU issues with anycast services**: Using higher than minimum MTU for anycasted services can cause issues because path MTU discovery doesn't work properly with anycast. Since different anycast points of presence (POPs) may be reached during discovery attempts, the path MTU detection can fail, leading to packet fragmentation or drops.
80
+## MTU, Anycast and TCP MSS Clamping
81
+
82
+In DN42, you are almost always running over tunnels (WireGuard, GRE, etc.). Different routers and tunnels may use different MTU. Combined with Anycast, this creates a specific failure mode for Path MTU Discovery (PMTUD).
83
+
84
+### The Anycast PMTUD Blackhole
85
+
86
+When a router sends an ICMPv6 "Packet Too Big" (PTB) message to an Anycast address, the network may route that ICMP packet to a different Anycast instance than the one that sent the original data. As a result, the actual sender never learns to reduce its packet size, leading to "hanging" connections that pass the TCP handshake but fail when transferring data.
87
+
88
+### The Solution: MSS Clamping
89
+
90
+To ensure stability, you must clamp the TCP Maximum Segment Size (MSS) on all nodes. This forces the TCP handshake to negotiate a segment size that fits within your smallest link MTU, bypassing the need for PMTUD.
91
+
92
+It is highly recommended to set this on both `FORWARD` and `OUTPUT` chains of all routers.
93
+
94
+Using nftables (recommended):
95
+```
96
+chain forward {
97
+ type filter hook forward priority filter; policy accept;
98
+ tcp flags syn tcp option maxseg size set rt mtu
99
+}
100
+chain output {
101
+ type filter hook output priority filter; policy accept;
102
+ tcp flags syn tcp option maxseg size set rt mtu
103
+}
104
+```
105
+
106
+Using ip(6)tables:
107
+```
108
+# For IPv4
109
+iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
110
+iptables -t mangle -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
111
+
112
+# For IPv6
113
+ip6tables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
114
+ip6tables -t mangle -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
115
+```
116
+
117
+## Other Non-Trivial Pitfalls
82 118
83 119
- **accept_local sysctl settings**: When running anycast services on routers, ensure the accept_local sysctl is enabled. Without this setting, a router might drop transit traffic from other origins that has the anycasted IP as the source address, breaking connectivity through your network for those services.
84 120