path MTU discovery, maximum segment size and idiots on the internet

Recently we ran into a seriously weird problem at home.

Just to clarify our setup: we have a cable-modem that does about absolutely nothing, connected to a Debian Stable box with 2 interfaces that acts as our gateway, firewall. NAT.

First, DNS queries were randomly dropped or had serious issues trying to resolve which is extremely annoying as it slows down your internet experience quite a bit.

That turned out to be an issue we still haven’t figured out completely, it seems to have something to do with rtorrent opening a ridiculous amount of sockets and keeping those open so the kernel just kills or throws away other sockets it can’t handle anymore which turned out to be our DNS requests.

The other, far more troubling problem was that some sites, for no apparent reason were excruciatingly slow or just would not load at all.

After some serious business tcpdumping with fortisbanking.com we came to the very simple conclusion that fortisbanking.com was dropping ICMP Type 3 code 4 (destination unreachable + fragmentation needed) packets from our side, basically telling it that our MTU was set to 1420 and that we couldn’t accept their replies since they had a size of 1500.

Why do we have an MTU set to 1420? Because we use a few 6to4 tunnels and due to the overhead that causes we need to deduce 80 bytes from the normal 1500 MTU to make sure those packets come through.

So, once we set our MTU to 1500, tah-dah, fortisbanking.com started working again.

We wondered though, how was it that with a MTU of 1420 fortisbnaking.com did redirect us from http to https but only then broke.

Some tcpdump’ing further and we discovered the ugly truth. For some reason yet unknown the HTTP-request’s MSS-size was clamped to MTU-40, which is the behavior you’d like. The HTTPS-request however wasn’t, sending an MSS-size of 1460 instead of 1380 which of course triggered a response packet-size from Fortis’ side of 1380 and since they dropped ICMP 3/4 we ended up loosing packets.

One way to fix this is a little trick in IP-tables called MSS-clamping. This basically forces the MSS of the packet to be the size of the MTU-40.

After we added that, everything started working like a charm.

The magic rule here is: /sbin/iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

For vuurmuur users such as me it’s even easier. Enter vuurmuur_conf, go to Interfaces, select your external/WAN/internet interface, hit ‘e’ to edit and then hit F7 to get the TCPMSS window and tick the box, hit OK and Apply Changes.

Though this just shouldn’t really be necessary it fixes the path MTU discovery issues we ran into and just makes sure everything works from our side.

In my opinion, people just shouldn’t drop stuff like ICMP’s dest. unreachable + fragmentation needed packets, however, as a friend pointed out, we don’t live in Utopia.

(We’ve now even extended our setup so that on the internal, fully gigabit-powered network our MTU is set to 9000, the size of a jumbo packet, externally we’re still clamping everything and it works).