So, you have a Netgear DGN1000 N150 router or possibly any other Netgear router, and you have it set up with:
- DSL – VPI=8, VCI=35, LLC, PPPOE
One day, as you are sending mail, you notice that mail larger than 300kb or so cannot be sent … on the wireless only. This is odd, so you capture the traffic on the server side and on the client side, and discover that the packets you are sending on the client side do not reach the server side. That’s really odd. So you dig.
After downloading the source code for your router (thanks Netgear!) it turns out that Netgear has the following widely recommended code to set up iptables:
# strings DGN1000B_VB1.00.45_GR_src/target/sbin/rc | grep -r pmtu -A TCP_MSS -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
This rule is correctly linked into the
FORWARD chain and will process both
So, what’s up? Why can I not send my mail?
This is what the server sees during the TCP session setup – SYN, SYN-ACK, ACK:
$ tcpdump -r server-smtpfail.pcap -v port 25 -c 3 -n reading from file server-smtpfail.pcap, link-type EN10MB (Ethernet) 14:23:24.621396 IP (tos 0x0, ttl 54, id 30211, offset 0, flags [DF], proto TCP (6), length 60) 188.8.131.52.55337 > 184.108.40.206.25: Flags [S], cksum 0x62c2 (correct), seq 994029838, win 14600, options [mss 1452,sackOK,TS val 55887629 ecr 0,nop,wscale 7], length 0 14:23:24.621502 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 220.127.116.11.25 > 18.104.22.168.55337: Flags [S.], cksum 0x1d7a (incorrect -> 0x26f2), seq 2508676892, ack 994029839, win 14480, options [mss 1460,sackOK,TS val 18433651 ecr 55887629,nop,wscale 6], length 0 14:23:24.844055 IP (tos 0x0, ttl 54, id 30212, offset 0, flags [DF], proto TCP (6), length 52) 22.214.171.124.55337 > 126.96.36.199.25: Flags [.], cksum 0x8da2 (correct), ack 1, win 115, options [nop,nop,TS val 55887685 ecr 18433651], length 0
This is what the client sees:
$ tcpdump -r client-smtpfail.pcap -v port 25 -c 3 -n reading from file client-smtpfail.pcap, link-type EN10MB (Ethernet) 14:23:23.814713 IP (tos 0x0, ttl 64, id 30211, offset 0, flags [DF], proto TCP (6), length 60) 192.168.64.4.55337 > 188.8.131.52.25: Flags [S], cksum 0xad23 (incorrect -> 0xd310), seq 994029838, win 14600, options [mss 1460,sackOK,TS val 55887629 ecr 0,nop,wscale 7], length 0 14:23:24.038449 IP (tos 0x0, ttl 48, id 0, offset 0, flags [DF], proto TCP (6), length 60) 184.108.40.206.25 > 192.168.64.4.55337: Flags [S.], cksum 0x9748 (correct), seq 2508676892, ack 994029839, win 14480, options [mss 1460,sackOK,TS val 18433651 ecr 55887629,nop,wscale 6], length 0 14:23:24.038491 IP (tos 0x0, ttl 64, id 30212, offset 0, flags [DF], proto TCP (6), length 52) 192.168.64.4.55337 > 220.127.116.11.25: Flags [.], cksum 0xad1b (incorrect -> 0xfdf8), ack 1, win 115, options [nop,nop,TS val 55887685 ecr 18433651], length 0
$ tcpdump -r client-smtpfail.pcap -v port 25 -c 3 -n | egrep -o 'mss.[0-9]+' reading from file client-smtpfail.pcap, link-type EN10MB (Ethernet) mss 1460 mss 1460 $ tcpdump -r server-smtpfail.pcap -v port 25 -c 3 -n | egrep -o 'mss.[0-9]+' reading from file server-smtpfail.pcap, link-type EN10MB (Ethernet) mss 1452 mss 1460
So here’s how it went down:
- Client says MTU is 1460 in SYN packet (1500 bytes ethernet packet, less 40 bytes IP header)
- ADSL router subtracts 8 and transmits it to the server as 1452
- Server says it can transmit TCP segments of 1460 bytes
- ADSL router forwards the server’s MSS size without alteration
- Client sees that server will accept MSS of 1460
During the data transmission (sending SMTP mail in this case):
- After about 200kB of data on a nice fast line the segment size climbs up to 1460
- The client transmits this packet to the server – it’s 1500 bytes across the wire, which is the limit.
- The server never gets the packet – after it adds 8 bytes of PPPOE headers to it, it is too big, so it gets dropped.
So what’s the bug? The ADSL router has the code to set the MSS to the path MTU. Here’s the thing:
- Path MTU for outgoing SYN is 1492 – this is the MTU of the PPP interface (correct)
- Path MTU for incoming SYN-ACK is 1500 – this is the MTU of the wireless interface (FAIL!)
- Workaround: on each client using the wireless connection, set the MTU to be a little lower
- Crummy fix: on the Netgear DSL router, the MTU of the wireless card can be set to 1492. I have no idea what this is going to do.
- Better fix: don’t rely on the path MTU, but set the MTU explicitly by changing the rol from
-m tcpmss --clamp-mss-to-pmtu
-m tcpmss --mss 1492:65535 -j TCPMSS --set-mss 1492
- Best fix: update the kernel to the version that checks the source and the destination MTU
I told netgear about it — case 19630614 on my.netgear.com :
Thu Oct 18 08:44:58 ~ $ xsel|grep expert Thank you for choosing NETGEAR. My name is Ashish and I will be your support expert. I appreciate the opportunity to assist you. Thank you for choosing NETGEAR. My name is Ashish and I will be your support expert. I appreciate the opportunity to assist you. Thank you for choosing NETGEAR. My name is Sajeev and I will be your support expert. I appreciate the opportunity to assist you. Thank you for choosing NETGEAR. My name is Amrit and I will be your support expert. I appreciate the opportunity to assist you.
The fourth reply says they’re bumping it up the chain. That’s nice.
Update Fri 7 Dec 2012: and they asked for the pcap files! Yay! Just as well I kept them. These are the pcap files:
- What the client sees on the wireless: client-smtpfail.pcap.
- What the server sees out on the internet: server-smtpfail.pcap
Update Fri 25 Jan 2013: another response, but cluelessness has returned: they say “cannot replicate” and want access to “the computer” via teamviewer. I said, “You can’t be serious! Teamviewer? It takes you 30 days for Netgear to decide to do anything! There is NO WAY that I’ll be able to keep teamviewer open for that long – there are children in the house! Besides all this, the Netgear router that you are trying to diagnose is providing the network access, and any attempt to replicate the problem will cause teamviewer to fail. Additionally, the desktop system that could possibly run Teamviewer is running Linux, and if you couldn’t do anything with the stuff I’ve sent you already, I don’t have any reason to believe you’ll do any better with a Linux desktop. Even if I gave you a root login on my system, you would not be able to simultaneously bring up the wireless and the ethernet AND perform a meaningful test. This is not a desktop support matter, but a subtle networking problem. Perhaps you can tell me what specific difficulty was encountered while reproducing the fault, e.g. you couldn’t find a device of the appropriate model, you couldn’t establish a 802.11g wireless link, you couldn’t run a sniffer, you couldn’t generate traffic, etc, or after all of that your test worked perfectly, and your sniffer confirmed that 1500 byte packets were sent and handled correctly?” Much as that’s a great response to a stupid idea, because I couldn’t do the crazy thing they asked, my expectations are rather low. I think my Netgear junk will never be fixed. “Connect forever” they said, “with easy and reliable products for your wireless home network” they said – just that the “reliable” is only for certain types of traffic.
Update Tuesday 2 April 2013:they have closed my ticket. No more correspondence is being entered into. I have given them a piece of my mind, which, it turns out, is rather angry about it all:
Bloody hell. In 6 months only one person had enough sense to ask for a .pcap file, and when I supplied it, nobody could respond to that. Hire some competent staff, or stop selling networking products. Instead you send it back over and over with no real update. It’s a simple problem, but you’ve moved on and you DO NOT SUPPORT OLD PRODUCTS (ie. something not under active development). It doesn’t help to be polite if you are not willing to apply your mind for more than 3 minutes in the course of 6 months. I have invested more time in this problem than you ever did. Clearly I care too much about your products, and I will try to fix that by never buying them again. You join my blacklist, consisting of Lexmark and Apple, and now also Netgear.
Yes, I have a personal blacklist of vendors. It’s still short:
- Lexmark: for suing people who made cartridges that work in their printers, and abandoning their older GDI printers without software support.
- Apple: for adding boot-loader encryption in the ipod Nano (without changing the model number) so that it cannot be customised (I still have it somewhere)
- Netgear: for abandonware: making a basic networking mistake and then providing only lip service support instead of technical support.
Bonus candidates for my personal vendor blacklist:
- Hewlett-Packard: for years of price gouging on inkjet cartridges, and failure to connect the IPMI BMC on their motherboards to the network port (but then offering an expensive add on that does just that). HP remains in candidate blacklist status because I have not yet paid any of my own money for their products …
- Microsoft: for their subversive and destructive approach to document standards, and for funding “Planned Parenthood”.