So I put all of these things together a while ago for bandwidth reporting:
- VMWare‘s netflow reporting – I configured a virtual distributed switch to send netflow reporting to a collector. Every time some machine runs up its internet usage, the bytes and things get tallied up and sent off for collection.
- NAT: in order to reach the collector, the netflow packets pass through a NAT gateway, which rewrite them because the source network address is on a private network. Okay, so that makes no sense if you don’t speak jargon, but that’s just tough. The packets went through this thing. Come to think of it, I think I didn’t even consider how the packets were getting routed from the source to the collector – they arrived in good order, so I was happy, and I thought no further.
- ipfix: VMware doesn’t send netflow packets, but sends something like netflow called ipfix.
And, for good measure:
- My preferred flow capture thing didn’t speak ipfix, so I used nfcapd to translate the packets.
And it worked wonderfully – one of those quick hacks that becomes permanent. It was perfect(ish) … until the tenth of this month. At that point, it started logging very odd bandwidth information:
srcIP dstIP prot srcPort dstPort octets packets 0.0.0.0 0.0.0.1 0 23047 38382 285327087 43 0.0.0.0 0.0.0.231 0 0 0 325 285327087
The famous IP address 0.0.0.0 sent 43 packets with a total of 285327087 bytes … that means that some of those packets were seriously huge – mega jumbo frames of 6.3Mb each. Then it sent 285327087 packets totalling 325 bytes. The famous NULL packet was detected! This information appeared in the flow-capture log … and investigation showed that it was in the nfcapd logs. Wireshark could not understand the captures, which left the prime suspect as VMware – surely they updated something and broke everything. Maybe a memory error on the collector, or maybe a memory error on the netflow originator … what could it be?
It turns out that at the bottom of this pile of steaming randomness, is that the ipfix/netflow streams from multiple VMware ESX servers together look like one stream from a single confused host, rather than discrete streams from a number of slightly less confused hosts. The collector sees them all having the same source IP address. Usually the data sent by different VMware hosts would be sent at different times, but as the load increased, and as time passed and things went in and out of sync, so it became more likely that there would be a collision, confusion and corruption.
All I needed to do was remove the NAT, and the problem evaporated.
The moral of the story is: netflow and NAT do not mix. Don’t cross the beams.
