« Building LIBGD on FreeBSD | Main | Agile Programming: How to doom your software project »

June 23, 2005

Things to look for in a gigabit switch

Several customers have been moving to gigabit Ethernet switches to run traffic around inside their networks, and I've been setting up MRTG monitoring of them. Virtually all networking devices implement SNMP's interface MIB, and they provide byte counters for each interface in both directions. Defined in the ifTable tree, ifInOctets.1 is the number of bytes received on port 1, and ifOutOctets.1 is the outgoing count. These counters are unsigned 32-bit integers, and the values are typically collected every five minutes: the difference between the two counters is taken and divided by the interval (300 seconds), yielding an average bandwidth during that period.

But difficulties arise as the speeds increase: a 32-bit counter is just not big enough to hold 5 minutes worth of traffic at speeds much above 100 mbit/sec. Though MRTG can deal with wraparound through zero, they cannot deal with full wraparound of a 32-bit counter - the data is simply lost. One needs to poll the device around twice per minute to monitor full gigabit speeds.

The solution is to use 64-bit counters instead; they have dramatically higher range and would take more than 4000 years to overflow even at full gigabit speeds. These are defined in a different part of the tree, in ifXTable, with several "HC" - High Capacity - counters that largely parallel the similar entries in ifTable. We'd expect to find ifHCInOctets.n and ifHCOutOctets.n here.

But Dell's Power Connect 5324 24-port managed gigabit switch simply doesn't support these 64-bit counters. I didn't see them when doing an snmpwalk, and because I just couldn't believe that a $1000 switch supported only 32-bit counters, so I must have been doing something wrong.

After hours of looking I found a note in the Dell community support forums that they are not supported and won't be - limitations of the ASIC.

Wow.

These devices are entirely unsuitable for any environment where monitoring is important, and I'm going to recommend that my customer return it and get a switch that does.

And now I know to put "Does the device support 64-bit octet counters?" on the shopping checklist.

Posted by steve at June 23, 2005 09:25 AM

Trackback Pings

TrackBack URL for this entry:
http://www.unixwiz.net/mt/trackback/41

Comments

Another thing to look for in a gigabit switch: an application that can actually use them. :)

Most people I've seen going Gig are doing so because "I want more speed" even though the network is no where near the bottleneck in their application perf today.

So I'm all for faster networks, but if you want faster systems, it takes some analysis in my book before you really have justified new NICs and switches end to end.

Posted by: Eric Fleischman at June 23, 2005 12:08 PM

In this case, they will use it; they have a raft of very fast servers and needed a 1gbit uplink to the ISP router. They have already had servers go above 100mbit of real traffic per server.

Posted by: Steve Friedl at June 23, 2005 12:10 PM

Cisco equipment does not only have 64bit counters, but also OIDs which directly delivers the five minute exponentially-decayed moving average of certain counters. For example, .1.3.6.1.4.1.9.2.2.1.1.6 (locIfInBitsSec) delivers that value for input octets.

otoh, cisco is notoriously known for misfiring interface counters.

Posted by: Marc 'Zugschlus' Haber at July 11, 2005 01:30 AM

I'm trying to monitor/graph a Cisco 3750 switch using MRTG and Cacti but I'm getting the dreaded wrap around issue (goes a little above 100mbit, then pops back down to zero). Does anyone know how to correctly monitor them?

Posted by: John Boyd at October 21, 2005 02:55 PM

in MRTG config file, use SNMP version 2
eg. public@202.18.1.1:::::2

Posted by: Ben at July 26, 2006 12:46 PM

FYI the Dell 5324s now have a firmware upgrade for to support 64bit counters and so do the 3000 series. The new 62xx series supports 64bit out of the factory

Posted by: Matthew Gillaspie at February 13, 2007 09:07 AM