Already wondered how you are going to find where these drops came from? Say no more, we are going to talk about the Span-on-drop feature, especially on Nexus 9000.

How is it working

Packets drops are very common in networks, and may be caused by a a lot of different things. In order to troubleshoot them, we can use SPAN on drop. It will copy the packet that is dropped to a destination interface, so we can analyze it. It is a very useful tool, especially when you see drops, but you don’t knopw where they come from! The ‘Span-on-Drop’ feature can pinpoint packets lost due to congestion or forwarding issues. When a packet is earmarked for dropping, the system duplicates it into a separate buffer, designates it as ‘Span-on-Drop’, and then send it to the pre-set ‘Span-on-Drop’ destination. The packet is then dropped as usual. The ‘Span-on-Drop’ feature is supported on (almost) all Cisco Nexus 9000 Series switches.

Use case

We recently found COPP drops on a bunch of n9k. We wanted to know what was the source of these drops, so we used the span-on-drop (SOD) feature, with this configuration :

#configure
#monitor erspan origin ip-address x.x.x.x global
#monitor session 1 type erspan-source 
#source forward-drops rx
#destination ip y.y.y.y
#erspan-id 1
#vrf default
#no shut
#exit  

We then used wireshark to analyze the packets, and found that the drops were caused by a lot of ARP requests.

Limitations

Unfortunately, only ERSPAN is supported as the destination for SPAN-on-drop. It means that you can’t use a physical interface as destination. Also, the source can only be a physical interface, not a VLAN or a port-channel.

More details are available here : https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/nexus-9000-span-drop-wp.pdf