So I did some tests - and would love to know if this is typical. Any CISCO experts reading this may be able to comment.
Testing using arping from linux, I could see that the CISCO would respond to only some of my ARP requests. Maybe one in five, but not very consistent. This is a tad odd, and may be down to some general ARP rate limiting perhaps.
On top of that, when it did respond, it did so after 2.99 seconds. This was very consistent - I had to use arping one ARP request at a time to confirm this.
I have to wonder what the hell it is doing! From a coding point of view, holding on to the ARP request or reply for that length of time is more work than just answering the ARP right away. I am at a loss as to what is going on.
For comparison, a FireBrick is timed by linux at 180us response and answered every ARP.
Anyway, it means I have had to tweak the way the ARP system renews ARPs to try a bit longer, otherwise every now and then the CISCO vanishes for a few seconds.
Oh, and yes, they still look like this with some arbitrary padding to min packet size for Ethernet.
09:40:20.688429 ARP, Request who-has 22.214.171.124 tell 126.96.36.199, length 46
0x0000: 0001 0800 0604 0001 0003 971d c009 5bf0 ..............[.
0x0010: b0fe 0000 0000 0000 5bf0 b001 474e 5520 ........[...GNU.
0x0020: 5465 7272 7950 7261 7463 6865 7474 TerryPratchett
P.S. It was CoPP, but we don't understand why it would delay ARPs 3s in that process.