Broadcast/Multicast Storm issue on P1010 and Marvell 88E1512 PHY with Linux 3.18.9-rt4 kernel

k_nagesh

I am using the Freescale P1010 Processor and Marvell 88E1512 PHY for Ethernet with Linux 3.18.9-rt4 kernel;

While performing Cyber Security tests using Achilles-box, we have found certain issues with IP Multicast Storm (generates identical, valid IP packets and multicasts them over the link at a specific rate) and UDP Broadcast Storm (generates identical, valid UDP packets and broadcasts them over the link at a specific rate), thus it was observed that the above threshold rate (above 90Mbps), the TCP and UDP ports went down and it requires a hard reboot to bring them back to the normal state.

Immediately after the broadcast storm starts, the message "NOHZ: local_softirq_pending 102" appears 10 times on the console. The reason it "only" appears 10 times is because the kernel limits the number of times it writes this message to 10.

About 11 seconds after the first "NOHZ: local_softirq_pending 102" message, a page allocation failure is reported. This means that system memory is depleted. Analysis of the call stack reveals that it is the driver (drivers/net/ethernet/freescale/gianfar.c and net/core/skbuff.c) for the Ethernet port, more precisely the soft-irq task which handles incoming frames, that fails to obtain more memory to receive buffers. Before the broadcast storm starts, there is more than 200MB of free memory, and all this is gone in 11 seconds. This renders the system unable to continue normal operation, even after the broadcast storm stops.

There seems to be a weakness in how Linux handles incoming Ethernet frames, leaving it vulnerable to storm attacks where all available memory is consumed, leaving the system in an unstable state after the attack is over.

Could anyone have faced such similar issue and does anybody have knowledge on how to fix / suggestion to find out why a broadcast/multicast storm is able to consume almost all CPU power of the P1010 processor.