[ag-automation] Adding OSADL patch to 2.6.33.7 increases latency and CPU usage?

[Grant Edwards]

I've been doing some interrupt latency testing using a board running a
400MHz ARM9 (an Atmel at19sam9g20).  I'm measuring interrupt latency
of an ISR attached to external interrupt IRQ0.

I built three kernels from scratch and ran the same test on all three:

 2.6.30          the kernel we've been using (no OSADL patches)
 2.6.33.7        without OSADL real-time patches
 2.6.33.7-30rt   with OSADL real-time patches

I don't have enough data to say much about worst-case latency, but
after looking a thousand or two samples for each...

 * 2.6.30 and 2.6.33.7 are the same: there is as much variation
   between test runs as there is between kernels.

 * Typical latency with OSADL patch is 3X the latency without it.

 * CPU idle time during test is 10% with OSADL patch and 30% without
   it.

IOW: the OSADL real-time patch for 2.6.33.7 makes both the typical
interrupt latency and the CPU usage significantly worse.

Typical latency on a heavily loaded system w/o the OSADL patch is
5-15us.

Typical latency on the same system w/ the OSADL patch is 15-50us (I've
never seen latency below 15us with the OSADL patch).

Is this the expected behavior?

I don't have enough data to state conclusively what the worst-case
latencies are, but it looks like in all cases they're 200-250us.

In all cases the kernel was configured to be the most preemptible it
could be.  In the 2.6.33.5-30rt kernel I don't have any of the tracers
enabled.

The kernel configurations are as identical as I could make them.  In
the 2.6.33.7-30rt case, I started with the .config file used to build
the other 2.6.33.7 kernel and ran old-config making sure to select the
"fully preemptible" option when it asked.

Why would a "vanilla" 2.6.33.7 AT91 kernel perform so much better than
one with the OSADL realtime patch?

-- 
Grant