[ag-automation] echo 1 > /sys/kernel/debug/tracing/latency_hist/enable/preemptirqsoff causes crash

[Carsten Emde]

On 11/10/2014 06:53 PM, "Dr.-Ing. Matthias Schöpfer" wrote:

> I am having a strange latency issue with my application. But I do not
> want to bore you with details. Funny is, I think, that:
> echo 1 > /sys/kernel/debug/tracing/latency_hist/enable/preemptirqsoff
> will immediately crash the machine. I have seen this with 3.12.26-rt40
> and 3.12.31-rt45 and, I think, with 3.14.12-rt9 (I can double check, if
> anyone is interested).
Yes. This is known. It is not funny.

> I have another machine where it works just fine, although different
> .config. Both are x86-Hardware, the one that crashes is x86_64.
Same here.

Currently, we are very busy to get the development of the real-time 
patches up and running again. Since this is priority #1 we have to 
postpone maintenance of the preemptirqsoff crash and other known 
problems to a later date.

In a first step of latency hunting, I would propose to run hwlatdetect 
to exclude SMIs as the origin of the observed latency. This is the most 
frequent single source of a latency.

If your system does not suffer from SMIs, try cyclictest with function 
tracing instead of preemptirqsoff - this normally works pretty well. 
When setting the break timeout, you should take in mind that function 
tracing slows down the system by a factor of 3 to 5. Fortunately, it 
slows down the latency too. If, for example, your system has a 
worst-case latency of 50 µs with sporadic latencies of 200 µs, you would run

   cyclictest -m -Sp99 -i1000 -d0 -fb500

This ensures that the normal system latency will not stop cyclictest but 
the abnormal latency will. Then locate the regular schedules of 
cyclictest in the trace output. They will have a distance of 
approximately 1000 µs with the exception of the last one. The culprit of 
the latency you are searching is the code that is executed at 1000 µs 
plus the time of the last but one cyclictest schedule.

Please come back, if this does not work for you.

Thanks,
Carsten.