Documentation/RCU/stallwarn.rst

1 .. SPDX-License-Identifier: GPL-2.0
9 options that can be used to fine-tune the detector's operation.  Finally,
20 -	A CPU looping in an RCU read-side critical section.
22 -	A CPU looping with interrupts disabled.
24 -	A CPU looping with preemption disabled.
26 -	A CPU looping with bottom halves disabled.
28 -	For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel
33 -	Booting Linux using a console connection that is too slow to
34 	keep up with the boot-time console-message rate.  For example,
35 	a 115Kbaud serial console can be -way- too slow to keep up
36 	with boot-time message rates, and will frequently result in
40 -	Anything that prevents RCU's grace-period kthreads from running.
41 	This can result in the "All QSes seen" console-log message.
44 	result in the ``rcu_.*kthread starved for`` console-log message,
47 -	A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might
48 	happen to preempt a low-priority task in the middle of an RCU
49 	read-side critical section.   This is especially damaging if
50 	that low-priority task is not permitted to run on any other CPU,
54 	memory, you might see stall-warning messages.
56 -	A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that
62 	CONFIG_PREEMPT_RCU case, you might see stall-warning
68 	can increase your system's context-switch rate and thus degrade
69 	performance.
71 -	A periodic interrupt whose handler takes longer than the time
74 	Note that certain high-overhead debugging options, for example
79 -	Testing a workload on a fast system, tuning the stall-warning
81 	running the same workload with the same stall-warning timeout on a
82 	slow system.  Note that thermal throttling and on-demand governors
85 -	A hardware or software issue shuts off the scheduler-clock
86 	interrupt on a CPU that is not in dyntick-idle mode.  This
90 -	A hardware or software issue that prevents time-based wakeups
97 -	A bug in the RCU implementation.
99 -	A hardware failure.  This is quite unlikely, but has occurred
105 The RCU, RCU-sched, and RCU-tasks implementations have CPU stall warning.
106 Note that SRCU does -not- have CPU stall warnings.  Please note that
123 Fine-Tuning the RCU CPU Stall Detector
129 but may be overridden via boot-time parameter or at runtime via sysfs.
134 ----------------------------
144 	So if you are 10 seconds into a 40-second stall, setting this
146 	-next- stall, or the following warning for the current stall
150 	Stall-warning messages may be enabled and disabled completely via
154 ---------------------
163 -------------------
166 	own warnings, as this often gives better-quality stack traces.
174 -------------------------------
176 	This boot/sysfs parameter controls the RCU-tasks stall warning
177 	interval.  A value of zero or less suppresses RCU-tasks stall
178 	warnings.  A positive value sets the stall-warning interval
179 	in seconds.  An RCU-tasks stall warning starts with the line:
184 	task stalling the current RCU-tasks grace period.
187 Interpreting RCU's CPU Stall-Detector "Splats"
190 For non-RCU-tasks flavors of RCU, when a CPU detects that it is stalling,
194 	2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0
195 	16-...: (0 ticks this GP) idle=81c/0/0 softirq=764/764 fqs=0
199 causing stalls, and that the stall was affecting RCU-sched.  This message
203 possible for an rcu_state stall to be caused by both CPUs -and- tasks,
208 ticks this GP)" indicates that this CPU has not taken any scheduling-clock
211 The "idle=" portion of the message prints the dyntick-idle state.
212 The hex number before the first "/" is the low-order 12 bits of the
213 dynticks counter, which will have an even-numbered value if the CPU
214 is in dyntick-idle mode and an odd-numbered value otherwise.  The hex
216 a small non-negative number if in the idle loop (as shown above) and a
224 example, if the CPU might have been in dyntick-idle mode for an extended
227 across repeated stall-warning messages, it is possible that RCU's softirq
229 the stalled CPU is spinning with interrupts are disabled, or, in -rt
230 kernels, if a high-priority process is starving RCU's softirq handler.
232 The "fqs=" shows the number of force-quiescent-state idle/offline
233 detection passes that the grace-period kthread has made across this
239 period (in this case 2603), the grace-period sequence number (7075), and
248 The "last_accelerate:" prints the low-order 16 bits (in hex) of the
251 rcu_prepare_for_idle(). "dyntick_enabled: 1" indicates that dyntick-idle
255 there will be a spurious stall-warning message, which will include
261 possible for a zero-jiffy stall to be flagged in this case, depending
262 on how the stall warning and the grace-period initialization happen to
268 grace period has nevertheless failed to end, the stall-warning splat
271 …n, last rcu_preempt kthread activity 23807 (4297905177-4297881370), jiffies_till_next_fqs=3, root …
274 since the grace-period kthread ran.  The "jiffies_till_next_fqs"
276 of jiffies between force-quiescent-state scans, in this case three,
278 ->qsmask field is printed, which will normally be zero.
280 If the relevant grace-period kthread has been unable to run prior to
284 	kthread starved for 23807 jiffies! g7075 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 ->cpu=5
286 Starving the grace-period kthreads of CPU time can of course result
289 grace-period sequence number, the "f" precedes the ->gp_flags command
290 to the grace-period kthread, the "RCU_GP_WAIT_FQS" indicates that the
292 task_struct ->state field, and the "cpu" indicates that the grace-period
299 If a stall lasts long enough, multiple stall-warning messages will be
312 	INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 7-... } 21119 jiffies s: 73 root: 0x2/.
323 grace-period sequence counter is 73.  The fact that this last value is