Lines Matching +full:fault +full:- +full:inject
2 Fault injection capabilities infrastructure
5 See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
8 Available fault injection capabilities
9 --------------------------------------
11 - failslab
15 - fail_page_alloc
19 - fail_usercopy
23 - fail_futex
25 injects futex deadlock and uaddr fault errors.
27 - fail_sunrpc
31 - fail_make_request
34 /sys/block/<device>/make-it-fail or
35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
37 - fail_mmc_request
42 - fail_function
48 - NVMe fault injection
50 inject NVMe status code and retry flag on devices permitted by setting
55 - Null test block driver fault injection
57 inject IO timeouts by setting config items under
59 inject requeue requests by setting config items under
61 inject init_hctx() errors by setting config items under
64 Configure fault-injection capabilities behavior
65 -----------------------------------------------
70 fault-inject-debugfs kernel module provides some debugfs entries for runtime
71 configuration of fault-injection capabilities.
73 - /sys/kernel/debug/fail*/probability:
79 Note that one-failure-per-hundred is a very high error rate
83 - /sys/kernel/debug/fail*/interval:
91 - /sys/kernel/debug/fail*/times:
93 specifies how many times failures may happen at most. A value of -1
96 - /sys/kernel/debug/fail*/space:
102 - /sys/kernel/debug/fail*/verbose
108 log line per failure; '2' will print a call trace too -- useful
109 to debug the problems revealed by fault injection.
111 - /sys/kernel/debug/fail*/task-filter:
117 /proc/<pid>/make-it-fail==1.
119 - /sys/kernel/debug/fail*/require-start,
120 /sys/kernel/debug/fail*/require-end,
121 /sys/kernel/debug/fail*/reject-start,
122 /sys/kernel/debug/fail*/reject-end:
131 - /sys/kernel/debug/fail*/stacktrace-depth:
134 for a caller within [require-start,require-end) OR
135 [reject-start,reject-end).
137 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
141 default is 'Y', setting it to 'N' will also inject failures into
144 - /sys/kernel/debug/failslab/ignore-gfp-wait:
145 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
149 default is 'Y', setting it to 'N' will also inject failures
152 - /sys/kernel/debug/fail_page_alloc/min-order:
157 - /sys/kernel/debug/fail_futex/ignore-private:
164 - /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
171 - /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
178 - /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
185 - /sys/kernel/debug/fail_function/inject:
187 Format: { 'function-name' | '!function-name' | '' }
194 - /sys/kernel/debug/fail_function/injectable:
199 - NULL: retval must be 0.
200 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
201 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
203 - /sys/kernel/debug/fail_function/<function-name>/retval:
205 specifies the "error" return value to inject to the given function.
209 $ printf %#x -12 > retval
214 In order to inject faults while debugfs is not available (early boot time),
227 - /proc/<pid>/fail-nth,
228 /proc/self/task/<tid>/fail-nth:
230 Write to this file of integer N makes N-th call in the task fail.
232 that the fault setup with a previous write to this file was injected.
233 A positive integer N indicates that the fault wasn't yet injected.
236 like probability, interval, times, etc. But per-capability settings
237 (e.g. fail_futex/ignore-private) take precedence over it.
244 --------------------------
252 Since the function-level error injection forcibly changes the code path
257 - The function returns an error code if it fails, and the callers must check
260 - The function does not execute any code which can change any state before
267 (free objects) functions are usually harder to inject errors than allocate
283 There are 4 types of errors defined in include/asm-generic/error-injection.h
290 This function will return an `-errno` error code if it fails. e.g. return
291 -EINVAL if the input is wrong. This will include the functions which will
292 return an address which encodes `-errno` by ERR_PTR() macro.
295 This function will return an `-errno` or `NULL` if it fails. If the caller
300 This function will return `true` (non-zero positive value) if it fails.
307 How to add new fault injection capability
308 -----------------------------------------
310 - #include <linux/fault-inject.h>
312 - define the fault attributes
316 Please see the definition of struct fault_attr in fault-inject.h
319 - provide a way to configure fault attributes
321 - boot option
323 If you need to enable the fault injection capability from boot time, you can
328 - debugfs entries
335 - module parameters
337 If the scope of the fault injection capability is limited to a
339 configure the fault attributes.
341 - add a hook to insert failures
343 Upon should_fail() returning true, client code should inject a failure:
348 --------------------
350 - Inject slab allocation failures into module init/exit code::
355 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
358 echo -1 > /sys/kernel/debug/$FAILTYPE/times
361 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
365 bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
368 if [ $# -eq 0 ]
380 faulty_system modprobe -r $m
383 ------------------------------------------------------------------------------
385 - Inject page allocation failures only for a specific module::
392 if [ -z $module ]
400 if [ ! -d /sys/module/$module/sections ]
406 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
407 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
409 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
412 echo -1 > /sys/kernel/debug/$FAILTYPE/times
415 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
416 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
417 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
424 ------------------------------------------------------------------------------
426 - Inject open_ctree error while btrfs mount::
430 rm -f testfile.img
432 DEVICE=$(losetup --show -f testfile.img)
433 mkfs.btrfs -f $DEVICE
434 mkdir -p tmpmnt
438 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
439 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
440 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
443 echo -1 > /sys/kernel/debug/$FAILTYPE/times
447 mount -t btrfs $DEVICE tmpmnt
448 if [ $? -ne 0 ]
456 echo > /sys/kernel/debug/$FAILTYPE/inject
459 losetup -d $DEVICE
464 ----------------------------------------------------
466 tools/testing/fault-injection/failcmd.sh. Please run a command
467 "./tools/testing/fault-injection/failcmd.sh --help" for more information and
472 Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
475 # ./tools/testing/fault-injection/failcmd.sh \
476 -- make -C tools/testing/selftests/ run_tests
481 # ./tools/testing/fault-injection/failcmd.sh --times=100 \
482 -- make -C tools/testing/selftests/ run_tests
484 Same as above except to inject page allocation failure instead of slab
488 ./tools/testing/fault-injection/failcmd.sh --times=100 \
489 -- make -C tools/testing/selftests/ run_tests
491 Systematic faults using fail-nth
492 ---------------------------------
494 The following code systematically faults 0-th, 1-st, 2-nd and so on
513 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
514 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
526 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
536 1-th fault Y: res=-1/23
537 2-th fault Y: res=-1/23
538 3-th fault Y: res=-1/12
539 4-th fault Y: res=-1/12
540 5-th fault Y: res=-1/23
541 6-th fault Y: res=-1/23
542 7-th fault Y: res=-1/23
543 8-th fault Y: res=-1/12
544 9-th fault Y: res=-1/12
545 10-th fault Y: res=-1/12
546 11-th fault Y: res=-1/12
547 12-th fault Y: res=-1/12
548 13-th fault Y: res=-1/12
549 14-th fault Y: res=-1/12
550 15-th fault Y: res=-1/12
551 16-th fault N: res=0/12