1Real-time application monitors 2============================== 3 4- Name: rtapp 5- Type: container for multiple monitors 6- Author: Nam Cao <namcao@linutronix.de> 7 8Description 9----------- 10 11Real-time applications may have design flaws such that they experience 12unexpected latency and fail to meet their time requirements. Often, these flaws 13follow a few patterns: 14 15 - Page faults: A real-time thread may access memory that does not have a 16 mapped physical backing or must first be copied (such as for copy-on-write). 17 Thus a page fault is raised and the kernel must first perform the expensive 18 action. This causes significant delays to the real-time thread 19 - Priority inversion: A real-time thread blocks waiting for a lower-priority 20 thread. This causes the real-time thread to effectively take on the 21 scheduling priority of the lower-priority thread. For example, the real-time 22 thread needs to access a shared resource that is protected by a 23 non-pi-mutex, but the mutex is currently owned by a non-real-time thread. 24 25The `rtapp` monitor detects these patterns. It aids developers to identify 26reasons for unexpected latency with real-time applications. It is a container of 27multiple sub-monitors described in the following sections. 28 29Monitor pagefault 30+++++++++++++++++ 31 32The `pagefault` monitor reports real-time tasks raising page faults. Its 33specification is:: 34 35 RULE = always (RT imply not PAGEFAULT) 36 37To fix warnings reported by this monitor, `mlockall()` or `mlock()` can be used 38to ensure physical backing for memory. 39 40This monitor may have false negatives because the pages used by the real-time 41threads may just happen to be directly available during testing. To minimize 42this, the system can be put under memory pressure (e.g. invoking the OOM killer 43using a program that does `ptr = malloc(SIZE_OF_RAM); memset(ptr, 0, 44SIZE_OF_RAM);`) so that the kernel executes aggressive strategies to recycle as 45much physical memory as possible. 46 47Monitor sleep 48+++++++++++++ 49 50The `sleep` monitor reports real-time threads sleeping in a manner that may 51cause undesirable latency. Real-time applications should only put a real-time 52thread to sleep for one of the following reasons: 53 54 - Cyclic work: real-time thread sleeps waiting for the next cycle. For this 55 case, only the `clock_nanosleep` syscall should be used with `TIMER_ABSTIME` 56 (to avoid time drift) and `CLOCK_MONOTONIC` (to avoid the clock being 57 changed). No other method is safe for real-time. For example, threads 58 waiting for timerfd can be woken by softirq which provides no real-time 59 guarantee. 60 - Real-time thread waiting for something to happen (e.g. another thread 61 releasing shared resources, or a completion signal from another thread). In 62 this case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one of 63 FUTEX_WAIT_*) should be used. Applications usually do not use futexes 64 directly, but use PI mutexes and PI condition variables which are built on 65 top of futexes. Be aware that the C library might not implement conditional 66 variables as safe for real-time. As an alternative, the librtpi library 67 exists to provide a conditional variable implementation that is correct for 68 real-time applications in Linux. 69 70Beside the reason for sleeping, the eventual waker should also be 71real-time-safe. Namely, one of: 72 73 - An equal-or-higher-priority thread 74 - Hard interrupt handler 75 - Non-maskable interrupt handler 76 77This monitor's warning usually means one of the following: 78 79 - Real-time thread is blocked by a non-real-time thread (e.g. due to 80 contention on a mutex without priority inheritance). This is priority 81 inversion. 82 - Time-critical work waits for something which is not safe for real-time (e.g. 83 timerfd). 84 - The work executed by the real-time thread does not need to run at real-time 85 priority at all. This is not a problem for the real-time thread itself, but 86 it is potentially taking the CPU away from other important real-time work. 87 88Application developers may purposely choose to have their real-time application 89sleep in a way that is not safe for real-time. It is debatable whether that is a 90problem. Application developers must analyze the warnings to make a proper 91assessment. 92 93The monitor's specification is:: 94 95 RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST)) 96 97 RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD) 98 and ((not WAKE) until RT_FRIENDLY_WAKE) 99 100 RT_VALID_SLEEP_REASON = FUTEX_WAIT 101 or RT_FRIENDLY_NANOSLEEP 102 103 RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP 104 and NANOSLEEP_TIMER_ABSTIME 105 and NANOSLEEP_CLOCK_MONOTONIC 106 107 RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO 108 or WOKEN_BY_HARDIRQ 109 or WOKEN_BY_NMI 110 or KTHREAD_SHOULD_STOP 111 112 ALLOWLIST = BLOCK_ON_RT_MUTEX 113 or FUTEX_LOCK_PI 114 or TASK_IS_RCU 115 or TASK_IS_MIGRATION 116 117Beside the scenarios described above, this specification also handle some 118special cases: 119 120 - `KERNEL_THREAD`: kernel tasks do not have any pattern that can be recognized 121 as valid real-time sleeping reasons. Therefore sleeping reason is not 122 checked for kernel tasks. 123 - `KTHREAD_SHOULD_STOP`: a non-real-time thread may stop a real-time kernel 124 thread by waking it and waiting for it to exit (`kthread_stop()`). This 125 wakeup is safe for real-time. 126 - `ALLOWLIST`: to handle known false positives with the kernel. 127 - `BLOCK_ON_RT_MUTEX` is included in the allowlist due to its implementation. 128 In the release path of rt_mutex, a boosted task is de-boosted before waking 129 the rt_mutex's waiter. Consequently, the monitor may see a real-time-unsafe 130 wakeup (e.g. non-real-time task waking real-time task). This is actually 131 real-time-safe because preemption is disabled for the duration. 132 - `FUTEX_LOCK_PI` is included in the allowlist for the same reason as 133 `BLOCK_ON_RT_MUTEX`. 134