1====================== 2Firmware-Assisted Dump 3====================== 4 5July 2011 6 7The goal of firmware-assisted dump is to enable the dump of 8a crashed system, and to do so from a fully-reset system, and 9to minimize the total elapsed time until the system is back 10in production use. 11 12- Firmware-Assisted Dump (FADump) infrastructure is intended to replace 13 the existing phyp assisted dump. 14- Fadump uses the same firmware interfaces and memory reservation model 15 as phyp assisted dump. 16- Unlike phyp dump, FADump exports the memory dump through /proc/vmcore 17 in the ELF format in the same way as kdump. This helps us reuse the 18 kdump infrastructure for dump capture and filtering. 19- Unlike phyp dump, userspace tool does not need to refer any sysfs 20 interface while reading /proc/vmcore. 21- Unlike phyp dump, FADump allows user to release all the memory reserved 22 for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem. 23- Once enabled through kernel boot parameter, FADump can be 24 started/stopped through /sys/kernel/fadump_registered interface (see 25 sysfs files section below) and can be easily integrated with kdump 26 service start/stop init scripts. 27 28Comparing with kdump or other strategies, firmware-assisted 29dump offers several strong, practical advantages: 30 31- Unlike kdump, the system has been reset, and loaded 32 with a fresh copy of the kernel. In particular, 33 PCI and I/O devices have been reinitialized and are 34 in a clean, consistent state. 35- Once the dump is copied out, the memory that held the dump 36 is immediately available to the running kernel. And therefore, 37 unlike kdump, FADump doesn't need a 2nd reboot to get back 38 the system to the production configuration. 39 40The above can only be accomplished by coordination with, 41and assistance from the Power firmware. The procedure is 42as follows: 43 44- The first kernel registers the sections of memory with the 45 Power firmware for dump preservation during OS initialization. 46 These registered sections of memory are reserved by the first 47 kernel during early boot. 48 49- When system crashes, the Power firmware will copy the registered 50 low memory regions (boot memory) from source to destination area. 51 It will also save hardware PTE's. 52 53 NOTE: 54 The term 'boot memory' means size of the low memory chunk 55 that is required for a kernel to boot successfully when 56 booted with restricted memory. By default, the boot memory 57 size will be the larger of 5% of system RAM or 256MB. 58 Alternatively, user can also specify boot memory size 59 through boot parameter 'crashkernel=' which will override 60 the default calculated size. Use this option if default 61 boot memory size is not sufficient for second kernel to 62 boot successfully. For syntax of crashkernel= parameter, 63 refer to Documentation/admin-guide/kdump/kdump.rst. If any 64 offset is provided in crashkernel= parameter, it will be 65 ignored as FADump uses a predefined offset to reserve memory 66 for boot memory dump preservation in case of a crash. 67 68- After the low memory (boot memory) area has been saved, the 69 firmware will reset PCI and other hardware state. It will 70 *not* clear the RAM. It will then launch the bootloader, as 71 normal. 72 73- The freshly booted kernel will notice that there is a new node 74 (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot 75 on OPAL platform) in the device tree, indicating that 76 there is crash data available from a previous boot. During 77 the early boot OS will reserve rest of the memory above 78 boot memory size effectively booting with restricted memory 79 size. This will make sure that this kernel (also, referred 80 to as second kernel or capture kernel) will not touch any 81 of the dump memory area. 82 83- User-space tools will read /proc/vmcore to obtain the contents 84 of memory, which holds the previous crashed kernel dump in ELF 85 format. The userspace tools may copy this info to disk, or 86 network, nas, san, iscsi, etc. as desired. 87 88- Once the userspace tool is done saving dump, it will echo 89 '1' to /sys/kernel/fadump_release_mem to release the reserved 90 memory back to general use, except the memory required for 91 next firmware-assisted dump registration. 92 93 e.g.:: 94 95 # echo 1 > /sys/kernel/fadump_release_mem 96 97Please note that the firmware-assisted dump feature 98is only available on POWER6 and above systems on pSeries 99(PowerVM) platform and POWER9 and above systems with OP940 100or later firmware versions on PowerNV (OPAL) platform. 101Note that, OPAL firmware exports ibm,opal/dump node when 102FADump is supported on PowerNV platform. 103 104On OPAL based machines, system first boots into an intermittent 105kernel (referred to as petitboot kernel) before booting into the 106capture kernel. This kernel would have minimal kernel and/or 107userspace support to process crash data. Such kernel needs to 108preserve previously crash'ed kernel's memory for the subsequent 109capture kernel boot to process this crash data. Kernel config 110option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel 111to ensure that crash data is preserved to process later. 112 113-- On OPAL based machines (PowerNV), if the kernel is build with 114 CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also 115 exported as /sys/firmware/opal/mpipl/core file. This procfs file is 116 helpful in debugging OPAL crashes with GDB. The kernel memory 117 used for exporting this procfs file can be released by echo'ing 118 '1' to /sys/firmware/opal/mpipl/release_core node. 119 120 e.g. 121 # echo 1 > /sys/firmware/opal/mpipl/release_core 122 123-- Support for Additional Kernel Arguments in Fadump 124 Fadump has a feature that allows passing additional kernel arguments 125 to the fadump kernel. This feature was primarily designed to disable 126 kernel functionalities that are not required for the fadump kernel 127 and to reduce its memory footprint while collecting the dump. 128 129 Command to Add Additional Kernel Parameters to Fadump: 130 e.g. 131 # echo "nr_cpus=16" > /sys/kernel/fadump/bootargs_append 132 133 The above command is sufficient to add additional arguments to fadump. 134 An explicit service restart is not required. 135 136 Command to Retrieve the Additional Fadump Arguments: 137 e.g. 138 # cat /sys/kernel/fadump/bootargs_append 139 140Note: Additional kernel arguments for fadump with HASH MMU is only 141 supported if the RMA size is greater than 768 MB. If the RMA 142 size is less than 768 MB, the kernel does not export the 143 /sys/kernel/fadump/bootargs_append sysfs node. 144 145Implementation details: 146----------------------- 147 148During boot, a check is made to see if firmware supports 149this feature on that particular machine. If it does, then 150we check to see if an active dump is waiting for us. If yes 151then everything but boot memory size of RAM is reserved during 152early boot (See Fig. 2). This area is released once we finish 153collecting the dump from user land scripts (e.g. kdump scripts) 154that are run. If there is dump data, then the 155/sys/kernel/fadump_release_mem file is created, and the reserved 156memory is held. 157 158If there is no waiting dump data, then only the memory required to 159hold CPU state, HPTE region, boot memory dump, and FADump header is 160usually reserved at an offset greater than boot memory size (see Fig. 1). 161This area is *not* released: this region will be kept permanently 162reserved, so that it can act as a receptacle for a copy of the boot 163memory content in addition to CPU state and HPTE region, in the case 164a crash does occur. 165 166Since this reserved memory area is used only after the system crash, 167there is no point in blocking this significant chunk of memory from 168production kernel. Hence, the implementation uses the Linux kernel's 169Contiguous Memory Allocator (CMA) for memory reservation if CMA is 170configured for kernel. With CMA reservation this memory will be 171available for applications to use it, while kernel is prevented from 172using it. With this FADump will still be able to capture all of the 173kernel memory and most of the user space memory except the user pages 174that were present in CMA region:: 175 176 o Memory Reservation during first kernel 177 178 Low memory Top of memory 179 0 boot memory size |<------ Reserved dump area ----->| | 180 | | | Permanent Reservation | | 181 V V | | V 182 +-----------+-----/ /---+---+----+-----------+-------+----+-----+ 183 | | |///|////| DUMP | HDR |////| | 184 +-----------+-----/ /---+---+----+-----------+-------+----+-----+ 185 | ^ ^ ^ ^ ^ 186 | | | | | | 187 \ CPU HPTE / | | 188 -------------------------------- | | 189 Boot memory content gets transferred | | 190 to reserved area by firmware at the | | 191 time of crash. | | 192 FADump Header | 193 (meta area) | 194 | 195 | 196 Metadata: This area holds a metadata structure whose 197 address is registered with f/w and retrieved in the 198 second kernel after crash, on platforms that support 199 tags (OPAL). Having such structure with info needed 200 to process the crashdump eases dump capture process. 201 202 Fig. 1 203 204 205 o Memory Reservation during second kernel after crash 206 207 Low memory Top of memory 208 0 boot memory size | 209 | |<------------ Crash preserved area ------------>| 210 V V |<--- Reserved dump area --->| | 211 +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+ 212 | |ELF| | |///|////| DUMP | HDR |/////| | 213 +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+ 214 | | | | | | 215 ----- ------------------------------ --------------- 216 \ | | 217 \ | | 218 \ | | 219 \ | ---------------------------- 220 \ | / 221 \ | / 222 \ | / 223 /proc/vmcore 224 225 226 +---+ 227 |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above 228 +---+ figures are not always present. For example, OPAL platform 229 does not have CPU & HPTE regions while Metadata region is 230 not supported on pSeries currently. 231 232 +---+ 233 |ELF| -> elfcorehdr, it is created in second kernel after crash. 234 +---+ 235 236 Note: Memory from 0 to the boot memory size is used by second kernel 237 238 Fig. 2 239 240 241Currently the dump will be copied from /proc/vmcore to a new file upon 242user intervention. The dump data available through /proc/vmcore will be 243in ELF format. Hence the existing kdump infrastructure (kdump scripts) 244to save the dump works fine with minor modifications. KDump scripts on 245major Distro releases have already been modified to work seamlessly (no 246user intervention in saving the dump) when FADump is used, instead of 247KDump, as dump mechanism. 248 249The tools to examine the dump will be same as the ones 250used for kdump. 251 252How to enable firmware-assisted dump (FADump): 253---------------------------------------------- 254 2551. Set config option CONFIG_FA_DUMP=y and build kernel. 2562. Boot into linux kernel with 'fadump=on' kernel cmdline option. 257 By default, FADump reserved memory will be initialized as CMA area. 258 Alternatively, user can boot linux kernel with 'fadump=nocma' to 259 prevent FADump to use CMA. 2603. Optionally, user can also set 'crashkernel=' kernel cmdline 261 to specify size of the memory to reserve for boot memory dump 262 preservation. 263 264NOTE: 265 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead 266 use 'crashkernel=' to specify size of the memory to reserve 267 for boot memory dump preservation. 268 2. If firmware-assisted dump fails to reserve memory then it 269 will fallback to existing kdump mechanism if 'crashkernel=' 270 option is set at kernel cmdline. 271 3. if user wants to capture all of user space memory and ok with 272 reserved memory not available to production system, then 273 'fadump=nocma' kernel parameter can be used to fallback to 274 old behaviour. 275 276Sysfs/debugfs files: 277-------------------- 278 279Firmware-assisted dump feature uses sysfs file system to hold 280the control files and debugfs file to display memory reserved region. 281 282Here is the list of files under kernel sysfs: 283 284 /sys/kernel/fadump_enabled 285 This is used to display the FADump status. 286 287 - 0 = FADump is disabled 288 - 1 = FADump is enabled 289 290 This interface can be used by kdump init scripts to identify if 291 FADump is enabled in the kernel and act accordingly. 292 293 /sys/kernel/fadump_registered 294 This is used to display the FADump registration status as well 295 as to control (start/stop) the FADump registration. 296 297 - 0 = FADump is not registered. 298 - 1 = FADump is registered and ready to handle system crash. 299 300 To register FADump echo 1 > /sys/kernel/fadump_registered and 301 echo 0 > /sys/kernel/fadump_registered for un-register and stop the 302 FADump. Once the FADump is un-registered, the system crash will not 303 be handled and vmcore will not be captured. This interface can be 304 easily integrated with kdump service start/stop. 305 306 /sys/kernel/fadump/mem_reserved 307 308 This is used to display the memory reserved by FADump for saving the 309 crash dump. 310 311 /sys/kernel/fadump_release_mem 312 This file is available only when FADump is active during 313 second kernel. This is used to release the reserved memory 314 region that are held for saving crash dump. To release the 315 reserved memory echo 1 to it:: 316 317 echo 1 > /sys/kernel/fadump_release_mem 318 319 After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region 320 file will change to reflect the new memory reservations. 321 322 The existing userspace tools (kdump infrastructure) can be easily 323 enhanced to use this interface to release the memory reserved for 324 dump and continue without 2nd reboot. 325 326Note: /sys/kernel/fadump_release_opalcore sysfs has moved to 327 /sys/firmware/opal/mpipl/release_core 328 329 /sys/firmware/opal/mpipl/release_core 330 331 This file is available only on OPAL based machines when FADump is 332 active during capture kernel. This is used to release the memory 333 used by the kernel to export /sys/firmware/opal/mpipl/core file. To 334 release this memory, echo '1' to it: 335 336 echo 1 > /sys/firmware/opal/mpipl/release_core 337 338Note: The following FADump sysfs files are deprecated. 339 340+----------------------------------+--------------------------------+ 341| Deprecated | Alternative | 342+----------------------------------+--------------------------------+ 343| /sys/kernel/fadump_enabled | /sys/kernel/fadump/enabled | 344+----------------------------------+--------------------------------+ 345| /sys/kernel/fadump_registered | /sys/kernel/fadump/registered | 346+----------------------------------+--------------------------------+ 347| /sys/kernel/fadump_release_mem | /sys/kernel/fadump/release_mem | 348+----------------------------------+--------------------------------+ 349 350Here is the list of files under powerpc debugfs: 351(Assuming debugfs is mounted on /sys/kernel/debug directory.) 352 353 /sys/kernel/debug/powerpc/fadump_region 354 This file shows the reserved memory regions if FADump is 355 enabled otherwise this file is empty. The output format 356 is:: 357 358 <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size> 359 360 and for kernel DUMP region is: 361 362 DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes 363 364 e.g. 365 Contents when FADump is registered during first kernel:: 366 367 # cat /sys/kernel/debug/powerpc/fadump_region 368 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0 369 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0 370 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0 371 372 Contents when FADump is active during second kernel:: 373 374 # cat /sys/kernel/debug/powerpc/fadump_region 375 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020 376 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000 377 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000 378 : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000 379 380 381NOTE: 382 Please refer to Documentation/filesystems/debugfs.rst on 383 how to mount the debugfs filesystem. 384 385 386TODO: 387----- 388 - Need to come up with the better approach to find out more 389 accurate boot memory size that is required for a kernel to 390 boot successfully when booted with restricted memory. 391 392Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> 393 394This document is based on the original documentation written for phyp 395 396assisted dump by Linas Vepstas and Manish Ahuja. 397