1.. SPDX-License-Identifier: GPL-2.0 2 3========== 4Netconsole 5========== 6 7 8started by Ingo Molnar <mingo@redhat.com>, 2001.09.17 9 102.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003 11 12IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013 13 14Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015 15 16Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023 17 18Userdata append support by Matthew Wood <thepacketgeek@gmail.com>, Jan 22 2024 19 20Sysdata append support by Breno Leitao <leitao@debian.org>, Jan 15 2025 21 22Please send bug reports to Matt Mackall <mpm@selenic.com> 23Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com> 24 25Introduction: 26============= 27 28This module logs kernel printk messages over UDP allowing debugging of 29problem where disk logging fails and serial consoles are impractical. 30 31It can be used either built-in or as a module. As a built-in, 32netconsole initializes immediately after NIC cards and will bring up 33the specified interface as soon as possible. While this doesn't allow 34capture of early kernel panics, it does capture most of the boot 35process. 36 37Sender and receiver configuration: 38================================== 39 40It takes a string configuration parameter "netconsole" in the 41following format:: 42 43 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr] 44 45 where 46 + if present, enable extended console support 47 r if present, prepend kernel version (release) to the message 48 src-port source for UDP packets (defaults to 6665) 49 src-ip source IP to use (interface address) 50 dev network interface (eth0) 51 tgt-port port for logging agent (6666) 52 tgt-ip IP address for logging agent 53 tgt-macaddr ethernet MAC address for logging agent (broadcast) 54 55Examples:: 56 57 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 58 59or:: 60 61 insmod netconsole netconsole=@/,@10.0.0.2/ 62 63or using IPv6:: 64 65 insmod netconsole netconsole=@/,@fd00:1:2:3::1/ 66 67It also supports logging to multiple remote agents by specifying 68parameters for the multiple agents separated by semicolons and the 69complete string enclosed in "quotes", thusly:: 70 71 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/" 72 73Built-in netconsole starts immediately after the TCP stack is 74initialized and attempts to bring up the supplied dev at the supplied 75address. 76 77The remote host has several options to receive the kernel messages, 78for example: 79 801) syslogd 81 822) netcat 83 84 On distributions using a BSD-based netcat version (e.g. Fedora, 85 openSUSE and Ubuntu) the listening port must be specified without 86 the -p switch:: 87 88 nc -u -l -p <port>' / 'nc -u -l <port> 89 90 or:: 91 92 netcat -u -l -p <port>' / 'netcat -u -l <port> 93 943) socat 95 96:: 97 98 socat udp-recv:<port> - 99 100Dynamic reconfiguration: 101======================== 102 103Dynamic reconfigurability is a useful addition to netconsole that enables 104remote logging targets to be dynamically added, removed, or have their 105parameters reconfigured at runtime from a configfs-based userspace interface. 106 107To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the 108netconsole module (or kernel, if netconsole is built-in). 109 110Some examples follow (where configfs is mounted at the /sys/kernel/config 111mountpoint). 112 113To add a remote logging target (target names can be arbitrary):: 114 115 cd /sys/kernel/config/netconsole/ 116 mkdir target1 117 118Note that newly created targets have default parameter values (as mentioned 119above) and are disabled by default -- they must first be enabled by writing 120"1" to the "enabled" attribute (usually after setting parameters accordingly) 121as described below. 122 123To remove a target:: 124 125 rmdir /sys/kernel/config/netconsole/othertarget/ 126 127The interface exposes these parameters of a netconsole target to userspace: 128 129 =============== ================================= ============ 130 enabled Is this target currently enabled? (read-write) 131 extended Extended mode enabled (read-write) 132 release Prepend kernel release to message (read-write) 133 dev_name Local network interface name (read-write) 134 local_port Source UDP port to use (read-write) 135 remote_port Remote agent's UDP port (read-write) 136 local_ip Source IP address to use (read-write) 137 remote_ip Remote agent's IP address (read-write) 138 local_mac Local interface's MAC address (read-only) 139 remote_mac Remote agent's MAC address (read-write) 140 transmit_errors Number of packet send errors (read-only) 141 =============== ================================= ============ 142 143The "enabled" attribute is also used to control whether the parameters of 144a target can be updated or not -- you can modify the parameters of only 145disabled targets (i.e. if "enabled" is 0). 146 147To update a target's parameters:: 148 149 cat enabled # check if enabled is 1 150 echo 0 > enabled # disable the target (if required) 151 echo eth2 > dev_name # set local interface 152 echo 10.0.0.4 > remote_ip # update some parameter 153 echo cb:a9:87:65:43:21 > remote_mac # update more parameters 154 echo 1 > enabled # enable target again 155 156You can also update the local interface dynamically. This is especially 157useful if you want to use interfaces that have newly come up (and may not 158have existed when netconsole was loaded / initialized). 159 160Netconsole targets defined at boot time (or module load time) with the 161`netconsole=` param are assigned the name `cmdline<index>`. For example, the 162first target in the parameter is named `cmdline0`. You can control and modify 163these targets by creating configfs directories with the matching name. 164 165Let's suppose you have two netconsole targets defined at boot time:: 166 167 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc 168 169You can modify these targets in runtime by creating the following targets:: 170 171 mkdir cmdline0 172 cat cmdline0/remote_ip 173 10.0.0.2 174 175 mkdir cmdline1 176 cat cmdline1/remote_ip 177 10.0.0.3 178 179Append User Data 180---------------- 181 182Custom user data can be appended to the end of messages with netconsole 183dynamic configuration enabled. User data entries can be modified without 184changing the "enabled" attribute of a target. 185 186Directories (keys) under `userdata` are limited to 53 character length, and 187data in `userdata/<key>/value` are limited to 200 bytes:: 188 189 cd /sys/kernel/config/netconsole && mkdir cmdline0 190 cd cmdline0 191 mkdir userdata/foo 192 echo bar > userdata/foo/value 193 mkdir userdata/qux 194 echo baz > userdata/qux/value 195 196Messages will now include this additional user data:: 197 198 echo "This is a message" > /dev/kmsg 199 200Sends:: 201 202 12,607,22085407756,-;This is a message 203 foo=bar 204 qux=baz 205 206Preview the userdata that will be appended with:: 207 208 cd /sys/kernel/config/netconsole/cmdline0/userdata 209 for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done 210 211If a `userdata` entry is created but no data is written to the `value` file, 212the entry will be omitted from netconsole messages:: 213 214 cd /sys/kernel/config/netconsole && mkdir cmdline0 215 cd cmdline0 216 mkdir userdata/foo 217 echo bar > userdata/foo/value 218 mkdir userdata/qux 219 220The `qux` key is omitted since it has no value:: 221 222 echo "This is a message" > /dev/kmsg 223 12,607,22085407756,-;This is a message 224 foo=bar 225 226Delete `userdata` entries with `rmdir`:: 227 228 rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux 229 230.. warning:: 231 When writing strings to user data values, input is broken up per line in 232 configfs store calls and this can cause confusing behavior:: 233 234 mkdir userdata/testing 235 printf "val1\nval2" > userdata/testing/value 236 # userdata store value is called twice, first with "val1\n" then "val2" 237 # so "val2" is stored, being the last value stored 238 cat userdata/testing/value 239 val2 240 241 It is recommended to not write user data values with newlines. 242 243CPU number auto population in userdata 244-------------------------------------- 245 246Inside the netconsole configfs hierarchy, there is a file called 247`cpu_nr` under the `userdata` directory. This file is used to enable or disable 248the automatic CPU number population feature. This feature automatically 249populates the CPU number that is sending the message. 250 251To enable the CPU number auto-population:: 252 253 echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr 254 255When this option is enabled, the netconsole messages will include an additional 256line in the userdata field with the format `cpu=<cpu_number>`. This allows the 257receiver of the netconsole messages to easily differentiate and demultiplex 258messages originating from different CPUs, which is particularly useful when 259dealing with parallel log output. 260 261Example:: 262 263 echo "This is a message" > /dev/kmsg 264 12,607,22085407756,-;This is a message 265 cpu=42 266 267In this example, the message was sent by CPU 42. 268 269.. note:: 270 271 If the user has set a conflicting `cpu` key in the userdata dictionary, 272 both keys will be reported, with the kernel-populated entry appearing after 273 the user one. For example:: 274 275 # User-defined CPU entry 276 mkdir -p /sys/kernel/config/netconsole/target1/userdata/cpu 277 echo "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value 278 279 Output might look like:: 280 281 12,607,22085407756,-;This is a message 282 cpu=1 283 cpu=42 # kernel-populated value 284 285 286Extended console: 287================= 288 289If '+' is prefixed to the configuration line or "extended" config file 290is set to 1, extended console support is enabled. An example boot 291param follows:: 292 293 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 294 295Log messages are transmitted with extended metadata header in the 296following format which is the same as /dev/kmsg:: 297 298 <level>,<sequnum>,<timestamp>,<contflag>;<message text> 299 300If 'r' (release) feature is enabled, the kernel release version is 301prepended to the start of the message. Example:: 302 303 6.4.0,6,444,501151268,-;netconsole: network logging started 304 305Non printable characters in <message text> are escaped using "\xff" 306notation. If the message contains optional dictionary, verbatim 307newline is used as the delimiter. 308 309If a message doesn't fit in certain number of bytes (currently 1000), 310the message is split into multiple fragments by netconsole. These 311fragments are transmitted with "ncfrag" header field added:: 312 313 ncfrag=<byte-offset>/<total-bytes> 314 315For example, assuming a lot smaller chunk size, a message "the first 316chunk, the 2nd chunk." may be split as follows:: 317 318 6,416,1758426,-,ncfrag=0/31;the first chunk, 319 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. 320 321Miscellaneous notes: 322==================== 323 324.. Warning:: 325 326 the default target ethernet setting uses the broadcast 327 ethernet address to send packets, which can cause increased load on 328 other systems on the same ethernet segment. 329 330.. Tip:: 331 332 some LAN switches may be configured to suppress ethernet broadcasts 333 so it is advised to explicitly specify the remote agents' MAC addresses 334 from the config parameters passed to netconsole. 335 336.. Tip:: 337 338 to find out the MAC address of, say, 10.0.0.2, you may try using:: 339 340 ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2 341 342.. Tip:: 343 344 in case the remote logging agent is on a separate LAN subnet than 345 the sender, it is suggested to try specifying the MAC address of the 346 default gateway (you may use /sbin/route -n to find it out) as the 347 remote MAC address instead. 348 349.. note:: 350 351 the network device (eth1 in the above case) can run any kind 352 of other network traffic, netconsole is not intrusive. Netconsole 353 might cause slight delays in other traffic if the volume of kernel 354 messages is high, but should have no other impact. 355 356.. note:: 357 358 if you find that the remote logging agent is not receiving or 359 printing all messages from the sender, it is likely that you have set 360 the "console_loglevel" parameter (on the sender) to only send high 361 priority messages to the console. You can change this at runtime using:: 362 363 dmesg -n 8 364 365 or by specifying "debug" on the kernel command line at boot, to send 366 all kernel messages to the console. A specific value for this parameter 367 can also be set using the "loglevel" kernel boot option. See the 368 dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst 369 for details. 370 371Netconsole was designed to be as instantaneous as possible, to 372enable the logging of even the most critical kernel bugs. It works 373from IRQ contexts as well, and does not enable interrupts while 374sending packets. Due to these unique needs, configuration cannot 375be more automatic, and some fundamental limitations will remain: 376only IP networks, UDP packets and ethernet devices are supported. 377