linux-3.3/Documentation/memory-hotplug.txt

2 Memory Hotplug
6 Add description of notifier of memory hotplug	Oct 11 2007
8 This document is about memory hotplug including how-to-use and current status.
9 Because Memory Hotplug is still under development, contents of this text will
13   1.1 purpose of memory hotplug
14   1.2. Phases of memory hotplug
15   1.3. Unit of Memory online/offline operation
17 3. sysfs files for memory hotplug
18 4. Physical memory hot-add phase
20   4.2 Notify memory hot-add event by hand
21 5. Logical Memory hot-add phase
22   5.1. State of memory
23   5.2. How to online memory
24 6. Logical memory remove
25   6.1 Memory offline and ZONE_MOVABLE
26   6.2. How to offline memory
27 7. Physical memory remove
28 8. Memory hotplug event notifier
31 Note(1): x86_64's has special implementation for memory hotplug.
40 1.1 purpose of memory hotplug
42 Memory Hotplug allows users to increase/decrease the amount of memory.
45 (A) For changing the amount of memory.
51 hardware which supports memory power management.
53 Linux memory hotplug is designed for both purpose.
56 1.2. Phases of memory hotplug
58 There are 2 phases in Memory Hotplug.
59   1) Physical Memory Hotplug phase
60   2) Logical Memory Hotplug phase.
63 environment for hotplugged memory. Basically, this phase is necessary
67 When memory is hotplugged, the kernel recognizes new memory, makes new memory
68 management tables, and makes sysfs files for new memory's operation.
70 If firmware supports notification of connection of new memory to OS,
75 Logical Memory Hotplug phase is to change memory state into
76 available/unavailable for users. Amount of memory from user's view is
77 changed by this phase. The kernel makes all memory in it as free pages
78 when a memory range is available.
82 Logical Memory Hotplug phase is triggered by write of sysfs file by system
85 (However, if you writes udev's hotplug scripts for memory hotplug, these
89 1.3. Unit of Memory online/offline operation
91 Memory hotplug uses SPARSEMEM memory model. SPARSEMEM divides the whole memory
98 /sys/devices/system/memory/block_size_bytes
105 To use memory hotplug feature, kernel must be compiled with following
108 - For all memory hotplug
109     Memory model -> Sparse Memory  (CONFIG_SPARSEMEM)
110     Allow for memory hot-add       (CONFIG_MEMORY_HOTPLUG)
112 - To enable memory removal, the followings are also necessary
113     Allow for memory hot remove    (CONFIG_MEMORY_HOTREMOVE)
116 - For ACPI memory hotplug, the followings are also necessary
117     Memory hotplug (under ACPI Support menu) (CONFIG_ACPI_HOTPLUG_MEMORY)
127 4 sysfs files for memory hotplug
130 a memory block under /sys/devices/system/memory as
132 /sys/devices/system/memory/memoryXXX
136 section contained in the memory block.  The files 'phys_index' and
138 for the memory block covered by the sysfs directory.  It is expected that all
139 memory sections in this range are present and no memory holes exist in the
140 range. Currently there is no way to determine if there is a memory hole, but
141 the existence of one should not affect the hotplug capabilities of the memory
144 For example, assume 1GiB section size. A device for a memory starting at
145 0x100000000 is /sys/device/system/memory/memory4
152 /sys/devices/system/memory/memoryXXX/start_phys_index
153 /sys/devices/system/memory/memoryXXX/end_phys_index
154 /sys/devices/system/memory/memoryXXX/phys_device
155 /sys/devices/system/memory/memoryXXX/state
156 /sys/devices/system/memory/memoryXXX/removable
159 		    in the memory block, same as XXX.
161 		    in the memory block.
163                     at read:  contains online/offline state of memory.
166 'phys_device'     : read-only: designed to show the name of physical memory
169                     whether the memory block is removable or not
170                     removable.  A value of 1 indicates that the memory
172                     it is not removable. A memory block is removable only if
176   These directories/files appear after physical memory hotplug phase.
182 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9
185 /sys/devices/system/memory/memory9/node0 -> ../../node/node0
188 4. Physical memory hot-add phase
193 On x86_64/ia64 platform, memory hotplug by ACPI is supported.
195 In general, the firmware (ACPI) which supports memory hotplug defines
196 memory class object of _HID "PNP0C80". When a notify is asserted to PNP0C80,
197 Linux's ACPI handler does hot-add memory to the system and calls a hotplug udev
200 But scripts for memory hotplug are not contained in generic udev package(now).
201 You may have to write it by yourself or online/offline memory by hand.
202 Please see "How to online memory", "How to offline memory" in this text.
207 If memory device is found, memory hotplug code will be called.
210 4.2 Notify memory hot-add event by hand
213 notify memory hotplug event to the kernel. For such environment, "probe"
221 /sys/devices/system/memory/probe
223 You can tell the physical address of new memory to the kernel by
225 % echo start_address_of_new_memory > /sys/devices/system/memory/probe
228 memory range is hot-added. In this case, hotplug script is not called (in
229 current implementation). You'll have to online memory by yourself.
230 Please see "How to online memory" in this text.
235 5. Logical Memory hot-add phase
238 5.1. State of memory
240 To see (online/offline) state of memory section, read 'state' file.
242 % cat /sys/device/system/memory/memoryXXX/state
245 If the memory section is online, you'll read "online".
246 If the memory section is offline, you'll read "offline".
249 5.2. How to online memory
251 Even if the memory is hot-added, it is not at ready-to-use state.
252 For using newly added memory, you have to "online" the memory section.
256 % echo online > /sys/devices/system/memory/memoryXXX/state
259 available memory will be increased.
261 Currently, newly added memory is added as ZONE_NORMAL (for powerpc, ZONE_DMA).
267 6. Logical memory remove
270 6.1 Memory offline and ZONE_MOVABLE
272 Memory offlining is more complicated than memory online. Because memory offline
273 has to make the whole memory section be unused, memory offline can fail if
274 the section includes memory which cannot be freed.
276 In general, memory offline can use 2 techniques.
278 (1) reclaim and free all memory in the section.
281 In the current implementation, Linux's memory offline uses method (2), freeing
292 Assume the system has "TOTAL" amount of memory at boot time, this boot option
296   Size of memory not for movable pages (not for offline) is YYYY.
297   Size of memory for movable pages (for offline) is TOTAL-YYYY.
300   Size of memory not for movable pages (not for offline) is TOTAL - ZZZZ.
301   Size of memory for movable pages (for offline) is ZZZZ.
308 6.2. How to offline memory
311 memory onlining.
313 % echo offline > /sys/devices/system/memory/memoryXXX/state
315 If offline succeeds, the state of the memory section is changed to be "offline".
318 If it doesn't contain 'unmovable' memory, you'll get success.
321 But under some busy state, it may return -EBUSY. Even if a memory section
328 Memory hotplug's design direction is to make the possibility of memory offlining
329 higher and to guarantee unplugging memory under any situation. But it needs
331 can decide to retry more or not by himself. Currently, memory offlining code
335 7. Physical memory remove
342 8. Memory hotplug event notifier
344 Memory hotplug has event notifer. There are 6 types of notification.
347   Generated before new memory becomes available in order to be able to
348   prepare subsystems to handle memory. The page allocator is still unable
349   to allocate from the new memory.
355   Generated when memory has successfully brought online. The callback may
356   allocate pages from the new memory.
359   Generated to begin the process of offlining memory. Allocations are no
360   longer possible from the memory but some of the memory to be offlined
361   is still in use. The callback can be used to free memory known to a
362   subsystem from the indicated memory section.
365   Generated if MEMORY_GOING_OFFLINE fails. Memory is available again from
369   Generated after offlining memory is complete.
383 start_pfn is start_pfn of online/offline memory.
384 nr_pages is # of pages of online/offline memory.
386 set/clear. It means a new(memoryless) node gets new memory by online and a
387 node loses all memory. If this is -1, then nodemask status is not changed.
394   - allowing memory hot-add to ZONE_MOVABLE. maybe we need some switch like
396   - showing memory section and physical device relationship.
397   - showing memory section is under ZONE_MOVABLE or not
398   - test and make it better memory offlining.
400   - memmap removing at memory offline.
401   - physical remove memory.