Lines Matching +full:memory +full:- +full:controller
1 Memory Resource Controller
3 NOTE: The Memory Resource Controller has generically been referred to as the
4 memory controller in this document. Do not confuse memory controller
5 used here with the memory controller that is used in hardware.
9 When we mention a cgroup (cgroupfs's directory) with memory controller,
10 we call it "memory cgroup". When you see git-log and source code, you'll
14 Benefits and Purpose of the memory controller
16 The memory controller isolates the memory behaviour of a group of tasks
18 uses of the memory controller. The memory controller can be used to
21 Memory hungry applications can be isolated and limited to a smaller
22 amount of memory.
23 b. Create a cgroup with limited amount of memory, this can be used
25 c. Virtualization solutions can control the amount of memory they want
27 d. A CD/DVD burner could control the amount of memory used by the
29 of available memory.
30 e. There are several other use cases, find one or use the controller just
33 Current Status: linux-2.6.34-mmotm(development version of 2010/April)
36 - accounting anonymous pages, file caches, swap caches usage and limiting them.
37 - private LRU and reclaim routine. (system's global LRU and private LRU
39 - optionally, memory+swap usage can be accounted and limited.
40 - hierarchical accounting
41 - soft limit
42 - moving(recharging) account at moving a task is selectable.
43 - usage threshold notifier
44 - oom-killer disable knob and oom-notifier
45 - Root cgroup has no limit controls.
47 Kernel memory support is work in progress, and the current version provides
55 memory.usage_in_bytes # show current res_counter usage for memory
57 memory.memsw.usage_in_bytes # show current res_counter usage for memory+Swap
59 memory.limit_in_bytes # set/show limit of memory usage
60 memory.memsw.limit_in_bytes # set/show limit of memory+Swap usage
61 memory.failcnt # show the number of memory usage hits limits
62 memory.memsw.failcnt # show the number of memory+Swap hits limits
63 memory.max_usage_in_bytes # show max memory usage recorded
64 memory.memsw.max_usage_in_bytes # show max memory+Swap usage recorded
65 memory.soft_limit_in_bytes # set/show soft limit of memory usage
66 memory.stat # show various statistics
67 memory.use_hierarchy # set/show hierarchical account enabled
68 memory.force_empty # trigger forced move charge to parent
69 memory.swappiness # set/show swappiness parameter of vmscan
71 memory.move_charge_at_immigrate # set/show controls of moving charges
72 memory.oom_control # set/show oom controls.
73 memory.numa_stat # show the number of memory usage per numa node
75 memory.kmem.tcp.limit_in_bytes # set/show hard limit for tcp buf memory
76 memory.kmem.tcp.usage_in_bytes # show current tcp buf memory allocation
80 The memory controller has a long history. A request for comments for the memory
81 controller was posted by Balbir Singh [1]. At the time the RFC was posted
82 there were several implementations for memory control. The goal of the
84 for memory control. The first RSS controller was posted by Balbir Singh[2]
86 RSS controller. At OLS, at the resource management BoF, everyone suggested
88 to allow user space handling of OOM. The current memory controller is
92 2. Memory Control
94 Memory is a unique resource in the sense that it is present in a limited
97 memory, the same physical memory needs to be reused to accomplish the task.
99 The memory controller implementation has been divided into phases. These
102 1. Memory controller
103 2. mlock(2) controller
104 3. Kernel user memory accounting and slab control
105 4. user mappings length controller
107 The memory controller is the first controller developed.
112 tracks the current memory usage and limit of the group of processes associated
113 with the controller. Each cgroup has a memory controller specific data
118 +--------------------+
121 +--------------------+
124 +---------------+ | +---------------+
127 +---------------+ | +---------------+
129 + --------------+
131 +---------------+ +------+--------+
132 | page +----------> page_cgroup|
134 +---------------+ +---------------+
139 Figure 1 shows the important aspects of the controller
150 If everything goes well, a page meta-data-structure called page_cgroup is
152 (*) page_cgroup structure is allocated at boot/memory-hotplug time.
162 inserted into inode (radix-tree). While it's mapped into the page tables of
166 unaccounted when it's removed from radix-tree. Even if RSS pages are fully
169 A swapped-in page is not accounted until it's mapped.
171 Note: The kernel does swapin-readahead and read multiple swaps at once.
172 This means swapped-in pages may contain pages for other tasks than a task
173 causing page fault. So, we avoid accounting at swap-in I/O.
177 Note: we just account pages-on-LRU because our purpose is to control amount
178 of used pages; not-on-LRU pages tend to be out-of-control from VM view.
186 the cgroup that brought it in -- this will happen on memory pressure).
189 When you do swapoff and make swapped-out pages of shmem(tmpfs) to
190 be backed into memory in force, charges for pages are accounted against the
196 Swap Extension allows you to record charge for swap. A swapped-in page is
200 - memory.memsw.usage_in_bytes.
201 - memory.memsw.limit_in_bytes.
203 memsw means memory+swap. Usage of memory+swap is limited by
206 Example: Assume a system with 4G of swap. A task which allocates 6G of memory
207 (by mistake) under 2G memory limitation will use all swap.
212 * why 'memory+swap' rather than swap.
213 The global LRU(kswapd) can swap out arbitrary pages. Swap-out means
214 to move account from memory to swap...there is no change in usage of
215 memory+swap. In other words, when we want to limit the usage of swap without
216 affecting global LRU, memory+swap limit is better than just limiting swap from
219 * What happens when a cgroup hits memory.memsw.limit_in_bytes
220 When a cgroup hits memory.memsw.limit_in_bytes, it's useless to do swap-out
221 in this cgroup. Then, swap-out will not be done by cgroup routine and file
222 caches are dropped. But as mentioned above, global LRU can do swapout memory
223 from it for sanity of the system's memory management state. You can't forbid
230 to reclaim memory from the cgroup so as to make space for the new
250 mapping->tree_lock.
254 mm->page_table_lock
255 zone->lru_lock
258 per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by
259 zone->lru_lock, it has no lock of its own.
261 2.7 Kernel Memory Extension (CONFIG_CGROUP_MEM_RES_CTLR_KMEM)
263 With the Kernel memory extension, the Memory Controller is able to limit
264 the amount of kernel memory used by the system. Kernel memory is fundamentally
265 different than user memory, since it can't be swapped out, which makes it
268 Kernel memory limits are not imposed for the root cgroup. Usage for the root
271 Currently no soft limit is implemented for kernel memory. It is future work
274 2.7.1 Current Kernel Memory resources accounted
276 * sockets memory pressure: some sockets protocols have memory pressure
277 thresholds. The Memory Controller allows them to be controlled individually
280 * tcp memory pressure: sockets memory pressure for the tcp protocol.
292 # mount -t tmpfs none /sys/fs/cgroup
293 # mkdir /sys/fs/cgroup/memory
294 # mount -t cgroup none /sys/fs/cgroup/memory -o memory
297 # mkdir /sys/fs/cgroup/memory/0
298 # echo $$ > /sys/fs/cgroup/memory/0/tasks
300 Since now we're in the 0 cgroup, we can alter the memory limit:
301 # echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytes
306 NOTE: We can write "-1" to reset the *.limit_in_bytes(unlimited).
309 # cat /sys/fs/cgroup/memory/0/memory.limit_in_bytes
313 # cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes
319 availability of memory on the system. The user is required to re-read
322 # echo 1 > memory.limit_in_bytes
323 # cat memory.limit_in_bytes
326 The memory.failcnt field gives the number of times that the cgroup limit was
329 The memory.stat file gives accounting information. Now, the number of
336 Performance test is also important. To see pure memory controller's overhead,
340 Page-fault scalability is also important. At measuring parallel
341 page fault test, multi-process test may be better than multi-thread
345 Trying usual test under memory controller is always helpful.
353 2. The user is using anonymous memory and swap is turned off or too low
380 rmdir() may return -EBUSY if freeing/moving fails. See 5.1 also.
390 memory.force_empty interface is provided to make cgroup's memory usage empty.
394 # echo 0 > memory.force_empty
396 Almost all pages tracked by this memory cgroup will be unmapped and freed.
397 Some pages cannot be freed because they are locked or in-use. Such pages are
398 moved to parent and this cgroup will be empty. This may return -EBUSY if
402 Because rmdir() moves all pages to parent, some out-of-use page caches can be
407 memory.stat file includes following statistics
409 # per-memory cgroup local status
410 cache - # of bytes of page cache memory.
411 rss - # of bytes of anonymous and swap cache memory.
412 mapped_file - # of bytes of mapped file (includes tmpfs/shmem)
413 pgpgin - # of charging events to the memory cgroup. The charging
416 pgpgout - # of uncharging events to the memory cgroup. The uncharging
418 swap - # of bytes of swap usage
419 inactive_anon - # of bytes of anonymous memory and swap cache memory on
421 active_anon - # of bytes of anonymous and swap cache memory on active
423 inactive_file - # of bytes of file-backed memory on inactive LRU list.
424 active_file - # of bytes of file-backed memory on active LRU list.
425 unevictable - # of bytes of memory that cannot be reclaimed (mlocked etc).
427 # status considering hierarchy (see memory.use_hierarchy settings)
429 hierarchical_memory_limit - # of bytes of memory limit with regard to hierarchy
430 under which the memory cgroup is
431 hierarchical_memsw_limit - # of bytes of memory+swap limit with regard to
432 hierarchy under which memory cgroup is.
434 total_cache - sum of all children's "cache"
435 total_rss - sum of all children's "rss"
436 total_mapped_file - sum of all children's "cache"
437 total_pgpgin - sum of all children's "pgpgin"
438 total_pgpgout - sum of all children's "pgpgout"
439 total_swap - sum of all children's "swap"
440 total_inactive_anon - sum of all children's "inactive_anon"
441 total_active_anon - sum of all children's "active_anon"
442 total_inactive_file - sum of all children's "inactive_file"
443 total_active_file - sum of all children's "active_file"
444 total_unevictable - sum of all children's "unevictable"
448 recent_rotated_anon - VM internal parameter. (see mm/vmscan.c)
449 recent_rotated_file - VM internal parameter. (see mm/vmscan.c)
450 recent_scanned_anon - VM internal parameter. (see mm/vmscan.c)
451 recent_scanned_file - VM internal parameter. (see mm/vmscan.c)
459 Only anonymous and swap cache memory is listed as part of 'rss' stat.
461 amount of physical memory used by the cgroup.
464 file_mapped is accounted only when the memory cgroup is owner of page
472 - root cgroup (uses /proc/sys/vm/swappiness).
473 - a cgroup which uses hierarchy and it has other cgroup(s) below it.
474 - a cgroup which uses hierarchy and not the root of hierarchy.
478 A memory cgroup provides memory.failcnt and memory.memsw.failcnt files.
480 hit its limit. When a memory cgroup hits a limit, failcnt increases and
481 memory under it will be reclaimed.
484 # echo 0 > .../memory.failcnt
488 For efficiency, as other kernel components, memory cgroup uses some optimization
490 method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz
492 If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
493 value in memory.stat(see 5.2).
497 This is similar to numa_maps but operates on a per-memcg basis. This is
503 We export "total", "file", "anon" and "unevictable" pages per-node for
504 each memcg. The ouput format of memory.numa_stat is:
515 The memory controller supports a deep hierarchy and hierarchical accounting.
528 In the diagram above, with hierarchical accounting enabled, all memory
530 that has memory.use_hierarchy enabled. If one of the ancestors goes over its
536 A memory cgroup by default disables the hierarchy feature. Support
537 can be enabled by writing 1 to memory.use_hierarchy file of the root cgroup
539 # echo 1 > memory.use_hierarchy
543 # echo 0 > memory.use_hierarchy
554 Soft limits allow for greater sharing of memory. The idea behind soft limits
555 is to allow control groups to use as much of the memory as needed, provided
557 a. There is no memory contention
560 When the system detects memory contention or low memory, control groups
563 sure that one control group does not starve the others of memory.
566 no guarantees, but it does its best to make sure that when memory is
567 heavily contended for, memory is allocated based on the soft limit
576 # echo 256M > memory.soft_limit_in_bytes
580 # echo 1G > memory.soft_limit_in_bytes
583 reclaiming memory for balancing between memory cgroups
597 writing to memory.move_charge_at_immigrate of the destination cgroup.
601 # echo (some positive value) > memory.move_charge_at_immigrate
605 Note: Charges are moved only when you move mm->owner, IOW, a leader of a thread
608 try to make space by reclaiming memory. Task migration may fail if we
614 # echo 0 > memory.move_charge_at_immigrate
621 memory cgroup.
624 -----+------------------------------------------------------------------------
628 -----+------------------------------------------------------------------------
629 1 | A charge of file pages(normal file, tmpfs file(e.g. ipc shared memory)
640 - Implement madvise(2) to let users decide the vma to be moved or not to be
642 - All of moving charge operations are done under cgroup_mutex. It's not good
645 9. Memory thresholds
647 Memory cgroup implements memory thresholds using cgroups notification
648 API (see cgroups.txt). It allows to register multiple memory and memsw
652 - create an eventfd using eventfd(2);
653 - open memory.usage_in_bytes or memory.memsw.usage_in_bytes;
654 - write string like "<event_fd> <fd of memory.usage_in_bytes> <threshold>" to
657 Application will be notified through eventfd when memory usage crosses
660 It's applicable for root and non-root cgroup.
664 memory.oom_control file is for OOM notification and other controls.
666 Memory cgroup implements OOM notifier using cgroup notification
671 - create an eventfd using eventfd(2)
672 - open memory.oom_control file
673 - write string like "<event_fd> <fd of memory.oom_control>" to
679 You can disable OOM-killer by writing "1" to memory.oom_control file, as:
681 #echo 1 > memory.oom_control
683 This operation is only allowed to the top cgroup of sub-hierarchy.
684 If OOM-killer is disabled, tasks under cgroup will hang/sleep
685 in memory cgroup's OOM-waitqueue when they request accountable memory.
687 For running them, you have to relax the memory cgroup's OOM status by
697 oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
698 under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
703 1. Add support for accounting huge pages (as a separate controller)
704 2. Make per-cgroup scanner reclaim not-shared pages first
705 3. Teach controller to account for shared-pages
711 Overall, the memory controller has been a stable controller and has been
716 1. Singh, Balbir. RFC: Memory Controller, http://lwn.net/Articles/206697/
717 2. Singh, Balbir. Memory Controller (RSS Control),
721 4. Emelianov, Pavel. RSS controller based on process cgroups (v2)
723 5. Emelianov, Pavel. RSS controller based on process cgroups (v3)
728 8. Singh, Balbir. RSS controller v2 test results (lmbench),
730 9. Singh, Balbir. RSS controller v2 AIM9 results
732 10. Singh, Balbir. Memory controller v6 test results,
734 11. Singh, Balbir. Memory controller introduction (v6),
736 12. Corbet, Jonathan, Controlling memory use in cgroups,