Documentation/vm/page_migration.rst

4 Page migration
7 Page migration allows moving the physical location of pages between
15 The main intent of page migration is to reduce the latency of memory accesses
19 Page migration allows a process to manually relocate the node on which its
25 Page migration functions are provided by the numactl package by Andi Kleen
28 which provides an interface similar to other NUMA functionality for page
31 proc(5) man page.
37 manual page migration support. Automatic page migration may be implemented
54 Page migration allows the preservation of the relative location of pages
60 Page migration occurs in several steps. First a high level
73    Calling isolate_lru_page() increases the references to the page
74    so that it cannot vanish while the page migration occurs.
76    the page.
80    how to allocate the correct new page given the old page.
84    the new page for each page that is considered for
90 migrate_pages() does several passes over its list of pages. A page is moved
91 if all references to a page are removable at the time. The page has
93 is increased so that the page cannot be freed while page migration occurs.
97 1. Lock the page to be migrated.
101 3. Lock the new page that we want to move to. It is locked so that accesses to
102    this (not yet up-to-date) page immediately block while the move is in progress.
104 4. All the page table references to the page are converted to migration
105    entries. This decreases the mapcount of a page. If the resulting
106    mapcount is not zero then we do not migrate the page. All user space
107    processes that attempt to access the page will now wait on the page lock
108    or wait for the migration page table entry to be removed.
111    to access the page via the mapping to block on the spinlock.
113 6. The refcount of the page is examined and we back out if references remain.
114    Otherwise, we know that we are the only one referencing this page.
117    page then we back out because someone else modified the radix tree.
119 8. The new page is prepped with some settings from the old page so that
120    accesses to the new page will discover a page with the correct settings.
122 9. The radix tree is changed to point to the new page.
124 10. The reference count of the old page is dropped because the address space
125     reference is gone. A reference to the new page is established because
126     the new page is referenced by the address space.
130     to sleeping on the locked new page.
132 12. The page contents are copied to the new page.
134 13. The remaining page flags are copied to the new page.
136 14. The old page flags are cleared to indicate that the page does
139 15. Queued up writeback on the new page is triggered.
141 16. If migration entries were inserted into the page table, then replace them
143     already waiting for the page lock.
145 17. The page locks are dropped from the old and new page.
146     Processes waiting on the page lock will redo their page faults
147     and will reach the new page.
149 18. The new page is moved to the LRU and can be scanned by the swapper,
152 Non-LRU page migration
167 To overcome the problem, VM supports non-LRU page migration which provides
174 1. ``bool (*isolate_page) (struct page *page, isolate_mode_t mode);``
177    if driver isolates the page successfully. On returning true, VM marks the page
178    as PG_isolated so concurrent isolation in several CPUs skip the page
179    for isolation. If a driver cannot isolate the page, it should return *false*.
181    Once page is successfully isolated, VM uses page.lru fields so driver
185 |	``struct page *newpage, struct page *oldpage, enum migrate_mode);``
187    After isolation, VM calls migratepage() of driver with the isolated page.
188    The function of migratepage() is to move the contents of the old page to the
189    new page
190    and set up fields of struct page newpage. Keep in mind that you should
193    MIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, driver
194    can return -EAGAIN. On -EAGAIN, VM will retry page migration in a short time
196    any error except -EAGAIN, VM will give up the page migration without
199    Driver shouldn't touch the page.lru field while in the migratepage() function.
201 3. ``void (*putback_page)(struct page *);``
203    If migration fails on the isolated page, VM should return the isolated page
204    to the driver so VM calls the driver's putback_page() with the isolated page.
205    In this function, the driver should put the isolated page back into its own data
208 4. non-LRU movable page flags
210    There are two page flags for supporting non-LRU movable page.
214      Driver should use the function below to make page movable under page_lock::
216 	void __SetPageMovable(struct page *page, struct address_space *mapping)
220      PG_movable is not a real flag of struct page. Rather, VM
221      reuses the page->mapping's lower bits to represent it::
224 	page->mapping = page->mapping | PAGE_MAPPING_MOVABLE;
226      so driver shouldn't access page->mapping directly. Instead, driver should
227      use page_mapping() which masks off the low two bits of page->mapping under
228      page lock so it can get the right struct address_space.
232      the page->mapping field is unified with other variables in struct page.
233      If the driver releases the page after isolation by VM, page->mapping
236      page is LRU or non-LRU movable once the page has been isolated because LRU
237      pages can never have PAGE_MAPPING_MOVABLE set in page->mapping. It is also
241      For guaranteeing non-LRU movable page, VM provides PageMovable() function.
242      Unlike __PageMovable(), PageMovable() validates page->mapping and
244      sudden destroying of page->mapping.
247      __ClearMovablePage() under page_lock() before the releasing the page.
251      To prevent concurrent isolation among several CPUs, VM marks isolated page
253      non-LRU movable page, it can skip it. Driver doesn't need to manipulate the
255      driver sees a PG_isolated page, it means the page has been isolated by the
256      VM so it shouldn't touch the page.lru field.
263 The following events (counters) can be used to monitor page migration.
265 1. PGMIGRATE_SUCCESS: Normal page migration success. Each count means that a
266    page was migrated. If the page was a non-THP page, then this counter is
267    increased by one. If the page was a THP, then this counter is increased by
272 2. PGMIGRATE_FAIL: Normal page migration failure. Same counting rules as for