Lines Matching full:split

629 DEFINE_MTHP_STAT_ATTR(split, MTHP_STAT_SPLIT);
1765 * under splitting since we don't split the page itself, only pmd to in copy_huge_pmd()
1784 /* Page maybe pinned: split and retry the fault on PTEs. */ in copy_huge_pmd()
1845 * folio_try_dup_anon_rmap_*() and split if duplicating fails. in copy_huge_pud()
2163 * If user want to discard part-pages of THP, split it so MADV_FREE in madvise_free_huge_pmd()
2278 * With split pmd lock we also need to move preallocated in pmd_move_must_withdraw()
2936 * the whole access to the hugepage during the split (which in __split_huge_pmd_locked()
2952 * remain set at all times on the pmd until the split is in __split_huge_pmd_locked()
2973 * Without "freeze", we'll simply split the PMD, propagating the in __split_huge_pmd_locked()
2982 * In case we cannot clear PageAnonExclusive(), split the PMD in __split_huge_pmd_locked()
3132 * contain an hugepage: check if we need to split an huge pmd. in split_huge_pmd_if_needed()
3145 /* Check if we need to split start first. */ in vma_adjust_trans_huge()
3148 /* Check if we need to split end next. */ in vma_adjust_trans_huge()
3151 /* If we're incrementing next->vm_start, we might need to split it. */ in vma_adjust_trans_huge()
3292 /* Racy check whether the huge page can be split */
3417 * @folio: the to-be-split folio
3418 * @new_order: the smallest order of the after split folios (since buddy
3419 * allocator like split generates folios with orders from @folio's
3421 * @split_at: in buddy allocator like split, the folio containing @split_at
3422 * will be split until its order becomes @new_order.
3424 * @list: the after split folios will be added to @list if it is not NULL,
3429 * @uniform_split: if the split is uniform or not (buddy allocator like split)
3432 * 1. uniform split: the given @folio into multiple @new_order small folios,
3435 * 2. buddy allocator like (non-uniform) split: the given @folio is split into
3436 * half and one of the half (containing the given page) is split into half
3441 * 1. uniform split: a single __split_folio_to_order() is called to split the
3445 * 2. non-uniform split: in general, folio_order - @new_order calls to
3446 * __split_folio_to_order() are made in a for loop to split the @folio
3449 * @page, which is split in next for loop.
3455 * 1. uniform split leaves @page (or the folio contains it) locked;
3456 * 2. buddy allocator like (non-uniform) split leaves @folio locked.
3460 * split. The caller needs to check the input folio.
3484 /* a swapcache folio can only be uniformly split to order-0 */ in __split_unmapped_folio()
3501 * split to new_order one order at a time. For uniform split, in __split_unmapped_folio()
3502 * folio is split to new_order directly. in __split_unmapped_folio()
3519 * uniform split has xas_split_alloc() called before in __split_unmapped_folio()
3521 * non-uniform split can handle ENOMEM. in __split_unmapped_folio()
3544 * Iterate through after-split folios and perform related in __split_unmapped_folio()
3545 * operations. But in buddy allocator like split, the folio in __split_unmapped_folio()
3553 * for buddy allocator like split, the folio containing in __split_unmapped_folio()
3554 * page will be split next and should not be released, in __split_unmapped_folio()
3570 * entries are updated with all the other after-split in __split_unmapped_folio()
3632 * For uniform split, it is left for caller to unlock. in __split_unmapped_folio()
3633 * For buddy allocator like split, the first after-split folio is left in __split_unmapped_folio()
3647 * of the tail pages after the split is complete. in __split_unmapped_folio()
3660 "Cannot split to order-1 folio"); in non_uniform_split_supported()
3665 * No split if the file system does not support large folio. in non_uniform_split_supported()
3671 "Cannot split file folio to non-0 order"); in non_uniform_split_supported()
3678 "Cannot split swapcache folio to non-0 order"); in non_uniform_split_supported()
3691 "Cannot split to order-1 folio"); in uniform_split_supported()
3697 "Cannot split file folio to non-0 order"); in uniform_split_supported()
3704 "Cannot split swapcache folio to non-0 order"); in uniform_split_supported()
3712 * __folio_split: split a folio at @split_at to a @new_order folio
3713 * @folio: folio to split
3717 * @list: after-split folios will be put on it if non NULL
3718 * @uniform_split: perform uniform split or not (non-uniform split)
3720 * It calls __split_unmapped_folio() to perform uniform and non-uniform split.
3721 * It is in charge of checking whether the split is supported or not and
3725 * split but not to @new_order, the caller needs to check)
3772 * is taken to serialise against parallel split or collapse in __folio_split()
3802 VM_WARN_ONCE(1, "Cannot split mapped folio below min-order: %u", in __folio_split()
3841 * Racy check if we can split the page, before unmap_folio() will in __folio_split()
3842 * split PMDs in __folio_split()
3878 * split will see list corruption when checking the in __folio_split()
3930 * @page can point to any page of the large folio to split. The split operation
3941 * GUP pins, will result in the folio not getting split; instead, the caller
3959 * Returns 0 if the huge page was split successfully.
3964 * Returns -EBUSY when trying to split the huge zeropage, if the folio is
3972 * Returns -EINVAL when trying to split to an order that is incompatible
3984 * folio_split: split a folio at @split_at to a @new_order folio
3985 * @folio: folio to split
3990 * split but not to @new_order, the caller needs to check)
3995 * Split a folio at @split_at to a new_order folio, leave the
3999 * After the split, there will be a group of folios with different orders and
4003 * After split, folio is left locked for caller.
4047 * therefore important to unqueue deferred split before changing folio memcg.
4095 * Exclude swapcache: originally to avoid a corrupt deferred split in deferred_split_folio()
4184 int split = 0, removed = 0; in deferred_split_scan() local
4227 split++; in deferred_split_scan()
4265 * Stop shrinker if we didn't split any page, but the queue is empty. in deferred_split_scan()
4268 if (!split && list_empty(&ds_queue->split_queue)) in deferred_split_scan()
4270 return split; in deferred_split_scan()
4280 unsigned long total = 0, split = 0; in split_huge_pages_all() local
4282 pr_debug("Split all THPs\n"); in split_huge_pages_all()
4312 split++; in split_huge_pages_all()
4321 pr_debug("%lu of %lu THP split\n", split, total); in split_huge_pages_all()
4337 unsigned long total = 0, split = 0; in split_huge_pages_pid() local
4358 pr_debug("Split huge pages in pid: %d, vaddr: [0x%lx - 0x%lx]\n", in split_huge_pages_pid()
4401 * will try to drop it before split and then check if the folio in split_huge_pages_pid()
4402 * can be split or not. So skip the check here. in split_huge_pages_pid()
4419 split++; in split_huge_pages_pid()
4424 split++; in split_huge_pages_pid()
4441 pr_debug("%lu of %lu THP split\n", split, total); in split_huge_pages_pid()
4457 unsigned long total = 0, split = 0; in split_huge_pages_in_file() local
4469 pr_debug("split file-backed THPs in file: %s, page offset: [0x%lx - 0x%lx]\n", in split_huge_pages_in_file()
4500 split++; in split_huge_pages_in_file()
4505 split++; in split_huge_pages_in_file()
4518 pr_debug("%lu of %lu file-backed THP split\n", split, total); in split_huge_pages_in_file()