LinuxKPI: cleanup: implement __free() and DEFINE_FREE(); use for kfree()

A wifi driver update needs this.

Sponsored by: The FreeBSD Foundation (initially)
MFC after: 3 days
Reviewed by: dumbbell
Di

LinuxKPI: cleanup: implement __free() and DEFINE_FREE(); use for kfree()

A wifi driver update needs this.

Sponsored by: The FreeBSD Foundation (initially)
MFC after: 3 days
Reviewed by: dumbbell
Differential Revision: https://reviews.freebsd.org/D52075

show more ...

linuxkpi: Call `lkpi_fpu_safe_exec()` in the implementation of kvmalloc()

`kvmalloc()` was a simple wrapper around the FreeBSD native `malloc()`.
Unlike the more involved implementation of `kmalloc(

linuxkpi: Call `lkpi_fpu_safe_exec()` in the implementation of kvmalloc()

`kvmalloc()` was a simple wrapper around the FreeBSD native `malloc()`.
Unlike the more involved implementation of `kmalloc()`, it didn't end
and being the FPU context around the actual call to `malloc()`.

This caused the following panic in the amdgup DRM driver:

panic: malloc: called with spinlock or critical section held

... triggered by the call:

struct dc_3dlut *lut = kvzalloc(sizeof(*lut), GFP_KERNEL);

(for the record, GFP_KERNEL is defined as M_WAITOK)

Replicating the same behaviour as `kmalloc()`, in other words, ending
the FPU context before the call to the underlying `malloc()`, and
beginning it again afterwards solves the problem.

Reviewed by: olce
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D51557

show more ...

LinuxKPI: Have kvzalloc() rely on kvmalloc(), not kmalloc()

Since commit 19df0c5abcb9d4e9 ("LinuxKPI: make __kmalloc() play by the
rules"), kmalloc() systematically allocates contiguous physical mem

LinuxKPI: Have kvzalloc() rely on kvmalloc(), not kmalloc()

Since commit 19df0c5abcb9d4e9 ("LinuxKPI: make __kmalloc() play by the
rules"), kmalloc() systematically allocates contiguous physical memory,
as it should. However, kvzalloc() was left defined in terms of
kmalloc(), which makes it allocate contiguous physical memory too. This
is a too stringent restriction, as kvzalloc() is supposed to be a simple
page-zeroing wrapper around kvmalloc().

According to Linux's documentation ("memory-allocation.rst"), kvmalloc()
first tries to allocate contiguous memory, falling back to
non-contiguous one if that fails. Thus, callers are already supposed to
deal with the possibility of non-contiguous memory being returned.

Reviewed by: bz
Fixes: 19df0c5abcb9 ("LinuxKPI: make __kmalloc() play by the rules")
MFC after: 10 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D51247

show more ...

LinuxKPI; cleanup slab.h a bit; move more free() into slab.c

Move kfree() into slab.c as an implementation and hide the private
function linux_kfree_async() entirely.

Remove a ; at the end of a def

LinuxKPI; cleanup slab.h a bit; move more free() into slab.c

Move kfree() into slab.c as an implementation and hide the private
function linux_kfree_async() entirely.

Remove a ; at the end of a define and sort some defines into place.

Remove extern from function declarations and move the closer to where
they belong.

Sort the functions into "base allocator/free" functions--these have
an implementation in slab.c and are ensuring contiguous physical
memory allocations.

Followed by inline functions using these base allocators to implement
their functionality; vmalloc/kvalloc, and misc functions.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: dumbbell
Differential Revision: https://reviews.freebsd.org/D49572

show more ...

LinuxKPI: implement krealloc() for memory being contiguous

Implement krealloc_array() using krealloc(). Implement krealloc()
doing the various size checks ourselves and use realloc() or kmalloc()
d

LinuxKPI: implement krealloc() for memory being contiguous

Implement krealloc_array() using krealloc(). Implement krealloc()
doing the various size checks ourselves and use realloc() or kmalloc()
depending on old and new allocation sizes.
This way we can ensure that allocated memory stays physically contiguous.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Suggested by: jhb (see D46657)
Reviewed by: jhb, markj
Differential Revision: https://reviews.freebsd.org/D49571

show more ...

LinuxKPI: switch mallocarray to an lkpi implementation using __kmalloc()

With mallocarray() we cannot guarantee that any size larger than
PAGE_SIZE will be contiguous. Switch kmalloc_array() and
km

LinuxKPI: switch mallocarray to an lkpi implementation using __kmalloc()

With mallocarray() we cannot guarantee that any size larger than
PAGE_SIZE will be contiguous. Switch kmalloc_array() and
kmalloc_array_node() to use __kmalloc()/lkpi___kmalloc_node() as their
underlying implementation which now does provide that guarantee.
Likewise adjust kcalloc_node() to use kmalloc_array_node().
This means we only have two (plain + _node) underlying allocation
routines for the entire category of functions.

Also adjust kvmalloc() and kvmalloc_array() to be a "mirrored"
implementation to their non-v counterparts. These may return
non-contiguous memory so can use malloc().

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: jhb
Extra thanks to: jhb for helping sorting this out
Differential Revision: https://reviews.freebsd.org/D46657

show more ...

LinuxKPI: move __kmalloc from slab.h to slab.c

In order to allow the allocator to change in the future move it into
the implementation file from being an inline function in the header.

While here f

LinuxKPI: move __kmalloc from slab.h to slab.c

In order to allow the allocator to change in the future move it into
the implementation file from being an inline function in the header.

While here factor out the size calculation and add a comment as-to why
this is done. We will need the size (_s) in the future to make a
decision on how to allocate.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D45815

show more ...

LinuxKPI: Add kmalloc_size_roundup function

kmalloc_size_roundup reports allocation bucket size for the given size.

Sponsored by: Serenity CyberSecurity, LLC
MFC after: 1 week
Reviewed by: manu, em

LinuxKPI: Add kmalloc_size_roundup function

kmalloc_size_roundup reports allocation bucket size for the given size.

Sponsored by: Serenity CyberSecurity, LLC
MFC after: 1 week
Reviewed by: manu, emaste
Differential Revision: https://reviews.freebsd.org/D45841

show more ...

LinuxKPI: Add kvrealloc to linux/slab.h

Sponsored by: Serenity Cyber Security, LLC
MFC after: 1 week
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D45616

LinuxKPI: Allow kmalloc to be called when FPU protection is enabled

Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This d

LinuxKPI: Allow kmalloc to be called when FPU protection is enabled

Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This does not work
on FreeBSD as its malloc function can not be run within a critical
section. Check this condition and temporally exit from FPU-protected
context to workaround issue and reduce source code patching.

Sponsored by: Serenity Cyber Security, LLC
Reviewed by: manu (previous version)
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42822

show more ...

Revert "linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now"

This change seems to break some drivers such as the mlx5*(4) drivers.

As kib@ says:
> According to the 'official' Linux kernel documentation, t

Revert "linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now"

This change seems to break some drivers such as the mlx5*(4) drivers.

As kib@ says:
> According to the 'official' Linux kernel documentation, the GFP_KERNEL
> flag implies sleepable context.

It was introduced while working on the new vt(4)/DRM integration [1].
During this work, doing sleepable allocations broke vt(4) and the DRM
drivers. However, I made further improvements and some locking-related
fixed to the new integration without revisiting the need for it.

After more testing, the improvements to the integration mentionned above
seems to make the change to `GFP_KERNEL` unneeded now. I can thus
revert it to restore expectations of other drivers.

This reverts commit 14dcd40983748596d116d91acb934a8a95ac76bc.

[1] https://github.com/freebsd/drm-kmod/pull/243

Reviewed by: kib
Approved by: kib
Differential Revision: https://reviews.freebsd.org/D42962

show more ...

linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now

... instead of `M_WAITOK`.

[Why]
The reason is that in some places in the DRM drivers (in particular, the
framebuffer management code), kmalloc() is cal

linuxkpi: `GFP_KERNEL` equals `M_NOWAIT` now

... instead of `M_WAITOK`.

[Why]
The reason is that in some places in the DRM drivers (in particular, the
framebuffer management code), kmalloc() is called from a non-sleepable
context, such as after a call to mtx_lock(8) with an MTX_DEF mutex.

If `GFP_KERNEL` is defined as `M_WAITOK`, we hit an assertion from
witness(4).

[How]
The definition of `GFP_KERNEL` is changed to `M_NOWAIT`. This means that
callers should verify the return value of kmalloc(). Fortunately, this
is always the case in Linux.

Reviewed by: bz, emaste, manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D42054

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

linuxkpi: Support non-NULL zero-size pointers

DRM drivers set some pointers to `ZERO_SIZE_PTR` directly (without
allocating anything), to treat pointers which were "initialized" (set to
`ZERO_SIZE_P

linuxkpi: Support non-NULL zero-size pointers

DRM drivers set some pointers to `ZERO_SIZE_PTR` directly (without
allocating anything), to treat pointers which were "initialized" (set to
`ZERO_SIZE_PTR`) with no memory allocation like really allocated
pointers. NULL isn't used because it represents a third state.

Reviewed by: emaste, manu
Approved by: emaste, manu
Differential Revision: https://reviews.freebsd.org/D39055

show more ...

linuxkpi: Define `ZERO_OR_NULL_PTR()` in <linux/slab.h>

On Linux, the `kmalloc()` family of functions returns a special value if
the size of the allocation is zero. This macro verifies if the pointe

linuxkpi: Define `ZERO_OR_NULL_PTR()` in <linux/slab.h>

On Linux, the `kmalloc()` family of functions returns a special value if
the size of the allocation is zero. This macro verifies if the pointer
is NULL (the allocation failed) or the size is 0 (the allocation was not
performed AFAIU). This special value can be passed to `kfree()`.

On FreeBSD, our `malloc(9)` functions don't return a special value for
0-size allocations. Therefore we can simply compare the result against
NULL.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D37367

show more ...

linuxkpi: Add `krealloc_array()`

In FreeBSD, this is a wrapper on top of `realloc()`.

V2: Check if `n * size` would overflow and return `NULL` if that's the
case. Suggested by hselasky@ and ema

linuxkpi: Add `krealloc_array()`

In FreeBSD, this is a wrapper on top of `realloc()`.

V2: Check if `n * size` would overflow and return `NULL` if that's the
case. Suggested by hselasky@ and emaste@.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D36959

show more ...

linuxkpi: Add stub kmem_cache_shrink

Needed by drm-kmod.

Reviewed by: bz
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D36108

LinuxKPI: Constantly use _LINUXKPI_ prefix in include guards

MFC after: 1 week
Reviewed by: bz, emaste, hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33562

LinuxKPI: Move kfree_async() functionality in to kfree()

Obsolete it usage but keep for a while for drm-kmod 5.4 compatibility

MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: h

LinuxKPI: Move kfree_async() functionality in to kfree()

Obsolete it usage but keep for a while for drm-kmod 5.4 compatibility

MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33298

show more ...

LinuxKPI: Add some typical header pollution

To reduce amount of drm-kmod patching

MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33297

LinuxKPI: Allow kmem_cache_free() to be called from critical sections

as it is required by i915kms driver from Linux kernel v 5.5.
This is done with asynchronous freeing of requested memory areas fr

LinuxKPI: Allow kmem_cache_free() to be called from critical sections

as it is required by i915kms driver from Linux kernel v 5.5.
This is done with asynchronous freeing of requested memory areas from
taskqueue thread. As memory to be freed is reused to store linked list
entry, backing UMA zone item size is rounded up to pointer size.

While here, make struct linux_kmem_cache private to LKPI to reduce amount
of BSD headers included by linux/slab.h and switch RCU code to usage of
LKPI's linux_irq_work_tq taskqueue to avoid injection of current into
system-wide taskqueue_fast thread context.

Submitted by: nc (initial version for drm-kmod)
Reviewed by: manu, nc
Differential revision: https://reviews.freebsd.org/D30760

show more ...

LinuxKPI: add kfree_sensitive() using zfree().

Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Reviewed by: hselasky
Differential Revision: https://reviews.freebsd.org/D30437

Implement basic support for allocating memory from a specific numa node
in the LinuxKPI.

Differential Revision: https://reviews.freebsd.org/D29077
Reviewed by: markj@ and kib@
MFC after: 1 week
Spon

Implement basic support for allocating memory from a specific numa node
in the LinuxKPI.

Differential Revision: https://reviews.freebsd.org/D29077
Reviewed by: markj@ and kib@
MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

show more ...

LinuxKPI: Reimplement irq_work queue on top of fast taskqueue

Summary:
Linux's irq_work queue was created for asynchronous execution of code from contexts where spin_lock's are not available like "h

LinuxKPI: Reimplement irq_work queue on top of fast taskqueue

Summary:
Linux's irq_work queue was created for asynchronous execution of code from contexts where spin_lock's are not available like "hardware interrupt context". FreeBSD's fast taskqueues was created for the same purposes.

Drm-kmod 5.4 uses irq_work_queue() at least in one place to schedule execution of task/work from the critical section that triggers following INVARIANTS-induced panic:

```
panic: acquiring blockable sleep lock with spinlock or critical section held (sleep mutex) linuxkpi_short_wq @ /usr/src/sys/kern/subr_taskqueue.c:281
cpuid = 6
time = 1605048416
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe006b538c90
vpanic() at vpanic+0x182/frame 0xfffffe006b538ce0
panic() at panic+0x43/frame 0xfffffe006b538d40
witness_checkorder() at witness_checkorder+0xf3e/frame 0xfffffe006b538f00
__mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe006b538f50
taskqueue_enqueue() at taskqueue_enqueue+0x42/frame 0xfffffe006b538f70
linux_queue_work_on() at linux_queue_work_on+0xe9/frame 0xfffffe006b538fb0
irq_work_queue() at irq_work_queue+0x21/frame 0xfffffe006b538fd0
semaphore_notify() at semaphore_notify+0xb2/frame 0xfffffe006b539020
__i915_sw_fence_notify() at __i915_sw_fence_notify+0x2e/frame 0xfffffe006b539050
__i915_sw_fence_complete() at __i915_sw_fence_complete+0x63/frame 0xfffffe006b539080
i915_sw_fence_complete() at i915_sw_fence_complete+0x8e/frame 0xfffffe006b5390c0
dma_i915_sw_fence_wake() at dma_i915_sw_fence_wake+0x4f/frame 0xfffffe006b539100
dma_fence_signal_locked() at dma_fence_signal_locked+0x105/frame 0xfffffe006b539180
dma_fence_signal() at dma_fence_signal+0x72/frame 0xfffffe006b5391c0
dma_fence_is_signaled() at dma_fence_is_signaled+0x80/frame 0xfffffe006b539200
dma_resv_add_shared_fence() at dma_resv_add_shared_fence+0xb3/frame 0xfffffe006b539270
i915_vma_move_to_active() at i915_vma_move_to_active+0x18a/frame 0xfffffe006b5392b0
eb_move_to_gpu() at eb_move_to_gpu+0x3ad/frame 0xfffffe006b539320
eb_submit() at eb_submit+0x15/frame 0xfffffe006b539350
i915_gem_do_execbuffer() at i915_gem_do_execbuffer+0x7d4/frame 0xfffffe006b539570
i915_gem_execbuffer2_ioctl() at i915_gem_execbuffer2_ioctl+0x1c1/frame 0xfffffe006b539600
drm_ioctl_kernel() at drm_ioctl_kernel+0xd9/frame 0xfffffe006b539670
drm_ioctl() at drm_ioctl+0x5cd/frame 0xfffffe006b539820
linux_file_ioctl() at linux_file_ioctl+0x323/frame 0xfffffe006b539880
kern_ioctl() at kern_ioctl+0x1f4/frame 0xfffffe006b5398f0
sys_ioctl() at sys_ioctl+0x12a/frame 0xfffffe006b5399c0
amd64_syscall() at amd64_syscall+0x121/frame 0xfffffe006b539af0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe006b539af0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800a6f09a, rsp = 0x7fffffffe588, rbp = 0x7fffffffe640 ---
KDB: enter: panic
```
Here, the dma_resv_add_shared_fence() performs a critical_enter() and following call of schedule_work() from semaphore_notify() triggers 'acquiring blockable sleep lock with spinlock or critical section held' panic.

Switching irq_work implementation to fast taskqueue fixes the panic for me.

Other report with the similar bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247166

Reviewed By: hselasky
Differential Revision: https://reviews.freebsd.org/D27171

show more ...

LinuxKPI: Implement ksize() function.

In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It al

LinuxKPI: Implement ksize() function.

In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It also maps LinuxKPI ksize() to newly created function.

ksize() function is used by drm-kmod.

Reviewed by: hselasky, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26215

show more ...