0bafd6e9 | 02-Apr-2025 |
Jack Wang <jinpu.wang@ionos.com> |
migration/rdma: Remove qemu_rdma_broken_ipv6_kernel
I hit following error which testing migration in pure RoCE env: "-incoming rdma:[::]:8089: RDMA ERROR: You only have RoCE / iWARP devices in your
migration/rdma: Remove qemu_rdma_broken_ipv6_kernel
I hit following error which testing migration in pure RoCE env: "-incoming rdma:[::]:8089: RDMA ERROR: You only have RoCE / iWARP devices in your systems and your management software has specified '[::]', but IPv6 over RoCE / iWARP is not supported in Linux.#012'."
In our setup, we use rdma bind on ipv6 on target host, while connect from source with ipv4, remove the qemu_rdma_broken_ipv6_kernel, migration just work fine.
Checking the git history, the function was added since introducing of rdma migration, which is more than 10 years ago. linux-rdma has improved support on RoCE/iWARP for ipv6 over past years. There are a few fixes back in 2016 seems related to the issue, eg: aeb76df46d11 ("IB/core: Set routable RoCE gid type for ipv4/ipv6 networks")
other fixes back in 2018, eg: 052eac6eeb56 RDMA/cma: Update RoCE multicast routines to use net namespace 8d20a1f0ecd5 RDMA/cma: Fix rdma_cm raw IB path setting for RoCE 9327c7afdce3 RDMA/cma: Provide a function to set RoCE path record L2 parameters 5c181bda77f4 RDMA/cma: Set default GID type as RoCE when resolving RoCE route 3c7f67d1880d IB/cma: Fix default RoCE type setting be1d325a3358 IB/core: Set RoCEv2 MGID according to spec 63a5f483af0e IB/cma: Set default gid type to RoCEv2
So remove the outdated function and it's usage.
Cc: Peter Xu <peterx@redhat.com> Cc: Li Zhijian <lizhijian@fujitsu.com> Cc: Yu Zhang <yu.zhang@ionos.com> Cc: Fabiano Rosas <farosas@suse.de> Cc: qemu-devel@nongnu.org Cc: linux-rdma@vger.kernel.org Cc: michael@flatgalaxy.com Signed-off-by: Jack Wang <jinpu.wang@ionos.com> Tested-by: Li zhijian <lizhijian@fujitsu.com> Reviewed-by: Michael Galaxy <mrgalaxy@nvidia.com> Link: https://lore.kernel.org/r/20250402051306.6509-1-jinpu.wang@ionos.com [peterx: some cosmetic changes] Signed-off-by: Peter Xu <peterx@redhat.com>
show more ...
|
20d82622 | 24-Apr-2025 |
Peter Xu <peterx@redhat.com> |
migration/postcopy: Spatial locality page hint for preempt mode
The preempt mode postcopy has been introduced for a while. From latency POV, it should always win the vanilla postcopy.
However ther
migration/postcopy: Spatial locality page hint for preempt mode
The preempt mode postcopy has been introduced for a while. From latency POV, it should always win the vanilla postcopy.
However there's one thing missing when preempt mode is enabled right now, which is the spatial locality hint when there're page requests from the destination side.
In vanilla postcopy, as long as a page request was unqueued, it will update the PSS of the precopy background stream, so that after a page request the background thread will move the pages after whatever was requested. It's pretty much a natural behavior when there's only one channel anyway, and one scanner to send the pages.
Preempt mode didn't follow that, because preempt mode has its own channel and its own PSS (which doesn't linearly scan the guest memory, but dedicated to resolve page requested from destination). So the page request process and the background migration process are completely separate.
This patch adds the hint explicitly for preempt mode. With that, whenever the preempt mode receives a page request on the source, it will service the remote page fault in the return path, then it'll provide a hint to the background thread so that we'll start sending the pages right after the requested ones in the background, assuming the follow up pages have a higher chance to be accessed later.
NOTE: since the background migration thread and return path thread run completely concurrently, it doesn't always mean the hint will be applied every single time. For example, it's possible that the return path thread receives multiple page requests in a row without the background thread getting the chance to consume one. In such case, the preempt thread only provide the hint if the previous hint has been consumed. After all, there's no point queuing hints when we only have one linear scanner.
This could measureably improve the simple sequential memory access pattern during postcopy (when preempt is on). For random accesses, I can measure a slight increase of remote page fault latency from ~500us -> ~600us, that could be a trade-off to have such hint mechanism, and after all that's still greatly improved comparing to vanilla postcopy on random (~10ms).
The patch is verified by our QE team in a video streaming test case, to reduce the pause of the video from ~1min to a few seconds when switching over to postcopy with preempt mode.
Reported-by: Xiaohui Li <xiaohli@redhat.com> Tested-by: Xiaohui Li <xiaohli@redhat.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20250424220705.195544-1-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com>
show more ...
|
ad8d82ff | 11-Apr-2025 |
Peter Xu <peterx@redhat.com> |
migration/ram: Implement save_postcopy_prepare()
Implement save_postcopy_prepare(), preparing for the enablement of both multifd and postcopy.
Signed-off-by: Peter Xu <peterx@redhat.com> Signed-off
migration/ram: Implement save_postcopy_prepare()
Implement save_postcopy_prepare(), preparing for the enablement of both multifd and postcopy.
Signed-off-by: Peter Xu <peterx@redhat.com> Signed-off-by: Prasad Pandit <pjp@fedoraproject.org> Reviewed-by: Fabiano Rosas <farosas@suse.de> Message-ID: <20250411114534.3370816-5-ppandit@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
1d481116 | 11-Apr-2025 |
Peter Xu <peterx@redhat.com> |
migration: Add save_postcopy_prepare() savevm handler
Add a savevm handler for a module to opt-in sending extra sections right before postcopy starts, and before VM is stopped.
RAM will start to us
migration: Add save_postcopy_prepare() savevm handler
Add a savevm handler for a module to opt-in sending extra sections right before postcopy starts, and before VM is stopped.
RAM will start to use this new savevm handler in the next patch to do flush and sync for multifd pages.
Note that we choose to do it before VM stopped because the current only potential user is not sensitive to VM status, so doing it before VM is stopped is preferred to enlarge any postcopy downtime.
It is still a bit unfortunate that we need to introduce such a new savevm handler just for the only use case, however it's so far the cleanest.
Signed-off-by: Peter Xu <peterx@redhat.com> Signed-off-by: Prasad Pandit <pjp@fedoraproject.org> Reviewed-by: Fabiano Rosas <farosas@suse.de> Message-ID: <20250411114534.3370816-4-ppandit@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
00f3fcef | 11-Apr-2025 |
Prasad Pandit <pjp@fedoraproject.org> |
migration: refactor channel discovery mechanism
The various logical migration channels don't have a standardized way of advertising themselves and their connections may be seen out of order by the m
migration: refactor channel discovery mechanism
The various logical migration channels don't have a standardized way of advertising themselves and their connections may be seen out of order by the migration destination. When a new connection arrives, the incoming migration currently make use of heuristics to determine which channel it belongs to.
The next few patches will need to change how the multifd and postcopy capabilities interact and that affects the channel discovery heuristic.
Refactor the channel discovery heuristic to make it less opaque and simplify the subsequent patches.
Signed-off-by: Prasad Pandit <pjp@fedoraproject.org> Reviewed-by: Fabiano Rosas <farosas@suse.de> Message-ID: <20250411114534.3370816-3-ppandit@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
56e3c89f | 11-Apr-2025 |
Prasad Pandit <pjp@fedoraproject.org> |
migration/multifd: move macros to multifd header
Move MULTIFD_ macros to the header file so that they are accessible from other source files.
Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off
migration/multifd: move macros to multifd header
Move MULTIFD_ macros to the header file so that they are accessible from other source files.
Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off-by: Prasad Pandit <pjp@fedoraproject.org> Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20250411114534.3370816-2-ppandit@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
b407c9e7 | 07-Apr-2025 |
Markus Armbruster <armbru@redhat.com> |
migration: Fix latent bug in migrate_params_test_apply()
migrate_params_test_apply() neglects to apply tls_authz. Currently harmless, because migrate_params_check() doesn't care. Fix it anyway.
S
migration: Fix latent bug in migrate_params_test_apply()
migrate_params_test_apply() neglects to apply tls_authz. Currently harmless, because migrate_params_check() doesn't care. Fix it anyway.
Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Message-ID: <20250407072833.2118928-1-armbru@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
5e7ca4a7 | 05-Mar-2025 |
Li Zhijian <lizhijian@fujitsu.com> |
migration: Unfold control_save_page()
control_save_page() is for RDMA only, unfold it to make the code more clear. In addition: - Similar to other branches style in ram_save_target_page(), involve
migration: Unfold control_save_page()
control_save_page() is for RDMA only, unfold it to make the code more clear. In addition: - Similar to other branches style in ram_save_target_page(), involve RDMA only if the condition 'migrate_rdma()' is true. - Further simplify the code by removing the RAM_SAVE_CONTROL_NOT_SUPP.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com> Message-ID: <20250305062825.772629-6-lizhijian@fujitsu.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
4ecd6bea | 05-Mar-2025 |
Li Zhijian <lizhijian@fujitsu.com> |
migration/rdma: Remove redundant migration_in_postcopy checks
Since we have disabled RDMA + postcopy, it's safe to remove the migration_in_postcopy() that follows the migrate_rdma().
Reviewed-by: P
migration/rdma: Remove redundant migration_in_postcopy checks
Since we have disabled RDMA + postcopy, it's safe to remove the migration_in_postcopy() that follows the migrate_rdma().
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com> Message-ID: <20250305062825.772629-5-lizhijian@fujitsu.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
103fa641 | 05-Mar-2025 |
Li Zhijian <lizhijian@fujitsu.com> |
migration: disable RDMA + postcopy-ram
It's believed that RDMA + postcopy-ram has been broken for a while. Rather than spending time re-enabling it, let's simply disable it as a trade-off.
Reviewed
migration: disable RDMA + postcopy-ram
It's believed that RDMA + postcopy-ram has been broken for a while. Rather than spending time re-enabling it, let's simply disable it as a trade-off.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com> Message-ID: <20250305062825.772629-4-lizhijian@fujitsu.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
e1d4ea53 | 26-Mar-2025 |
Cédric Le Goater <clg@redhat.com> |
vfio: Introduce a new header file for external migration services
The migration core subsystem makes use of the VFIO migration API to collect statistics on the number of bytes transferred. These ser
vfio: Introduce a new header file for external migration services
The migration core subsystem makes use of the VFIO migration API to collect statistics on the number of bytes transferred. These services are declared in "hw/vfio/vfio-common.h" which also contains VFIO internal declarations. Move the migration declarations into a new header file "hw/vfio/vfio-migration.h" to reduce the exposure of VFIO internals.
While at it, use a 'vfio_migration_' prefix for these services.
To be noted, vfio_migration_add_bytes_transferred() is a VFIO migration internal service which we will be moved in the subsequent patches.
Cc: Kirti Wankhede <kwankhede@nvidia.com> Cc: Avihai Horon <avihaih@nvidia.com> Reviewed-by: Prasad Pandit <pjp@fedoraproject.org> Reviewed-by: John Levon <john.levon@nutanix.com> Reviewed-by: Avihai Horon <avihaih@nvidia.com> Link: https://lore.kernel.org/qemu-devel/20250326075122.1299361-4-clg@redhat.com Signed-off-by: Cédric Le Goater <clg@redhat.com>
show more ...
|
baa41af1 | 05-Mar-2025 |
Li Zhijian <lizhijian@fujitsu.com> |
migration: Prioritize RDMA in ram_save_target_page()
Address an error in RDMA-based migration by ensuring RDMA is prioritized when saving pages in `ram_save_target_page()`.
Previously, the RDMA pro
migration: Prioritize RDMA in ram_save_target_page()
Address an error in RDMA-based migration by ensuring RDMA is prioritized when saving pages in `ram_save_target_page()`.
Previously, the RDMA protocol's page-saving step was placed after other protocols due to a refactoring in commit bc38dc2f5f3. This led to migration failures characterized by unknown control messages and state loading errors destination: (qemu) qemu-system-x86_64: Unknown control message QEMU FILE qemu-system-x86_64: error while loading state section id 1(ram) qemu-system-x86_64: load of migration failed: Operation not permitted source: (qemu) qemu-system-x86_64: RDMA is in an error state waiting migration to abort! qemu-system-x86_64: failed to save SaveStateEntry with id(name): 1(ram): -1 qemu-system-x86_64: rdma migration: recv polling control error! qemu-system-x86_64: warning: Early error. Sending error. qemu-system-x86_64: warning: rdma migration: send polling control error
RDMA migration implemented its own protocol/method to send pages to destination side, hand over to RDMA first to prevent pages being saved by other protocol.
Fixes: bc38dc2f5f3 ("migration: refactor ram_save_target_page functions") Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com> Message-ID: <20250305062825.772629-2-lizhijian@fujitsu.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|
094a3dbc | 27-Feb-2025 |
Steve Sistare <steven.sistare@oracle.com> |
migration: ram block cpr blockers
Unlike cpr-reboot mode, cpr-transfer mode cannot save volatile ram blocks in the migration stream file and recreate them later, because the physical memory for the
migration: ram block cpr blockers
Unlike cpr-reboot mode, cpr-transfer mode cannot save volatile ram blocks in the migration stream file and recreate them later, because the physical memory for the blocks is pinned and registered for vfio. Add a blocker for volatile ram blocks.
Also add a blocker for RAM_GUEST_MEMFD. Preserving guest_memfd may be sufficient for CPR, but it has not been tested yet.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: David Hildenbrand <david@redhat.com> Message-ID: <1740667681-257312-1-git-send-email-steven.sistare@oracle.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
show more ...
|