qemu/docs/rdma.txt

4 Wiki: https://wiki.qemu.org/Features/RDMALiveMigration
5 Github: git@github.com:hinesmr/qemu.git, 'rdma' branch
10 linked on the QEMU wiki above.
31 data copies by bypassing the host networking stack. In particular, a TCP-based
32 migration, under certain types of memory-bound workloads, may take a more
38 over Converged Ethernet) as well as Infiniband-based. This implementation of
47 for a working build of QEMU to run successfully using RDMA Migration.
56 of RDMA migration may in fact be harmful to co-located VMs or other
65 bulk-phase round of the migration and can be enabled for extremely
66 high-performance RDMA hardware using the following command:
68 QEMU Monitor Command:
69 $ migrate_set_capability rdma-pin-all on # disabled by default
91 QEMU Monitor Command:
92 $ migrate_set_parameter max-bandwidth 40g # or whatever is the MAX of your RDMA device
94 Next, on the destination machine, add the following to the QEMU command line:
96 qemu ..... -incoming rdma:host:port
100 QEMU Monitor Command:
101 $ migrate -d rdma:host:port
107 Using a 40gbps infiniband link performing a worst-case stress test,
111 $ apt-get install stress
112 $ stress --vm-bytes 7500M --vm 1 --vm-keep
123 1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
124 2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
132 the bulk round and does not need to be re-registered during the successive
175 To begin the migration, the initial connection setup is
176 as follows (migration-rdma.c):
197 so that the protocol is compatible against multiple versions of QEMU.
201 The maximum number of repeats is hard-coded to 4096. This is a conservative
208      3. Ready                      (control-channel is available)
209      4. QEMU File                  (for sending non-live device state)
210      5. RAM Blocks request         (used right after connection setup)
211      6. RAM Blocks result          (used right after connection setup)
223 After connection setup, message 5 & 6 are used to exchange ram block
226 After ram block exchange is completed, we have two protocol-level
227 functions, responsible for communicating control-channel commands
240 5. Verify that the command-type and version received matches the one we expected.
266 1. After connection setup, RAMBlock information is exchanged using
277    when transmitting non-live state, such as devices or to send
295 at connection-setup time before any infiniband traffic is generated.
322 Finally: Negotiation happens with the Flags field: If the primary-VM
324 will return a zero-bit for that flag and the primary-VM will understand
326 capability on the primary-VM side.
337 describe above to deliver bytes without changing the upper-level
344 to hold on to the bytes received from control-channel's SEND
347 Each time we receive a complete "QEMU File" control-channel
355 listed above and issue another "QEMU File" protocol command,
356 asking for a new SEND message to re-fill the buffer.
361 At the beginning of the migration, (migration-rdma.c),
368 addresses and possibly includes pre-registered RDMA keys in case dynamic
369 page registration was disabled on the server-side, otherwise not.
374 Pages are migrated in "chunks" (hard-coded to 1 Megabyte right now).
392 Error-handling:
406 socket is broken during a non-RDMA based migration.
410 1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
415 3. Also, some form of balloon-device usage tracking would also
417 4. Use LRU to provide more fine-grained direction of UNREGISTER
419 5. Expose UNREGISTER support to the user by way of workload-specific