qemu/docs/rdma.txt

1 (RDMA: Remote Direct Memory Access)
2 RDMA Live Migration Specification, Version # 1
5 Github: git@github.com:hinesmr/qemu.git, 'rdma' branch
18 * RDMA Migration Protocol Description
28 RDMA helps make your migration more deterministic under heavy load because
30 because the RDMA I/O architecture reduces the number of interrupts and
31 data copies by bypassing the host networking stack. In particular, a TCP-based
32 migration, under certain types of memory-bound workloads, may take a more
37 RDMA currently comes in two flavors: both Ethernet based (RoCE, or RDMA
38 over Converged Ethernet) as well as Infiniband-based. This implementation of
39 migration using RDMA is capable of using both technologies because of
43 Refer to openfabrics.org or your respective RDMA hardware vendor for
47 for a working build of QEMU to run successfully using RDMA Migration.
52 Use of RDMA during migration requires pinning and registering memory
56 of RDMA migration may in fact be harmful to co-located VMs or other
59 use of RDMA is discouraged and it is recommended to use standard TCP migration.
65 bulk-phase round of the migration and can be enabled for extremely
66 high-performance RDMA hardware using the following command:
69 $ migrate_set_capability rdma-pin-all on # disabled by default
84 still gain from the benefits of advanced pinning with RDMA.
92 $ migrate_set_parameter max-bandwidth 40g # or whatever is the MAX of your RDMA device
96 qemu ..... -incoming rdma:host:port
101 $ migrate -d rdma:host:port
106 Here is a brief summary of total migration time and downtime using RDMA:
107 Using a 40gbps infiniband link performing a worst-case stress test,
111 $ apt-get install stress
112 $ stress --vm-bytes 7500M --vm 1 --vm-keep
123 1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
124 2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
127 you have to migrate using RDMA.
132 the bulk round and does not need to be re-registered during the successive
135 RDMA Protocol Description:
138 Migration with RDMA is separated into two parts:
140 1. The transmission of the pages using RDMA
148 The only difference between a SEND message and an RDMA
151 infiniband receiver side, whereas RDMA messages (used
161 RDMA messages are much easier to deal with. Once the memory
167 completes, given that RDMA migrations are very fast.)
176 as follows (migration-rdma.c):
201 The maximum number of repeats is hard-coded to 4096. This is a conservative
208      3. Ready                      (control-channel is available)
209      4. QEMU File                  (for sending non-live device state)
226 After ram block exchange is completed, we have two protocol-level
227 functions, responsible for communicating control-channel commands
240 5. Verify that the command-type and version received matches the one we expected.
259    hold the rkey need to perform RDMA. Note that the virtual address
271    before performing the RDMA operations.
273    be sent with RDMA, the registration commands are used to ask the
277    when transmitting non-live state, such as devices or to send
295 at connection-setup time before any infiniband traffic is generated.
322 Finally: Negotiation happens with the Flags field: If the primary-VM
324 will return a zero-bit for that flag and the primary-VM will understand
325 that as not being an available capability and will thus disable that
326 capability on the primary-VM side.
337 describe above to deliver bytes without changing the upper-level
344 to hold on to the bytes received from control-channel's SEND
347 Each time we receive a complete "QEMU File" control-channel
356 asking for a new SEND message to re-fill the buffer.
361 At the beginning of the migration, (migration-rdma.c),
368 addresses and possibly includes pre-registered RDMA keys in case dynamic
369 page registration was disabled on the server-side, otherwise not.
372 but is instead migrated with normal RDMA Write operations.
374 Pages are migrated in "chunks" (hard-coded to 1 Megabyte right now).
381 After pinning, an RDMA Write is generated and transmitted
390 and helps keep the hardware busy performing RDMA operations.
392 Error-handling:
397 we use for RDMA migration.
401 cleanup all the RDMA descriptors and unregister all
406 socket is broken during a non-RDMA based migration.
410 1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
414    the use of KSM and ballooning while using RDMA.
415 3. Also, some form of balloon-device usage tracking would also
417 4. Use LRU to provide more fine-grained direction of UNREGISTER
419 5. Expose UNREGISTER support to the user by way of workload-specific