devel/migration/compatibility.rst

5 ---------------------------------
15 - QEMU version
16 - machine type version
20 - qemu-system-x86_64 (v5.2), from now on qemu-5.2.
21 - qemu-system-x86_64 (v5.1), from now on qemu-5.1.
26 - pc-q35-5.2 (newer one in qemu-5.2) from now on pc-5.2
27 - pc-q35-5.1 (newer one in qemu-5.1) from now on pc-5.1
33 except for a few cases where the backend features influence frontend
40 1 - qemu-5.2 -M pc-5.2  -> migrates to -> qemu-5.2 -M pc-5.2
45 2 - qemu-5.1 -M pc-5.1  -> migrates to -> qemu-5.1 -M pc-5.1
54 we have the same QEMU version in both sides (qemu-5.2) but we are using
55 the latest machine type for that version (pc-5.2) but one of an older
56 QEMU version, in this case pc-5.1.
58 3 - qemu-5.2 -M pc-5.1  -> migrates to -> qemu-5.2 -M pc-5.1
60   It needs to use the definition of pc-5.1 and the devices as they
63   idea of what the pc-5.1 machine is.
65 4 - qemu-5.1 -M pc-5.2  -> migrates to -> qemu-5.1 -M pc-5.2
67   This combination is not possible as the qemu-5.1 doesn't understand
68   pc-5.2 machine type.  So nothing to worry here.
71 different.  Notice also that the machine type needs to be pc-5.1,
72 because we have the limitation than qemu-5.1 doesn't know pc-5.2.  So
75 5 - qemu-5.2 -M pc-5.1  -> migrates to -> qemu-5.1 -M pc-5.1
79   migration to qemu-5.1.  Notice that we can't make updates to
80   qemu-5.1 to understand whatever qemu-5.2 decides to change, so it is
81   in qemu-5.2 side to make the relevant changes.
83 6 - qemu-5.1 -M pc-5.1  -> migrates to -> qemu-5.2 -M pc-5.1
86   than we are able to receive migrations from qemu-5.1. The problem is
89 If qemu-5.1 and qemu-5.2 were the same, there will not be any
90 compatibility problems.  But the reason that we create qemu-5.2 is to
91 get new features, devices, defaults, etc.
93 If we get a device that has a new feature, or change a default value,
97 So we need a way to tell qemu-5.2 that when we are using machine type
98 pc-5.1, it needs to **not** use the feature, to be able to migrate to
99 real qemu-5.1.
101 And the equivalent part when migrating from qemu-5.1 to qemu-5.2.
102 qemu-5.2 has to expect that it is not going to get data for the new
103 feature, because qemu-5.1 doesn't know about it.
108 If we change a default value, we need to put back the old value on
114 DEFINE_PROP_*() macros. See include/hw/qdev-properties.h to find the
115 macros that exist.  With it, we set the default value for that
122 - name_device
123 - name_property
124 - value
128 In qemu-5.2 virtio-blk-device got multi queue support.  This is a
129 change that is not backward compatible.  In qemu-5.1 it has one
130 queue. In qemu-5.2 it has the same number of queues as the number of
138 Similar problem when we migrate from qemu-5.1 that has only one queue
139 to qemu-5.2, we only sent information for one queue, but destination
143 So, how can we address this problem.  Easy, just convince qemu-5.2
144 that when it is running pc-5.1, it needs to set the number of queues
145 for virtio-blk-devices to 1.
149 5 - qemu-5.2 -M pc-5.1  -> migrates to -> qemu-5.1 -M pc-5.1
151     qemu-5.2 -M pc-5.1 sets number of queues to be 1.
152     qemu-5.1 -M pc-5.1 expects number of queues to be 1.
156 6 - qemu-5.1 -M pc-5.1  -> migrates to -> qemu-5.2 -M pc-5.1
158     qemu-5.1 -M pc-5.1 sets number of queues to be 1.
159     qemu-5.2 -M pc-5.1 expects number of queues to be 1.
165 3 - qemu-5.2 -M pc-5.1  -> migrates to -> qemu-5.2 -M pc-5.1
175     A -> migrates -> B -> migrates -> C
179     A: qemu-5.1 -M pc-5.1
180     B: qemu-5.2 -M pc-5.1
181     C: qemu-5.2 -M pc-5.1
183     migration A -> B is case 6, so number of queues needs to be 1.
185     migration B -> C is case 3, so we don't care.  But actually we
186     care because we haven't started the guest in qemu-5.2, it came
187     migrated from qemu-5.1.  So to be in the safe place, we need to
188     always use number of queues 1 when we are using pc-5.1.
197   virtio-blk-pci: default num_queues to -smp N
201     @@ -1281,7 +1284,8 @@ static const Property virtio_blk_properties[] = {
203          DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
205     -    DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
206     +    DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues,
208          DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256),
210 It changes the default value of num_queues.  But it fishes it for old
213     @@ -31,6 +31,7 @@
216     +    { "virtio-blk-device", "num-queues", "1"},
220 A device with different features on both sides
221 ----------------------------------------------
225 different features on both sides of the migration.  That can be
227 devices have different features, whatever.
230 "theoretically" easy.  You have to get the features that the device
231 has in the source of the migration.  The features that the device has
233 features of both sides, and that is the way that you should launch
242 bad at being able to be launched with only some features enabled.
247 https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html
262 If we try to migrate without any care from host A to host B, it will
270 $ qemu-system-x86_64 -cpu host
274 $ qemu-system-x86_64 -cpu host
276 When both hosts have different cpu features this is guaranteed to
277 fail.  Especially if Host B has less features than host A.  If host A
278 has less features than host B, sometimes it works.  Important word of
281 So, forgetting about cpu models and continuing with the -cpu host
283 B have the following features:
285 Features:   'pcid'  'stibp' 'taa-no'
294 $ qemu-system-x86_64 -cpu host,pcid=off,stibp=off
298 $ qemu-system-x86_64 -cpu host,taa-no=off
303 of features in general.
305 Notice that we don't recommend to use -cpu host for migration.  It is
308 Other devices have worse control about individual features.  If they
309 want to be able to migrate between hosts that show different features,
318 -----------------------------------------------------------
327 After the release of qemu-8.0 we found a problem when doing migration
328 of the machine type pc-7.2.
330 - $ qemu-7.2 -M pc-7.2  ->  qemu-7.2 -M pc-7.2
334 - $ qemu-8.0 -M pc-7.2  ->  qemu-8.0 -M pc-7.2
338 - $ qemu-8.0 -M pc-7.2  ->  qemu-7.2 -M pc-7.2
342 - $ qemu-7.2 -M pc-7.2  ->  qemu-8.0 -M pc-7.2
346 So clearly something fails when migration between qemu-7.2 and
347 qemu-8.0 with machine type pc-7.2.  The error messages, and git bisect
350 In qemu-8.0 we got this commit::
361     --- a/hw/pci/pcie_aer.c
363     @@ -112,6 +112,10 @@ int pcie_aer_init(PCIDevice *dev,
365          pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
367     +    pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
369     +    pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
372          pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
383     Date:   Tue May 2 21:27:02 2023 -0300
394     diff --git a/hw/pci/pci.c b/hw/pci/pci.c
396     --- a/hw/pci/pci.c
398     @@ -79,6 +79,8 @@ static const Property pci_props[] = {
401          DEFINE_PROP_UINT32("acpi-index",  PCIDevice, acpi_index, 0),
402     +    DEFINE_PROP_BIT("x-pcie-err-unc-mask", PCIDevice, cap_present,
411     diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
413     --- a/hw/pci/pcie_aer.c
415     @@ -112,10 +112,13 @@ int pcie_aer_init(PCIDevice *dev, uint8_t cap_ver,
418          pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
420     -    pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
421     -                 PCI_ERR_UNC_MASK_DEFAULT);
422     -    pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
423     -                 PCI_ERR_UNC_SUPPORTED);
425     +    if (dev->cap_present & QEMU_PCIE_ERR_UNC_MASK) {
426     +        pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
428     +        pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
432          pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
436 qemu-8.0.  If the property bit is not set, we configure it as it was in 7.2.
441     diff --git a/hw/core/machine.c b/hw/core/machine.c
443     --- a/hw/core/machine.c
445     @@ -48,6 +48,7 @@ GlobalProperty hw_compat_7_2[] = {
446          { "e1000e", "migrate-timadj", "off" },
447          { "virtio-mem", "x-early-migration", "false" },
448          { "migration", "x-preempt-pre-7-2", "true" },
449     +    { TYPE_PCI_DEVICE, "x-pcie-err-unc-mask", "off" },
453 And now, when qemu-8.0.1 is released with this fix, all combinations
456 - $ qemu-7.2 -M pc-7.2  ->  qemu-7.2 -M pc-7.2 (works)
457 - $ qemu-8.0.1 -M pc-7.2  ->  qemu-8.0.1 -M pc-7.2 (works)
458 - $ qemu-8.0.1 -M pc-7.2  ->  qemu-7.2 -M pc-7.2 (works)
459 - $ qemu-7.2 -M pc-7.2  ->  qemu-8.0.1 -M pc-7.2 (works)
467 - $ qemu-7.2 -M pc-7.2  ->  qemu-7.2 -M pc-7.2
468 - $ qemu-8.0 -M pc-7.2  ->  qemu-8.0 -M pc-7.2
469 - $ qemu-8.0.1 -M pc-7.2  ->  qemu-8.0.1 -M pc-7.2
475 - $ qemu-7.2 -M pc-7.2  ->  qemu-8.0 -M pc-7.2
476 - $ qemu-8.0 -M pc-7.2  ->  qemu-7.2 -M pc-7.2
479 change in qemu-8.0.1 release was to fix this issue:
481 - $ qemu-7.2 -M pc-7.2  ->  qemu-8.0.1 -M pc-7.2
482 - $ qemu-8.0.1 -M pc-7.2  ->  qemu-7.2 -M pc-7.2
484 But now we found that qemu-8.0 neither can migrate to qemu-7.2 not
485 qemu-8.0.1.
487 - $ qemu-8.0 -M pc-7.2  ->  qemu-8.0.1 -M pc-7.2
488 - $ qemu-8.0.1 -M pc-7.2  ->  qemu-8.0 -M pc-7.2
490 So, if we start a pc-7.2 machine in qemu-8.0 we can't migrate it to
491 anything except to qemu-8.0.
497 - $ qemu-8.0 -M pc-7.2  ->  qemu-8.0.1 -M pc-7.2
501   --device...,x-pci-e-err-unc-mask=on
505 property for pc-7.2.  Notice that we need to remember, it is not
506 enough to know that the source of the migration is qemu-8.0.  Think of
509 $ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 -> qemu-8.2 -M pc-7.2
511 In the second migration, the source is not qemu-8.0, but we still have