14ed49febSFabiano RosasMapped-ram 24ed49febSFabiano Rosas========== 34ed49febSFabiano Rosas 44ed49febSFabiano RosasMapped-ram is a new stream format for the RAM section designed to 54ed49febSFabiano Rosassupplement the existing ``file:`` migration and make it compatible 64ed49febSFabiano Rosaswith ``multifd``. This enables parallel migration of a guest's RAM to 74ed49febSFabiano Rosasa file. 84ed49febSFabiano Rosas 94ed49febSFabiano RosasThe core of the feature is to ensure that RAM pages are mapped 104ed49febSFabiano Rosasdirectly to offsets in the resulting migration file. This enables the 114ed49febSFabiano Rosas``multifd`` threads to write exclusively to those offsets even if the 124ed49febSFabiano Rosasguest is constantly dirtying pages (i.e. live migration). Another 134ed49febSFabiano Rosasbenefit is that the resulting file will have a bounded size, since 144ed49febSFabiano Rosaspages which are dirtied multiple times will always go to a fixed 154ed49febSFabiano Rosaslocation in the file, rather than constantly being added to a 164ed49febSFabiano Rosassequential stream. Having the pages at fixed offsets also allows the 174ed49febSFabiano Rosasusage of O_DIRECT for save/restore of the migration stream as the 184ed49febSFabiano Rosaspages are ensured to be written respecting O_DIRECT alignment 198d60280eSFabiano Rosasrestrictions. 204ed49febSFabiano Rosas 214ed49febSFabiano RosasUsage 224ed49febSFabiano Rosas----- 234ed49febSFabiano Rosas 244ed49febSFabiano RosasOn both source and destination, enable the ``multifd`` and 254ed49febSFabiano Rosas``mapped-ram`` capabilities: 264ed49febSFabiano Rosas 274ed49febSFabiano Rosas ``migrate_set_capability multifd on`` 284ed49febSFabiano Rosas 294ed49febSFabiano Rosas ``migrate_set_capability mapped-ram on`` 304ed49febSFabiano Rosas 314ed49febSFabiano RosasUse a ``file:`` URL for migration: 324ed49febSFabiano Rosas 334ed49febSFabiano Rosas ``migrate file:/path/to/migration/file`` 344ed49febSFabiano Rosas 354ed49febSFabiano RosasMapped-ram migration is best done non-live, i.e. by stopping the VM on 364ed49febSFabiano Rosasthe source side before migrating. 374ed49febSFabiano Rosas 388d60280eSFabiano RosasFor best performance enable the ``direct-io`` parameter as well: 398d60280eSFabiano Rosas 408d60280eSFabiano Rosas ``migrate_set_parameter direct-io on`` 418d60280eSFabiano Rosas 424ed49febSFabiano RosasUse-cases 434ed49febSFabiano Rosas--------- 444ed49febSFabiano Rosas 454ed49febSFabiano RosasThe mapped-ram feature was designed for use cases where the migration 464ed49febSFabiano Rosasstream will be directed to a file in the filesystem and not 47*381d2c36SPaolo Bonziniimmediately restored on the destination VM\ [#alternatives]_. These could be 484ed49febSFabiano Rosasthought of as snapshots. We can further categorize them into live and 494ed49febSFabiano Rosasnon-live. 504ed49febSFabiano Rosas 514ed49febSFabiano Rosas- Non-live snapshot 524ed49febSFabiano Rosas 534ed49febSFabiano RosasIf the use case requires a VM to be stopped before taking a snapshot, 544ed49febSFabiano Rosasthat's the ideal scenario for mapped-ram migration. Not having to 554ed49febSFabiano Rosastrack dirty pages, the migration will write the RAM pages to the disk 564ed49febSFabiano Rosasas fast as it can. 574ed49febSFabiano Rosas 584ed49febSFabiano RosasNote: if a snapshot is taken of a running VM, but the VM will be 594ed49febSFabiano Rosasstopped after the snapshot by the admin, then consider stopping it 604ed49febSFabiano Rosasright before the snapshot to take benefit of the performance gains 614ed49febSFabiano Rosasmentioned above. 624ed49febSFabiano Rosas 634ed49febSFabiano Rosas- Live snapshot 644ed49febSFabiano Rosas 654ed49febSFabiano RosasIf the use case requires that the VM keeps running during and after 664ed49febSFabiano Rosasthe snapshot operation, then mapped-ram migration can still be used, 674ed49febSFabiano Rosasbut will be less performant. Other strategies such as 684ed49febSFabiano Rosasbackground-snapshot should be evaluated as well. One benefit of 694ed49febSFabiano Rosasmapped-ram in this scenario is portability since background-snapshot 704ed49febSFabiano Rosasdepends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not 714ed49febSFabiano Rosassupported outside of Linux. 724ed49febSFabiano Rosas 73*381d2c36SPaolo Bonzini.. [#alternatives] While this same effect could be obtained with the usage of 744ed49febSFabiano Rosas snapshots or the ``file:`` migration alone, mapped-ram provides 754ed49febSFabiano Rosas a performance increase for VMs with larger RAM sizes (10s to 764ed49febSFabiano Rosas 100s of GiBs), specially if the VM has been stopped beforehand. 774ed49febSFabiano Rosas 784ed49febSFabiano RosasRAM section format 794ed49febSFabiano Rosas------------------ 804ed49febSFabiano Rosas 814ed49febSFabiano RosasInstead of having a sequential stream of pages that follow the 824ed49febSFabiano RosasRAMBlock headers, the dirty pages for a RAMBlock follow its header 834ed49febSFabiano Rosasinstead. This ensures that each RAM page has a fixed offset in the 844ed49febSFabiano Rosasresulting migration file. 854ed49febSFabiano Rosas 864ed49febSFabiano RosasA bitmap is introduced to track which pages have been written in the 874ed49febSFabiano Rosasmigration file. Pages are written at a fixed location for every 884ed49febSFabiano Rosasramblock. Zero pages are ignored as they'd be zero in the destination 894ed49febSFabiano Rosasmigration as well. 904ed49febSFabiano Rosas 914ed49febSFabiano Rosas:: 924ed49febSFabiano Rosas 934ed49febSFabiano Rosas Without mapped-ram: With mapped-ram: 944ed49febSFabiano Rosas 954ed49febSFabiano Rosas --------------------- -------------------------------- 964ed49febSFabiano Rosas | ramblock 1 header | | ramblock 1 header | 974ed49febSFabiano Rosas --------------------- -------------------------------- 984ed49febSFabiano Rosas | ramblock 2 header | | ramblock 1 mapped-ram header | 994ed49febSFabiano Rosas --------------------- -------------------------------- 1004ed49febSFabiano Rosas | ... | | padding to next 1MB boundary | 1014ed49febSFabiano Rosas --------------------- | ... | 1024ed49febSFabiano Rosas | ramblock n header | -------------------------------- 1034ed49febSFabiano Rosas --------------------- | ramblock 1 pages | 1044ed49febSFabiano Rosas | RAM_SAVE_FLAG_EOS | | ... | 1054ed49febSFabiano Rosas --------------------- -------------------------------- 1064ed49febSFabiano Rosas | stream of pages | | ramblock 2 header | 1074ed49febSFabiano Rosas | (iter 1) | -------------------------------- 1084ed49febSFabiano Rosas | ... | | ramblock 2 mapped-ram header | 1094ed49febSFabiano Rosas --------------------- -------------------------------- 1104ed49febSFabiano Rosas | RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary | 1114ed49febSFabiano Rosas --------------------- | ... | 1124ed49febSFabiano Rosas | stream of pages | -------------------------------- 1134ed49febSFabiano Rosas | (iter 2) | | ramblock 2 pages | 1144ed49febSFabiano Rosas | ... | | ... | 1154ed49febSFabiano Rosas --------------------- -------------------------------- 1164ed49febSFabiano Rosas | ... | | ... | 1174ed49febSFabiano Rosas --------------------- -------------------------------- 1184ed49febSFabiano Rosas | RAM_SAVE_FLAG_EOS | 1194ed49febSFabiano Rosas -------------------------------- 1204ed49febSFabiano Rosas | ... | 1214ed49febSFabiano Rosas -------------------------------- 1224ed49febSFabiano Rosas 1234ed49febSFabiano Rosaswhere: 1244ed49febSFabiano Rosas - ramblock header: the generic information for a ramblock, such as 1254ed49febSFabiano Rosas idstr, used_len, etc. 1264ed49febSFabiano Rosas 1274ed49febSFabiano Rosas - ramblock mapped-ram header: the information added by this feature: 1284ed49febSFabiano Rosas bitmap of pages written, bitmap size and offset of pages in the 1294ed49febSFabiano Rosas migration file. 1304ed49febSFabiano Rosas 1314ed49febSFabiano RosasRestrictions 1324ed49febSFabiano Rosas------------ 1334ed49febSFabiano Rosas 1344ed49febSFabiano RosasSince pages are written to their relative offsets and out of order 1354ed49febSFabiano Rosas(due to the memory dirtying patterns), streaming channels such as 1364ed49febSFabiano Rosassockets are not supported. A seekable channel such as a file is 1374ed49febSFabiano Rosasrequired. This can be verified in the QIOChannel by the presence of 1384ed49febSFabiano Rosasthe QIO_CHANNEL_FEATURE_SEEKABLE. 1394ed49febSFabiano Rosas 1404ed49febSFabiano RosasThe improvements brought by this feature apply only to guest physical 1414ed49febSFabiano RosasRAM. Other types of memory such as VRAM are migrated as part of device 1424ed49febSFabiano Rosasstates. 143