Merge tag 'pull-block-2022-03-22' of https://gitlab.com/hreitz/qemu into staging

Block patches for 7.0-rc1:
- iotest fixes:
- Fix some iotests for riscv targets
- Use GNU sed in more places wher

Merge tag 'pull-block-2022-03-22' of https://gitlab.com/hreitz/qemu into staging

Block patches for 7.0-rc1:
- iotest fixes:
- Fix some iotests for riscv targets
- Use GNU sed in more places where required
- Meson-related fixes (i.e. to print errors when they occur)
- Have qemu-img calls (from Python tests) generally raise nicely
formattable exceptions on errors
- Fix iotest 207
- Allow RBD images to be growable by writing zeroes past the end of
file, fixing qcow2 on rbd

# gpg: Signature made Tue 22 Mar 2022 11:51:10 GMT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF

* tag 'pull-block-2022-03-22' of https://gitlab.com/hreitz/qemu: (25 commits)
iotests/207: Filter host fingerprint
iotests.py: Filters for VM.run_job()
iotests: make qemu_img_log and img_info_log raise on error
iotests: remove qemu_img_pipe_and_status()
iotests: replace qemu_img_log('create', ...) calls
iotests: use qemu_img() in has_working_luks()
iotests: remove remaining calls to qemu_img_pipe()
iotests/149: Remove qemu_img_pipe() call
iotests: replace unchecked calls to qemu_img_pipe()
iotests: change supports_quorum to use qemu_img
iotests: add qemu_img_map() function
iotests/remove-bitmap-from-backing: use qemu_img_info()
iotests: add qemu_img_info()
iotests: use qemu_img_json() when applicable
iotests: add qemu_img_json()
iotests: fortify compare_images() against crashes
iotests: make qemu_img raise on non-zero rc by default
iotests: Remove explicit checks for qemu_img() == 0
python/utils: add VerboseProcessError
python/utils: add add_visual_margin() text decoration utility
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: remove remaining calls to qemu_img_pipe()

As part of moving all python iotest invocations of qemu-img onto a
single qemu_img() implementation, remove a few lingering uses of
qemu_img_pipe()

iotests: remove remaining calls to qemu_img_pipe()

As part of moving all python iotest invocations of qemu-img onto a
single qemu_img() implementation, remove a few lingering uses of
qemu_img_pipe() from outside of iotests.py itself.

Several cases here rely on the knowledge that qemu_img_pipe() suppresses
*all* output on a successful case when the command being issued is
'create'.

065: This call's output is inspected, but it appears as if it's expected
to succeed. Replace this call with the checked qemu_img() variant
instead to get better diagnostics if/when qemu-img itself fails.

237: "create" call output isn't actually logged. Use qemu_img_create()
instead, which checks the return code. Remove the empty lines from
the test output.

296: Two calls;
-create: Expected to succeed. Like other create calls, the output
isn't actually logged. Switch to a checked variant
(qemu_img_create) instead. The output for this test is
a mixture of both test styles, so actually replace the
blank line for readability.
-amend: This is expected to fail. Log the output.

After this patch, the only uses of qemu_img_pipe are internal to
iotests.py and will be removed in subsequent patches.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220321201618.903471-15-jsnow@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration bl

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration block node permissions
- iotests fixes (051 and mirror-ready-cancel-error)
- Remove an outdated comment

# gpg: Signature made Tue 01 Feb 2022 13:34:54 GMT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF

* remotes/hreitz-gitlab/tags/pull-block-2022-02-01: (24 commits)
block.h: remove outdated comment
iotests/migration-permissions: New test
block-backend: Retain permissions after migration
iotests: declare lack of support for compresion_type in IMGOPTS
iotest 214: explicit compression type
iotests 60: more accurate set dirty bit in qcow2 header
iotests: bash tests: filter compression type
iotest 39: use _qcow2_dump_header
iotests: massive use _qcow2_dump_header
iotests/common.rc: introduce _qcow2_dump_header helper
qcow2: simple case support for downgrading of qcow2 images with zstd
iotest 302: use img_info_log() helper
iotests.py: filter compression type out
iotests.py: filter out successful output of qemu-img create
iotest 065: explicit compression type
iotest 303: explicit compression type
iotests.py: rewrite default luks support in qemu_img
iotests: drop qemu_img_verbose() helper
iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd'
iotests: specify some unsupported_imgopts for python iotests
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests.py: filter out successful output of qemu-img create

The only "feature" of this "Formatting ..." line is that we have to
update it every time we add new option. Let's drop it.

Signed-off-by:

iotests.py: filter out successful output of qemu-img create

The only "feature" of this "Formatting ..." line is that we have to
update it every time we add new option. Let's drop it.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20211223160144.1097696-10-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2021-10-27' into staging

QAPI patches patches for 2021-10-27

# gpg: Signature made Wed 27 Oct 2021 08:21:54 AM PDT
# gpg:

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2021-10-27' into staging

QAPI patches patches for 2021-10-27

# gpg: Signature made Wed 27 Oct 2021 08:21:54 AM PDT
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]

* remotes/armbru/tags/pull-qapi-2021-10-27:
qapi: Implement deprecated-input={reject,crash} for enum values
qapi: Move compat policy from QObject to generic visitor
qapi: Add feature flags to enum members
qapi: Enable enum member introspection to show more than name
qapi: Improve input_type_enum()'s error message

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

qapi: Improve input_type_enum()'s error message

The error message claims the parameter is invalid:

$ qemu-system-x86_64 -object qom-type=nonexistent
qemu-system-x86_64: -object qom-type=non

qapi: Improve input_type_enum()'s error message

The error message claims the parameter is invalid:

$ qemu-system-x86_64 -object qom-type=nonexistent
qemu-system-x86_64: -object qom-type=nonexistent: Invalid parameter 'nonexistent'

What's wrong is actually the *value* 'nonexistent'. Improve the
message to

qemu-system-x86_64: -object qom-type=nonexistent: Parameter 'qom-type' does not accept value 'nonexistent'

Fixes: https://gitlab.com/qemu-project/qemu/-/issues/608
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20211020180231.434071-1-armbru@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- qemu-storage-daemon: add --pidfile option
- qemu-storage-daemon: CLI error messages include the op

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- qemu-storage-daemon: add --pidfile option
- qemu-storage-daemon: CLI error messages include the option name now
- vhost-user-blk export: Misc fixes
- docs: Improvements for qemu-storage-daemon documentation
- parallels: load bitmap extension
- backup-top: Don't crash on post-finalize accesses
- Improve error messages related to node-name options
- iotests improvements

# gpg: Signature made Mon 08 Mar 2021 17:01:41 GMT
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (30 commits)
blockdev: Clarify error messages pertaining to 'node-name'
block: Clarify error messages pertaining to 'node-name'
docs: qsd: Explain --export nbd,name=... default
MAINTAINERS: update parallels block driver
iotests: add parallels-read-bitmap test
iotests.py: add unarchive_sample_image() helper
parallels: support bitmap extension for read-only mode
block/parallels: BDRVParallelsState: add cluster_size field
parallels.txt: fix bitmap L1 table description
qcow2-bitmap: make bytes_covered_by_bitmap_cluster() public
block/export: port virtio-blk read/write range check
block/export: port virtio-blk discard/write zeroes input validation
block/export: fix vhost-user-blk export sector number calculation
block/export: use VIRTIO_BLK_SECTOR_BITS
block/export: fix blk_size double byteswap
libqtest: add qtest_remove_abrt_handler()
libqtest: add qtest_kill_qemu()
libqtest: add qtest_socket_server()
vhost-user-blk: fix blkcfg->num_queues endianness
docs: replace insecure /tmp examples in qsd docs
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: Clarify error messages pertaining to 'node-name'

Some error messages contain ambiguous representations of the 'node-name'
parameter. This can be particularly confusing when exchanging QMP
mes

block: Clarify error messages pertaining to 'node-name'

Some error messages contain ambiguous representations of the 'node-name'
parameter. This can be particularly confusing when exchanging QMP
messages (C = client, S = server):

C: {"execute": "block_resize", "arguments": { "device": "my_file", "size": 26843545600 }}
S: {"error": {"class": "GenericError", "desc": "Cannot find device=my_file nor node_name="}}
^^^^^^^^^

This error message suggests one could send a message with a key called
'node_name':

C: {"execute": "block_resize", "arguments": { "node_name": "my_file", "size": 26843545600 }}
^^^^^^^^^

but using the underscore is actually incorrect, the parameter should be
'node-name':

S: {"error": {"class": "GenericError", "desc": "Parameter 'node_name' is unexpected"}}

This behavior was uncovered in bz1651437, but I ended up going down a
rabbit hole looking for other areas where this miscommunication might
occur and changing those accordingly as well.

Fixes: https://bugzilla.redhat.com/1651437
Signed-off-by: Connor Kuehl <ckuehl@redhat.com>
Message-Id: <20210305151929.1947331-2-ckuehl@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- iotests: Fix output of qemu-io related tests
- Don't ignore bdrv_set_aio_context() for nodes with

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- iotests: Fix output of qemu-io related tests
- Don't ignore bdrv_set_aio_context() for nodes with bs->drv = NUL
- vmdk: Set vmdk parent backing_format to vmdk
- qcow2: Preallocation fixes (especially for external data files)
- Add linear-buffer-based APIs (as wrappers around qiov-based ones)
- Various code cleanups and small corner case fixes

# gpg: Signature made Tue 30 Apr 2019 16:35:09 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (27 commits)
block/qed: add missed coroutine_fn markers
iotests: Check that images are in read-only mode after block-commit
commit: Make base read-only if there is an early failure
qemu-img: use buffer-based io
block/stream: use buffer-based io
block/commit: use buffer-based io
block/backup: use buffer-based io
block/parallels: use buffer-based io
block/qed: use buffer-based io
block/qcow: use buffer-based io
block/qcow2: use buffer-based io
block: introduce byte-based io helpers
qcow2: Fix error handling in the compression code
qcow2: Fix qcow2_make_empty() with external data file
qemu-img: Make create hint at protocol options
iotests: Perform the correct test in 082
qcow2: Fix full preallocation with external data file
qcow2: Add errp to preallocate_co()
qcow2: Avoid COW during metadata preallocation
qemu-img: Saner printing of large file sizes
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-img: Saner printing of large file sizes

Disk sizes close to INT64_MAX cause overflow, for some pretty
ridiculous output:

$ ./nbdkit -U - memory size=$((2**63 - 512)) --run 'qemu-img info $nb

qemu-img: Saner printing of large file sizes

Disk sizes close to INT64_MAX cause overflow, for some pretty
ridiculous output:

$ ./nbdkit -U - memory size=$((2**63 - 512)) --run 'qemu-img info $nbd'
image: nbd+unix://?socket=/tmp/nbdkitHSAzNz/socket
file format: raw
virtual size: -8388607T (9223372036854775296 bytes)
disk size: unavailable

But there's no reason to have two separate implementations of integer
to human-readable abbreviation, where one has overflow and stops at
'T', while the other avoids overflow and goes all the way to 'E'. With
this patch, the output now claims 8EiB instead of -8388607T, which
really is the correct rounding of largest file size supported by qemu
(we could go 511 bytes larger if we used byte-accurate sizing instead
of rounding up to the next sector boundary, but that wouldn't change
the human-readable result).

Quite a few iotests need updates to expected output to match.

Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Tested-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Block graph change fixes (avoid loops, cope with non-tree graphs)
- bdrv_set_aio_context() related

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Block graph change fixes (avoid loops, cope with non-tree graphs)
- bdrv_set_aio_context() related fixes
- HMP snapshot commands: Use only tag, not the ID to identify snapshots
- qmeu-img, commit: Error path fixes
- block/nvme: Build fix for gcc 9
- MAINTAINERS updates
- Fix various issues with bdrv_refresh_filename()
- Fix various iotests
- Include LUKS overhead in qemu-img measure for qcow2
- A fix for vmdk's image creation interface

# gpg: Signature made Mon 25 Feb 2019 14:18:15 GMT
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (71 commits)
iotests: Skip 211 on insufficient memory
vmdk: false positive of compat6 with hwversion not set
iotests: add LUKS payload overhead to 178 qemu-img measure test
qcow2: include LUKS payload overhead in qemu-img measure
iotests.py: s/_/-/g on keys in qmp_log()
iotests: Let 045 be run concurrently
iotests: Filter SSH paths
iotests.py: Filter filename in any string value
iotests.py: Add is_str()
iotests: Fix 207 to use QMP filters for qmp_log
iotests: Fix 232 for LUKS
iotests: Remove superfluous rm from 232
iotests: Fix 237 for Python 2.x
iotests: Re-add filename filters
iotests: Test json:{} filenames of internal BDSs
block: BDS options may lack the "driver" option
block/null: Generate filename even with latency-ns
block/curl: Implement bdrv_refresh_filename()
block/curl: Harmonize option defaults
block/nvme: Fix bdrv_refresh_filename()
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'mreitz/tags/pull-block-2019-02-25' into queue-block

Block patches:
- Fix various issues with bdrv_refresh_filename()
- Fix various iotests
- Include LUKS overhead in qe

Merge remote-tracking branch 'mreitz/tags/pull-block-2019-02-25' into queue-block

Block patches:
- Fix various issues with bdrv_refresh_filename()
- Fix various iotests
- Include LUKS overhead in qemu-img measure for qcow2
- A fix for vmdk's image creation interface

# gpg: Signature made Mon Feb 25 15:13:43 2019 CET
# gpg: using RSA key F407DB0061D5CF40
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* mreitz/tags/pull-block-2019-02-25: (45 commits)
iotests: Skip 211 on insufficient memory
vmdk: false positive of compat6 with hwversion not set
iotests: add LUKS payload overhead to 178 qemu-img measure test
qcow2: include LUKS payload overhead in qemu-img measure
iotests.py: s/_/-/g on keys in qmp_log()
iotests: Let 045 be run concurrently
iotests: Filter SSH paths
iotests.py: Filter filename in any string value
iotests.py: Add is_str()
iotests: Fix 207 to use QMP filters for qmp_log
iotests: Fix 232 for LUKS
iotests: Remove superfluous rm from 232
iotests: Fix 237 for Python 2.x
iotests: Re-add filename filters
iotests: Test json:{} filenames of internal BDSs
block: BDS options may lack the "driver" option
block/null: Generate filename even with latency-ns
block/curl: Implement bdrv_refresh_filename()
block/curl: Harmonize option defaults
block/nvme: Fix bdrv_refresh_filename()
...

Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

iotests.py: s/_/-/g on keys in qmp_log()

This follows what qmp() does, so the output will correspond to the
actual QMP command.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 201902101457

iotests.py: s/_/-/g on keys in qmp_log()

This follows what qmp() does, so the output will correspond to the
actual QMP command.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190210145736.1486-11-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- vmdk: Support for blockdev-create
- block: Apply auto-read-only for ro-whitelist drivers
- virtio-

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- vmdk: Support for blockdev-create
- block: Apply auto-read-only for ro-whitelist drivers
- virtio-scsi: Fixes related to attaching/detaching iothreads
- scsi-disk: Fixed erroneously detected multipath setup with multiple
disks created with node-names. Added device_id property.
- block: Fix hangs in synchronous APIs with iothreads
- block: Fix invalidate_cache error path for parent activation
- block-backend, mirror, qcow2, vpc, vdi, qemu-iotests:
Minor fixes and code improvements

# gpg: Signature made Fri 01 Feb 2019 15:23:10 GMT
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (27 commits)
scsi-disk: Add device_id property
scsi-disk: Don't use empty string as device id
qtest.py: Wait for the result of qtest commands
block: Fix invalidate_cache error path for parent activation
iotests/236: fix transaction kwarg order
iotests: Filter second BLOCK_JOB_ERROR from 229
virtio-scsi: Forbid devices with different iothreads sharing a blockdev
scsi-disk: Acquire the AioContext in scsi_*_realize()
virtio-scsi: Move BlockBackend back to the main AioContext on unplug
block: Eliminate the S_1KiB, S_2KiB, ... macros
block: Remove blk_attach_dev_legacy() / legacy_dev code
block: Apply auto-read-only for ro-whitelist drivers
uuid: Make qemu_uuid_bswap() take and return a QemuUUID
block/vdi: Don't take address of fields in packed structs
block/vpc: Don't take address of fields in packed structs
vmdk: Reject excess extents in blockdev-create
iotests: Add VMDK tests for blockdev-create
iotests: Filter cid numbers in VMDK extent info
vmdk: Implement .bdrv_co_create callback
vmdk: Refactor vmdk_create_extent
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

vmdk: Reject excess extents in blockdev-create

Clarify that the number of extents provided in BlockdevCreateOptionsVmdk
must match the number of extents that will actually be used. Providing
more ex

vmdk: Reject excess extents in blockdev-create

Clarify that the number of extents provided in BlockdevCreateOptionsVmdk
must match the number of extents that will actually be used. Providing
more extents will result in an error now.

This requires adapting the test case to provide the right number of
extents.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>

show more ...

iotests: Add VMDK tests for blockdev-create

Signed-off-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>