Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- iotests: Fix output of qemu-io related tests
- Don't ignore bdrv_set_aio_context() for nodes with

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- iotests: Fix output of qemu-io related tests
- Don't ignore bdrv_set_aio_context() for nodes with bs->drv = NUL
- vmdk: Set vmdk parent backing_format to vmdk
- qcow2: Preallocation fixes (especially for external data files)
- Add linear-buffer-based APIs (as wrappers around qiov-based ones)
- Various code cleanups and small corner case fixes

# gpg: Signature made Tue 30 Apr 2019 16:35:09 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (27 commits)
block/qed: add missed coroutine_fn markers
iotests: Check that images are in read-only mode after block-commit
commit: Make base read-only if there is an early failure
qemu-img: use buffer-based io
block/stream: use buffer-based io
block/commit: use buffer-based io
block/backup: use buffer-based io
block/parallels: use buffer-based io
block/qed: use buffer-based io
block/qcow: use buffer-based io
block/qcow2: use buffer-based io
block: introduce byte-based io helpers
qcow2: Fix error handling in the compression code
qcow2: Fix qcow2_make_empty() with external data file
qemu-img: Make create hint at protocol options
iotests: Perform the correct test in 082
qcow2: Fix full preallocation with external data file
qcow2: Add errp to preallocate_co()
qcow2: Avoid COW during metadata preallocation
qemu-img: Saner printing of large file sizes
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-img: Saner printing of large file sizes

Disk sizes close to INT64_MAX cause overflow, for some pretty
ridiculous output:

$ ./nbdkit -U - memory size=$((2**63 - 512)) --run 'qemu-img info $nb

qemu-img: Saner printing of large file sizes

Disk sizes close to INT64_MAX cause overflow, for some pretty
ridiculous output:

$ ./nbdkit -U - memory size=$((2**63 - 512)) --run 'qemu-img info $nbd'
image: nbd+unix://?socket=/tmp/nbdkitHSAzNz/socket
file format: raw
virtual size: -8388607T (9223372036854775296 bytes)
disk size: unavailable

But there's no reason to have two separate implementations of integer
to human-readable abbreviation, where one has overflow and stops at
'T', while the other avoids overflow and goes all the way to 'E'. With
this patch, the output now claims 8EiB instead of -8388607T, which
really is the correct rounding of largest file size supported by qemu
(we could go 511 bytes larger if we used byte-accurate sizing instead
of rounding up to the next sector boundary, but that wouldn't change
the human-readable result).

Quite a few iotests need updates to expected output to match.

Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Tested-by: Max Reitz <mreitz@redhat.com>

show more ...

tests/qemu-iotests: Fix output of qemu-io related tests

One of the recent commits changed the way qemu-io prints out its
errors and warnings - they are now prefixed with the program name.
We've got

tests/qemu-iotests: Fix output of qemu-io related tests

One of the recent commits changed the way qemu-io prints out its
errors and warnings - they are now prefixed with the program name.
We've got to adapt the iotests accordingly to prevent that they
are failing.

Fixes: 99e98d7c9fc1a1639fad ("qemu-io: Use error_[gs]et_progname()")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2017-02-12' into staging

Block patches

# gpg: Signature made Sun 12 Feb 2017 01:26:20 GMT
# gpg: using RSA key 0xF407DB

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2017-02-12' into staging

Block patches

# gpg: Signature made Sun 12 Feb 2017 01:26:20 GMT
# gpg: using RSA key 0xF407DB0061D5CF40
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2017-02-12: (21 commits)
qemu-img: Avoid setting ret to unused value in img_convert()
qemu-img: Use qemu_strtoul() rather than raw strtoul()
qemu-io: don't allow I/O operations larger than BDRV_REQUEST_MAX_BYTES
qcow2: Optimize the refcount-block overlap check
qemu-io: Add failure regression tests
qemu-iotests: Add _unsupported_fmt helper
qemu-io: Return non-zero exit code on failure
block/nfs: fix naming of runtime opts
block/nfs: fix NULL pointer dereference in URI parsing
block: bdrv_invalidate_cache: invalidate children first
block/qapi: reduce the execution time of qmp_query_blockstats
block/qapi: reduce the coupling between the bdrv_query_stats and bdrv_query_bds_stats
qemu-iotest: test to lookup protocol-based image with relative backing
qemu-iotests: Don't create fifos / pidfiles with protocol paths
block: check full backing filename when searching protocol filenames
block/vmdk: Fix the endian problem of buf_len and lba
iotests: record separate timings per format,protocol pair
iotests: Fix reference output for 059
qapi: Tweak error message of bdrv_query_image_info
qemu-img: Improve commit invalid base message
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-io: Return non-zero exit code on failure

The result of openfile was not checked, leading to failure deep in the
actual command with confusing error message, and exiting with exit code 0.

Here

qemu-io: Return non-zero exit code on failure

The result of openfile was not checked, leading to failure deep in the
actual command with confusing error message, and exiting with exit code 0.

Here is a simple example - trying to read with the wrong format:

$ touch file
$ qemu-io -f qcow2 -c 'read -P 1 0 1024' file; echo $?
can't open device file: Image is not in qcow2 format
no file open, try 'help open'
0

With this patch, we fail earlier with exit code 1:

$ ./qemu-io -f qcow2 -c 'read -P 1 0 1024' file; echo $?
can't open device file: Image is not in qcow2 format
1

Failing earlier, we don't log this error now:

no file open, try 'help open'

But some tests expected it; the line was removed from the test output.

Signed-off-by: Nir Soffer <nirsof@gmail.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20170201003120.23378-2-nirsof@gmail.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-error-2016-01-13' into staging

Error reporting patches for 2016-01-13

# gpg: Signature made Wed 13 Jan 2016 14:21:48 GMT using RSA key ID EB91

Merge remote-tracking branch 'remotes/armbru/tags/pull-error-2016-01-13' into staging

Error reporting patches for 2016-01-13

# gpg: Signature made Wed 13 Jan 2016 14:21:48 GMT using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-error-2016-01-13: (41 commits)
checkpatch: Detect newlines in error_report and other error functions
error: Consistently name Error * objects err, and not errp
s390/sclp: Simplify control flow in sclp_realize()
hw/s390x: Rename local variables Error *l_err to just err
error: Clean up errors with embedded newlines (again)
vhdx: Fix "log that needs to be replayed" error message
pci-assign: Clean up "Failed to assign" error messages
vmdk: Clean up "Invalid extent lines" error message
vmdk: Clean up control flow in vmdk_parse_extents() a bit
error: Strip trailing '\n' from error string arguments (again)
qemu-io qemu-nbd: Use error_report() etc. instead of fprintf()
migration: Use error_reportf_err() instead of monitor_printf()
spapr: Use error_reportf_err()
error: Use error_prepend() where it makes obvious sense
error: Use error_reportf_err() where it makes obvious sense
error: Don't decorate original error message when adding to it
error: New error_prepend(), error_reportf_err()
test-throttle: Simplify qemu_init_main_loop() error handling
qemu-nbd: Clean up "Failed to load snapshot" error message
block: Clean up "Could not create temporary overlay" error message
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

vhdx: Fix "log that needs to be replayed" error message

The arguments of error_setg_errno() should yield a short error string
without newlines.

Here, we try to append additional help to the error m

vhdx: Fix "log that needs to be replayed" error message

The arguments of error_setg_errno() should yield a short error string
without newlines.

Here, we try to append additional help to the error message by
embedding newlines in the error string. That's nice, but it's doesn't
play nicely with the errno part. tests/qemu-iotests/070.out shows the
resulting mess:

can't open device TEST_DIR/iotest-dirtylog-10G-4M.vhdx: VHDX image file 'TEST_DIR/iotest-dirtylog-10G-4M.vhdx' opened read-only, but contains a log that needs to be replayed. To replay the log, execute:
qemu-img check -r all 'TEST_DIR/iotest-dirtylog-10G-4M.vhdx': Operation not permitted

Switch to error_setg() and error_append_hint(). Result:

can't open device TEST_DIR/iotest-dirtylog-10G-4M.vhdx: VHDX image file 'TEST_DIR/iotest-dirtylog-10G-4M.vhdx' opened read-only, but contains a log that needs to be replayed
To replay the log, run:
qemu-img check -r all 'TEST_DIR/iotest-dirtylog-10G-4M.vhdx'

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <1450452927-8346-21-git-send-email-armbru@redhat.com>

show more ...

qemu-io qemu-nbd: Use error_report() etc. instead of fprintf()

Just three instances left.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id

qemu-io qemu-nbd: Use error_report() etc. instead of fprintf()

Just three instances left.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <1450452927-8346-16-git-send-email-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

# gpg: Signature made Sat 04 Oct 2014 21:24:46 BST using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan H

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

# gpg: Signature made Sat 04 Oct 2014 21:24:46 BST using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg: aka "Stefan Hajnoczi <stefanha@gmail.com>"

* remotes/stefanha/tags/block-pull-request: (23 commits)
blockdev-test: Test device_del after drive_del
blockdev-test: Factor out some common code into helpers
blockdev-test: Simplify by using g_assert_cmpstr()
blockdev-test: Clean up bogus drive_add argument
blockdev-test: Use single rather than double quotes in QMP
drive_del-test: Merge of qdev-monitor-test, blockdev-test
iotests: qemu-img info output for corrupt image
qapi: Add corrupt field to ImageInfoSpecificQCow2
iotests: Use _img_info
util: Emancipate id_wellformed() from QemuOpts
q35/ahci: Pick up -cdrom and -hda options
qtest/bios-tables: Correct Q35 command line
ide: Update ide_drive_get to be HBA agnostic
pc/vl: Add units-per-default-bus property
blockdev: Allow overriding if_max_dev property
blockdev: Orphaned drive search
qemu-iotests: Fix supported cache modes for 052
make check-block: Use default cache modes
Modify qemu_opt_rename to realize renaming all items in opts
vmdk: Fix integer overflow in offset calculation
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Use _img_info

qemu-img info should only be used directly if the format-specific
information or the name of the format is relevant (some tests explicitly
test format-specific information; te

iotests: Use _img_info

qemu-img info should only be used directly if the format-specific
information or the name of the format is relevant (some tests explicitly
test format-specific information; test 082 uses qcow2-specific settings
to test the qemu-img interface); otherwise, tests should always use
_img_info instead.

Test 082 was touched only partially. It does test the qemu-img
interface; however, its invocations of qemu-img info are not real tests
but rather verifications, so if format-specific information is not
important for the test, there is no reason not to use _img_info. In
contrast to directly invoking qemu-img info, "qcow2" is replaced by
"IMGFMT"; but as "qcow2" is only mentioned once in test 082 (in
_supported_fmt), I consider this an improvement.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1412105489-7681-2-git-send-email-mreitz@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Mon 19 May 2014 15:21:14 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Ke

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Mon 19 May 2014 15:21:14 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream: (22 commits)
block: optimize zero writes with bdrv_write_zeroes
blockdev: add a function to parse enum ids from strings
util: add qemu_iovec_is_zero
qcow1: Stricter backing file length check
qcow1: Validate image size (CVE-2014-0223)
qcow1: Validate L2 table size (CVE-2014-0222)
qcow1: Check maximum cluster size
qcow1: Make padding in the header explicit
curl: Add usage documentation
curl: Add sslverify option
curl: Remove broken parsing of options from url
curl: Fix build when curl_multi_socket_action isn't available
qemu-iotests: Fix blkdebug in VM drive in 030
qemu-iotests: Fix core dump suppression in test 039
iotests: Add test for the JSON protocol
block: Allow JSON filenames
check-qdict: Add test for qdict_join()
qdict: Add qdict_join()
block: add test for vhdx image created by Disk2VHD
block: vhdx - account for identical header sections
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: add test for vhdx image created by Disk2VHD

This adds a test for VHDX images created by Microsoft's tool, Disk2VHD.

VHDX images created by this tool have 2 identical header sections, with
id

block: add test for vhdx image created by Disk2VHD

This adds a test for VHDX images created by Microsoft's tool, Disk2VHD.

VHDX images created by this tool have 2 identical header sections, with
identical sequence numbers. This makes sure we detect VHDX images with
identical headers, and do not flag them as corrupt.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Block pull request

# gpg: Signature made Fri 14 Feb 2014 17:26:30 GMT using RSA key ID 81AB73C8
# gpg: Good sign

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Block pull request

# gpg: Signature made Fri 14 Feb 2014 17:26:30 GMT using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg: aka "Stefan Hajnoczi <stefanha@gmail.com>"
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 8695 A8BF D3F9 7CDA AC35 775A 9CA4 ABB3 81AB 73C8

* remotes/stefanha/tags/block-pull-request:
block: Open by reference will try device then node_name.
block: Relax bdrv_lookup_bs constraints.
blockdev: Fix wrong usage of QDECREF causing snapshoted quorum to crash on close.
block: mirror - use local_err to avoid NULL errp
qemu-iotests: Don't run 005 on vmdk split formats
block: qemu-iotests - add vhdx log replay tests for qemu-img
block: qemu-iotests - fix test 070 (vhdx)
block: Don't throw away errno via error_setg
block: Add notes to iSCSI's .bdrv_open and .bdrv_reopen_prepare
blockdev: Remove 'type' parameter from blockdev_init()
sdhci: Drop unnecessary #include

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: qemu-iotests - add vhdx log replay tests for qemu-img

VHDX logs can now be replayed via 'qemu-img check -r all'. Add
tests to verify that the log replay is successful when using qemu-img.

S

block: qemu-iotests - add vhdx log replay tests for qemu-img

VHDX logs can now be replayed via 'qemu-img check -r all'. Add
tests to verify that the log replay is successful when using qemu-img.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

block: qemu-iotests - fix test 070 (vhdx)

VHDX test 070 failed, due to different output from qemu-io / qemu
when opening an image read-only that contains a log file. Filter
the output, and update t

block: qemu-iotests - fix test 070 (vhdx)

VHDX test 070 failed, due to different output from qemu-io / qemu
when opening an image read-only that contains a log file. Filter
the output, and update the expected results to match the correct
output.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

Merge remote-tracking branch 'stefanha/block' into staging

# By Jeff Cody (26) and others
# Via Stefan Hajnoczi
* stefanha/block: (37 commits)
block: Round up total_sectors
block: vhdx qemu-iote

Merge remote-tracking branch 'stefanha/block' into staging

# By Jeff Cody (26) and others
# Via Stefan Hajnoczi
* stefanha/block: (37 commits)
block: Round up total_sectors
block: vhdx qemu-iotest - log replay of data sector
block: qemu-iotests for vhdx, add write test support
block: vhdx - update _make_test_img() to filter out vhdx options
block: vhdx - add .bdrv_create() support
block: vhdx - fix comment typos in header, fix incorrect struct fields
block: vhdx - break out code operations to functions
block: vhdx - move more endian translations to vhdx-endian.c
block: vhdx - remove BAT file offset bit shifting
block: vhdx write support
block: vhdx - add log write support
block: vhdx - add region overlap detection for image files
block: vhdx - log parsing, replay, and flush support
block: vhdx code movement - move vhdx_close() above vhdx_open()
block: vhdx - update log guid in header, and first write tracker
block: vhdx - break endian translation functions out
block: vhdx - log support struct and defines
block: vhdx code movement - VHDXMetadataEntries and BDRVVHDXState to header.
block: vhdx - add header update capability.
block: vhdx - minor comments and typo correction.
...

Message-id: 1383905551-16411-1-git-send-email-stefanha@redhat.com
Signed-off-by: Anthony Liguori <aliguori@amazon.com>

show more ...

block: vhdx qemu-iotest - log replay of data sector

This tests the replay of a data sector in a VHDX image file.

The image file is a 10G dynamic image, with 4MB block size. The
image was created w

block: vhdx qemu-iotest - log replay of data sector

This tests the replay of a data sector in a VHDX image file.

The image file is a 10G dynamic image, with 4MB block size. The
image was created with qemu-img, and the log left unplayed by
modification of the vhdx image format driver.

It was verified under both QEMU and Hyper-V that the image file,
post log replay, matched.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...