Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-10-22-v3' into staging

QAPI patches for 2019-10-22

# gpg: Signature made Tue 22 Oct 2019 15:56:36 BST
# gpg: using RS

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-10-22-v3' into staging

QAPI patches for 2019-10-22

# gpg: Signature made Tue 22 Oct 2019 15:56:36 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2019-10-22-v3:
qapi: Allow introspecting fix for savevm's cooperation with blockdev
tests/qapi-schema: Cover feature documentation comments
tests: qapi: Test 'features' of commands
qapi: Add feature flags to commands
tests/qapi-schema: Tidy up test output indentation
qapi: Clear scripts/qapi/doc.py executable bits again
qapi: Split up scripts/qapi/common.py
qapi: Move gen_enum(), gen_enum_lookup() back to qapi/types.py
qapi: Speed up frontend tests
qapi: Eliminate accidental global frontend state
qapi: Store pragma state in QAPISourceInfo, not global state
qapi: Don't suppress doc generation without pragma doc-required

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Speed up frontend tests

"make check-qapi-schema" takes around 10s user + system time for me.
With -j, it takes a bit over 3s real time. We have worse tests. It's
still annoying when you work

qapi: Speed up frontend tests

"make check-qapi-schema" takes around 10s user + system time for me.
With -j, it takes a bit over 3s real time. We have worse tests. It's
still annoying when you work on the QAPI generator.

Some 1.4s user + system time is consumed by make figuring out what to
do, measured by making a target that does nothing. There's nothing I
can do about that right now. But let's see what we can do about the
other 8s.

Almost 7s are spent running test-qapi.py for every test case, the rest
normalizing and diffing test-qapi.py output. We have 190 test cases.

If I downgrade to python2, it's 4.5s, but python2 is a goner.

Hacking up test-qapi.py to exit(0) without doing anything makes it
only marginally faster. The problem is Python startup overhead.

Our configure puts -B into $(PYTHON). Running without -B is faster:
4.4s.

We could improve the Makefile to run test cases only when the test
case or the generator changed. But I'm after improvement in the case
where the generator changed.

test-qapi.py is designed to be the simplest possible building block
for a shell script to do the complete job (it's actually a Makefile,
not a shell script; no real difference). Python is just not meant for
that. It's for bigger blocks.

Move the post-processing and diffing into test-qapi.py, and make it
capable of testing multiple schema files. Set executable bits while
there.

Running it once per test case now takes slightly longer than 8s. But
running it once for all of them takes under 0.2s.

Messing with the Makefile to run it only on the tests that need
retesting is clearly not worth the bother.

Expected error output changes because the new normalization strips off
$(SRCDIR)/tests/qapi-schema/ instead of just $(SRCDIR)/.

The .exit files go away, because there is no exit status to test
anymore.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-5-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-09-28' into staging

QAPI patches for 2019-09-28

# gpg: Signature made Sat 28 Sep 2019 16:18:13 BST
# gpg: using RSA k

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-09-28' into staging

QAPI patches for 2019-09-28

# gpg: Signature made Sat 28 Sep 2019 16:18:13 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2019-09-28: (27 commits)
qapi: Improve source file read error handling
qapi: Improve reporting of redefinition
qapi: Improve reporting of missing documentation comment
qapi: Eliminate check_keys(), rename check_known_keys()
qapi: Improve reporting of invalid 'if' further
qapi: Avoid redundant definition references in error messages
qapi: Improve reporting of missing / unknown definition keys
qapi: Improve reporting of invalid flags
qapi: Improve reporting of invalid 'if' errors
qapi: Move context-free checking to the proper place
qapi: Move context-sensitive checking to the proper place
qapi: Inline check_name() into check_union()
qapi: Plumb info to the QAPISchemaMember
qapi: Make check_type()'s array case a bit more obvious
qapi: Move check for reserved names out of add_name()
qapi: Report invalid '*' prefix like any other invalid name
qapi: Use check_name_str() where it suffices
qapi: Improve reporting of invalid name errors
qapi: Reorder check_FOO() parameters for consistency
qapi: Improve reporting of member name clashes
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Avoid redundant definition references in error messages

Many error messages refer to the offending definition even though
they're preceded by an "in definition" line. Rephrase them.

Signed-o

qapi: Avoid redundant definition references in error messages

Many error messages refer to the offending definition even though
they're preceded by an "in definition" line. Rephrase them.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190927134639.4284-22-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

qapi: Move check for reserved names out of add_name()

The checks for reserved names are spread far and wide. Move one from
add_name() to new check_defn_name_str(). This is a first step towards
col

qapi: Move check for reserved names out of add_name()

The checks for reserved names are spread far and wide. Move one from
add_name() to new check_defn_name_str(). This is a first step towards
collecting them all in dedicated name checking functions next to
check_name().

While there, drop the quotes around the meta-type in
check_name_str()'s error messages: "'command' uses ... name 'NAME'"
becomes "command uses ... name 'NAME'".

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-13-armbru@redhat.com>

show more ...

qapi: Improve reporting of invalid name errors

Split check_name() into check_name_is_str() and check_name_str(), keep
check_name() as a wrapper.

Move add_name()'s call into its caller check_exprs()

qapi: Improve reporting of invalid name errors

Split check_name() into check_name_is_str() and check_name_str(), keep
check_name() as a wrapper.

Move add_name()'s call into its caller check_exprs(), and inline.

This permits delaying check_name_str() there, so its error message
gains an "in definition" line.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-10-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-03-16' into staging

QAPI patches for 2017-03-16

# gpg: Signature made Thu 16 Mar 2017 06:18:38 GMT
# gpg: using RSA k

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-03-16' into staging

QAPI patches for 2017-03-16

# gpg: Signature made Thu 16 Mar 2017 06:18:38 GMT
# gpg: using RSA key 0x3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2017-03-16: (49 commits)
qapi: Fix a misleading parser error message
qapi: Make pylint a bit happier
qapi: Drop unused .check_clash() parameter schema
qapi: union_types is a list used like a dict, make it one
qapi: struct_types is a list used like a dict, make it one
qapi: enum_types is a list used like a dict, make it one
qapi: Factor add_name() calls out of the meta conditional
qapi: Simplify what gets stored in enum_types
qapi: Drop unused variable events
qapi: Eliminate check_docs() and drop QAPIDoc.expr
qapi: Fix detection of bogus member documentation
tests/qapi-schema: Improve coverage of bogus member docs
tests/qapi-schema: Rename doc-bad-args to doc-bad-command-arg
qapi: Move empty doc section checking to doc parser
qapi: Improve error message on @NAME: in free-form doc
qapi: Move detection of doc / expression name mismatch
qapi: Fix detection of doc / expression mismatch
tests/qapi-schema: Improve doc / expression mismatch coverage
qapi2texi: Use category "Object" for all object types
qapi2texi: Generate descriptions for simple union tags
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Back out doc comments added just to please qapi.py

This reverts commit 3313b61's changes to tests/qapi-schema/, except
for tests/qapi-schema/doc-*.

We could keep some of these doc comments to

qapi: Back out doc comments added just to please qapi.py

This reverts commit 3313b61's changes to tests/qapi-schema/, except
for tests/qapi-schema/doc-*.

We could keep some of these doc comments to serve as positive test
cases. However, they don't actually add to what we get from doc
comment use in actual schemas, as we we don't test output matches
expectations, and don't systematically cover doc comment features.
Proper positive test coverage would be nice.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <1489582656-31133-4-git-send-email-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-01-16' into staging

QAPI patches for 2017-01-16

# gpg: Signature made Mon 16 Jan 2017 09:26:49 GMT
# gpg: using RSA k

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-01-16' into staging

QAPI patches for 2017-01-16

# gpg: Signature made Mon 16 Jan 2017 09:26:49 GMT
# gpg: using RSA key 0x3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2017-01-16: (180 commits)
build-sys: add qapi doc generation targets
build-sys: add txt documentation rules
build-sys: use a generic TEXI2MAN rule
build-sys: remove dvi doc generation
build-sys: use --no-split for info
docs: add qemu logo to pdf
qapi: add qapi2texi script
qmp-events: move 'MIGRATION_PASS' doc to schema
qmp-events: move 'DUMP_COMPLETED' doc to schema
qmp-events: move 'MEM_UNPLUG_ERROR' doc to schema
qmp-events: move 'VSERPORT_CHANGE' doc to schema
qmp-events: move 'QUORUM_REPORT_BAD' doc to schema
qmp-events: move 'QUORUM_FAILURE' doc to schema
qmp-events: move 'GUEST_PANICKED' doc to schema
qmp-events: move 'BALLOON_CHANGE' doc to schema
qmp-events: move 'ACPI_DEVICE_OST' doc to schema
qmp-events: move 'MIGRATION' doc to schema
qmp-events: move 'SPICE_MIGRATE_COMPLETED' doc to schema
qmp-events: move 'SPICE_DISCONNECTED' doc to schema
qmp-events: move 'SPICE_INITIALIZED' doc to schema
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: add qapi2texi script

As the name suggests, the qapi2texi script converts JSON QAPI
description into a texi file suitable for different target
formats (info/man/txt/pdf/html...).

It parses the

qapi: add qapi2texi script

As the name suggests, the qapi2texi script converts JSON QAPI
description into a texi file suitable for different target
formats (info/man/txt/pdf/html...).

It parses the following kind of blocks:

Free-form:

##
# = Section
# == Subsection
#
# Some text foo with *emphasis*
# 1. with a list
# 2. like that
#
# And some code:
# | $ echo foo
# | -> do this
# | <- get that
#
##

Symbol description:

##
# @symbol:
#
# Symbol body ditto ergo sum. Foo bar
# baz ding.
#
# @param1: the frob to frobnicate
# @param2: #optional how hard to frobnicate
#
# Returns: the frobnicated frob.
# If frob isn't frobnicatable, GenericError.
#
# Since: version
# Notes: notes, comments can have
# - itemized list
# - like this
#
# Example:
#
# -> { "execute": "quit" }
# <- { "return": {} }
#
##

That's roughly following the following EBNF grammar:

api_comment = "##\n" comment "##\n"
comment = freeform_comment | symbol_comment
freeform_comment = { "# " text "\n" | "#\n" }
symbol_comment = "# @" name ":\n" { member | tag_section | freeform_comment }
member = "# @" name ':' [ text ] "\n" freeform_comment
tag_section = "# " ( "Returns:", "Since:", "Note:", "Notes:", "Example:", "Examples:" ) [ text ] "\n" freeform_comment
text = free text with markup

Note that the grammar is ambiguous: a line "# @foo:\n" can be parsed
both as freeform_comment and as symbol_comment. The actual parser
recognizes symbol_comment.

See docs/qapi-code-gen.txt for more details.

Deficiencies and limitations:
- the generated QMP documentation includes internal types
- union type support is lacking
- type information is lacking in generated documentation
- doc comment error message positions are imprecise, they point
to the beginning of the comment.
- a few minor issues, all marked TODO/FIXME in the code

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20170113144135.5150-16-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[test-qapi.py tweaked to avoid trailing empty lines in .out]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2015-11-02' into staging

QAPI patches

# gpg: Signature made Mon 02 Nov 2015 09:07:23 GMT using RSA key ID EB918653
# gpg: Good signature

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2015-11-02' into staging

QAPI patches

# gpg: Signature made Mon 02 Nov 2015 09:07:23 GMT using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qapi-2015-11-02: (25 commits)
qapi: Simplify gen_struct_field()
qapi: Reserve 'u' member name
qapi: Finish converting to new qapi union layout
tpm: Convert to new qapi union layout
memory: Convert to new qapi union layout
input: Convert to new qapi union layout
char: Convert to new qapi union layout
net: Convert to new qapi union layout
sockets: Convert to new qapi union layout
block: Convert to new qapi union layout
tests: Convert to new qapi union layout
qapi-visit: Convert to new qapi union layout
qapi: Start converting to new qapi union layout
qapi-visit: Remove redundant functions for flat union base
qapi: Unbox base members
qapi: Prefer typesafe upcasts to qapi base classes
qapi-types: Refactor base fields output
qapi-visit: Split off visit_type_FOO_fields forward decl
vnc: Hoist allocation of VncBasicInfo to callers
qapi: Reserve 'q_*' and 'has_*' member names
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Reserve 'q_*' and 'has_*' member names

c_name() produces names starting with 'q_' when protecting a
dictionary member name that would fail to directly compile, but
in doing so can cause clashe

qapi: Reserve 'q_*' and 'has_*' member names

c_name() produces names starting with 'q_' when protecting a
dictionary member name that would fail to directly compile, but
in doing so can cause clashes with any member name already
beginning with 'q-' or 'q_'. Likewise, we create a C name 'has_'
for any optional member that can clash with any member name
beginning with 'has-' or 'has_'.

Technically, rather than blindly reserving the namespace,
we could try to complain about user names only when an actual
collision occurs, or even teach c_name() how to munge names
to avoid collisions. But it is not trivial, especially when
collisions can occur across multiple types (such as via
inheritance or flat unions). Besides, no existing .json
files are trying to use these names. So it's easier to just
outright forbid the potential for collision. We can always
relax things in the future if a real need arises for QMP to
express member names that have been forbidden here.

'has_' only has to be reserved for struct/union member names,
while 'q_' is reserved everywhere (matching the fact that
only members can be optional, while we use c_name() for munging
both members and entities). Note that we could relax 'q_'
restrictions on entities independently from member names; for
example, c_name('qmp_' + 'unix') would result in a different
function name than our current 'qmp_' + c_name('unix').

Update and add tests to cover the new error messages.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1445898903-12082-6-git-send-email-eblake@redhat.com>
[Consistently pass protect=False to c_name(); commit message tweaked
slightly]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

tests/qapi-schema: Test for reserved names, empty struct

Add some testsuite coverage to ensure future patches are on
the right track:

Our current C representation of qapi arrays is done by appendin

tests/qapi-schema: Test for reserved names, empty struct

Add some testsuite coverage to ensure future patches are on
the right track:

Our current C representation of qapi arrays is done by appending
'List' to the element name; but we are not preventing the
creation of an object type with the same name. Add
reserved-type-list.json to test this. Then rename
enum-union-clash.json to reserved-type-kind.json to cover the
reservation that we DO detect, and shorten it to match the fact
that the name is reserved even if there is no clash.

We are failing to detect a collision between a dictionary member
and the implicit 'has_*' flag for another optional member. The
easiest fix would be for a future patch to reserve the entire
"has[-_]" namespace for member names (the collision is also
possible for branch names within flat unions, but only as long as
branch names can collide with (non-variant) members; however,
since future patches are about to remove that, it is not worth
testing here). Add reserved-member-has.json to test this.

A similar collision exists between a dictionary member where
c_name() munges what might otherwise be a reserved name to start
with 'q_', and another member explicitly starts with "q[-_]".
Again, the easiest solution for a future patch will be reserving
the entire namespace, but here for commands as well as members.
Add reserved-member-q.json and reserved-command-q.json to test
this; separate tests since arguably our munging of command 'unix'
to 'qmp_q_unix()' could be done without a q_, which is different
than the munging of a member 'unix' to 'foo.q_unix'.

Finally, our testsuite does not have any compilation coverage
of struct inheritance with empty qapi structs. Update
qapi-schema-test.json to test this.

Note that there is currently no technical reason to forbid type
name patterns from member names, or member name patterns from
types, since the two are not in the same namespace in C and
won't collide; but it's not worth adding positive tests of these
corner cases at this time, especially while there is other churn
pending in patches that rearrange which collisions actually
happen.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1445898903-12082-2-git-send-email-eblake@redhat.com>
[Commit message tweaked slightly]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...