Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2016-03-18' into staging

QAPI patches for 2016-03-18

# gpg: Signature made Fri 18 Mar 2016 09:54:57 GMT using RSA key ID EB918653
# gpg:

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2016-03-18' into staging

QAPI patches for 2016-03-18

# gpg: Signature made Fri 18 Mar 2016 09:54:57 GMT using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qapi-2016-03-18:
qapi: Use anonymous bases in QMP flat unions
qapi: Allow anonymous base for flat union
qapi: Make BlockdevOptions doc example closer to reality
qapi: Don't special-case simple union wrappers
qapi: Drop unused c_null()
qapi: Inline gen_visit_members() into lone caller
qapi-commands: Inline single-use helpers of gen_marshal()
qapi-commands: Utilize implicit struct visits
qapi-event: Utilize implicit struct visits
qapi-event: Drop qmp_output_get_qobject() null check
qapi: Emit implicit structs in generated C
qapi: Adjust names of implicit types
qapi: Make c_type() more OO-like
qapi: Fix command with named empty argument type
qapi: Assert in places where variants are not handled

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Adjust names of implicit types

The original choice of ':obj-' as the prefix for implicit types
made it obvious that we weren't going to clash with any user-defined
names, which cannot contain

qapi: Adjust names of implicit types

The original choice of ':obj-' as the prefix for implicit types
made it obvious that we weren't going to clash with any user-defined
names, which cannot contain ':'. But now we want to create structs
for implicit types, to get rid of special cases in the generators,
and our use of ':' in implicit names needs a tweak to produce valid
C code.

We could transliterate ':' to '_', except that C99 mandates that
"identifiers that begin with an underscore are always reserved for
use as identifiers with file scope in both the ordinary and tag name
spaces". So it's time to change our naming convention: we can
instead use the 'q_' prefix that we reserved for ourselves back in
commit 9fb081e0. Technically, since we aren't planning on exposing
the empty type in generated code, we could keep the name ':empty',
but renaming it to 'q_empty' makes the check for startswith('q_')
cover all implicit types, whether or not code is generated for them.

As long as we don't declare 'empty' or 'obj' ticklish, it shouldn't
clash with c_name() prepending 'q_' to the user's ticklish names.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1458254921-17042-5-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2015-12-17' into staging

QAPI patches for 2015-12-17

# gpg: Signature made Thu 17 Dec 2015 07:33:41 GMT using RSA key ID EB918653
# gpg:

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2015-12-17' into staging

QAPI patches for 2015-12-17

# gpg: Signature made Thu 17 Dec 2015 07:33:41 GMT using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qapi-2015-12-17: (40 commits)
qapi: Detect base class loops
qapi: Move duplicate collision checks to schema check()
qapi: Enforce (or whitelist) case conventions on qapi members
qapi: Track enum values by QAPISchemaMember, not string
qapi: Prepare new QAPISchemaMember base class
qapi: Shorter visits of optional fields
qapi: Simplify visits of optional fields
qapi: Fix alternates that accept 'number' but not 'int'
qapi: Inline _make_implicit_tag()
qapi-types: Drop unnedeed ._fwdefn
qapi: Simplify visiting of alternate types
qapi: Convert QType into QAPI built-in enum type
qobject: Rename qtype_code to QType
qobject: Simplify QObject
qapi: Change munging of CamelCase enum values
qapi: Add alias for ErrorClass
cpu: Convert CpuInfo into flat union
qapi: Remove obsolete tests for MAX collision
qapi: Don't let implicit enum MAX member collide
qapi: Tighten the regex on valid names
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Convert QType into QAPI built-in enum type

What's more meta than using qapi to define qapi? :)

Convert QType into a full-fledged[*] builtin qapi enum type, so
that a subsequent patch can then

qapi: Convert QType into QAPI built-in enum type

What's more meta than using qapi to define qapi? :)

Convert QType into a full-fledged[*] builtin qapi enum type, so
that a subsequent patch can then use it as the discriminator
type of qapi alternate types. Fortunately, the judicious use of
'prefix' in the qapi definition avoids churn to the spelling of
the enum constants.

To avoid circular definitions, we have to flip the order of
inclusion between "qobject.h" vs. "qapi-types.h". Back in commit
28770e0, we had the latter include the former, so that we could
use 'QObject *' for our implementation of 'any'. But that usage
also works with only a forward declaration, whereas the
definition of QObject requires QType to be a complete type.

[*] The type has to be builtin, rather than declared in
qapi/common.json, because we want to use it for alternates even
when common.json is not included. But since it is the first
builtin enum type, we have to add special cases to qapi-types
and qapi-visit to only emit definitions once, even when two
qapi files are being compiled into the same binary (the way we
already handled builtin list types like 'intList'). We may
need to revisit how multiple qapi files share common types,
but that's a project for another day.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1449033659-25497-4-git-send-email-eblake@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2015-09-21' into staging

qapi: QMP introspection

# gpg: Signature made Mon 21 Sep 2015 08:59:17 BST using RSA key ID EB918653
# gpg: Good

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2015-09-21' into staging

qapi: QMP introspection

# gpg: Signature made Mon 21 Sep 2015 08:59:17 BST using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qapi-2015-09-21: (26 commits)
qapi-introspect: Hide type names
qapi: New QMP command query-qmp-schema for QMP introspection
qapi: Pseudo-type '**' is now unused, drop it
qapi-schema: Fix up misleading specification of netdev_add
qom: Don't use 'gen': false for qom-get, qom-set, object-add
qapi: Introduce a first class 'any' type
qapi: Make output visitor return qnull() instead of NULL
qapi: Improve built-in type documentation
qapi-commands: De-duplicate output marshaling functions
qapi: De-duplicate parameter list generation
qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()
qapi-commands: Rearrange code
qapi-visit: Rearrange code a bit
qapi: Clean up after recent conversions to QAPISchemaVisitor
qapi: Replace dirty is_c_ptr() by method c_null()
qapi-event: Convert to QAPISchemaVisitor, fixing data with base
qapi-event: Eliminate global variable event_enum_value
qapi: De-duplicate enum code generation
qapi-commands: Convert to QAPISchemaVisitor
qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: New QMP command query-qmp-schema for QMP introspection

qapi/introspect.json defines the introspection schema. It's designed
for QMP introspection, but should do for similar uses, such as QGA.

qapi: New QMP command query-qmp-schema for QMP introspection

qapi/introspect.json defines the introspection schema. It's designed
for QMP introspection, but should do for similar uses, such as QGA.

The introspection schema does not reflect all the rules and
restrictions that apply to QAPI schemata. A valid QAPI schema has an
introspection value conforming to the introspection schema, but the
converse is not true.

Introspection lowers away a number of schema details, and makes
implicit things explicit:

* The built-in types are declared with their JSON type.

All integer types are mapped to 'int', because how many bits we use
internally is an implementation detail. It could be pressed into
external interface service as very approximate range information,
but that's a bad idea. If we need range information, we better do
it properly.

* Implicit type definitions are made explicit, and given
auto-generated names:

- Array types, named by appending "List" to the name of their
element type, like in generated C.

- The enumeration types implicitly defined by simple union types,
named by appending "Kind" to the name of their simple union type,
like in generated C.

- Types that don't occur in generated C. Their names start with ':'
so they don't clash with the user's names.

* All type references are by name.

* The struct and union types are generalized into an object type.

* Base types are flattened.

* Commands take a single argument and return a single result.

Dictionary argument or list result is an implicit type definition.

The empty object type is used when a command takes no arguments or
produces no results.

The argument is always of object type, but the introspection schema
doesn't reflect that.

The 'gen': false directive is omitted as implementation detail.

The 'success-response' directive is omitted as well for now, even
though it's not an implementation detail, because it's not used by
QMP.

* Events carry a single data value.

Implicit type definition and empty object type use, just like for
commands.

The value is of object type, but the introspection schema doesn't
reflect that.

* Types not used by commands or events are omitted.

Indirect use counts as use.

* Optional members have a default, which can only be null right now

Instead of a mandatory "optional" flag, we have an optional default.
No default means mandatory, default null means optional without
default value. Non-null is available for optional with default
(possible future extension).

* Clients should *not* look up types by name, because type names are
not ABI. Look up the command or event you're interested in, then
follow the references.

TODO Should we hide the type names to eliminate the temptation?

New generator scripts/qapi-introspect.py computes an introspection
value for its input, and generates a C variable holding it.

It can generate awfully long lines. Marked TODO.

A new test-qmp-input-visitor test case feeds its result for both
tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a
QmpInputVisitor to verify it actually conforms to the schema.

New QMP command query-qmp-schema takes its return value from that
variable. Its reply is some 85KiBytes for me right now.

If this turns out to be too much, we have a couple of options:

* We can use shorter names in the JSON. Not the QMP style.

* Optionally return the sub-schema for commands and events given as
arguments.

Right now qmp_query_schema() sends the string literal computed by
qmp-introspect.py. To compute sub-schema at run time, we'd have to
duplicate parts of qapi-introspect.py in C. Unattractive.

* Let clients cache the output of query-qmp-schema.

It changes only on QEMU upgrades, i.e. rarely. Provide a command
query-qmp-schema-hash. Clients can have a cache indexed by hash,
and re-query the schema only when they don't have it cached. Even
simpler: put the hash in the QMP greeting.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

tests/qapi-schema: Convert test harness to QAPISchemaVisitor

The old code prints the result of parsing (list of expression
dictionaries), and partial results of semantic analysis (list of enum
dicti

tests/qapi-schema: Convert test harness to QAPISchemaVisitor

The old code prints the result of parsing (list of expression
dictionaries), and partial results of semantic analysis (list of enum
dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends
are going to use. Built-in and array types are omitted, because
they're boring.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qmp-2015-05-05' into staging

drop qapi nested structs

# gpg: Signature made Tue May 5 17:40:40 2015 BST using RSA key ID EB918653
# gpg: Good

Merge remote-tracking branch 'remotes/armbru/tags/pull-qmp-2015-05-05' into staging

drop qapi nested structs

# gpg: Signature made Tue May 5 17:40:40 2015 BST using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qmp-2015-05-05: (40 commits)
qapi: Check for member name conflicts with a base class
qapi: Support (subset of) \u escapes in strings
qapi: Tweak doc references to QMP when QGA is also meant
qapi: Drop dead visitor code related to nested structs
qapi: Drop support for inline nested types
qapi: Drop inline nested structs in query-pci
qapi: Drop inline nested struct in query-version
qapi: Drop tests for inline nested structs
qapi: Merge UserDefTwo and UserDefNested in tests
qapi: Forbid 'type' in schema
qapi: Use 'struct' instead of 'type' in schema
qapi: Document 'struct' metatype
qapi: Prefer 'struct' over 'type' in generator
qapi: More rigorous checking for type safety bypass
qapi: Whitelist commands that don't return dictionary
qapi: Require valid names
qapi: More rigourous checking of types
qapi: Add some type check tests
qapi: Unify type bypass and add tests
qapi: Allow true, false and null in schema json
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Better error messages for bad expressions

The previous commit demonstrated that the generator overlooked some
fairly basic broken expressions:
- missing metataype
- metatype key has a non-stri

qapi: Better error messages for bad expressions

The previous commit demonstrated that the generator overlooked some
fairly basic broken expressions:
- missing metataype
- metatype key has a non-string value
- unknown key in relation to the metatype
- conflicting metatype (this patch treats the second metatype as an
unknown key of the first key visited, which is not necessarily the
first key the user typed)

Add check_keys to cover these situations, and update testcases to
match. A couple other tests (enum-missing-data, indented-expr) had
to change since the validation added here occurs so early.
Conversely, changes to ident-with-escape results show that we still
have problems where our handling of escape sequences differs from
true JSON, which will matter down the road if we allow arbitrary
default string values for optional parameters (but for now is not
too bad, as we currently can avoid unicode escaping as we don't
need to represent anything beyond C identifier material).

While valid .json files won't trigger any of these cases, we might
as well be nicer to developers that make a typo while trying to add
new QAPI code.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

qapi.py: Restructure lexer and parser

The parser has a rather unorthodox structure:

Until EOF:

Read a section:

Generator function get_expr() yields one section after the

qapi.py: Restructure lexer and parser

The parser has a rather unorthodox structure:

Until EOF:

Read a section:

Generator function get_expr() yields one section after the
other, as a string. An unindented, non-empty line that
isn't a comment starts a new section.

Lexing:

Split section into a list of tokens (strings), with help
of generator function tokenize().

Parsing:

Parse the first expression from the list of tokens, with
parse(), throw away any remaining tokens.

In parse_schema(): record value of an enum, union or
struct key (if any) in the appropriate global table,
append expression to the list of expressions.

Return list of expressions.

Known issues:

(1) Indentation is significant, unlike in real JSON.

(2) Neither lexer nor parser have any idea of source positions. Error
reporting is hard, let's go shopping.

(3) The one error we bother to detect, we "report" via raise.

(4) The lexer silently ignores invalid characters.

(5) If everything in a section gets ignored, the parser crashes.

(6) The lexer treats a string containing a structural character exactly
like the structural character.

(7) Tokens trailing the first expression in a section are silently
ignored.

(8) The parser accepts any token in place of a colon.

(9) The parser treats comma as optional.

(10) parse() crashes on unexpected EOF.

(11) parse_schema() crashes when a section's expression isn't a JSON
object.

Replace this piece of original art by a thoroughly unoriginal design.
Takes care of (1), (2), (5), (6) and (7), and lays the groundwork for
addressing the others. Generated source files remain unchanged.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1374939721-7876-4-git-send-email-armbru@redhat.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

show more ...

tests: QAPI schema parser tests

The parser handles erroneous input badly. To be improved shortly.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-id: 1374939721-7876-2-git-send-email-

tests: QAPI schema parser tests

The parser handles erroneous input badly. To be improved shortly.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-id: 1374939721-7876-2-git-send-email-armbru@redhat.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

show more ...