Documentation/dev-tools/ktap.rst

4 The Kernel Test Anything Protocol (KTAP), version 1
7 TAP, or the Test Anything Protocol is a format for specifying test results used
8 by a number of projects. It's website and specification are found at this `link
11 which don't align with the original TAP specification. Thus, a "Kernel TAP"
16 KTAP test results describe a series of tests (which may be nested: i.e., test
18 lines -- and a final result. The test structure and results are
30 there is a stagnant draft specification for TAP14, KTAP diverges from this in
31 a couple of places (notably the "Subtest" header), which are described where
37 All KTAP-formatted results begin with a "version line" which specifies which
41 - "KTAP version 1"
45 Note that, in KTAP, subtests also begin with a version line, which denotes the
46 start of the nested test results. This differs from TAP14, which uses a
49 While, going forward, "KTAP version 1" should be used by compliant tests, it
56 A test plan provides the number of tests (or subtests) in the KTAP output.
58 Plan lines must follow the format of "1..N" where N is the number of tests or subtests.
68 Test case result lines indicate the final status of a test.
79 have the number 1 and the number then must increase by 1 for each additional
82 The description is a description of the test, generally the name of
83 the test, and can be any string of characters other than # or a
87 must follow a hash sign, "#".
89 A directive is a keyword that indicates a different outcome for a test other
90 than passed and failed. The directive is optional, and consists of a single
91 keyword preceding the diagnostic data. In the event that a parser encounters
92 a directive it doesn't support, it should fall back to the "ok" / "not ok"
97 - "SKIP", which indicates a test was skipped (note the result of the test case
99 - "TODO", which indicates that a test is not expected to pass at the moment,
102 - "XFAIL", which indicates that a test is expected to fail. This is similar
104 - “TIMEOUT”, which indicates a test has timed out (note the result of the test
106 - “ERROR”, which indicates that the execution of a test has failed due to a
110 The diagnostic data is a plain-text field which contains any additional details
112 or failed tests, or a description of missing dependencies for a SKIP result.
114 The diagnostic data field is optional, and results which have neither a
120 	ok 1 test_case_name
126 	not ok 1 test_case_name
132 	ok 1 test # SKIP necessary dependency unavailable
139 	not ok 1 test # TIMEOUT 30 seconds
160 output. As a rule, diagnostic lines regarding a test are directly before the
164 even if they do not start with a "#": this is to capture any other useful
166 that tests always prefix any diagnostic output they have with a "#" character.
176 to the system console or a log file.  Both of these destinations may contain
179 invoked by the test likely is not aware that a test is in progress and
180 thus can not print the message as a diagnostic message.
185 In KTAP, tests can be nested. This is done by having a test include within its
194 Additionally, all lines in a subtest should be indented. One level of
198 "Unknown lines" are not considered to be lines in a subtest and thus are
201 An example of a test with two nested subtests:
205 	KTAP version 1
206 	1..1
207 	  KTAP version 1
208 	  1..2
209 	  ok 1 test_1
212 	not ok 1 example
218 	KTAP version 1
219 	1..2
220 	  KTAP version 1
221 	  1..2
222 	    KTAP version 1
223 	    1..2
224 	    not ok 1 test_1
226 	  not ok 1 test_3
228 	not ok 1 example_test_1
246 nested version line, uses a line of the form
253 	KTAP version 1
254 	1..1
255 	  KTAP version 1
256 	  1..3
257 	    KTAP version 1
258 	    1..1
260 	    ok 1 test_1
261 	  ok 1 example_test_1
262 	    KTAP version 1
263 	    1..2
264 	    ok 1 test_1 # SKIP test_1 skipped
267 	    KTAP version 1
268 	    1..3
269 	    ok 1 test_1
274 	not ok 1 main_test
278 A single test called "main_test", which fails, and has three subtests:
296 rules for "bubbling up" test results: a test fails if any of its subtests fail.
298 makes sense for a test to be marked skipped if _all_ of its subtests have been