diff --git a/index.md b/index.md index de39e91..4cf2647 100644 --- a/index.md +++ b/index.md @@ -49,11 +49,13 @@ TODO review needed - [TAP specification](/tap-specification.html) - [TAP version 13 specification](/tap-version-13-specification.html) +- [TAP version 14 specification](/tap-version-14-specification.html) ## External Resources -- [Wikipedia article on TAP](http://en.wikipedia.org/wiki/Test_Anything_Protocol) -- [The TAP subreddit](http://www.reddit.com/r/testanythingprotocol) +- [Wikipedia article on TAP](http://en.wikipedia.org/wiki/Test_Anything_Protocol) +- [The TAP subreddit](http://www.reddit.com/r/testanythingprotocol) +- [The TAP GitHub Group](https://github.com/TestAnything) {% comment %} TODO review needed diff --git a/tap-version-14-specification.md b/tap-version-14-specification.md new file mode 100644 index 0000000..05636ff --- /dev/null +++ b/tap-version-14-specification.md @@ -0,0 +1,741 @@ +--- +layout: default +title: TAP 14 specification +--- + +# NAME + +TAP14 - The Test Anything Protocol v14 + +## SYNOPSIS + +TAP, the Test Anything Protocol, is a simple text-based interface +between testing modules a test harness. TAP started life as part of +the test harness for Perl but now has implementations in C/C++, +Python, PHP, Perl, JavaScript, MatLab, and probably others by the time +you read this. + +This document describes version 14 of TAP. + +## THE TAP14 FORMAT + +A TAP stream is a UTF-8 encoded text stream representing a set of +tests. It should be parsed line by line. Lines are separated by +`\n`. + +TAP14's general format is: + +``` +TAPDocument -> + Version Plan Body | + Version Body Plan +``` + +The Version is always `TAP version 14`. + +The Body is a collection of lines representing a test set. + +The Plan reports on the number of tests included in the Body. + +Any line matching the pattern `/^\s*#.*$/` is treated as a diagnostic +comment, and MAY be reported to the user or ignored, but MUST be +ignored for the purposes of interpreting test output semantics. + +For example: + +``` +TAP version 14 +1..4 +ok 1 Description # Directive +# Diagnostic + --- + message: 'Failure message' + severity: fail + data: + found: + - 1 + - 3 + - 2 + wanted: + - 1 + - 2 + - 3 + ... +ok 2 Description +ok 3 Description + # Subtest: name of the child test + ok 1 this is ok + ok 2 this is also ok + not ok 3 this is a failure + --- + message: 'Child Failure' + severity: fail + ... + 1..3 +not ok 4 name of the child test +``` + +For example, a test file's output might look like: + +``` +TAP version 14 +1..4 +ok 1 - Input file opened +not ok 2 - First line of the input valid + --- + message: 'First line invalid' + severity: fail + data: + got: 'Flirble' + expect: 'Fnible' + ... +ok 3 - Read the rest of the file +not ok 4 - Summarized correctly # TODO Not written yet + --- + message: "Can't make summary yet" + severity: todo + ... +``` + +## HARNESS BEHAVIOR + +In this document, the "harness" is any program analyzing TAP output. + +A harness interpreting a subprocess MUST only read TAP output from +standard output and not from standard error. + +Lines written to standard output matching `/^(not)? ok\b/` must be +interpreted as test lines. After a test line a block of lines starting +with `/^ {2}---$/` and ending with `/^ {2}\.\.\.$/` will be interpreted as an +inline YAML document providing extended diagnostic information about +the preceding test. + +All other lines must not be considered test output. + +## TESTS LINES AND THE PLAN + +### The Version + +To indicate that this is TAP14 the first non-comment, non-whitespace +line must be: + +``` +TAP version 14 +``` + +A compliant TAP14 Harness SHOULD interpret TAP streams with `TAP +version 13` according to TAP13 specification. + +TAP streams lacking a version number SHOULD be interpreted according to +the TAP12 specification. + +In order to facilitate forward compatibility, a compliant TAP14 +Harness MAY interpret test data according to TAP14, regardless of +stated version number. For example, it may accept subtests in TAP12 +or TAP13 streams, even though these were not supported in those +versions of TAP. + +Any stated version number less than 13 is an error. + +### The Plan + +The Plan tells how many tests will be run, or how many tests have run. +It's a check that the test file hasn't stopped prematurely. + +The Plan MUST appear exactly once, EITHER: + +- the first line of TAP output after the Version, or +- the last line of TAP output prior to the end of the TAP stream. + +The Plan matches the regular expression: `/^1\.\.(0|[1-9][0-9]*)( #.*)?$/`. + +The Plan specifies how many test points are expected in the test +stream. For example, + +``` +1..10 +``` + +means you plan on running 10 tests. This is a safeguard in case your +test file dies silently in the middle of its run. + +In certain instances a test file may not know how many test points it +will ultimately be running. In this case the Plan should be the last +non-diagnostic line in the output. + +A test set missing a Plan MUST be interpreted as a failure by the +harness. + +### The test line + +The core of TAP is the test line. A test file prints one test line +test point executed. There must be at least one test line in TAP +output. Each test line comprises the following elements: + +- ok or not ok + +This tells whether the test point passed or failed. It must be at the +beginning of the line. /^not ok/ indicates a failed test point. /^ok/ +is a successful test point. This is the only mandatory part of the +line. Note that unlike the Directives below, ok and not ok are +case-sensitive. + +- Test number + +TAP expects the ok or not ok to be followed by a test point number, +separated by 1 or more space characters from the `/^(not )?ok/`. + +For example: + +``` +1..3 +ok 1 +not ok 2 +ok 3 +``` + +If there is no number the harness must maintain its own counter until +the script supplies test numbers again. So the following test output + +``` +1..6 +not ok +ok +not ok +ok +ok +``` + +has five tests. The sixth is missing. For example, in Perl, +`Test::Harness` will generate + +``` +FAILED tests 1, 3, 6 +Failed 3/6 tests, 50.00% okay +``` + +- Description + +An optional description may be provided, separated from the ok/not-ok +and test number by one or more space characters, terminated by either +a `#` character, or the end of the line. + +If the Description starts with a hyphen `-` and any amount of +whitespace, it SHOULD be discarded by the Harness. For example, these +test lines should all be identical: + +``` +ok 5 some description +ok 5 - some description +ok 5 - some description # and some comment +``` + +The harness may do whatever it wants with the description. + +- Directive + +The test point may include a directive, following a hash on the test +line. + +There are currently two directives allowed: TODO and SKIP. These are +discussed below. Any other directive is ignored. + +#### Test Line Summary + +To summarize: + +- ok/not ok (required) +- Test number (recommended) +- Description (recommended) +- Directive (only when necessary) + +The regular expression pattern for a test line is: + +``` +/^(not )?ok( +[1-9][0-9]*)?( +(- +)[\u0001-\u0009\u000B-\u000C\u000E-\u0022\u0024-\u00FF]+)?( #(.*))?$/ +``` + +### YAML blocks + +If a test line is immediately followed by a 2-space indented block +beginning with the line `/^ {2}---$/` and ending with the line +`/^ {2}\.\.\.$/` that block will be interpreted as an inline YAML +document. + +The YAML encodes a data structure that provides more detailed +information about the preceding failed test. The YAML document is +indented with 2 spaces to make it visually distinct from the +surrounding test results and to make it easier for the parser to +recover if the trailing `/^ {2}\.\.\.$/` terminator is missing. + +For example: + +``` +not ok 3 Resolve address + --- + message: "Failed with error 'hostname peebles.example.com not found'" + severity: fail + data: + got: + hostname: 'peebles.example.com' + address: ~ + expected: + hostname: 'peebles.example.com' + address: '85.193.201.85' + ... +``` + +The corresponding data structure in JSON would look like this: + +```json +{ + "message": "Failed with error 'hostname peebles.example.com not found'", + "severity": "fail", + "data": { + "got": { + "hostname": "peebles.example.com", + "address": null + }, + "expected": { + "hostname": "peebles.example.com", + "address": "85.193.201.85" + } + } +} +``` + +If a line occurs within the YAML document that is not indented, or if +the TAP stream ends before the `/^ {2}\.\.\.$/` terminator line, or if the +YAML document does not parse as YAML, then it MUST be treated as +non-compliant data, and handled by the Harness the same as any other +non-TAP lines. + +Currently (2015-10-06) the format of the data structure represented by +a YAML block has not been standardized. It is likely that a future +version of TAP will standardize meanings for some of the fields in the +YAML diagnostics. + +In order to adequately interpret and report the data within a test +output's YAML block, a full YAML parser is required. Specifying the +YAML language is outside the scope of this document. Harneses MAY +ignore YAML blocks entirely if the inclusion of a full YAML parser +would be overly difficult. + +## DIRECTIVES + +Directives are special notes that follow a # on the test line. Only +two are currently defined: TODO and SKIP. Note that these two keywords +are not case-sensitive. + +### TODO tests + +If the directive starts with `# TODO`, the test is counted as a todo +test, and the text after TODO is the explanation. + +``` +not ok 13 # TODO bend space and time +``` + +Note that if the TODO has an explanation it must be separated from +TODO by a space. These tests represent a feature to be implemented or +a bug to be fixed and act as something of an executable "things to do" +list. They are not expected to succeed. Should a todo test point begin +succeeding, the harness should report it as a bonus. This indicates +that whatever you were supposed to do has been done and you should +promote this to a normal test point. + +### Skipping tests + +If the directive starts with `# SKIP`, the test is counted as having +been skipped. If the whole test file succeeds, the count of skipped +tests is included in the generated output. The harness should report +the text after # SKIP\S*\s+ as a reason for skipping. + +``` +ok 23 # skip Insufficient flogiston pressure. +``` + +Similarly, one can include an explanation in a Plan line, emitted if +the test file is skipped completely: + +``` +1..0 # SKIP WWW::Mechanize not installed +``` + +## SUBTESTS + +A Subtest contains a block of test points grouped together. This is +useful for the output of test runners that collect several child +processes' TAP streams, and output the result in TAP for another +harness to interpret, or for tests with a predictable number of +blocks, but where each block might not have a predictable number of +tests. + +A Subtest is a TAP14 stream indented 4 spaces. It has the following +format: + +- A comment line with the name of the subtest: `/^ {4}# Subtest: (.*)$/` +- A TAP14 stream, indented 4 spaces. +- A (non-indented) test point indicating the success or failure of the + subtest, with a description matching the Subtest's name. + +For example: + +``` +TAP version 14 +1..2 +ok 1 - About to do a subtest + # Subtest: this is the subtest + ok 1 - This is ok + ok 2 - I am ok with how things are proceeding + not ok 3 - infinite loops # TODO Solve halting problem + not ok 4 - this is just a failure + 1..4 +not ok 2 - this is the subtest + --- + plan: 4 + count: 4 + passing: 2 + todo: 1 + failed: 1 + exitCode: 1 + ... +``` + +If not bailed out, a Subtest MUST be introduced by the indented +`# Subtest: ...` comment, and MUST be followed with a test point whose +description matches the Subtest name. If the Subtest is not +successful, then the Subtest test point MUST be a `not ok` test point. + +A `Bail out!` in a Subtest SHOULD be immediately followed by a +`Bail out!` in the parent test. + +If a Subtest TAP stream does not include a version number, it MUST be +interpreted as a TAP14 stream anyway. This is a backwards-compatible +affordance for pre-TAP14 parsers that complain about seeing a `TAP +version` line multiple times within a TAP stream. + +## OTHER LINES + +### Bail out! + +As an emergency measure a test script can decide that further tests +are useless (e.g. missing dependencies) and testing should stop +immediately. In that case the test script prints the magic words + +``` +Bail out! +``` + +to standard output. Any message after these words must be displayed by +the interpreter as the reason why testing must be stopped, as in + +``` +Bail out! MySQL is not running. +``` + +A Harness SHOULD ignore all lines that come after a bail out. + +### Diagnostics + +Additional information may be put into the testing output on separate +lines. Diagnostic lines should begin with a #, which the harness must +ignore, at least as far as analyzing the test results. The harness is +free, however, to display the diagnostics. + +YAML blocks are intended to replace diagnostics for most purposes but +consumers should maintain backwards compatibility by supporting them. + +### Pragma + +A Pragma line indicates that the parser should switch its behavior in +some way by setting a boolean field. Pragmas may occur anywhere +within the TAP stream, similar to Diagnostic comments. + +A Pragma starts with the word `pragma`, followed by a space character, +and a space-separated list of field names, each prefixed with either a +`+` (true), or a `-` (false). + +All Pragmas are false by default. + +Pragmas set in Subtests do not affect the state of the pragma field in +the parent test set. + +The only Pragma that is currently specified is `strict`. Additional +Pragmas may be added in the future. + +To set the strict pragma, use: + +``` +pragma +strict +``` + +To disable the strict pragma, use: + +``` +pragma -strict +``` + +- `strict` When set, any non-parsing lines are reported to the user as + an error, and cause the test set to be considered a failure. When + disabled, non-parsing lines may be reported to the user in some way, + but do not cause the test set to be considered a failure. + +### Blank Lines + +For the purposes of this specification, a "blank" line is any line +consisting exclusively of zero or more whitespace characters. + +Blank lines within YAML blocks MUST be preserved as part of the YAML +document, because line breaks have semantic meaning in YAML documents. +For example, multiline folded scalar values use `\n\n` to denote line +breaks. + +Blank lines outside of YAML blocks MUST be ignored by the Harness. + +### Anything else + +Any output that is not a version, a plan, a test line, a YAML block, a +subtest, a diagnostic, a pragma, a blank line, or a bail out is +incorrect. + +When the `pragma +strict` is enabled, incorrect test lines SHOULD +result in the test set being considered a failure, even if the test +set is otherwise valid. When `pragma -strict` is set, incorrect test +lines MUST NOT result in the test set being considered a failure if +the test set is otherwise valid. + +How or if the incorrect line is displayed to the user by the Harness +is undefined. + +- `Test::Harness` silently ignores incorrect lines, but will become more + stringent in the future. +- `TAP::Harness` reports TAP syntax errors at the end of a test run. +- `node-tap` prints incorrect lines to standard output, but otherwise + ignores them for the purposes of parsing or determining test + success. + +## EXAMPLES + +All names, places, and events depicted in any example are wholly +fictitious and bear no resemblance to, connection with, or relation to +any real entity. Any such similarity is purely coincidental, +unintentional, and unintended. + +### Common with explanation + +The following TAP listing declares that six tests follow as well as +provides handy feedback as to what the test is about to do. All six +tests pass. + +``` +TAP version 14 +1..6 +# +# Create a new Board and Tile, then place +# the Tile onto the board. +# +ok 1 - The object isa Board +ok 2 - Board size is zero +ok 3 - The object isa Tile +ok 4 - Get possible places to put the Tile +ok 5 - Placing the tile produces no error +ok 6 - Board size is 1 +``` + +### Unknown amount and failures + +This hypothetical test program ensures that a handful of servers are +online and network-accessible. Because it retrieves the hypothetical +servers from a database, it doesn't know exactly how many servers it +will need to ping. Thus, the test count is declared at the bottom +after all the test points have run. Also, two of the tests fail. The +YAML block following each failure gives additional information about +the failure that may be displayed by the harness. + +``` +TAP version 14 +1..2 + # Subtest: get list of servers + 1..3 + ok 1 - connect to database + ok 2 - retrieving list of servers + ok 3 - list of servers is an array of strings +ok 1 - get list of servers + # Subtest: ping servers + ok 1 - pinged diamond + ok 2 - pinged ruby + not ok 3 - pinged saphire + --- + message: 'hostname "saphire" unknown' + severity: fail + ... + ok 4 - pinged onyx + not ok 5 - pinged quartz + --- + message: 'timeout' + severity: fail + ... + ok 6 - pinged gold + 1..6 +not ok 2 - ping servers + --- + servers_offline: + - saphire + - quartz + ... +``` + +### Giving up + +This listing reports that a pile of tests are going to be run. +However, the first test fails, reportedly because a connection to the +database could not be established. The program decided that continuing +was pointless and exited. + +``` +TAP version 14 +1..573 +not ok 1 - database handle +Bail out! Couldn't connect to database. +``` + +This listing reports a pile of tests where a child test bails out. +The Harness MAY continue to run subsequent tests, but in most cases, +SHOULD bubble the bail out up to the parent test. + +``` +TAP version 14 +1..54 + # Subtest: 00-setup.js + not ok 1 - database handle + Bail out! Couldn't connect to database. +Bail out! Coulnd't connect to database. +``` + +### Skipping a few + +The following listing plans on running 5 tests. However, our program +decided to not run tests 2 thru 5 at all. To properly report this, the +tests are marked as being skipped. + +``` +TAP version 14 +1..5 +ok 1 - approved operating system +# $^0 is solaris +ok 2 - # SKIP no /sys directory +ok 3 - # SKIP no /sys directory +ok 4 - # SKIP no /sys directory +ok 5 - # SKIP no /sys directory +``` + +### Skipping everything + +This listing shows that the entire listing is a skip. No tests were run. + +``` +TAP version 14 +1..0 # skip English-to-French translator isn't installed +``` + +## Got spare tuits? + +The following example reports that four tests are run and the last two +tests failed. However, because the failing tests are marked as things +to do later, they are considered successes. Thus, a harness should +report this entire listing as a success. + +``` +TAP version 14 +1..1 + # Subtest: child program + 1..4 + ok 1 - Creating test program + ok 2 - Test program runs, no error + not ok 3 - infinite loop # TODO halting problem unsolved + not ok 4 - infinite loop 2 # TODO halting problem unsolved +ok 1 - child program +``` + +## Creative liberties + +This listing shows an alternate output where the test numbers aren't +provided. The test also reports the state of a ficticious board game +as a YAML block. Finally, the test count is reported at the end. + +``` +TAP version 14 +ok - created Board +ok +ok +ok +ok +ok +ok +ok + --- + message: "Board layout" + severity: comment + dump: + board: + - ' 16G 05C ' + - ' G N C C C G ' + - ' G C + ' + - '10C 01G 03C ' + - 'R N G G A G C C C ' + - ' R G C + ' + - ' 01G 17C 00C ' + - ' G A G G N R R N R ' + - ' G R G ' + ... +ok - board has 7 tiles + starter tile +1..9 +``` + +## CHANGES FROM TAP13 + +- YAML blocks standardized to 2 space indentation +- Subtests specified to behavior of `Test::More` and `node-tap`. +- Normative advice regarding exit code for harness programs +- Examples and usage comments made language-agnostic. +- Clarification of whitespace and hyphens in test lines. +- Clarification of handling of incorrect lines. +- Specification of Pragma lines + +## BUGS + +Please report issues with this specification to the +[TestAnything/Specification +project](https://github.com/TestAnything/Specification). + +## AUTHORS + +The original TAP documentation (of which this is a hacked about +version) was written by Andy Lester, based on the original +`Test::Harness` documentation by Michael Schwern. + +TAP13 documentation written by Andy Armstrong. + +TAP14 documentation written by Isaac Z. Schlueter. + +## ACKNOWLEDGEMENTS + +Thanks to Pete Krawczyk, Paul Johnson, Ian Langworth and Nik Clayton +for help and contributions on this document. The basis for the TAP +format was created by Larry Wall in the original test script for Perl +1. Tim Bunce and Andreas Koenig developed it further with their +modifications to `Test::Harness`. + +As of this writing, the TAP specification is now managed by the +[TestAnything organization](https://github.com/TestAnything). + +## COPYRIGHT + +Copyright Michael G Schwern , Andy Lester +, Andy Armstrong , Isaac Z. +Schlueter . + +This program is free software; you can redistribute it and/or modify +it under the same terms as Perl itself. See +[http://www.perl.com/perl/misc/artistic.html](http://www.perl.com/perl/misc/artistic.html).