[BREAKING CHANGE] Spec: Combine the "Test" and "Suite" concepts

== Status quo ==

The TAP 13 specification does not standardise a way of describing parent-child
relationships between tests, nor does it standardise how to group tests.

Yet, all major test frameworks have a way to group tests (e.g. QUnit module,
and Mocha suite) and/or allow nesting tests inside of other tests (like tape,
and node-tap). While the CRI draft provided a way to group tests, it did not
accomodate Tap. They would either need to flatten the tests with a separator
symbol in the test name, or to create an implied "Suite" for every test that
has non-zero children and then come up with an ad-hoc naming scheme for it.

Note that the TAP 13 reporter we ship, even after this change, still ends up
flattening the tests by defaut using the greater than `>` symbol, but at least
the event model itself recognises the relationships so that other output formats
can make use of it, and in the future TAP 14 hopefully will recognise it as
well, which we can then make use of.

Ref TestAnything/testanything.github.io#36.

== Summary of changes ==

See the diff of `test/integration/reference-data.js` for the concrete changes
this makes to the consumable events.

- Remove `suiteStart` and `suiteEnd` events.

  Instead, the spec now says that tests are permitted to have children.

  The link from child to parent remains the same as before, using the `fullName`
  field which is now a stack of test names. Previously, it was a stack of suite
  names with a test name at the end.

- Remove all "downward" links from parent to child. Tests don't describe
  their children upfront in detail, and neither does `runStart`. This was
  information was very repetitive and tedious to satisy for implementors, and
  encouraged or required inefficient use of memory.

  I do recognise that a common use case might be to generate a single output
  file or stream where real-time updates are not needed, in which case you
  may want a convenient tree that is ready to traverse without needing to
  listen for async events and put it together. For this purpose, I have added a
  built-in reporter that simply listens to the new events and outputs a "summary"
  event with an object that is similar to the old "runEnd" event object where
  the entire run is described in a single large object.

- New "SummaryReporter" for simple use cases of non-realtime traversing of
  single structure after the test has completed.

== Caveats ==

- A test with the "failed" status is no longer expected to always have
  an error directly associated with it.

  Now that tests aggregate into other tests rather than into suites,
  this means tests that merely have other tests as children do still
  have to send a full testEnd event, and thus an `errors` and `assertions`
  array.

  I considered specifying that errors have to propagate but this seemed
  messy and could lead to duplicate diagnostic output in  reporters, as well
  ambiguity or uncertainty over where errors originated.

- A suite containing only "skipped" tests now aggregates as "passed"
  instead of "skipped". Given we can't know whether a suite is its own
  test with its own assertions, we also can't assume that if a test parent
  has only "skipped" children that the parent was also skipped.

  This applies to our built-in adapters, but individual frameworks, if they
  know that a suite was skipped in its entirety, can of course still set the
  status of parents however they see fit.

- Graphical reporters (such as QUnit and Mocha's HTML reporters) may no
  longer assume that a test parent has either assertions/errors or other
  tests. A test parente can now have both its own assertions/errors, as well
  as other tests beneath it.

  This restricts the freedom and possibilities for visualisation.
  My recommendation is that, if a visual reporter wants to keep using different
  visual shapes for "group of assertions" and "group of tests", that they
  buffer the information internally such that they can first render all the
  tests's own assertions, and then render the children, even if they originally
  ran interleaved and/or the other way around.
  Ref #126.

- The "Console" reporter that comes with js-reporter now no longer
  uses `console.group()` for collapsing nested tests.

== Misc ==

- Add definitions for the "Adapter" and "Producer" terms.

- Use terms "producer" and "reporter" consistently, instead of
  "framework", "runner", or "adapter".

- Remove mention that the spec is for reporting information from
  "JavaScript test frameworks". CRI can be used to report information
  about any kind of test that can be represented in CRI's event model,
  including linting and end-to-end tests for JS programs, as well as
  non-JS programs. It describes a JS interface for reporters, but the
  information can come from anywhere.

  This further solifies that CRI is not meant to be used for "hooking"
  into a framework, and sets no expectation about timing or run-time
  environment being shared with whatever is executing tests in some
  form or another. This was already the intent originally, since it could
  be used to report information from other processes or from a cloud-based
  test runner like BrowserStack, but this removes any remaining confusion
  or doubt there may have been.

Fixes #126.

Author: Krinkle

README.md

@@ -52,26 +52,25 @@ Would _you_ be interested in discussing this with us further? Please join in!
 Listen to the events and receive the emitted data:
 
 ```js
-// Attach one of the exiting adapters.
+// Use automatic discovery of the framework adapter.
 const runner = JsReporters.autoRegister();
 
-// Listen to the same events for any testing framework.
-runner.on('testEnd', function (test) {
-  console.log('Test %s has errors:', test.fullname.join(' '), test.errors);
+// Listen to standard events, from any testing framework.
+runner.on('testEnd', (test) => {
+  console.log('Test %s has errors:', test.fullName.join(' '), test.errors);
 });
 
-runner.on('runEnd', function (globalSuite) {
-  const testCounts = globalSuite.testCounts;
-
-  console.log('Testsuite status: %s', globalSuite.status);
+runner.on('runEnd', (run) => {
+  const counts = run.counts;
 
+  console.log('Testsuite status: %s', run.status);
   console.log('Total %d tests: %d passed, %d failed, %d skipped',
-    testCounts.total,
-    testCounts.passed,
-    testCounts.failed,
-    testCounts.skipped);
-
-  console.log('Total duration: %d', globalSuite.runtime);
+    counts.total,
+    counts.passed,
+    counts.failed,
+    counts.skipped
+  );
+  console.log('Total duration: %d', run.runtime);
 });
 
 // Or use one of the built-in reporters.

index.js

@@ -4,6 +4,7 @@ const JasmineAdapter = require('./lib/adapters/JasmineAdapter.js');
 const MochaAdapter = require('./lib/adapters/MochaAdapter.js');
 const TapReporter = require('./lib/reporters/TapReporter.js');
 const ConsoleReporter = require('./lib/reporters/ConsoleReporter.js');
+const SummaryReporter = require('./lib/reporters/SummaryReporter.js');
 const {
   collectSuiteStartData,
   collectSuiteEndData,
@@ -18,6 +19,7 @@ module.exports = {
   MochaAdapter,
   TapReporter,
   ConsoleReporter,
+  SummaryReporter,
   EventEmitter,
   collectSuiteStartData,
   collectSuiteEndData,

lib/adapters/JasmineAdapter.js

@@ -13,9 +13,8 @@ module.exports = class JasmineAdapter extends EventEmitter {
     // NodeJS or browser
     this.env = jasmine.env || jasmine.getEnv();
 
-    this.suiteStarts = {};
     this.suiteChildren = {};
-    this.suiteEnds = {};
+    this.suiteEnds = [];
     this.testStarts = {};
     this.testEnds = {};
 
@@ -56,93 +55,90 @@ module.exports = class JasmineAdapter extends EventEmitter {
 
     return {
       name: testStart.name,
-      suiteName: testStart.suiteName,
+      parentName: testStart.parentName,
       fullName: testStart.fullName.slice(),
       status: (result.status === 'pending') ? 'skipped' : result.status,
+      // TODO: Jasmine 3.4+ has result.duration, use it.
+      // Note that result.duration uses 0 instead of null for a 'skipped' test.
       runtime: (result.status === 'pending') ? null : (new Date() - this.startTime),
       errors,
       assertions
     };
   }
 
   /**
-   * Convert a Jasmine SuiteResult for CRI 'runStart' or 'suiteStart' event data.
+   * Traverse the Jasmine structured returned by `this.env.topSuite()`
+   * in order to extract the child-parent relations and full names.
    *
-   * Jasmine provides details about childSuites and tests only in the structure
-   * returned by "this.env.topSuite()".
    */
-  createSuiteStart (result, parentNames) {
+  processSuite (result, parentNames, parentIds) {
     const isGlobalSuite = (result.description === 'Jasmine__TopLevel__Suite');
 
     const name = isGlobalSuite ? null : result.description;
     const fullName = parentNames.slice();
-    const tests = [];
-    const childSuites = [];
 
     if (!isGlobalSuite) {
-      fullName.push(result.description);
+      fullName.push(name);
     }
 
+    parentIds.push(result.id);
+    this.suiteChildren[result.id] = [];
+
     result.children.forEach((child) => {
+      this.testStarts[child.id] = {
+        name: child.description,
+        parentName: name,
+        fullName: [...fullName, child.description]
+      };
+
       if (child.id.indexOf('suite') === 0) {
-        childSuites.push(this.createSuiteStart(child, fullName));
+        this.processSuite(child, fullName.slice(), parentIds.slice());
       } else {
-        const testStart = {
-          name: child.description,
-          suiteName: name,
-          fullName: [...fullName, child.description]
-        };
-        tests.push(testStart);
-        this.testStarts[child.id] = testStart;
+        // Update flat list of test children
+        parentIds.forEach((id) => {
+          this.suiteChildren[id].push(child.id);
+        });
       }
     });
-
-    const helperData = helpers.collectSuiteStartData(tests, childSuites);
-    const suiteStart = {
-      name,
-      fullName,
-      tests,
-      childSuites,
-      testCounts: helperData.testCounts
-    };
-    this.suiteStarts[result.id] = suiteStart;
-    this.suiteChildren[result.id] = result.children.map(child => child.id);
-    return suiteStart;
   }
 
-  createSuiteEnd (suiteStart, result) {
-    const tests = [];
-    const childSuites = [];
-    this.suiteChildren[result.id].forEach((childId) => {
-      if (childId.indexOf('suite') === 0) {
-        childSuites.push(this.suiteEnds[childId]);
-      } else {
-        tests.push(this.testEnds[childId]);
-      }
-    });
+  createSuiteEnd (testStart, result) {
+    const tests = this.suiteChildren[result.id].map((testId) => this.testEnds[testId]);
 
-    const helperData = helpers.collectSuiteEndData(tests, childSuites);
+    const helperData = helpers.aggregateTests(tests);
     return {
-      name: suiteStart.name,
-      fullName: suiteStart.fullName,
-      tests,
-      childSuites,
+      name: testStart.name,
+      parentName: testStart.parentName,
+      fullName: testStart.fullName,
       // Jasmine has result.status, but does not propagate 'todo' or 'skipped'
       status: helperData.status,
-      testCounts: helperData.testCounts,
-      // Jasmine 3.4+ has result.duration, but uses 0 instead of null
-      // when 'skipped' is skipped.
-      runtime: helperData.status === 'skipped' ? null : (result.duration || helperData.runtime)
+      runtime: result.duration || helperData.runtime,
+      errors: [],
+      assertions: []
     };
   }
 
   onJasmineStarted () {
-    this.globalSuite = this.createSuiteStart(this.env.topSuite(), []);
-    this.emit('runStart', this.globalSuite);
+    this.processSuite(this.env.topSuite(), [], []);
+
+    let total = 0;
+    this.env.topSuite().children.forEach(function countChild (child) {
+      total++;
+      if (child.id.indexOf('suite') === 0) {
+        child.children.forEach(countChild);
+      }
+    });
+
+    this.emit('runStart', {
+      name: null,
+      counts: {
+        total: total
+      }
+    });
   }
 
   onSuiteStarted (result) {
-    this.emit('suiteStart', this.suiteStarts[result.id]);
+    this.emit('testStart', this.testStarts[result.id]);
   }
 
   onSpecStarted (result) {
@@ -156,11 +152,20 @@ module.exports = class JasmineAdapter extends EventEmitter {
   }
 
   onSuiteDone (result) {
-    this.suiteEnds[result.id] = this.createSuiteEnd(this.suiteStarts[result.id], result);
-    this.emit('suiteEnd', this.suiteEnds[result.id]);
+    const suiteEnd = this.createSuiteEnd(this.testStarts[result.id], result);
+    this.suiteEnds.push(suiteEnd);
+    this.emit('testEnd', suiteEnd);
   }
 
   onJasmineDone (doneInfo) {
-    this.emit('runEnd', this.createSuiteEnd(this.globalSuite, this.env.topSuite()));
+    const topSuite = this.env.topSuite();
+    const tests = this.suiteChildren[topSuite.id].map((testId) => this.testEnds[testId]);
+    const helperData = helpers.aggregateTests([...tests, ...this.suiteEnds]);
+    this.emit('runEnd', {
+      name: null,
+      status: helperData.status,
+      counts: helperData.counts,
+      runtime: helperData.runtime
+    });
   }
 };

lib/adapters/MochaAdapter.js

@@ -6,6 +6,14 @@ module.exports = class MochaAdapter extends EventEmitter {
     super();
 
     this.errors = null;
+    this.finalRuntime = 0;
+    this.finalCounts = {
+      passed: 0,
+      failed: 0,
+      skipped: 0,
+      todo: 0,
+      total: 0
+    };
 
     // Mocha will instantiate the given function as a class, even if you only need a callback.
     // As such, it can't be an arrow function as those throw TypeError when instantiated.
@@ -27,40 +35,37 @@ module.exports = class MochaAdapter extends EventEmitter {
   convertToSuiteStart (mochaSuite) {
     return {
       name: mochaSuite.title,
-      fullName: this.titlePath(mochaSuite),
-      tests: mochaSuite.tests.map(this.convertTest.bind(this)),
-      childSuites: mochaSuite.suites.map(this.convertToSuiteStart.bind(this)),
-      testCounts: {
-        total: mochaSuite.total()
-      }
+      parentName: (mochaSuite.parent && !mochaSuite.parent.root) ? mochaSuite.parent.title : null,
+      fullName: this.titlePath(mochaSuite)
     };
   }
 
   convertToSuiteEnd (mochaSuite) {
     const tests = mochaSuite.tests.map(this.convertTest.bind(this));
     const childSuites = mochaSuite.suites.map(this.convertToSuiteEnd.bind(this));
-    const helperData = helpers.collectSuiteEndData(tests, childSuites);
+    const helperData = helpers.aggregateTests([...tests, ...childSuites]);
+
     return {
       name: mochaSuite.title,
+      parentName: (mochaSuite.parent && !mochaSuite.parent.root) ? mochaSuite.parent.title : null,
       fullName: this.titlePath(mochaSuite),
-      tests,
-      childSuites,
       status: helperData.status,
-      testCounts: helperData.testCounts,
-      runtime: helperData.runtime
+      runtime: helperData.runtime,
+      errors: [],
+      assertions: []
     };
   }
 
   convertTest (mochaTest) {
-    let suiteName;
+    let parentName;
     let fullName;
     if (!mochaTest.parent.root) {
-      suiteName = mochaTest.parent.title;
+      parentName = mochaTest.parent.title;
       fullName = this.titlePath(mochaTest.parent);
       // Add also the test name.
       fullName.push(mochaTest.title);
     } else {
-      suiteName = null;
+      parentName = null;
       fullName = [mochaTest.title];
     }
 
@@ -73,21 +78,23 @@ module.exports = class MochaAdapter extends EventEmitter {
         message: error.message || error.toString(),
         stack: error.stack
       }));
+      const status = (mochaTest.state === undefined) ? 'skipped' : mochaTest.state;
+      const runtime = (mochaTest.duration === undefined) ? null : mochaTest.duration;
 
       return {
         name: mochaTest.title,
-        suiteName,
+        parentName,
         fullName,
-        status: (mochaTest.state === undefined) ? 'skipped' : mochaTest.state,
-        runtime: (mochaTest.duration === undefined) ? null : mochaTest.duration,
+        status,
+        runtime,
         errors,
         assertions: errors
       };
     } else {
       // It is a "test start".
       return {
         name: mochaTest.title,
-        suiteName,
+        parentName,
         fullName
       };
     }
@@ -112,15 +119,24 @@ module.exports = class MochaAdapter extends EventEmitter {
   }
 
   onStart () {
-    const globalSuiteStart = this.convertToSuiteStart(this.runner.suite);
-    globalSuiteStart.name = null;
-
-    this.emit('runStart', globalSuiteStart);
+    // total is all tests + all suites
+    // each suite gets a CRI "test" wrapper
+    let total = this.runner.suite.total();
+    this.runner.suite.suites.forEach(function addSuites (suite) {
+      total++;
+      suite.suites.forEach(addSuites);
+    });
+    this.emit('runStart', {
+      name: null,
+      counts: {
+        total: total
+      }
+    });
   }
 
   onSuite (mochaSuite) {
     if (!mochaSuite.root) {
-      this.emit('suiteStart', this.convertToSuiteStart(mochaSuite));
+      this.emit('testStart', this.convertToSuiteStart(mochaSuite));
     }
   }
 
@@ -148,19 +164,29 @@ module.exports = class MochaAdapter extends EventEmitter {
     // and status are already attached to the test, but the errors are not.
     mochaTest.errors = this.errors;
 
-    this.emit('testEnd', this.convertTest(mochaTest));
+    const testEnd = this.convertTest(mochaTest);
+    this.emit('testEnd', testEnd);
+    this.finalCounts.total++;
+    this.finalCounts[testEnd.status]++;
+    this.finalRuntime += testEnd.runtime || 0;
   }
 
   onSuiteEnd (mochaSuite) {
     if (!mochaSuite.root) {
-      this.emit('suiteEnd', this.convertToSuiteEnd(mochaSuite));
+      const testEnd = this.convertToSuiteEnd(mochaSuite);
+      this.emit('testEnd', testEnd);
+      this.finalCounts.total++;
+      this.finalCounts[testEnd.status]++;
+      this.finalRuntime += testEnd.runtime || 0;
     }
   }
 
-  onEnd () {
-    const globalSuiteEnd = this.convertToSuiteEnd(this.runner.suite);
-    globalSuiteEnd.name = null;
-
-    this.emit('runEnd', globalSuiteEnd);
+  onEnd (details) {
+    this.emit('runEnd', {
+      name: null,
+      status: this.finalCounts.failed > 0 ? 'failed' : 'passed',
+      counts: this.finalCounts,
+      runtime: this.finalRuntime
+    });
   }
 };