Implementing possibility to attach files to the json reports inside the afterEach hook

[artem-sokorskyi]

Current behavior

attach fn will fail with error message if invoked inside afterEach.
> Unexpected state in createStringAttachmentHandler: test-finished (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor)

Desired behavior

Attaching successfully the file to the json reports inside the afterEach hook. I suppose the attached file could go under the last executed step of the scenario in the json report

Test code to reproduce

dummy-hook.js

import { attach } from '@badeball/cypress-cucumber-preprocessor';

const catchFailedTestcase = (done) => {
  try {
    done();
  } catch (err) {}
};

afterEach((done) => {
  const test = Cypress.mocha.getRunner().test.ctx.currentTest;
  // const caseStartTimeUnixMs = test.wallClockStartedAt.getTime();
  // hardcoded the unix timestamp to be earlier added to the "cypress/downloads" directory than your scenario will be executed
  const caseStartTimeUnixMs = 1693648794962;
  const { platform } = window.Cypress;
  const downloadFolderGlobPath = `${Cypress.config('downloadsFolder')}/**/*.*`;

  cy.task('getFileSinceTime', {
    globFolderPath: downloadFolderGlobPath,
    platform,
    startingFromTimeUnixMs: caseStartTimeUnixMs,
  }).then((downloadPathes) => {
    const availableMimeTypes = {
      csv: {
        mimeType: 'text/csv',
        encoding: 'utf-8',
      },
      jpeg: {
        mimeType: 'base64:image/jpeg',
        encoding: 'base64',
      },
      jpg: {
        mimeType: 'base64:image/jpeg',
        encoding: 'base64',
      },
      json: {
        mimeType: 'application/json',
        encoding: 'utf-8',
      },
      png: {
        mimeType: 'base64:image/png',
        encoding: 'base64',
      },
      pdf: {
        mimeType: 'base64:application/pdf',
        encoding: 'base64',
      },
      txt: {
        mimeType: 'text/plain',
        encoding: 'utf-8',
      },
      xls: {
        mimeType: 'application/vnd.ms-excel',
        encoding: 'utf-8',
      },
      xlsx: {
        mimeType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
        encoding: 'utf-8',
      },
      xml: {
        mimeType: 'application/xml',
        encoding: 'utf-8',
      },
      zip: {
        mimeType: 'application/zip',
        encoding: 'utf-8',
      },
    };
    downloadPathes.forEach((downloadedItem) => {
      const fileExtension = downloadedItem.match(/\.(.+)$/)[1];

      const { mimeType, encoding } = availableMimeTypes[fileExtension];
      cy.readFile(downloadedItem, encoding).then((content) => {
        attach(content, mimeType);
      });
    });
  });
  cy.then(() => catchFailedTestcase(done));
});

cypress.config.js (remember to disable the "trashAssetsBeforeRuns" option inside the config object)

import path from 'path';
import { globSync } from 'glob';


const getRelativePathFromAbsolute = (absolutePath) => path.relative(process.cwd(), absolutePath);
const getUnixPathFromWin = (winPath) => winPath.split(path.sep).join(path.posix.sep);

  on('task', {
    getFileSinceTime({ globFolderPath, platform, startingFromTimeUnixMs }) {
      const relativeGlobPath = getRelativePathFromAbsolute(globFolderPath);
      const parsedGlobFolderPath = platform.includes('win') ? getUnixPathFromWin(relativeGlobPath) : globFolderPath;
      return globSync(parsedGlobFolderPath).filter(
        (name) => Date.parse(fs.statSync(name).ctime) - startingFromTimeUnixMs >= 0,
      );
    },
  });

Next steps:

• Enable json reports in your preprocessor config
• Put dummy pdf file into the "cypress/downloads" directory
• Create a scenario with one simple step (no matter what logic, can be cy.wrap(1).should('eq', 1)
• Execute the scenario in run mode
• Pay attention to the error message described in the actual behaviour above

Logs

DevTools listening on ws://127.0.0.1:63318/devtools/browser/53533b1d-ab11-4706-8209-6ae1725ba69b
  cypress-cucumber-preprocessor resolved environment overrides {} +0ms
  cypress-cucumber-preprocessor resolved explicit user configuration {
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: undefined },
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false
  cypress-cucumber-preprocessor } +1ms
  cypress-cucumber-preprocessor resolved configuration {
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: 'cucumber-report.html' },
  cypress-cucumber-preprocessor   pretty: { enabled: false },
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false,
  cypress-cucumber-preprocessor   implicitIntegrationFolder: '/'
  cypress-cucumber-preprocessor } +0ms

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        12.17.3                                                                        │
  │ Browser:        Chrome 116 (headless)                                                          │
  │ Node Version:   v18.17.1 (/usr/local/bin/node)                                                 │
  │ Specs:          1 found (Login.feature)                                                        │
  │ Searched:       cypress/e2e/features/User Identification/Login.feature                         │
  │ Experiments:    experimentalInteractiveRunEvents=true,experimentalOriginDependencies=true      │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

  cypress-cucumber-preprocessor beforeRunHandler() +328ms
  cypress-cucumber-preprocessor resolved environment overrides {} +2ms
  cypress-cucumber-preprocessor resolved explicit user configuration {
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: undefined },
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false
  cypress-cucumber-preprocessor } +0ms
  cypress-cucumber-preprocessor resolved configuration {
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: 'cucumber-report.html' },
  cypress-cucumber-preprocessor   pretty: { enabled: false },
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false,
  cypress-cucumber-preprocessor   implicitIntegrationFolder: '/'
  cypress-cucumber-preprocessor } +0ms

────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  Login.feature                                                                   (1 of 1)
  cypress-cucumber-preprocessor beforeSpecHandler() +1s
  cypress-cucumber-preprocessor resolved environment overrides {} +48ms
  cypress-cucumber-preprocessor resolved explicit user configuration {
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: undefined },
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false
  cypress-cucumber-preprocessor } +0ms
  cypress-cucumber-preprocessor resolved configuration {
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: 'cucumber-report.html' },
  cypress-cucumber-preprocessor   pretty: { enabled: false },
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false,
  cypress-cucumber-preprocessor   implicitIntegrationFolder: '/Users/artemSokorskyi/Desktop/Workspace/work projects/olympics-hrp-qa/cypress/e2e/features'
  cypress-cucumber-preprocessor } +1ms
  cypress-cucumber-preprocessor resolving step definitions using template(s) [ 'cypress/e2e/**/stepDefinitions/**/*.js' ] +0ms
  cypress-cucumber-preprocessor replacing [filepath] with 'User Identification/Login' +0ms
  cypress-cucumber-preprocessor replacing [filepart] with [ 'User Identification/Login', 'User Identification' ] +0ms
  cypress-cucumber-preprocessor for 'cypress/e2e/features/User Identification/Login.feature' yielded patterns [ 'cypress/e2e/**/stepDefinitions/**/*.js' ] +0ms
  cypress-cucumber-preprocessor found step definitions [
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/NavigateSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/GenericSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/DropdownSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/AssertFilteringSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/ApiSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/PageSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/MemorySteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/InputSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/ClickerSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/CheckSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/Hooks/clearTestStateHook.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/Hooks/IntegrationHooks.js'
  cypress-cucumber-preprocessor ] +7ms


  Login
  cypress-cucumber-preprocessor specEnvelopesHandler() +2s
    - Successful login with valid uppercase/lowercase email (example #1)
    - Successful login with valid uppercase/lowercase email (example #2)
    - Login with invalid email format
    - Login with empty fields
    - Login with invalid credentials
  cypress-cucumber-preprocessor testCaseStartedHandler() +31ms
  cypress-cucumber-preprocessor testStepStartedHandler() +5ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +2s
  cypress-cucumber-preprocessor testStepStartedHandler() +4ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +3s
  cypress-cucumber-preprocessor testStepStartedHandler() +5ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +242ms
  cypress-cucumber-preprocessor testStepStartedHandler() +10ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +5s
  cypress-cucumber-preprocessor testStepStartedHandler() +31ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +112ms
  cypress-cucumber-preprocessor testStepStartedHandler() +5ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +575ms
  cypress-cucumber-preprocessor testStepStartedHandler() +25ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +52ms
  cypress-cucumber-preprocessor testCaseFinishedHandler() +22ms
  cypress-cucumber-preprocessor createStringAttachmentHandler() +35ms
  cypress-cucumber-preprocessor afterScreenshotHandler() +120ms
    1) "after each" hook for "Dummy Scenario"


  0 passing (12s)
  5 pending
  1 failing

  1) Login
       "after each" hook for "Dummy Scenario":
     CypressError: `cy.task('cypress-cucumber-preprocessor:create-string-attachment')` failed with the following error:

> Unexpected state in createStringAttachmentHandler: test-finished (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor)

https://on.cypress.io/api/task

Because this error occurred during a `after each` hook we are skipping all of the remaining tests.
      at <unknown> (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:151171:78)
      at tryCatcher (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:18744:23)
      at Promise._settlePromiseFromHandler (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16679:31)
      at Promise._settlePromise (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16736:18)
      at Promise._settlePromise0 (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16781:10)
      at Promise._settlePromises (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16857:18)
      at _drainQueueStep (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13451:12)
      at _drainQueue (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13444:9)
      at ../../node_modules/bluebird/js/release/async.js.Async._drainQueues (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13460:5)
      at Async.drainQueues (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13330:14)
  From Your Spec Code:
      at createStringAttachment (webpack:///./node_modules/@badeball/cypress-cucumber-preprocessor/dist/entrypoint-browser.js:94:0)
      at attach (webpack:///./node_modules/@badeball/cypress-cucumber-preprocessor/dist/entrypoint-browser.js:100:0)
      at Context.eval (webpack:///./cypress/e2e/project/stepDefinitions/Hooks/IntegrationHooks.js:280:14)
  
  From Node.js Internals:
    Error: Unexpected state in createStringAttachmentHandler: test-finished (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor)
        at createError (/Users/artemSokorskyi/Desktop/Workspace/work projects/olympics-hrp-qa/node_modules/@badeball/cypress-cucumber-preprocessor/dist/helpers/error.js:9:12)
        at createStringAttachmentHandler (/Users/artemSokorskyi/Desktop/Workspace/work projects/olympics-hrp-qa/node_modules/@badeball/cypress-cucumber-preprocessor/dist/plugin-event-handlers.js:434:43)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)



  cypress-cucumber-preprocessor afterSpecHandler() +1s
  Hook failures can't be represented in any reports (messages / json / html), thus none is created for cypress/e2e/features/User Identification/Login.feature.

Versions

• Cypress version: 12.17.3
• Preprocessor version: 18.0.4
• Node version: 18.17.1

Checklist

• I've read the FAQ.
• I've read instructions for logging issues.
• I'm not using cypress-cucumber-preprocessor@4.3.1 (package name has changed and it is no longer the most recent version, see #689).

[hammzj]

See issue #1051

[badeball]

Sorry for taking so long to get back to you, @artem-sokorskyi and @hammzj. As mentioned, this has come up before. I was not entirely convinced of the use-cases, but adding downloads to the report seems akin to screenshots. Ideally I would have liked to have a download API in Cypress and have native support for adding downloads to reports, but based on previous experience I'd say there's little chance of ever seeing this added.

I've been trying to come up with the best way of supporting all of the use-cases so far, and I think having a after step handler in the node environment, providing an attach function, will suffice, Something along the lines of that shown below.

Adding logs from cypress-terminal-report (@hammzj :

await addCucumberPreprocessorPlugin(on, config, {
  onAfterStep({ wasLastStep, attach }) {
    if (wasLastStep) {
      const logs = ""; // Retrieve logs from  cypress-terminal-report.

      attach(logs);
    }
  }
});

Adding downloads (@artem-sokorskyi):

await addCucumberPreprocessorPlugin(on, config, {
  onAfterStep({ wasLastStep, attach }) {
    const downloads = []; // Retrieve recent downloads.
    
    for (const download of downloads) {
      attach(download);
    }
  }
});

Adding resized screenshots (@MDG-JHowley):

await addCucumberPreprocessorPlugin(on, config, {
  async onAfterStep({ wasLastStep, attach }) {
    const screenshots = []; // Retrieve recent screenshots.
    
    for (const screenshot of screenshots) {
      attach(resize(screenshot));
    }
  }
});

[artem-sokorskyi]

cool stuff!!!

[MDG-JHowley]

@badeball yes this would be great - as you say, we really just need the attach method available in the node environment. onAfterStep seems like a good place to add that hook but what happens if the test fails? Would this hook still be run?

[badeball]

but what happens if the test fails? Would this hook still be run?

This, contrary to others, can in fact be run despite test failures.

[hammzj]

So, onAfterStep is a node hook, but runs like this:

• after:step event (Node env)
• onAfterStep function (Cypress env)

If this is the case, this is really cool. I wonder, could this will also get around After hooks not running on test failures since it's able to call back into the browserified Cypress environment?

[badeball]

No, this would run purely in the node environment and it cannot call back into the browser environment.

[badeball]

I've released the above-mentioned API with v19.0.0. You can read more about it here. I hope this gives you the ability to attach files "post test" (albeit as part of the last step, as there are no "test scoped" attachments in Cucumber).

[hammzj]

This is awesome! I'll need to test it out in the next week or so. Thanks for adding this

[MDG-JHowley]

@badeball is there anyway to pass the current test details into this hook?

[badeball]

What do you have in mind specifically?

[MDG-JHowley]

Well ideally the feature/spec and scenario/test names and whether the test has failed at that point.

I'm trying to attach logs (from cypress-terminal-report) along with screenshots/videos (I can't use the built in screenshot handler because I want to embed an html <img> tag referencing an s3 bucket where the images will ultimately reside) to the report.

For the logs and the videos I need to know the current Feature/spec. For the screenshots the local file path is based on feature/spec, scenario/test and test failure state.

My current hack is a custom cy.task in a before hook which writes a file I can later read from in the onAfterStep hook.