cypress-cucumber-example not working with bundled features

[jcundill]

Seems bundled features breaks step resolution where there is more than one instance of a matching step

Steps to reproduce:

Updated the cypress-cucumber-example project to include
cypress-cucumber-preprocessor: ^1.15.1

Added All.features file to cypress/integration folder

npx cypress run --spec cypress/integration/All.features

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    3.3.0                                                                              │
  │ Browser:    Electron 61 (headless)                                                             │
  │ Specs:      1 found (All.features)                                                             │
  │ Searched:   cypress/integration/All.features                                                   │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running: All.features...                                                                 (1 of 1) 


  The Google
    ✓ Opening a Google network page (1128ms)
    ✓ Different kind of opening (3074ms)

  The Facebook
    ✓ Opening a social network page (2442ms)
    1) Different kind of opening

  The Twitter
    ✓ Opening Twitter (3343ms)
    ✓ Opening Twitter again (3009ms)


  5 passing (32s)
  1 failing

  1) The Facebook Different kind of opening:
     CypressError: Timed out retrying: expected 'Facebook – log in or sign up' to include 'Google'
      at Object.cypressErr (https://www.facebook.com/__cypress/runner/cypress_runner.js:83139:11)
      at Object.throwErr (https://www.facebook.com/__cypress/runner/cypress_runner.js:83104:18)
      at Object.throwErrByPath (https://www.facebook.com/__cypress/runner/cypress_runner.js:83131:17)
      at retry (https://www.facebook.com/__cypress/runner/cypress_runner.js:76579:16)
      at https://www.facebook.com/__cypress/runner/cypress_runner.js:68565:18
      at tryCatcher (https://www.facebook.com/__cypress/runner/cypress_runner.js:132036:23)
      at Promise._settlePromiseFromHandler (https://www.facebook.com/__cypress/runner/cypress_runner.js:130054:31)
      at Promise._settlePromise (https://www.facebook.com/__cypress/runner/cypress_runner.js:130111:18)
      at Promise._settlePromise0 (https://www.facebook.com/__cypress/runner/cypress_runner.js:130156:10)
      at Promise._settlePromises (https://www.facebook.com/__cypress/runner/cypress_runner.js:130231:18)
      at Async._drainQueue (https://www.facebook.com/__cypress/runner/cypress_runner.js:126960:16)
      at Async._drainQueues (https://www.facebook.com/__cypress/runner/cypress_runner.js:126970:10)
      at Async.drainQueues (https://www.facebook.com/__cypress/runner/cypress_runner.js:126844:14)
      at <anonymous>




  (Results)

  ┌────────────────────────────┐
  │ Tests:        6            │
  │ Passing:      5            │
  │ Failing:      1            │
  │ Pending:      0            │
  │ Skipped:      0            │
  │ Screenshots:  1            │
  │ Video:        true         │
  │ Duration:     32 seconds   │
  │ Spec Ran:     All.features │
  └────────────────────────────┘


  (Screenshots)

  - /Users/jcundill/code/cypress-cucumber-example/cypress/screenshots/All.features/The Facebook -- Different kind of opening (failed).png (1280x720)


  (Video)

  - Started processing:   Compressing to 32 CRF
  - Finished processing:  /Users/jcundill/code/cypress-cucumber-example/cypress/videos/All.features.mp4 (6 seconds)


====================================================================================================

  (Run Finished)


      Spec                                                Tests  Passing  Failing  Pending  Skipped 
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖ All.features                              00:32        6        5        1        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    1 of 1 failed (100%)                        00:32        6        5        1        -        -  

[jcundill]

This is probably a result of

https://github.com/TheBrainFamily/cypress-cucumber-preprocessor/blob/1c6f1267313e16fa3c130aecae54ad4c23ad2175/lib/resolveStepDefinition.js#L38

and the fact that the stepDefinitionRegistry has been loaded with all the run-wide steps in bundled mode.

[jcundill]

Also, anecdotally, the performance gains from bundling all of the tests into describe blocks on a single page are not too great for the real world features and scenarios we have implemented at the place I work for.

We seem to get around a 10% improvement in test run time across a generally 14 mins acceptance test run by adopting the All.features approach.

Setup is - Jenkins CI pipeline invoking the cypress docker container to run tests against the SUT as another docker instance on a local network.

This run includes the snapshots of failing tests and we let cypress produce the after test video in both cases.

Seems, at least for us, that our test run duration timings are not overly related to cypress stopping and restarting the 'world' between each test. They are more 'application' latency issues - but this is on a local network and the SUT is actually quite responsive during the test run.

As mentioned, this is anecdotal, other people may see more impressive speed-ups than we do.

[lgandecki]

That makes sense. Looks like we should clear the hooks registry between runs.

As per the performance gains, thanks for sharing. It does seem like it would be dependent on the case. And definitely if the tests themselves take >10 s each then the difference will not be as significant. But, if someone uses cypress to, for example, test react components directly, then the overhead of restarting everything between tests would be significant, and our solution would make the runs blazingly fast. I plan to start doing that, (and also work on some public/open source examples) once hooks support is there:
cypress-io/cypress-react-unit-test#51

[laiscoolblue]

I'd like to point out that in our project, the impact of bundling was amazing, it reduced the running time in almost 50%.

Regarding the examples repository, if you pull the latest change, it should work because I've changed the step name, but anyway, I think this situation should be handled.

[lgandecki]

🎉 This issue has been resolved in version 1.16.2 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀