Total tests time increased after updating from v20 to v21

[KarineBrandelli]

Current behavior

My tests were running way faster at the 20.1.2 version. After updating to the 21.0.2 version, the waiting time increased significantly.

At 20.1.2 version: 372.15 seconds

At 21.0.2 version: 3521.46 seconds

Which is odd, cause the tests separately have the time quite similar, but I was marking here on the clock, and it took all the time at the final.

Also, after updating, this warning is shown at the terminal:
Warning: terminator_CreateInstance: Failed to CreateInstance in ICD 2. Skipping ICD.

Desired behavior

Not affect the test time.

Versions

• Cypress version: 20.1.2 - 21.0.2
• Preprocessor version:
• Node version: 20.15.0

Checklist

• I've read the FAQ.
• I've read instructions for logging issues.
• I'm not using cypress-cucumber-preprocessor@4.3.1 (package name has changed and it is no longer the most recent version, see #689).

[badeball]

Hey @KarineBrandelli. Did you by any chance update any other dependencies simultaneously, like your Cypress version? If so, can you revert said update, update only @badeball/cypress-cucumber-preprocessor and see if the performance regression persists?

If it does, try each version between 20.1.2 and 21.0.2, to pinpoint exactly which version that introduces it.

Furthermore, since I'm not seeing this myself, you will probably have to provide a reproducible example that illustrates the behavior.

Lastly, why did you not do any of this before opening up an issue?

[mestevezcl]

I experienced the same issue. After upgrading from v20.1.2 to 21.0.2, it takes much longer on “Running: test.feature (1 of 1)”.

As I figured out, it has to do with the number of defined steps.

Having a simple project with a single feature and step definition, as defined below, works fine. Test execution takes a few seconds. Here is a sample of a simple feature:

Feature: Test Feature

    @test
    Scenario: 01 - Test sample
        Then User access to the application successfully

This is the step definition for the feature:

import { Then } from '@badeball/cypress-cucumber-preprocessor';

Then(/User access to the application successfully/, () => {
    expect(true).to.be.true;
})

As I mentioned before, this configuration works fine. However, if we add more steps in the step definition, the test execution time increases accordingly. For example, if we implement the following modification in the step definitions, the same execution takes half a minute, and it takes that extra time for each Feature:

import { Then } from '@badeball/cypress-cucumber-preprocessor';

Then(/User access to the application successfully/, () => {
    expect(true).to.be.true;
})

for (let iStep = 0; iStep < 2000; iStep++) {
    Then(`User access ${iStep} times to the application successfully`, () => {
        expect(true).to.be.true;
    })
}

I’ve tried the same code with v20.1.2 and it works fine. When I switch back to 21.0.2, the issue arises again.

[badeball]

@mestevezcl, I'm not seeing the same behavior using your code snippets. Can you share a complete example?

[KarineBrandelli]

@badeball

I have a sandbox, but I don't think is going to be very helpful.
Since there is no 'migration guide' or significant changes from v20 to v21, maybe is a machine configuration? Chromium perhaps?

On the sandbox with v21:

On the sandbox with v20:

Downgrade the version on the package.json, run yarn then run yarn cy:run.

https://codesandbox.io/p/devbox/cypress-cucumber-3y4jwl

[badeball]

Since there is no 'migration guide' or significant changes from v20 to v21, maybe is a machine configuration? Chromium perhaps?

There was a lot of internal refactoring made during v21.0.0.

I have a sandbox, but I don't think is going to be very helpful.

I did help and I can now reproduce the behavior. A complete (ie. runnable) example contains a tremendous amount of information. It shows me what loader you're using and that you're writing component tests, neither which is apparent from your original post. This is exactly why I ask everyone to provide this up front and without me having to ask for it anymore than I already have (this is specifically explained in the issue template and the contributing guidelines).

I'm not sure if I can get to this now, but I'll have a fix ready shortly-ish.

[KarineBrandelli]

@badeball As I said, I thought it was a machine setting, thus the runnable exemple not being very useful. Forgive me for that.

I appreciate the time you took to look into this. Any updates let me know.

[badeball]

This has been fixed with v21.0.3.

[mestevezcl]

With v21.0.3 it works fast again on my repo! Many thanks for the quick fix :)