Have the global timeout take into account `t.timeout()` durations

[spectranaut]

What you're trying to do

I appreciate that having a global timeout in order to catch unexpected behavior in a test and not consume too much CI resources. However, some tests take a long time intentionally, and I would like to be able to use t.timeout() to explicitly override the global timeouts for just one test. Right now, you can only override the global timeouts if then t.timeout() is shorter than the global timeout. It would be nice if it overrides when t.timeout() is longer as well.

I think I am not the only one, as I found this question referring to the same problem.

Why you can't use AVA for this

There is no way to have a global timeout that is something like 10s and a timeout for a particular test that is longer, like 30s.

And maybe how you think AVA could handle this

I'm not sure how the timeouts are coded, it might be a one-liner :)

[novemberborn]

The global timeout is a "stop tests in case they're stuck" kind of timeout.

I suppose we could interpret that as "stop tests in case they're stuck, but don't worry for the next 30 seconds since the test will fail if it's stuck anyway".

The workers will have to send a message to the main process whenever a per-test timeout starts, with the timeout duration. And then in the main process we'd have to make sure we don't stop tests because they're stuck until that duration has passed.

This should work with multiple t.timeout() calls, of course.

I don't think it's quite a one-liner, but this approach is consistent with what the global timeout is there for, so let's do it.

[sramam]

How is described behavior different from removing the global timeout and setting a default timeout on every test?

Basically,

• remove global timeout
• by default, t.timeout(10000)
• advertise this and let the user modify as necessary.

The net effect is the same IMHO.

[novemberborn]

When tests run concurrently, within and across worker processes, their duration may vary wildly. The global timeout (misnamed as it may be) is only concerned with making sure something is happening in your test suite.

Individual timeouts can be useful in certain cases, but far from all.

[sramam]

Disclaimer: I have now found my peace with the nested timeout and can live with the timeout change.

However, it seems my initial concerns are being experienced/reported by others.
Hopefully, the example below provides an insight.

The output below is meant to be slightly hyperbolic, but convey the confusion I sensed
with the timeout change on first blush.

$ npx ava dist/sdk/test/*.js

⠴ Server listening at http://127.0.0.1:54369

  ✖ Timed out while running tests

  3 tests were pending in dist/sdk/test/01_core-wait-for-registry.js

  ◌ 01_core-wait-for-registry › inProgress
  ◌ 01_core-wait-for-registry › isErrored
  ◌ 01_core-wait-for-registry › isAlive

  18 tests were pending in dist/sdk/test/01_core.js

  ◌ 01_core › waitFor server-state tests
  ◌ 01_core › getAllProviderMetadata !refresh
  ◌ 01_core › getAllProviderMetadata
  ◌ 01_core › getProviderMetadata aws
  ◌ 01_core › getRelativePath 0.0.2-rc1
  ◌ 01_core › getRelativePath abra
  ◌ 01_core › getRelativePath git-url#0.0.2-rc1
  ◌ 01_core › getRelativePath git-url#hash
  ◌ 01_core › getRelativePath file#hash
  ◌ 01_core › getRelativePath non-existant version
  ◌ 01_core › getRelativePath non-existant hash
  ◌ 01_core › getProviderMetadata aws w/ refresh
  ◌ 01_core › getProviderMetadata bad provider
  ◌ 01_core › getProviderMetadata github@0.0.1
  ◌ 01_core › getProviderMetadata aws@1.23.0 (non-existant version)
  ◌ 01_core › publish invalid type
  ◌ 01_core › shutdown server - work from cache after
  ◌ 01_core › getProviderMetadata github from cache

This output is completely useless in diagnosing any problem that might exist.
I know there is a timeout, I just have no idea where to look for the cause.

The only way to debug this:

 npx ava dist/sdk/test/*.js -T 30000

 npx ava dist/sdk/test/*.js -T 120000

and so on, with the hope that it wasn't an intermittent failure.

As a thought experiment, with per test timeouts, the output could look like:

  3 tests were pending in dist/sdk/test/01_core-wait-for-registry.js

  ◌ 01_core-wait-for-registry › inProgress
  ◌ 01_core-wait-for-registry › isErrored
  ◌ 01_core-wait-for-registry › isAlive

  18 tests were pending in dist/sdk/test/01_core.js

  ✖ 01_core › waitFor server-state tests  - Timed out after 10 seconds
  ◌ 01_core › getAllProviderMetadata !refresh
  ◌ 01_core › getAllProviderMetadata
  ◌ 01_core › getProviderMetadata aws
  ◌ 01_core › getRelativePath 0.0.2-rc1
  ◌ 01_core › getRelativePath abra
  ◌ 01_core › getRelativePath git-url#0.0.2-rc1
  ◌ 01_core › getRelativePath git-url#hash
  ◌ 01_core › getRelativePath file#hash
  ◌ 01_core › getRelativePath non-existant version
  ◌ 01_core › getRelativePath non-existant hash
  ◌ 01_core › getProviderMetadata aws w/ refresh
  ◌ 01_core › getProviderMetadata bad provider
  ◌ 01_core › getProviderMetadata github@0.0.1
  ◌ 01_core › getProviderMetadata aws@1.23.0 (non-existant version)
  ◌ 01_core › publish invalid type
  ◌ 01_core › shutdown server - work from cache after
  ◌ 01_core › getProviderMetadata github from cache

In most cases, that simple fault isolation in the middle,

  ✖ 01_core › waitFor server-state tests  - Timed out after 10 seconds

will result in a substantial quality of life improvement.

I appreciate that schedulers managing workers is a serious and complex problem,
but this nested timer business makes it harder to grok from the test writer point of view.

Hopefully this helps. I'll go back to holding my peace....