Increase envd timeout

[djeebus]

This timeout is used when waiting for the sandbox to become healthy. Before this point, there are parts of the process where things should fail quickly. At this point, however, we're waiting for customer-provided code to become healthy, and we don't control this duration. Let's give them as much time as we can, while still providing some room for a retry if it takes too long. The client SDK defaults to a 60 second timeout, so 40 seconds gives them enough time to try twice, although the second attempt won't have as long as the first to complete successfully.

[jakubno]

Should we think about making this a feature flag so we can change it on the fly without redeployment? wdyt?

[djeebus]

This is the first time (to my knowledge) that we're modifying this. I suspect it's not worth the effort, given how rarely it's going to be modified, but if we want to start tuning at some point, it'd be worth updating it then.

[jakubno]

I see this more as a guess; maybe we'll find out that after 20 seconds, it never succeeds. Also, if the main issue isn't time, it's possible that the success rate could be lower than the current solution. In those cases, we won't be able to reduce the timeout without redeploying.

[djeebus]

I see this more as a guess; maybe we'll find out that after 20 seconds, it never succeeds. Also, if the main issue isn't time, it's possible that the success rate could be lower than the current solution. In those cases, we won't be able to reduce the timeout without redeploying.

@ValentaTomas has some evidence that the sandbox in question would succeed after ~30-35 seconds. I agree that it's still a guess though. This feels more like a drive-by tweak; if we focus on it, agree we should invest.