Db retries #146
Db retries #146
Conversation
|
|
||
| go 1.22.0 | ||
|
|
||
| toolchain go1.25.0 |
There was a problem hiding this comment.
Not needed for template
| type DBOSError struct { | ||
| Message string // Human-readable error message | ||
| Code DBOSErrorCode // Error type code for programmatic handling | ||
| IsBase bool // Internal errors that shouldn't be caught by user code |
This reverts commit 9127d2c.
| AutoRemove: true, | ||
| Binds: []string{ | ||
| fmt.Sprintf("%s:%s", hostPgDataVolumeName, pgData), | ||
| }, |
There was a problem hiding this comment.
It turns out that an explicit volume binding is required. On MacOS, exporting PGDATA is enough to instruct Docker desktop to mount the right thing. Docker desktop uses a light VM to run postgres, and persist this volume across container restarts.
On the github action runner -- and likely on other environments too? The mount is not persisted across containers restarts.
| wfID, err := ctx.GetWorkflowID() | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to get workflow ID: %w", err) | ||
| } | ||
| err = ctx.(*dbosContext).systemDB.updateWorkflowOutcome(WithoutCancel(ctx), updateWorkflowOutcomeDBInput{ | ||
| workflowID: wfID, | ||
| status: WorkflowStatusError, | ||
| err: newWorkflowUnexpectedInputType(fqn, fmt.Sprintf("%T", typedInput), fmt.Sprintf("%T", input)), | ||
| }) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to record unexpected input type error: %w", err) | ||
| } |
There was a problem hiding this comment.
This is not required because typedErasedWorkflow is always call within RunWorkflow, in a goroutine that update the workflow outcome.
| mockCtx.AssertExpectations(t) | ||
| mockChildHandle.AssertExpectations(t) | ||
| mockGenericHandle.AssertExpectations(t) |
There was a problem hiding this comment.
A mock built with mocks.NewMoc... already checks expectations before destroying the object.
kraftp
left a comment
kraftp
left a comment
There was a problem hiding this comment.
Looks good, important tests and fixes!
Address #93
This PR adds a
retrymethod that can take in another function for retry until a specific condition is met.retrycan be configured with functional options and has aretryWithResultgeneric counterpart for functions returning a value.All system database methods are now called within
retryorretryWithResult.For example
retry performs exponential backoff with jitter. It defaults to infinite retries.
The default condition for retrying is
isRetryablePGError, which matches the function returned error with a set of connection errors (postgres codes, pgx connection errors,net.Error)Note: in the particular case of RunWorkflow, which does manage transactions, the entire function has to be retried. This is because the transaction object becomes invalid as soon as Commit/Rollback have been called, regardless of whether the operation was successful. See https://github.com/jackc/pgx/blob/61d3c965ad442cc14d6b0e39e0ab3821f3684c03/tx.go#L180
This PR also improves
DBOSErrorwith the ability to unwrap the underlying error, if any.