GitHub actions regression test #499
Conversation
| HIGHER_IS_BETTER = ["QPS", "Recall@10"] | ||
| LOWER_IS_BETTER = ["Mean Latency"] | ||
|
|
||
| class BenchmarkData: |
There was a problem hiding this comment.
I suggest leaving two blank lines between top-level functions and two blank lines between the import statements and other code: https://stackoverflow.com/questions/2953250/python-pep8-blank-lines-convention
There was a problem hiding this comment.
Done and commited
| @@ -0,0 +1,183 @@ | |||
| /* | |||
There was a problem hiding this comment.
Should we tidy up the main example directory? IMO, it is getting polluted. Maybe we should add a "tools" directory or similar. We can do a separate PR for this, no need to do it here as there are other files that should probably go there too and that have nothing to do with this PR.
There was a problem hiding this comment.
Noted and agreed
|
I suggest editing the description of this PR with a high-level explanation of what it contains/does. It would be great if it also contained an example of the output. |
This PR introduces a new github workflow and associated code to perform automated regression testing across branches. The workflow can be triggered manually with the branches to compare included as inputs. It is also triggered to run automatically when a PR to main is opened, in which case it will run a regression test comparing the requested branch with main.
Metrics compared include QPR, average latency, and recall at k10. See attached for an example of the output.
benchmark_report.zip