Skip to content

[Backend Tester] Clean up report output #13306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 129 commits into from
Aug 13, 2025
Merged

Conversation

GregoryComer
Copy link
Member

@GregoryComer GregoryComer commented Aug 12, 2025

Update the report output to be a bit cleaner. I've made the following changes:

  • Round decimal values to 3 decimal places.
  • Remove mean signed deviation and L2 norm. We have SNR, mean absolute error, and max element-wise error. I think that's sufficient.
  • Rename SQNR to SNR. Not all tests are quantized and we're using it as a general measure of tensor-wise error.
  • Split out the result column into pass/skip/fail and a detailed reason. This should be easier to parse at a glance.

Example output (not sure markdown formatting is weird here...):

Test ID Test Case Flow Params Result Result Detail Delegated Quantize Time (s) Lower Time (s) Delegated Nodes Undelegated Nodes Delegated Ops Undelegated Ops PTE Size (Kb) Output 0 Error Max Output 0 Error MAE Output 0 SNR Output 1 Error Max Output 1 Error MAE Output 1 SNR
test_add_dtype_float32_xnnpack test_add_dtype xnnpack {'dtype': torch.float32} Pass   TRUE   3.747 1 0 {'aten::add.Tensor': 1} {} 1.6 0 0 inf      
test_add_dtype_float32_xnnpack_static_int8_per_channel test_add_dtype xnnpack_static_int8_per_channel {'dtype': torch.float32} Pass   TRUE 0.66 0.818 7 0 {'aten::add.Tensor': 1, 'quantized_decomposed::dequantize_per_tensor': 3, 'quantized_decomposed::quantize_per_tensor': 3} {} 2 0 0 inf      

[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
Base automatically changed from gh/GregoryComer/116/head to main August 13, 2025 18:53
[ghstack-poisoned]
@GregoryComer GregoryComer added the release notes: none Do not include this in the release notes label Aug 13, 2025
@GregoryComer GregoryComer merged commit 8b04295 into main Aug 13, 2025
104 of 105 checks passed
@GregoryComer GregoryComer deleted the gh/GregoryComer/119/head branch August 13, 2025 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: none Do not include this in the release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants