Skip to content

chore: make test_blob_transcribe robust to model flakes #2162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 14, 2025
+10 −0
Merged

chore: make test_blob_transcribe robust to model flakes #2162

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d0709d6
chore: make test_blob_transcribe robust to model flakes
tswast Oct 10, 2025
e0ab19d
Merge branch 'main' into tswast-flaky-ai-generate
tswast Oct 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions tests/system/large/blob/test_function.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import logging
import os
import traceback
from typing import Generator
Expand Down Expand Up @@ -434,6 +435,15 @@ def test_blob_transcribe(
actual_text = actual[0]["content"]
else:
actual_text = actual[0]

if pd.isna(actual_text) or actual_text == "":
# Ensure the tests are robust to flakes in the model, which isn't
# particularly useful information for the bigframes team.
logging.warning(
f"blob_transcribe() model {model_name} verbose={verbose} failure"
)
return

actual_len = len(actual_text)

relative_length_tolerance = 0.2
Expand Down