ci: add features to assist in troubleshooting release test runs #879

pau-hedgehog · 2025-08-14T10:52:03Z

Add pause_on_fail input parameter for debugging failed release tests
Add -rt suffix to release test job names for better identification
Add regex (and inverse flag) for release test filtering input parameter

Copilot

Pull Request Overview

This pull request adds a new "pause on fail" feature for debugging VLAB release tests. The feature allows pausing test execution when a failure occurs, with different behavior for interactive vs CI environments.

Adds a PauseOnFail boolean option to VLAB configuration and CLI
Implements smart pause behavior: interactive prompt for terminal sessions, automatic 60-minute pause for CI
Updates GitHub Actions workflows to support the new pause-on-fail functionality

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/hhfab/vlabrunner.go	Adds PauseOnFail field to VLABRunOpts struct and passes it to release test configuration
pkg/hhfab/release.go	Implements pauseOnFail function with terminal detection and different pause behaviors
cmd/hhfab/main.go	Adds CLI flag for pause-on-fail option with alias "-p"
.github/workflows/run-vlab.yaml	Adds pause_on_fail input parameter and updates workflow to use it conditionally
.github/workflows/ci.yaml	Adds pause_on_fail input parameter and passes it to the run-vlab workflow

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.}

cmd/hhfab/main.go

Frostman

I think great improvements to allow pause on fail, regex/inverts as flags in CLI, but we shouldn't do that for the CI, and I'm not sure about the script

Frostman · 2025-09-14T06:31:19Z

.github/workflows/ci.yaml

        description: "Run release tests in vlab/hlab"
        required: false
        default: false
+      release_test_regex:


I think we shouldn't allow this in CI, it'll be very deceptive as when you start release tests you expect all to run

I barely use it but I added it to eventually target a specific flaky test on CI. It is tested and works.

The problem with deceptiveness is more a lack of visibility of what we are actually running (unless you dig into the logs) as the current summary just gives you the counts:

#529 (comment)

Frostman · 2025-09-14T06:31:25Z

.github/workflows/ci.yaml

+        description: "Run only release tests matched by regex"
+        required: false
+        default: ""
+      release_test_invert_regex:


Frostman · 2025-09-14T06:32:44Z

.github/workflows/ci.yaml

        description: "Run the build with tmate debugging enabled (https://github.com/marketplace/actions/debugging-with-tmate)"
        required: false
        default: false
+      pause_on_fail:


We shouldn't pause on failure especially on hlab as it's a very expensive resource. If debug is required, I suggest to disable worker and run bunch of e.g. release tests manually to have full control as debugging through runner is anyways always limited by timeouts and impossible to restart things.

We can exclude hlab or add an option in case you want to target a flaky test hitting there. As it's an expensive resource being hit continuously by flaky tests is also a waste of CI time, thus the whole idea of this PR, to be able to hunt issues actively. This proved useful in the TPCM Sonic VS issue. It is how I could nail it down.

By default no pipeline will be paused unless you trigger it specifically to. If you cordon the runner you will end up affecting more jobs than with this PR approach, IMHO

Frostman · 2025-09-14T06:33:44Z

.github/workflows/ci.yaml

    needs:
      - test-build
+
+    name: "${{ matrix.hybrid && 'h' || 'v' }}-${{ matrix.upgradefrom && 'up' || '' }}${{ matrix.upgradefrom }}${{ matrix.upgradefrom && '-' || '' }}${{ matrix.fabricmode == 'collapsed-core' && 'cc-' || '' }}${{ matrix.mesh && 'mesh-' || '' }}${{ matrix.gateway && 'gw-' || '' }}${{ matrix.includeonie && 'onie-' || '' }}${{ matrix.buildmode }}-${{ matrix.vpcmode }}${{ (inputs.releasetest == true || contains(github.event.pull_request.labels.*.name, 'ci:+release')) && '-rt' || '' }}"


We don't need to have -rt in a job name as it represents a topo and deployment type, not how it's tested. Detailed flags of what is running within it is visible in the beginning of the job already

Frostman · 2025-09-14T06:34:24Z

.github/workflows/run-vlab.yaml

 env:
  # global workflow configs
-  slug: "${{ inputs.hybrid && 'h' || 'v' }}-${{ inputs.upgradefrom }}${{ inputs.upgradefrom && '-' || '' }}${{ inputs.fabricmode == 'collapsed-core' && 'cc-' || '' }}${{ inputs.mesh && 'mesh-' || '' }}${{ inputs.gateway && 'gw-' || '' }}${{ inputs.includeonie && 'onie-' || '' }}${{ inputs.buildmode }}-${{ inputs.vpcmode }}"
+  slug: "${{ inputs.hybrid && 'h' || 'v' }}-${{ inputs.upgradefrom }}${{ inputs.upgradefrom && '-' || '' }}${{ inputs.fabricmode == 'collapsed-core' && 'cc-' || '' }}${{ inputs.mesh && 'mesh-' || '' }}${{ inputs.gateway && 'gw-' || '' }}${{ inputs.includeonie && 'onie-' || '' }}${{ inputs.buildmode }}-${{ inputs.vpcmode }}${{ inputs.releasetest && '-rt' || '' }}"


Frostman · 2025-09-14T06:45:39Z

hack/monitor-ci-paused-jobs.sh

@@ -0,0 +1,232 @@
+#!/bin/bash


That looks like quite a big and generic script to maintain in the fabricator repo. You can put it in some lab/internal repo if you think it makes sense to have in git somewhere.

Yes, I was doubting to add it here or lab repo, where I have a related script to jump into the runners. I was proposing here because it could be handy for anyone to get a compact overview of running Ci workflows without having to drill down into the GHA Web UI:

⏳ Workflow: CI (87m) - Branch: pau/ci_reltest_pause 🟡 Workflow: CI (93m) - Branch: dev/frostman/stability ✅ v-mesh-iso-l3vni (Runner: 'vlab-6bggx-runner-bgqj2') ✅ v-gw-onie-usb-l2vni (Runner: 'vlab-6bggx-runner-ww25s') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770734 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492869 🟡 v-up25.04-iso-l3vni (Runner: 'vlab-6bggx-runner-9xptf') (47m) ✅ v-manual-l2vni (Runner: 'vlab-6bggx-runner-mc8sn') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770738 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492878 ✅ v-iso-l3vni (Runner: 'vlab-6bggx-runner-2pnfz') 🟡 v-iso-l3vni (Runner: 'vlab-6bggx-runner-v5v5d') (40m) → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770740 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492879 🟡 v-mesh-iso-l2vni (Runner: 'vlab-6bggx-runner-rj4bx') (34m) ✅ v-gw-onie-iso-l2vni (Runner: 'vlab-6bggx-runner-xc272') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770764 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492882 ✅ v-gw-iso-l3vni (Runner: 'vlab-6bggx-runner-gz6km') ✅ v-gw-iso-l3vni (Runner: 'vlab-6bggx-runner-wlssw') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770765 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492886 ⏳ h-iso-l2vni 🟡 h-iso-l2vni (Runner: 'hlab-gbvbd-runner-4djxz') (85m) → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770767 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492898 ✅ v-gw-onie-usb-l2vni (Runner: 'vlab-6bggx-runner-p29gz') ✅ v-up25.04-iso-l3vni (Runner: 'vlab-6bggx-runner-9jjl6') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770770 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492902 🟡 v-usb-l2vni (Runner: 'vlab-6bggx-runner-cnssw') (29m) ✅ v-usb-l2vni (Runner: 'vlab-6bggx-runner-qbpgc') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770781 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492922 ✅ v-manual-l2vni (Runner: 'vlab-6bggx-runner-8c4nd') ✅ v-cc-iso-l2vni (Runner: 'vlab-6bggx-runner-67jds') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770785 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492929 ✅ v-cc-iso-l2vni (Runner: 'vlab-6bggx-runner-t4mj9') ✅ v-mesh-iso-l2vni (Runner: 'vlab-6bggx-runner-rlfhk') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770787 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365492949 ✅ v-iso-l2vni (Runner: 'vlab-6bggx-runner-v6dkl') ✅ v-mesh-iso-l3vni (Runner: 'vlab-6bggx-runner-wxp6t') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770788 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365493006 🟡 v-gw-onie-iso-l2vni (Runner: 'vlab-6bggx-runner-kvh42') (26m) ✅ v-iso-l2vni (Runner: 'vlab-6bggx-runner-knwth') → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770806 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365493094 🟡 v-up25.04-iso-l2vni (Runner: 'vlab-6bggx-runner-4lvjq') (33m) 🟡 v-up25.04-iso-l2vni (Runner: 'vlab-6bggx-runner-8jvng') (68m) → https://github.com/githedgehog/fabricator/actions/runs/17725549258/job/50365770807 → https://github.com/githedgehog/fabricator/actions/runs/17725413073/job/50365493112 === Updated: 11:00:25 | ⏳ Queued 🟡 Running ✅ Success ❌ Failed 🚫 Cancelled 🚨 Paused ===

This will save me hours of browsing the UI. I will move it to the lab repo

Frostman · 2025-09-14T06:46:22Z

pkg/hhfab/show-tech/switch.sh

+    echo "=== System Information and Status ==="
    run_sonic_cmd "show version"
    run_sonic_cmd "show uptime"
+    run_sonic_cmd "show system status brief"


It's better to run status detail as brief would be limited to the system readiness summary

Frostman · 2025-09-14T06:48:03Z

pkg/hhfab/release.go

-	var input string
-	_, _ = fmt.Scanln(&input)
-	slog.Info("Continuing...")
+	if !term.IsTerminal(int(os.Stdin.Fd())) {


We shouldn't make CI pause, especially for 60 minutes. It'll be still almost impossible to catch as it's a very short window of opportunity plus what I said in a first comment

github-actions · 2025-09-15T21:39:05Z

Test Results

12 files 48 suites 2h 59m 39s ⏱️
20 tests 6 ✅ 14 💤 0 ❌
240 runs 63 ✅ 177 💤 0 ❌

Results for commit 3fb39b5.

♻️ This comment has been updated with latest results.

Signed-off-by: Pau Capdevila <[email protected]>

pau-hedgehog · 2025-09-19T13:58:17Z

@Frostman , please comment if you find this useful (I may keep rebasing the branch for personal use when in need for debugging) or not and if shall i remove the rt and regex commits or not. Thanks in advance

Copilot

Pull Request Overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-30T20:55:32Z

pkg/hhfab/release.go

 func pauseOnFail() {
-	// pause until the user presses enter
-	slog.Warn("Test failed, pausing execution. Note that reverts might still need to apply, so if you intend to continue, please make sure to leave the environment in the same state as you found it")
-	slog.Info("Press enter to continue...")
-	var input string
-	_, _ = fmt.Scanln(&input)
-	slog.Info("Continuing...")
+	if !term.IsTerminal(int(os.Stdin.Fd())) {


Consider adding error handling for os.Stdin.Fd() as it can fail in some environments where stdin is not available or accessible.

Suggested change

func pauseOnFail() {

// pause until the user presses enter

slog.Warn("Test failed, pausing execution. Note that reverts might still need to apply, so if you intend to continue, please make sure to leave the environment in the same state as you found it")

slog.Info("Press enter to continue...")

var input string

_, _ = fmt.Scanln(&input)

slog.Info("Continuing...")

if !term.IsTerminal(int(os.Stdin.Fd())) {

func isStdinTerminal() bool {

defer func() {

if r := recover(); r != nil {

// If Fd() panics, treat as non-terminal

}

}()

return term.IsTerminal(int(os.Stdin.Fd()))

}

func pauseOnFail() {

if !isStdinTerminal() {

Copilot · 2025-09-30T20:55:32Z

pkg/hhfab/release.go

-	slog.Info("Continuing...")
+	if !term.IsTerminal(int(os.Stdin.Fd())) {
+		// CI environment - pause for a long time to allow debugging
+		pauseDuration := 60 * time.Minute


The hardcoded 60-minute pause duration should be configurable or at least defined as a constant to make it easier to adjust for different debugging scenarios.

Copilot · 2025-09-30T20:55:32Z

.github/workflows/ci.yaml

+    name: "${{ matrix.hybrid && 'h' || 'v' }}-${{ matrix.upgradefrom && 'up' || '' }}${{ matrix.upgradefrom }}${{ matrix.upgradefrom && '-' || '' }}${{ matrix.fabricmode == 'collapsed-core' && 'cc-' || '' }}${{ matrix.mesh && 'mesh-' || '' }}${{ matrix.gateway && 'gw-' || '' }}${{ matrix.includeonie && 'onie-' || '' }}${{ matrix.buildmode }}-${{ matrix.vpcmode }}${{ (inputs.releasetest == true || contains(github.event.pull_request.labels.*.name, 'ci:+release')) && '-rt' || '' }}"
+


This job name expression is extremely long and complex. Consider breaking it into multiple variables or using a more readable format to improve maintainability.

Suggested change

name: "${{ matrix.hybrid && 'h' || 'v' }}-${{ matrix.upgradefrom && 'up' || '' }}${{ matrix.upgradefrom }}${{ matrix.upgradefrom && '-' || '' }}${{ matrix.fabricmode == 'collapsed-core' && 'cc-' || '' }}${{ matrix.mesh && 'mesh-' || '' }}${{ matrix.gateway && 'gw-' || '' }}${{ matrix.includeonie && 'onie-' || '' }}${{ matrix.buildmode }}-${{ matrix.vpcmode }}${{ (inputs.releasetest == true || contains(github.event.pull_request.labels.*.name, 'ci:+release')) && '-rt' || '' }}"

env:

HYBRID_PREFIX: ${{ matrix.hybrid && 'h' || 'v' }}

UPGRADE_PREFIX: ${{ matrix.upgradefrom && 'up' || '' }}

UPGRADE_VALUE: ${{ matrix.upgradefrom }}

UPGRADE_DASH: ${{ matrix.upgradefrom && '-' || '' }}

FABRICMODE_PREFIX: ${{ matrix.fabricmode == 'collapsed-core' && 'cc-' || '' }}

MESH_PREFIX: ${{ matrix.mesh && 'mesh-' || '' }}

GW_PREFIX: ${{ matrix.gateway && 'gw-' || '' }}

ONIE_PREFIX: ${{ matrix.includeonie && 'onie-' || '' }}

BUILD_MODE: ${{ matrix.buildmode }}

VPC_MODE: ${{ matrix.vpcmode }}

RELEASE_TEST: ${{ (inputs.releasetest == true || contains(github.event.pull_request.labels.*.name, 'ci:+release')) && '-rt' || '' }}

name: "${{ env.HYBRID_PREFIX }}-${{ env.UPGRADE_PREFIX }}${{ env.UPGRADE_VALUE }}${{ env.UPGRADE_DASH }}${{ env.FABRICMODE_PREFIX }}${{ env.MESH_PREFIX }}${{ env.GW_PREFIX }}${{ env.ONIE_PREFIX }}${{ env.BUILD_MODE }}-${{ env.VPC_MODE }}${{ env.RELEASE_TEST }}"

pau-hedgehog self-assigned this Aug 14, 2025

pau-hedgehog force-pushed the pau/ci_reltest_pause branch 7 times, most recently from d00a2f8 to 6564811 Compare August 14, 2025 21:27

pau-hedgehog requested a review from Copilot August 14, 2025 22:19

Copilot AI reviewed Aug 14, 2025

View reviewed changes

cmd/hhfab/main.go Show resolved Hide resolved

pau-hedgehog force-pushed the pau/ci_reltest_pause branch from 6564811 to f9a9b54 Compare August 15, 2025 00:41

pau-hedgehog changed the title ~~Pau/ci reltest pause~~ ci: add pause-on-fail flag and split connectivity-test from release-test Aug 15, 2025

pau-hedgehog force-pushed the pau/ci_reltest_pause branch 4 times, most recently from 1b281b8 to b337470 Compare August 15, 2025 01:37

pau-hedgehog changed the title ~~ci: add pause-on-fail flag and split connectivity-test from release-test~~ ci: add pause-on-fail flag add suffix for release-test in jobs/artifacts Aug 16, 2025

pau-hedgehog force-pushed the pau/ci_reltest_pause branch 4 times, most recently from 3fd4c53 to f99a064 Compare August 17, 2025 21:14

pau-hedgehog force-pushed the pau/ci_reltest_pause branch 4 times, most recently from 5009b1d to 962c3ba Compare September 10, 2025 08:18

pau-hedgehog force-pushed the pau/ci_reltest_pause branch from 962c3ba to 0754bc2 Compare September 10, 2025 23:44

pau-hedgehog marked this pull request as ready for review September 10, 2025 23:44

pau-hedgehog requested review from a team as code owners September 10, 2025 23:44

pau-hedgehog force-pushed the pau/ci_reltest_pause branch from 0754bc2 to fa70dfc Compare September 11, 2025 06:25

pau-hedgehog changed the title ~~ci: add pause-on-fail flag add suffix for release-test in jobs/artifacts~~ ci: add features to assist in troubleshooting release test runs Sep 11, 2025

pau-hedgehog force-pushed the pau/ci_reltest_pause branch from fa70dfc to 4ed61eb Compare September 13, 2025 17:38

pau-hedgehog requested a review from a team as a code owner September 13, 2025 17:38

pau-hedgehog force-pushed the pau/ci_reltest_pause branch from 4ed61eb to 941cf11 Compare September 13, 2025 18:04

Frostman requested changes Sep 14, 2025

View reviewed changes

pau-hedgehog force-pushed the pau/ci_reltest_pause branch from 848f049 to b1237b0 Compare September 15, 2025 16:35

pau-hedgehog added 5 commits September 15, 2025 23:46

ci: add -rt to release test job names

d6b1da9

Signed-off-by: Pau Capdevila <[email protected]>

ci: add workflow dispatch flag for pause-on-fail

59021be

Signed-off-by: Pau Capdevila <[email protected]>

chore: add pause on fail to on-ready commands

052b923

Signed-off-by: Pau Capdevila <[email protected]>

ci: allow release test regexes in workflow dispatch

411d64b

Signed-off-by: Pau Capdevila <[email protected]>

ci: add workflow dispatch option to select hlab

3fb39b5

Signed-off-by: Pau Capdevila <[email protected]>

pau-hedgehog force-pushed the pau/ci_reltest_pause branch from b1237b0 to 3fb39b5 Compare September 15, 2025 21:46

Frostman requested a review from Copilot September 30, 2025 20:54

Copilot AI reviewed Sep 30, 2025

View reviewed changes

		name: "${{ matrix.hybrid && 'h' \|\| 'v' }}-${{ matrix.upgradefrom && 'up' \|\| '' }}${{ matrix.upgradefrom }}${{ matrix.upgradefrom && '-' \|\| '' }}${{ matrix.fabricmode == 'collapsed-core' && 'cc-' \|\| '' }}${{ matrix.mesh && 'mesh-' \|\| '' }}${{ matrix.gateway && 'gw-' \|\| '' }}${{ matrix.includeonie && 'onie-' \|\| '' }}${{ matrix.buildmode }}-${{ matrix.vpcmode }}${{ (inputs.releasetest == true \|\| contains(github.event.pull_request.labels.*.name, 'ci:+release')) && '-rt' \|\| '' }}"

-    name: "${{ matrix.hybrid && 'h' || 'v' }}-${{ matrix.upgradefrom && 'up' || '' }}${{ matrix.upgradefrom }}${{ matrix.upgradefrom && '-' || '' }}${{ matrix.fabricmode == 'collapsed-core' && 'cc-' || '' }}${{ matrix.mesh && 'mesh-' || '' }}${{ matrix.gateway && 'gw-' || '' }}${{ matrix.includeonie && 'onie-' || '' }}${{ matrix.buildmode }}-${{ matrix.vpcmode }}${{ (inputs.releasetest == true || contains(github.event.pull_request.labels.*.name, 'ci:+release')) && '-rt' || '' }}"
+    env:
+      HYBRID_PREFIX: ${{ matrix.hybrid && 'h' || 'v' }}
+      UPGRADE_PREFIX: ${{ matrix.upgradefrom && 'up' || '' }}
+      UPGRADE_VALUE: ${{ matrix.upgradefrom }}
+      UPGRADE_DASH: ${{ matrix.upgradefrom && '-' || '' }}
+      FABRICMODE_PREFIX: ${{ matrix.fabricmode == 'collapsed-core' && 'cc-' || '' }}
+      MESH_PREFIX: ${{ matrix.mesh && 'mesh-' || '' }}
+      GW_PREFIX: ${{ matrix.gateway && 'gw-' || '' }}
+      ONIE_PREFIX: ${{ matrix.includeonie && 'onie-' || '' }}
+      BUILD_MODE: ${{ matrix.buildmode }}
+      VPC_MODE: ${{ matrix.vpcmode }}
+      RELEASE_TEST: ${{ (inputs.releasetest == true || contains(github.event.pull_request.labels.*.name, 'ci:+release')) && '-rt' || '' }}
+    name: "${{ env.HYBRID_PREFIX }}-${{ env.UPGRADE_PREFIX }}${{ env.UPGRADE_VALUE }}${{ env.UPGRADE_DASH }}${{ env.FABRICMODE_PREFIX }}${{ env.MESH_PREFIX }}${{ env.GW_PREFIX }}${{ env.ONIE_PREFIX }}${{ env.BUILD_MODE }}-${{ env.VPC_MODE }}${{ env.RELEASE_TEST }}"

Uh oh!

ci: add features to assist in troubleshooting release test runs #879

Are you sure you want to change the base?

ci: add features to assist in troubleshooting release test runs #879

Uh oh!

Conversation

pau-hedgehog commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Frostman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pau-hedgehog Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

pau-hedgehog commented Sep 19, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pau-hedgehog commented Aug 14, 2025 •

edited

Loading

pau-hedgehog Sep 15, 2025 •

edited

Loading

github-actions bot commented Sep 15, 2025 •

edited

Loading