Skip to content

Commit fed0108

Browse files
josh-ramerjosh-ramer
authored
Scripting & documenting debugging one test without anything else in the loop. (#7096)
* A little documentation that shares my quick tips for working in the repository. * Update startup-testing-debugging.md * script that shows a menu of tests to pick from & run the debugger on * debug-test.sh: Refactor CLI help message * debug-test.sh: documentation update * debug-test.sh: CLI Help output corrections * debug-test.sh: minor doc fix --------- authored-by: Josh Ramer <[email protected]> Assisted-by: brian khuu <[email protected]>
1 parent 72c177c commit fed0108

File tree

2 files changed

+205
-0
lines changed

2 files changed

+205
-0
lines changed

‎docs/debugging-tests.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Debugging Tests Tips
2+
3+
## How to run & debug a specific test without anything else to keep the feedback loop short?
4+
5+
There is a script called debug-test.sh in the scripts folder whose parameter takes a REGEX and an optional test number.
6+
7+
For example, running the following command will output an interactive list from which you can select a test. It takes this form:
8+
9+
`debug-test.sh [OPTION]... <test_regex> <test_number>`
10+
11+
It will then build & run in the debugger for you.
12+
13+
```bash
14+
./scripts/debug-test.sh test-tokenizer
15+
16+
# Once in the debugger, i.e. at the chevrons prompt, setting a breakpoint could be as follows:
17+
>>> b main
18+
```
19+
20+
For further reference use `debug-test.sh -h` to print help.
21+
22+
&nbsp;
23+
24+
### How does the script work?
25+
If you want to be able to use the concepts contained in the script separately, the important ones are briefly outlined below.
26+
27+
#### Step 1: Reset and Setup folder context
28+
29+
From base of this repository, let's create `build-ci-debug` as our build context.
30+
31+
```bash
32+
rm -rf build-ci-debug && mkdir build-ci-debug && cd build-ci-debug
33+
```
34+
35+
#### Step 2: Setup Build Environment and Compile Test Binaries
36+
37+
Setup and trigger a build under debug mode. You may adapt the arguments as needed, but in this case these are sane defaults.
38+
39+
```bash
40+
cmake -DCMAKE_BUILD_TYPE=Debug -DLLAMA_CUDA=1 -DLLAMA_FATAL_WARNINGS=ON ..
41+
make -j
42+
```
43+
44+
#### Step 3.1: Identify Test Command for Debugging
45+
46+
The output of this command will give you the command & arguments needed to run GDB.
47+
48+
* `-R test-tokenizer` : looks for all the test files named `test-tokenizer*` (R=Regex)
49+
* `-N` : "show-only" disables test execution & shows test commands that you can feed to GDB.
50+
* `-V` : Verbose Mode
51+
52+
```bash
53+
ctest -R "test-tokenizer" -V -N
54+
```
55+
56+
This may return output similar to below (focusing on key lines to pay attention to):
57+
58+
```bash
59+
...
60+
1: Test command: ~/llama.cpp/build-ci-debug/bin/test-tokenizer-0 "~/llama.cpp/tests/../models/ggml-vocab-llama-spm.gguf"
61+
1: Working Directory: .
62+
Labels: main
63+
Test #1: test-tokenizer-0-llama-spm
64+
...
65+
4: Test command: ~/llama.cpp/build-ci-debug/bin/test-tokenizer-0 "~/llama.cpp/tests/../models/ggml-vocab-falcon.gguf"
66+
4: Working Directory: .
67+
Labels: main
68+
Test #4: test-tokenizer-0-falcon
69+
...
70+
```
71+
72+
So for test #1 we can tell these two pieces of relevant information:
73+
* Test Binary: `~/llama.cpp/build-ci-debug/bin/test-tokenizer-0`
74+
* Test GGUF Model: `~/llama.cpp/tests/../models/ggml-vocab-llama-spm.gguf`
75+
76+
#### Step 3.2: Run GDB on test command
77+
78+
Based on the ctest 'test command' report above we can then run a gdb session via this command below:
79+
80+
```bash
81+
gdb --args ${Test Binary} ${Test GGUF Model}
82+
```
83+
84+
Example:
85+
86+
```bash
87+
gdb --args ~/llama.cpp/build-ci-debug/bin/test-tokenizer-0 "~/llama.cpp/tests/../models/ggml-vocab-llama-spm.gguf"
88+
```

‎scripts/debug-test.sh‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
#!/bin/bash
2+
test_suite=${1:-}
3+
test_number=${2:-}
4+
5+
PROG=${0##*/}
6+
build_dir="build-ci-debug"
7+
8+
if [ x"$1" = x"-h" ] || [ x"$1" = x"--help" ]; then
9+
echo "Usage: $PROG [OPTION]... <test_regex> (test_number)"
10+
echo "Debug specific ctest program."
11+
echo
12+
echo "Options:"
13+
echo " -h, --help Display this help and exit"
14+
echo
15+
echo "Arguments:"
16+
echo " <test_regex> (Mandatory) Supply one regex to the script to filter tests"
17+
echo " (test_number) (Optional) Test number to run a specific test"
18+
echo
19+
echo "Example:"
20+
echo " $PROG test-tokenizer"
21+
echo " $PROG test-tokenizer 3"
22+
echo
23+
exit 0
24+
fi
25+
26+
# Function to select and debug a test
27+
function select_test() {
28+
test_suite=${1:-test}
29+
test_number=${2:-}
30+
31+
# Sanity Check If Tests Is Detected
32+
printf "\n\nGathering tests that fit REGEX: ${test_suite} ...\n"
33+
tests=($(ctest -R ${test_suite} -V -N | grep -E " +Test +#[0-9]+*" | cut -d':' -f2 | awk '{$1=$1};1'))
34+
if [ ${#tests[@]} -eq 0 ]
35+
then
36+
echo "No tests avaliable... check your compliation process..."
37+
echo "Exiting."
38+
exit 1
39+
fi
40+
41+
if [ -z $test_number ]
42+
then
43+
# List out avaliable tests
44+
printf "Which test would you like to debug?\n"
45+
id=0
46+
for s in "${tests[@]}"
47+
do
48+
echo "Test# ${id}"
49+
echo " $s"
50+
((id++))
51+
done
52+
53+
# Prompt user which test they wanted to run
54+
printf "\nRun test#? "
55+
read test_number
56+
else
57+
printf "\nUser Already Requested #${test_number}"
58+
fi
59+
60+
# Start GDB with the requested test binary and arguments
61+
printf "Debugging(GDB) test: ${tests[test_number]}\n"
62+
# Change IFS (Internal Field Separator)
63+
sIFS=$IFS
64+
IFS=$'\n'
65+
66+
# Get test args
67+
gdb_args=($(ctest -R ${test_suite} -V -N | grep "Test command" | cut -d':' -f3 | awk '{$1=$1};1' ))
68+
IFS=$sIFS
69+
printf "Debug arguments: ${gdb_args[test_number]}\n\n"
70+
71+
# Expand paths if needed
72+
args=()
73+
for x in $(echo ${gdb_args[test_number]} | sed -e 's/"\/\<//' -e 's/\>"//')
74+
do
75+
args+=($(echo $x | sed -e 's/.*\/..\//..\//'))
76+
done
77+
78+
# Execute debugger
79+
echo "gdb args: ${args[@]}"
80+
gdb --args ${args[@]}
81+
}
82+
83+
# Step 0: Check the args
84+
if [ -z "$test_suite" ]
85+
then
86+
echo "Usage: $PROG [OPTION]... <test_regex> (test_number)"
87+
echo "Supply one regex to the script to filter tests,"
88+
echo "and optionally a test number to run a specific test."
89+
echo "Use --help flag for full instructions"
90+
exit 1
91+
fi
92+
93+
# Step 1: Reset and Setup folder context
94+
## Sanity check that we are actually in a git repo
95+
repo_root=$(git rev-parse --show-toplevel)
96+
if [ ! -d "$repo_root" ]; then
97+
echo "Error: Not in a Git repository."
98+
exit 1
99+
fi
100+
101+
## Reset folder to root context of git repo
102+
pushd "$repo_root" || exit 1
103+
104+
## Create and enter build directory
105+
rm -rf "$build_dir" && mkdir "$build_dir" || exit 1
106+
107+
# Step 2: Setup Build Environment and Compile Test Binaries
108+
cmake -B "./$build_dir" -DCMAKE_BUILD_TYPE=Debug -DLLAMA_CUDA=1 -DLLAMA_FATAL_WARNINGS=ON || exit 1
109+
pushd "$build_dir" && make -j || exit 1
110+
111+
# Step 3: Debug the Test
112+
select_test "$test_suite" "$test_number"
113+
114+
# Step 4: Return to the directory from which the user ran the command.
115+
popd || exit 1
116+
popd || exit 1
117+
popd || exit 1

0 commit comments

Comments
 (0)