More fine-grained coverage output control

[davidanthoff]

The VS Code test framework is going to get coverage support at some point, and I'd love to use that in the test item framework. For that to work, I would need a much more fine-grained control over coverage output in Julia base, though :)

In particular, Julia processes are re-used across multiple test runs in the test item framework, and I would need to get individual coverage results per test-run. So the model of having just one global coverage output per Julia instance doesn't really work.

I could think of various different ways to solve this.

One could be an API that deletes all accumulated coverage data in the current process and starts writing new coverage data into a new file (a bit like Profile.clear()):

restart_coverage_collection(new_output_filename)

Or maybe even more elegant would be a scoped coverage collection a la:

collect_coverage(output_filename) do
   # Code for which coverage should be collected goes here
end

Or maybe it would even be possible to not write coverage data to disk at all but rather collect in-memory?

coverage_data = collect_coverage() do
     # Code for which coverage should be collected goes here
end

Not sure how big coverage data gets and whether that is a good idea or not.

I think for my purposes any of these would work well, so maybe the first one would be the simplest?

[MilesCranmer]

Reading through the code after your post on the discourse thread (here).

It looks like the coverage data is stored in:

julia/src/coverage.cpp

Line 46 in da6892f

static logdata_t coverageData;

And this gets written to disk here:

julia/src/coverage.cpp

Line 205 in da6892f

write_log_data(coverageData, stm.c_str());

which gets called only when Julia is exiting, here:

julia/src/init.c

Line 289 in da6892f

jl_write_coverage_data(jl_options.output_code_coverage);

So the first order mechanism for live monitoring would be to have some hook so this coverage variable is accessible to the Julia runtime?

[davidanthoff]

Very cool!

Two other thoughts:

• it would be amazing if there was something like a flag that one could flip at runtime to pause and resume coverage collection.
• look at the code that @MilesCranmer identified, it also suggests that one could add an API to get access to the coverage information without going through a file on disc, which would also be nice for the VS Code use case.

[Seelengrab]

Related #50834

One difficulty with just adding a coverage flip is that you'll invalidate everything in your session, as far as I'm aware. This is the reason why every vendored stdlib/sysimage is compiled twice, once with coverage hooks and once without. This can greatly influence performance, effects etc - see #49978 for one such example.

[davidanthoff]

For now it looks like #54358 solved this for the VS Code extension, thanks @MilesCranmer :)