Merging collector reports with `IDataCollectorAttachmentProcessor`

[daveMueller]

I want to extend our data collector to merge reports if someone is calling dotnet test --collect:"MyCollector" on solution level. I try to follow the description here 0031-Test-Run-Attachments-Processing but I'm having issues implementing it.

I implemented a specific IDataCollectorAttachmentProcessor and also registered it properly to the collector. I can also see that the method ProcessAttachmentSetsAsync is called every time a testproject finishes and the report is being generated.

But every time the post processor is triggered and I iterate over the ICollection<AttachmentSet> I can only see the report of the current test project. Thus I'm not really able to merge the reports of the whole solution.

I appreciate all the help that I can get on this topic 🙏 .

[nohwnd]

@MarcoRossignoli please respond here.

[MarcoRossignoli]

Hi @daveMueller I did a sample here

merge.zip

...\merge>dotnet test --collect "MyDataCollector"  --test-adapter-path "....\MyDataCollector\bin\Debug\netstandard2.0" --diag log.txt

In the processor I've added the code to attach in case we're called with 2 attachments (one per test).
The processor is called also once per project "in process". So in my sample 3 times.

[daveMueller]

OK thanks a lot for that. I made a mistake with the return value of ProcessAttachmentSetsAsync that led to the false behavior. I'll get back to that again in a minute. I still have some small questions:

1. I still don't fully understand when the post processing is triggered. As described here, it says that it is triggered after any two test executions are done. But from my experiments and also in your sample it additionally is triggered after finalizing each project report?

2. Does the newly merged report needs to be passed back to the test platform in addition to the input attachments? Something like:

       public Task<ICollection<AttachmentSet>> ProcessAttachmentSetsAsync(XmlElement configurationElement, ICollection<AttachmentSet> attachments, IProgress<int> progressReporter,
           IMessageLogger logger, CancellationToken cancellationToken)
       {
           if(attachments.Count <= 1) return Task.FromResult(attachments);
   
           // merge reports to new report
   
           attachments = new[] { new AttachmentSet(new Uri(filePath), report.fileName) }.Concat(attachments).ToList();
           return Task.FromResult(attachments);
       }

   This would also mean that the actual merge operation could be executed multiple times with intermediate merge results depending on the amount of test projects? Like also seen in here.

   If it is like that, I would love to know the last time the post processing is triggered to clean up the intermediate merge result files. Or can I do this in another way?

3. Is there a specific operation needed to tell the platform that there is a new post processed attachment? Like for instance inside the collector Microsoft.VisualStudio.TestPlatform.ObjectModel.DataCollection.DataCollectionSink.SendFileAsync(...)?

[MarcoRossignoli]

I still don't fully understand when the post processing is triggered. As described here, it says that it is triggered after any two test executions are done. But from my experiments and also in your sample it additionally is triggered after finalizing each project report?

Yes, the merge works the same way inside VS and there when we run multi-tfm the same "server" is running more modules, this is not the case for dotnet test. For that reason on dotnet test you'll see a call at the end of every project and another call at the end of all projects. From your question I suppose your doubt is if it's possible use the json format as intermediate and the real one at the end. I suppose that this is not possible so my suggestion on coverlet repo could be wrong. It's possible we need to merge every format separately.

Does the newly merged report needs to be passed back to the test platform in addition to the input attachments? Something like:

The processor select "your" assets(using the uris GetExtensionUris) and the input will be composed by that assets, you can elaborate and return a new set(one or more) you don't need to return the input if are no more useful and you can discard those.

If it is like that, I would love to know the last time the post processing is triggered to clean up the intermediate merge result files. Or can I do this in another way?

Unfortunately nope, we don't have at the moment a signal of "last merge", I suppose this is linked to the intermediate format as I've discussed above.

Is there a specific operation needed to tell the platform that there is a new post processed attachment? Like for instance inside the collector Microsoft.VisualStudio.TestPlatform.ObjectModel.DataCollection.DataCollectionSink.SendFileAsync(...)?

This question is not clear, why you need it? Post processing is done at the end of "the execution", after the data collectors are shootdown.

[daveMueller]

This question is not clear, why you need it? Post processing is done at the end of "the execution", after the data collectors are shootdown.

I was just thinking about where to store the final result then. For the collectors the file storage location is selected by the platform. For the post processing we might need to add another runsettins option to specify the location of the merged result.

[MarcoRossignoli]

I was just thinking about where to store the final result then. For the collectors the file storage location is selected by the platform. For the post processing we might need to add another runsettins option to specify the location of the merged result.

I would expect that all the attachments you receive inside the post processor come from the same result folder(default or user specified), so you can take the first one directory and create the final file there and return.

[daveMueller]

OK thanks for that. I guess everything is clear now.