Experiments

Experiments of 6 code search methods in the paper

We performed experiments on Cosbench using 6 methods, and the details are published in the paper "Are the Code Snippets What We Are Searching for? A Benchmark and an Empirical Study on Code Search with Natural-Language Queries".

The experimental raw data of our experiments.

Experiments of 6 code search methods on current QASet

Since the QAset is incomplete, the potential answers will continue to increase. The experimental results based on the current QAset are here.

For the convenience of Evaluation, we provide the Top-20 search answers given by the 6 code search methods in the folder. You can obtain the Evaluation results by modifying the file.

It should be noted that due to various factors (for example, different versions of the library used and the external data set), the Top-20 search answers obtained by the search method may be different. We only provide Top-20 search answers used in our experiments.

Experiments of new code search methods

If you want to use Cosbench to evaluate your code search method, The next steps may be helpful.

1. you may need to add more potential answers in the search answer obtained by your method to the QAset before evaluation since the current QAset is incomplete.

2. Clone CSES_IR.

3. Add a file in test/resources/searchResult which contained the code ids of search answers returned by your methods. One query per line in this file. Each line contains code ids of top-k search answers for this query.

4. You can obtain the Evaluation results in test/resources/evaluateResult folder by modifying and running the test/java/CS/Evaluate/EvaluateResult file.