Tips for modifying this for YoloV5

[Transigent]

Hi I was excited to find this project, I hoped that YoloV4 and YoloV5 were similar enough that I could successfully use it to run my YoloV5 ONNX-exported model. But apparently it's not as simple as changing the path to the model and recompiling... :) Unfortunately I don't know a lot about the key model properties that you use in this code.

The first thing I noted was that Netron reports different shapes for the inputs and outputs. The YoloV4 model input is shaped
Input { 1, 416, 416, 3 } and outputs { 1, 52, 52, 3, 85 }, { 1, 26, 26, 3, 85 }, { 1, 13, 13, 3, 85 }
and the YoloV5 model (YOLOV5l) is shaped:
Input { 1, 3, 640, 640 } and outputs { 1, 3, 80, 80, 19 }, { 1, 3, 40, 40, 19 }, { 1, 3, 20, 20, 19 }

I changed the code to reflect these differences along with the column names for the inputs and outputs.

I also changed the anchor figures according to what I found in lines 8-10 under anchors here: https://github.com/ultralytics/yolov5/blob/master/models/yolov5l.yaml
as well as the SHAPES constants.

I changed all references to 416 to 640 as thats the default pixel dimension.

It appeared to run but at the line
var results = predict.GetResults(classesNames, 0.3f, 0.7f);
I got 11,000 results. Clearly my changes were not enough.

I havent changed the XYSCALE constants as I am not sure what this is.

Do you have any thoughts about how I might get this to work? It it actually likely to work at all, ie. is YoloV5 too different to work with this code at all?

Thanks for any tips.

[BobLd]

Hi @Transigent, do you have an onnx version of the model?

[Transigent]

Hi BobLd

Thanks very much for the response, I really appreciate it!

Here is an ONNX export of the original YOLOV5l model pretrained on COCO with the default 80 classes. Unlike my version with 14 classes it will work with everyday images, plus I haven't contaminated it with training.
https://drive.google.com/drive/folders/18PpGCnQ4Ca4vRMBQbvSta3chqtuh2zXV?usp=sharing

Let me know if you need anything else.

[BobLd]

Thanks @Transigent.

A first comment concerning the input size: because the color dimension appears first in your model ( { 1, *3*, 640, 640 } ), I think you will need to set the interleavePixelColors to false to have something like this:
.Append(mlContext.Transforms.ExtractPixels(outputColumnName: "images", scaleImage: 1f / 255f, interleavePixelColors: false))

Also, can you give the link to the part where the image is pre-processed in the python library? I need it in order to understand if you need to scale the image by 1f / 255f or not.

I'll have a look soon at the rest.

EDIT: same remark for the output layers, the color channel comes first. I think the offset will need to be changed accordingly.

[Transigent]

Great, once again thanks so much for your thoughts. I will sit down with the code after work tomorrow when I get a second, and try to see how it works. I'm having trouble understanding it at a glance.

The closest things that might have relevance are here and here

Also the anchor constants are here

Again thanks for all your time!

[BobLd]

@Transigent Any progress on your side? If not, I'll try to have a look soon

[Transigent]

Hi @BobLd , sorry I haven't been in touch, we had to farewell our cat due to illness. It has been difficult. When the situation permits I will get back to this.

[Awakawaka]

Hi, @BobLd, currently I'm struggling to do the same as @Transigent was trying to do:(
I'm using yolov4-tiny which is trained on my custom dataset. I've converted this model to onnx format by using a python script which I grabbed from this repository https://github.com/Tianxiaomo/pytorch-YOLOv4. I don't really understand how VectorType attr is working and googling wasn't helped me at all. For example, my model output shape is [1,2535,1,4], how this will be represented in a one-dimensional array?
I would really appreciate your help and if needed I'll move this topic to another isssue

[BobLd]

Hi @Awakawaka,

You need your output as a float[]. Try creating a predictionEngine like I did here.

Then, if your output is of size [1, 2535, 1, 4], you can basically skip the 1s, leaving you with a 2D array of shape [2535, 4]. ML.Net will give you a 1D array representation of size 2535 x 4 = 10140.

My guess is that the size 4 is for a single bounding box (4 elements: x1, y1, x2, y2), and you have 2535 bounding boxes.

Try the following:

// output is your model's 1D prediction array of size 2535 * 4 = 10140
List<(float, float, float, float)> bboxes = new List<(float, float, float, float)>();
for (int i = 0; i < 2535 * 4; i+=4)
{
    bboxes.Add((output[i], output[i+1], output[i+2], output[i+3]));
}

=> bboxes will contain all your bounding box predictions. Then you will certainly need to do some post-processing of the bounding boxes coordinates.

Hope this helps

[ESEricWare]

Hi guys,

I'm trying to do the same.
Is there any progress from your side?

Thanks a lot

[BobLd]

Hi all,

I did some progress, the categories now seems to be correct but not the bounding boxes yet. The part you are missing is that Detect() needs to be implemented in C# (it is not present in the onnx model).

Please have a look here: https://github.com/BobLd/YOLOv4MLNet/tree/yolo-v5 for the latest progress.

Please find more info here:

• Use export.py to generate yolov5s.onnx will get a negative number. ultralytics/yolov5#343 (comment)
• https://github.com/ultralytics/yolov5/blob/a1c8406af3eac3e20d4dd5d327fd6cbd4fbb9752/models/yolo.py#L29-L36

post-processing is also done here:

• https://github.com/ultralytics/yolov5/blob/68e6ab668b30a6014215b94e399151f8c76e471a/detect.py#L18
• https://github.com/ultralytics/yolov5/blob/68e6ab668b30a6014215b94e399151f8c76e471a/utils/general.py#L261

Any help welcome on the missing bits!

[kellansteele]

Hi, I'm trying to use YOLOv5 with ML.Net and have a few questions about where your project is so far!

1. Say I was to export my YOLOv5s.pt model to ONNX format with the Detect() layer included (following ultralytics/yolov5#343 (comment). In your code, have you implemented any of the Detect() layer so far? If so, where?
2. Further down in ultralytics/yolov5#343 (comment), they mention the following:

In both cases, you do miss the following:

- filtering results with objectness lower than some threshold
- NMS
- conversion from xc, yc, w, h to x1, y1, x2, y2

I've had a look at your code and it looks like you've implemented both the filtering results with objectness and conversion steps but not NMS. Is this correct?

Thanks so much!

[deanbennettdeveloper]

I've managed to get something working, but my bounding boxes are slightly off all the time. Would this be the NMS routine too maybe. I've taken the raw output by switching the export=False and simply using this output (I did something similar with Yolov4).

As it happens I'm not interested in the exact boxes for my application, just objects and their confidence rating so this works perfectly for me and super quick even on a CPU.

[deanbennettdeveloper]

What I'm seeing is exactly what is mentioned here to be fair. So I think its definitely that I'm missing a final set of steps with the bounding boxes, but like I say I'm only interested in confidences and labels at the moment.

https://towardsdatascience.com/object-detection-part1-4dbe5147ad0a

[BobLd]

@deanbennettdeveloper
Thanks for your help, I will have a look at the code you provided here. Did you modify anything else in the code? Your code basically replaces my NMS code, right?

@kellansteele:

• Would you be able to share a link to a YoloV5 model with Detect included?
• Detect (not exactly following the same structure, a bit messy):

  YOLOv4MLNet/YOLOv4MLNet/DataStructures/YoloV4Prediction.cs

  Lines 62 to 114 in c9e1f42

  	for (int i = 0; i < results.Length; i++)
  	{
  	var pred = results[i];
  	var outputSize = shapes[i];
  	for (int boxY = 0; boxY < outputSize; boxY++)
  	{
  	for (int boxX = 0; boxX < outputSize; boxX++)
  	{
  	for (int a = 0; a < anchorsCount; a++)
  	{
  	var offset = (boxY * outputSize * (classesCount + 5) * anchorsCount) + (boxX * (classesCount + 5) * anchorsCount) + a * (classesCount + 5);
  	var predBbox = pred.Skip(offset).Take(classesCount + 5).Select(x => Sigmoid(x)).ToArray(); // y = x[i].sigmoid()
  	// more info at
  	// https://github.com/ultralytics/yolov5/issues/343#issuecomment-658021043
  	// https://github.com/ultralytics/yolov5/blob/a1c8406af3eac3e20d4dd5d327fd6cbd4fbb9752/models/yolo.py#L29-L36
  	// postprocess_bbbox()
  	var predXywh = predBbox.Take(4).ToArray();
  	var predConf = predBbox[4];
  	var predProb = predBbox.Skip(5).ToArray();
  	var rawDx = predXywh[0];
  	var rawDy = predXywh[1];
  	var rawDw = predXywh[2];
  	var rawDh = predXywh[3];
  	float predX = ((rawDx * 2f) - 0.5f + boxX) * STRIDES[i];
  	float predY = ((rawDy * 2f) - 0.5f + boxY) * STRIDES[i];
  	float predW = (float)Math.Pow(rawDw * 2, 2) * ANCHORS[i][a][0];
  	float predH = (float)Math.Pow(rawDh * 2, 2) * ANCHORS[i][a][1];
  	// postprocess_boxes
  	// (1) (x, y, w, h) --> (xmin, ymin, xmax, ymax)
  	var box = Xywh2xyxy(new float[] { predX, predY, predW, predH });
  	float predX1 = box[0]; //predX - predW * 0.5f;
  	float predY1 = box[1]; //predY - predH * 0.5f;
  	float predX2 = box[2]; //predX + predW * 0.5f;
  	float predY2 = box[3]; //predY + predH * 0.5f;
  	// (2) (xmin, ymin, xmax, ymax) -> (xmin_org, ymin_org, xmax_org, ymax_org)
  	float org_h = ImageHeight;
  	float org_w = ImageWidth;
  	float inputSize = 640f;
  	float resizeRatio = Math.Min(inputSize / org_w, inputSize / org_h);
  	float dw = (inputSize - resizeRatio * org_w) / 2f;
  	float dh = (inputSize - resizeRatio * org_h) / 2f;
  	var orgX1 = 1f * (predX1 - dw) / resizeRatio; // left
  	var orgX2 = 1f * (predX2 - dw) / resizeRatio; // right
  	var orgY1 = 1f * (predY1 - dh) / resizeRatio; // top
  	var orgY2 = 1f * (predY2 - dh) / resizeRatio; // bottom

• NMS (Non max suppresion) - As you can see, the line 165 is commented to deactivate the NMS for testing purpose. Just uncomment the line to reactivate it:

  YOLOv4MLNet/YOLOv4MLNet/DataStructures/YoloV4Prediction.cs

  Lines 139 to 169 in c9e1f42

  	// Non-maximum Suppression
  	postProcesssedResults = postProcesssedResults.OrderByDescending(x => x[4]).ToList(); // sort by confidence
  	List<YoloV4Result> resultsNms = new List<YoloV4Result>();
  	int f = 0;
  	while (f < postProcesssedResults.Count)
  	{
  	var res = postProcesssedResults[f];
  	if (res == null)
  	{
  	f++;
  	continue;
  	}
  	var conf = res[4];
  	string label = categories[(int)res[5]];
  	resultsNms.Add(new YoloV4Result(res.Take(4).ToArray(), label, conf));
  	postProcesssedResults[f] = null;
  	var iou = postProcesssedResults.Select(bbox => bbox == null ? float.NaN : BoxIoU(res, bbox)).ToList();
  	for (int i = 0; i < iou.Count; i++)
  	{
  	if (float.IsNaN(iou[i])) continue;
  	if (iou[i] > iouThres)
  	{
  	//postProcesssedResults[i] = null; // deactivated for debugging
  	}
  	}
  	f++;
  	}