Nucleus instance segmentation classification performance very poor

[deeplearningmaniac]

• TIA Toolbox version: 1.4.1
• Python version: 3.10.12
• Operating System: Ubuntu 22.04.3 LTS

Description

I've been trying to use some of the pretrained nucleus instance segmentation models to segment, in particular, epithelial cells and lymphocytes. However, from some of my testing, all of the models perform very poorly. I was hoping you might be able to help me figure out if I'm going wrong somewhere.

What I Did

I downloaded some data from the NuCLS dataset, ran the different pretrained models on some patches, and visually compared the results with the ground truth. All three models I tried -- PanNuke, MoNuSAC, and CoNSeP -- were able to segment the nuclei to varying degrees, but none of them were able to properly classify any epithelial cells, and MoNuSAC wasn't able to classify lymphocytes either.

Here is a Google Colab notebook replicating this: https://colab.research.google.com/drive/1D3h4A2R7LvMUhUbel8YullU3gkL4AVbm?usp=sharing

Am I doing something wrong? I've checked my code over and over again and don't see any obvious bugs. Any help would be appreciated.

If you'd like to try some patches other than the two that I included with the Google Drive download (which I custom uploaded for simplicity), you can find them here: https://drive.google.com/drive/folders/1eGlF9Dgu3WMEik4fqj0wJ13LKVufsfZ0 under the rgb/ and csv/ folders.

[shaneahmed]

Thanks @deeplearningmaniac, we will investigate this. It should work with PanNuke. There is a model trained on much larger dataset i.e., Conic that should help for now https://github.com/TissueImageAnalytics/CoNIC/blob/main/notebooks/baseline.ipynb

[Taylor-Kelce]

I noticed that hovernet from consep and panuke give very different classification. For example, I used sample image provided by example notebook 08 with the two different hovernets. The majority of the purple coloured nuclei are labeled as neoplastic epithelial by Pannuake:

On the other hand, they are labeled as background with HoverNet trained from Consep:
.

[shaneahmed]

Please note that models trained on different datasets will produce different results. We do not claim that the models we have trained would be flawless but these models should be able to produce results as listed in the documentation https://tia-toolbox.readthedocs.io/en/latest/_autosummary/tiatoolbox.models.architecture.hovernet.HoVerNet.html.

The best model in our opinion, so far, is the one provided with the CONIC challenge https://github.com/TissueImageAnalytics/CoNIC/tree/main/notebooks. You can directly load the model from CONIC using the notebooks provided.

We are also further clarifying the classes for the models in #751

[shaneahmed]

I noticed that hovernet from consep and panuke give very different classification. For example, I used sample image provided by example notebook 08 with the two different hovernets. The majority of the purple coloured nuclei are labeled as neoplastic epithelial by Pannuake:

On the other hand, they are labeled as background with HoverNet trained from Consep: .

The classification labels might be incorrect in this case. We are adding the documentation in #751