Skip to content

✨ Support for Additional Foundation Models #906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 21 commits into from
Mar 7, 2025
Merged

✨ Support for Additional Foundation Models #906

merged 21 commits into from
Mar 7, 2025

Conversation

@adamshephard
Copy link
Contributor

@adamshephard adamshephard commented Jan 22, 2025 •
edited
Loading

  • Add support for additional foundation models as feature extractors using the TimmBackbone.
  • Added models include: UNI2, Virchow, Virchow2, kaiko and H-optimus-1.
  • Add more information to docstrings.
  • Allow foundation models with additional parameters.

@adamshephard
Update vanilla.py
d123b6c 
Add UNI2 foundation model
@adamshephard adamshephard changed the title Add extra foundation models ✨Add extra foundation models Jan 22, 2025
@codecov
Copy link

codecov bot commented Jan 23, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.90%. Comparing base (2416ba9) to head (580573f).
Report is 16 commits behind head on develop.

Additional details and impacted files 
@@           Coverage Diff            @@
##           develop     #906   +/-   ##
========================================
  Coverage    99.90%   99.90%           
========================================
  Files           70       70           
  Lines         8736     8738    +2     
  Branches      1149     1150    +1     
========================================
+ Hits          8728     8730    +2     
  Misses           3        3           
  Partials         5        5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shaneahmed shaneahmed changed the title ✨Add extra foundation models ✨Add Support for UNI-2 Jan 23, 2025
@shaneahmed shaneahmed added this to the Release v1.7.0 milestone Jan 24, 2025
@shaneahmed shaneahmed added the enhancement New feature or request label Jan 24, 2025
adamshephard and others added 3 commits January 24, 2025 11:36
@adamshephard
Update vanilla.py
2b584e0 
Add extra models including Virchow, Virchow2
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
9f8277e 
for more information, see https://pre-commit.ci
@adamshephard
Update vanilla.py
10ec832 
Add kaiko
@shaneahmed shaneahmed changed the title ✨Add Support for UNI-2 🆕 Add Support for UNI-2 Jan 24, 2025
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
adamshephard
adamshephard commented Feb 24, 2025
View reviewed changes
@shaneahmed shaneahmed changed the title 🆕 Add Support for UNI-2 ✨ Support for Additional Foundation Models Feb 24, 2025
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Feb 24, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Mar 4, 2025
View reviewed changes
shaneahmed
shaneahmed reviewed Mar 4, 2025
View reviewed changes
Copy link
Member

@shaneahmed shaneahmed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adamshephard This looks good. Just a minor comment in the docstring. Please can you update the Jupyter Notebook to incorporate kwargs in timm model architecture? It should be ready to merge after this.

@adamjtaylor
Copy link

adamjtaylor commented Mar 6, 2025

Great to see this addition. Supporting H0-mini would be greatly appreciated as well.

@adamshephard
Copy link
Contributor Author

adamshephard commented Mar 6, 2025

Great to see this addition. Supporting H0-mini would be greatly appreciated as well.

Thanks for the suggestion Adam, I have now added this also.

shaneahmed
shaneahmed approved these changes Mar 7, 2025
View reviewed changes
Copy link
Member

@shaneahmed shaneahmed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adamshephard

@shaneahmed shaneahmed merged commit 01948a9 into develop Mar 7, 2025
15 checks passed
@shaneahmed shaneahmed deleted the add-extra-foundation-models branch March 7, 2025 15:50
mbasheer04 added a commit to mbasheer04/tiatoolbox that referenced this pull request Mar 27, 2025
@mbasheer04
Preparing for PR
7b403c9 
Removing large files

Removing unneccessary files

Fixing pre-commit

[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Fixing PR issues

:technologist: pre-commit autoupdate (TissueImageAnalytics#910)

* 🧑‍💻 pre-commit autoupdate

updates:
- [github.com/executablebooks/mdformat: 0.7.21 → 0.7.22](hukkin/mdformat@0.7.21...0.7.22)
- [github.com/astral-sh/ruff-pre-commit: v0.8.6 → v0.9.4](astral-sh/ruff-pre-commit@v0.8.6...v0.9.4)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 📌 Update `ruff` dependency

* 🔥 TIAToolbox does not support Python > 3.12 yet
- There is no need for this check as this will be tested while upgrading to Python 3.13

* ♻️ Refactor `typing` to `type_hints`.

* 🐛 Fix `mypy` workflow

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <[email protected]>

📝 Update Documentation Structure (TissueImageAnalytics#909)

- Use `Python 3.12` for docs build
- Update `copyright` year to `2025`
- Landing page now shows text from README
- Update documentation structure
- Update `readthedocs` Build
- Remove `usage.rst`
- Rename Jupyter Notebooks to Usage Examples
- Show README for Usage Examples instead of TOC
- Reduce TOC depth for basic functionalities and pipelines
- Improve `README` quality.

🐛 Fix in `test_arch_mapde` and `test_arch_sccnn` (TissueImageAnalytics#911)

- If cuda is available model should be moved to cuda otherwise tests will fail as test data is moved to cuda.

[skip ci] 📝 Improve Documentation (TissueImageAnalytics#913)

- Update CONTRIBUTING.rst
- Bug fix in conf.py to fix notebook links
- Update `examples/README.md`
- Update `docs/installation.rst`
- Update `docs/visualization.rst`

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: adamshephard <[email protected]>

:bug: Fix `MapDe` `dist_filter` Shape  (TissueImageAnalytics#914)

- Fix `dist_filter` in `MapDe` model for multi-class output.

Explanation:
Previously, if we set `num_class` to more than 1, the model would still output 1 channel. This was because the `dist_filter` always had size of 1 in its first dimension, however the first dimension determines the number of output channels in the tensor produced by `torch.functional.F.conv2d`.
This PR changes this by repeating the filters the match the number of output classes.

:technologist: pre-commit autoupdate (TissueImageAnalytics#916)

* 🧑‍💻 pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.9](astral-sh/ruff-pre-commit@v0.9.4...v0.9.9)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 🔨 Update `ruff` version

* 🔨 Update noqa for Unused static method argument

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <[email protected]>

Add FsspecJsonWSIReader class. (TissueImageAnalytics#897)

The `FsspecJsonWSIReader` reads fsspec json file which represents SVS or TIFF whole slide image. The images are accessible by HTTP range requests, eg:

`https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27`

The whole image can be downloaded like:

`curl -C - -o TCGA-22-1017-01Z-00-DX1.9562FE79-A261-42D3-B394-F3E0E2FF7DDA.svs   https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27`

The `FsspecJsonWSIReader` class has a `_zarr_store` field which is created by reading json file using `fsspec`:

```
mapper = fsspec.get_mapper(
            "reference://", fo=str(input_img), target_protocol="file"
        )
self._zarr_array = zarr.open(mapper, mode="r")
self._zarr_store = self._zarr_array.store

self._zarr_lru_cache = zarr.LRUStoreCache(self._zarr_store, max_size=cache_size)
self._zarr_group = zarr.open(self._zarr_lru_cache)
```

This is equivalent to `TIFFWSIReader` code:

```
self._zarr_store = tifffile.imread(
            self.input_path,
            series=self.series_n,
            aszarr=True,
        )
        self._zarr_lru_cache = zarr.LRUStoreCache(self._zarr_store, max_size=cache_size)
        self._zarr_group = zarr.open(self._zarr_lru_cache)
```

Both FsspecJsonWSIReader and TIFFWSIReader forward calls to `read_bounds` and  `read_rect` methods of the`TIFFWSIReaderDelegate` delegate instance.

The method `_info` of the`TIFFWSIReaderDelegate` reads SVS metadata which is stored in the root group metadata like:
```
{
  ".zattrs": {
    "multiscales": [
      {
        "metadata": {
          "objective_power": 40,
          "vendor": "Aperio",
          "mpp": [0.2525, 0.2525]
        }
      }
    ]
  }
}
```

To test, execute from the root dir:
```
pip install -r requirements/requirements_dev.txt
mkdir -p samples/slides
mkdir -p samples/fsspec
cd samples/slides
curl -C - -o TCGA-22-1017-01Z-00-DX1.9562FE79-A261-42D3-B394-F3E0E2FF7DDA.svs   https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27
cd ../../
cp tiatoolbox/utils/tiff_to_fsspec.py .
python tiff_to_fsspec.py "samples/slides/TCGA-22-1017-01Z-00-DX1.9562FE79-A261-42D3-B394-F3E0E2FF7DDA.svs"  "samples/fsspec/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27_fsspec.json" "https://api.gdc.cancer.gov/data/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27"
```

Create `tileserver.py` inside of the project root:

```
from flask_cors import CORS

from tiatoolbox.visualization import TileServer
from tiatoolbox.wsicore.wsireader import FsspecJsonWSIReader

wsi = FsspecJsonWSIReader.open(
    "./samples/fsspec/73c69d24-6f9e-44e2-bfe5-a608d4cf5c27_fsspec.json"
)

tile_server = TileServer(
    title="Tiatoolbox TileServer",
    layers={"layer": wsi},
)
CORS(tile_server, send_wildcard=True)

tile_server.run(host="127.0.0.1", port=5000)
```

Open `http://127.0.0.1:5000/` and verify that it works.
---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <[email protected]>

✨ Support for Additional Foundation Models (TissueImageAnalytics#906)

- Add support for additional foundation models as feature extractors using the TimmBackbone.
- Added models include: UNI2, Virchow, Virchow2, kaiko and H-optimus-1.
- Add more information to docstrings.
- Allow foundation models with additional parameters.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Shan E Ahmed Raza <[email protected]>

Fixing PR issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shaneahmed shaneahmed shaneahmed approved these changes

@YijieZhu15 YijieZhu15 Awaiting requested review from YijieZhu15

Assignees

@adamshephard adamshephard

Labels

enhancement New feature or request

Projects

None yet

Milestone

Release v1.7.0

Development

Successfully merging this pull request may close these issues.

4 participants

@adamshephard @adamjtaylor @shaneahmed