Optim-WIP: Preliminary transform unit tests & more #521

ProGamerGov · 2020-11-11T00:02:50Z

I've added the first batch of unit tests. I'm not sure if they need to be more in depth / test more stuff? Other unit tests are currently on hold due to potential changes to how the objective, optimization system, and ImageTensor will work. Once @greentfrapp finishes those changes, then we'll write tests for them.

Added unit tests for all of the optim transform classes and functions.
Added unit tests for FFTImage.
Added unit tests for PixelImage.
Added unit tests for loading and running pretrained InceptionV1 model.
Fixed a number of the optim transforms that hadn't been tested thoroughly yet.
Added warning about tqdm package being required. It's pretty much the exact same as we added for PIL / Pillow.
Moved some of the general model functions & classes to their own file. These functions and classes can be used on / with other models, so it makes sense to separate them from the Inception 5h model.
Added unit tests for all of the _utils/models functions & classes.
Added function to get all hookable model layers.
Added RedirectedReLU example to Torchvision notebook tutorial.
Added a ton of missing type hints.
Added functions for calculating the Karhunen-Loève (KLT) matrix of custom image datasets. The current ToRGB transform uses an ImageNet KLT matrix, but users may not always be using models trained on ImageNet. Added new tests the new functions.
Readded Ludwig's Conv2dSame with tests and type hints.

* No idea if this will work. * Not sure how I'm supposed to be writing these tests, how extensive they need to be, or if what I have so far is okay. * Still need to add some tests for GaussianSmoothing with different dimensions.

ProGamerGov · 2020-11-11T02:57:17Z

==================================== ERRORS ====================================
_______________ ERROR collecting tests/optim/test_transforms.py ________________
ImportError while importing test module '/home/circleci/project/tests/optim/test_transforms.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/optim/test_transforms.py:7: in <module>
    from captum.optim._param.image import transform
captum/optim/__init__.py:3: in <module>
    from captum.optim._core import objectives  # noqa: F401
captum/optim/_core/objectives.py:9: in <module>
    from tqdm.auto import tqdm
E   ModuleNotFoundError: No module named 'tqdm'

So, I added a try-except block to deal with the missing tqdm import. But it'll have to be dealt with more in-depth for objective and optimization tests.

ProGamerGov · 2020-11-11T17:29:51Z

@NarineK Is adding tqdm and pillow (if it's not already included) to the test suite possible? We might be able to find a workaround for tqdm, but Pillow would be required for testing real image loading. @greentfrapp

This PR doesn't required PIL / Pillow or tqdm, so this is more about future test PRs.

ProGamerGov · 2020-11-13T19:53:29Z

There's a slight issue with indexing tensors with named dimensions. This may present an issue for future PRs until it's fixed.

pytorch/pytorch#47931

ProGamerGov · 2020-11-13T20:58:09Z

@NarineK There shouldn't be anymore major changes to this PR right now, so you can review and merge it!

NarineK · 2020-11-17T19:05:49Z

@NarineK There shouldn't be anymore major changes to this PR right now, so you can review and merge it!

Thank you very much for working on this, @ProGamerGov! I'll make a pass this week.

ProGamerGov · 2020-11-20T23:55:29Z

@NarineK I added type hinting to all the optim forward() and __init___ functions. I also noticed that most of the __init___ & forward() functions in Captum were missing some of or all of their type hints, despite everything else being type hinted. I'm not sure why that is?

I made a pull request to add the ending type hint stuff to all the __init___ functions that were missing them in the master branch: #535 The forward() functions can't as easily be mass fixed.

NarineK

Thank you for addressing the comments @ProGamerGov!
I made couple more nit comments.

NarineK · 2020-12-03T04:33:53Z

captum/optim/_utils/models.py

+    A workaround when there is no gradient flow from an initial random input.
+    ReLU layers will block the gradient flow during backpropagation when their
+    input is less than 0. This means that it can be impossible to visualize a
+    target without allowing negative values to pass through ReLU layers during


Thank you for adding description, @ProGamerGov! Is this replacement necessary if the target layer is a ReLU or is this necessary in general for all ReLU layers in the model?

It is necessary for all ReLU layers in the model, regardless of what the target layer is.

Thank you for the explanation, @ProGamerGov! Is this related to dead ReLU problem that Ludwig mentioned in his original PR ?

Yeah, it's a solution to the dead ReLU problem!

NarineK · 2020-12-03T05:51:27Z

captum/optim/_utils/models.py

+    def backward(self, grad_output: torch.Tensor) -> torch.Tensor:
+        (input_tensor,) = self.saved_tensors
+        grad_input = grad_output.clone()
+        grad_input[input_tensor < 0] = grad_input[input_tensor < 0] * 1e-1


nit: why is it necessary to multiple by 0.1 here ?

I'm not 100% sure why it's done so I've reached out to @greentfrapp for exactly why gradient scaling is required.

I also ran some tests and found that removing the scaling seemed to allow features that shouldn't be in the visualization (tests are from the InceptionV1 tutorial):

So, it looks like gradient scaling is required if we remove the normal backward ReLU code, but I'm not sure why scaling by 0.1 was chosen vs other numbers.

So, @greentfrapp just chose 0.1 as a way of handling the gradient flow in a way that wouldn't mess things up. In Lucid, Lubwig let everything through with the same weight / scale for the first 16 optimization steps.

I think @greentfrapp is working on a more advanced version of RedirectedReLU that more closely resembles the one in Lucid, but for right now the current version is sufficient.

Thank you for the explanation @ProGamerGov! Perhaps, we can write a sentence documentation about it so that we remember.
In the example above, does each image correspond to a step in the process of the optimization without using 0.1 multiplier ?

@NarineK The images were just the same ones as in the InceptionV1 tutorial notebook as I reran all the cells in the notebook without multiplying by 0.1 in RedirectedReLU. They aren't different steps. They are different targets and parameters from the tutorial notebook.

@greentfrapp has put together a more a better and advanced version of RedirectedReLU in his pull request, so I'll leave the RedirectedReLU version in this pull request as it currently is for merging. #552

NarineK · 2020-12-03T05:55:02Z

tests/optim/param/test_transforms.py

+
+from captum.optim._param.image import transform
+from tests.helpers.basic import BaseTest
+


I was thinking that if those transforms already exist in numpy or scipy them we could compare our implementation with theirs but we can do that later.

NarineK · 2020-12-03T06:05:20Z

tests/optim/utils/test_model_utils.py

+        rr_loss = rr_layer(x * 1).mean()
+        rr_loss.backward()
+
+        assertTensorAlmostEqual(self, t_grad_input[0], t_grad_output[0], 0)


Why do we need t_grad_input and t_grad_output as arrays if we are using only the first element ?

Lists can be appended from inside the hook function whereas other variables cannot, I think.

tests/optim/utils/test_model_utils.py

tests/optim/utils/test_images_utils.py

ProGamerGov · 2020-12-03T21:48:48Z

@NarineK I added NumPy comparison tests for 4 of the transforms! The other tests are not easily recreated in NumPy or there is no need for NumPy versions.

…asets

captum/optim/_utils/datasets.py

ProGamerGov · 2020-12-09T03:15:00Z

@NarineK @greentfrapp and I had to make some changes to the FFTImage classes to handle the PyTorch fft depreciations, but it should be ready for merging once you've finished reviewing the code!

NarineK · 2020-12-09T05:17:04Z

Thank you, @ProGamerGov ! I'm merging now in order not to not make other PRs to wait. I will take another look later into more details.

ProGamerGov added 12 commits November 10, 2020 14:44

Create __init__.py

04ecebb

First batch of transform tests

3a039be

RandomSpatialJitter is easier to test this way

66a7820

Add CenterCrop Test

43140a9

Fix CenterCrop

b7fb341

Fix BlendAlpha

0fdc885

Add Alpha tests

7e4b17b

remove unneeded operation

2a99db9

Add GaussianSmoothing test

90923e0

Update test_transforms.py

9cc0bdb

Add unit test stuff?

8e39eb5

Add unit test stuff?

2c51fdc

* No idea if this will work. * Not sure how I'm supposed to be writing these tests, how extensive they need to be, or if what I have so far is okay. * Still need to add some tests for GaussianSmoothing with different dimensions.

facebook-github-bot added the cla signed label Nov 11, 2020

Lint fix

f43a11d

ProGamerGov changed the title ~~Optim-WIP: Preliminary transform unit tests~~ Optim-WIP: Preliminary transform unit tests & more Nov 13, 2020

ProGamerGov mentioned this pull request Nov 13, 2020

Optim wip - Move & restructure loss objectives #527

Merged

Optim wip fix alpha channel support

6f3d9ee

ProGamerGov force-pushed the tests branch 2 times, most recently from da78e9a to f43a11d Compare November 27, 2020 01:12

ProGamerGov added 2 commits November 26, 2020 18:15

Merge branch 'tests' into optim-wip

ced29db

Merge optim-wip updates

ce12efb

ProGamerGov force-pushed the tests branch from 65ea3b3 to ce12efb Compare November 27, 2020 01:31

ProGamerGov and others added 2 commits November 27, 2020 01:47

Merge commits into a single commit

a61fad5

Delete old file and fix lint error

9e2061c

NarineK approved these changes Dec 3, 2020

View reviewed changes

ProGamerGov added 2 commits December 3, 2020 08:23

Changes based on additional comments

112f4bf

Add append

aabbd62

ProGamerGov requested a review from NarineK December 3, 2020 18:57

ProGamerGov force-pushed the tests branch 4 times, most recently from f11fdb4 to aabbd62 Compare December 4, 2020 19:41

Replace some tests with NumPy comparison tests

9a79c78

ProGamerGov force-pushed the tests branch 2 times, most recently from f8da94d to 9a79c78 Compare December 5, 2020 00:17

ProGamerGov added 2 commits December 4, 2020 17:18

Replace some image tests with NumPy comparison tests

e738462

Add some additional tests

01a414a

ProGamerGov force-pushed the tests branch from 295a6bc to 01a414a Compare December 5, 2020 23:55

ProGamerGov added 2 commits December 5, 2020 17:04

Add functions for calculating the KLT transform matrix for custom dat…

fdace77

…asets

Readd Ludwig's Conv2dSame with tests

93275f9

ProGamerGov mentioned this pull request Dec 6, 2020

Optim wip - General fixes, SharedImage, & Weight Visualization #543

Merged

ProGamerGov force-pushed the tests branch 2 times, most recently from bb64aef to 93275f9 Compare December 6, 2020 23:40

NarineK reviewed Dec 8, 2020

View reviewed changes

captum/optim/_utils/datasets.py Show resolved Hide resolved

Change datasets.py to image_dataset.py

4a753a5

ProGamerGov force-pushed the tests branch from 852bb33 to 4a753a5 Compare December 8, 2020 23:05

ProGamerGov added 4 commits December 8, 2020 16:22

Support both new & old fft functions

74bb28c

Fix FFTImage init

0e57949

Fix FFTImage & NumPy version

bf9faaf

Fix NumPy FFTImage output shape

aba9c5c

NarineK merged commit e319690 into meta-pytorch:optim-wip Dec 9, 2020


		from captum.optim._param.image import transform
		from tests.helpers.basic import BaseTest

Optim-WIP: Preliminary transform unit tests & more #521

Optim-WIP: Preliminary transform unit tests & more #521

Uh oh!

Conversation

ProGamerGov commented Nov 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ProGamerGov commented Nov 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ProGamerGov commented Nov 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ProGamerGov commented Nov 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ProGamerGov commented Nov 13, 2020

Uh oh!

NarineK commented Nov 17, 2020

Uh oh!

ProGamerGov commented Nov 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NarineK left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ProGamerGov Dec 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ProGamerGov Dec 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ProGamerGov Dec 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ProGamerGov Dec 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ProGamerGov commented Dec 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ProGamerGov commented Dec 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NarineK commented Dec 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ProGamerGov commented Nov 11, 2020 •

edited

Loading

ProGamerGov commented Nov 11, 2020 •

edited

Loading

ProGamerGov commented Nov 11, 2020 •

edited

Loading

ProGamerGov commented Nov 13, 2020 •

edited

Loading

ProGamerGov commented Nov 20, 2020 •

edited

Loading

ProGamerGov Dec 3, 2020 •

edited

Loading

ProGamerGov Dec 3, 2020 •

edited

Loading

ProGamerGov Dec 9, 2020 •

edited

Loading

ProGamerGov Dec 3, 2020 •

edited

Loading

ProGamerGov commented Dec 3, 2020 •

edited

Loading

ProGamerGov commented Dec 9, 2020 •

edited

Loading