fix: prelu perf gap on Unet

[zewenli98]

Description

In Monai/Unet, I observed that Torch-TRT's prelu is slower than ONNX-TRT, and what is worse is that using prelu converter is even slower than decomposing it.

Update: The perf issue is because to_copy forces to insert Cast Layer.

Fixes #3715 #3723

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the style guidelines of this project (You can use the linters)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas and hacks
• I have made corresponding changes to the documentation
• I have added tests to verify my fix or my feature
• New and existing unit tests pass locally with my changes
• I have added the relevant labels to my PR in so that relevant reviewers are notified

[narendasan]

LGTM

[narendasan]

If possible consider adding a test, we talked a bit about a new INetwork verification test based on expected numbers of ops. Might be a simple case to build the harness for

[zewenli98]

@narendasan I'll merge this PR now and cherrypick to the release 2.8, and then work on the number of ops check in another PR for the next release. sounds like a plan?

[zewenli98]

there's an error. will merge after fixing

[zewenli98]

def forward(self, x):
    x_1 = x + 1
    y = torch.ops.aten._to_copy.default(x_1, dtype=torch.float)
    z = torch.ops.aten._to_copy.default(x_1, dtype=torch.float)
    return y, z

this case fails because self.engine.num_io_tensors=2, len(self.input_names)=1, len(self.output_names)=2, which doesn't meet the assertion:

assert self.engine.num_io_tensors == len(self.input_names) + len(self.output_names)

self.input_names: ['x']
self.output_names: ['output0', 'output1']

Since output0 is invalid, it gets self.engine.num_io_tensors=2.

[narendasan]

Where does this id function come from?

[zewenli98]

id() is a built-in function that returns the "identity" of an object.

[peri044]

I think there might be cases where we might need to actually force_layer=True. Do you know when that would be useful?. Also consider adding a comment here conveying that force_layer=False results in better performance.

[zewenli98]

Do you know when we need force_layer=True? My understanding is that 1) Since cast_trt_tensor would be called even if the force_layer=False, it only adds Cast Layer if necessary. 2) If an operator satisfies not is_only_operator_on_placeholder, that means there's other operator on the placeholder or the lowering pass repair_input_as_output to promise "input is not output", so we don't have to explicitly add Cast Layer.