Remove reference interfaces of chained controllers if those are unloaded.

[destogl]

I think we should remove interfaces already if possible

Originally posted by @bmagyar in #667 (comment)

[github-actions]

This issue is being labeled as stale because it has been open 45 days with no activity. It will be automatically closed after another 45 days without follow-ups.

[github-actions]

This issue was closed because it has been stalled for 45 days with no activity.