💥 The model.py will be released once the paper is accepted 💥

Degradation Oriented and Regularized Network for
Real-World Depth Super-Resolution

Zhengxue Wang¹, Zhiqiang Yan✉¹, Jinshan Pan¹, Guangwei Gao², Kai Zhang³, Jian Yang✉¹

¹Nanjing University of Science and Technology   
²Nanjing University of Posts and Telecommunications    ³Nanjing University   

Overview of DORNet. Given $\boldsymbol D_{up}$ as input, the degradation learning first encodes it to produce degradation representations $\boldsymbol {\tilde{D}}$ and $\boldsymbol D $. Then, $\boldsymbol {\tilde{D}}$, $\boldsymbol D $, $\boldsymbol D_{lr} $, and $\boldsymbol I_{r}$ are fed into multiple degradation-oriented feature transformation (DOFT) modules, generating the HR depth $\boldsymbol D_{hr}$. Finally, $\boldsymbol D$ and $\boldsymbol D_{hr}$ are sent to the degradation regularization to obtain $\boldsymbol D_{d}$, which is used as input for the degradation loss $\mathcal L_{deg}$ and the contrastive loss $\mathcal L_{cont}$. The degradation regularization only applies during training and adds no extra overhead in testing.

Dependencies

Python==3.11.5
PyTorch==2.1.0
numpy==1.23.5 
torchvision==0.16.0
scipy==1.11.3
Pillow==10.0.1
tqdm==4.65.0
scikit-image==0.21.0
mmcv-full==1.7.2

Datasets

RGB-D-D

TOFDSR

NYU-v2

Models

Pretrained models on RGB-D-D can be found in checkpoints (The remaining pre-trained models will be released once the paper is accepted).

Training

DORNet

Train on real-world RGB-D-D and TOFDSR
> python train.py
Train on synthetic NYU-v2
> python train.py --scale 4

DORNet-T

Train on real-world RGB-D-D and TOFDSR
> python train.py --tiny_model
Train on synthetic NYU-v2
> python train.py --scale 4 --tiny_model

Testing

SPFNet

Test on real-world RGB-D-D and TOFDSR
> python test.py
Test on synthetic NYU-v2
> python test.py --scale 4

SPFNet-T

Test on real-world RGB-D-D and TOFDSR
> python test.py --tiny_model
Test on synthetic NYU-v2
> python test.py --scale 4 --tiny_model

Experiments

Quantitative comparison

Complexity on RGB-D-D (w/o Noisy) tested by a 4090 GPU. A larger circle diameter indicates a higher inference time.

Visual comparison

Visual results on the real-world RGB-D-D dataset (w/o Noise).