Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning

This repository is the official implementation for accepted paper: "Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning" in NeurIPS 2020

Experiments

Arch	Optimizer	Dataset	FRNPF(II)	FRNPF(DI)	DNFPL	FLNPF	ReLU
FC	SGD	MNIST	95.85±0.10	95.85±0.17	97.86±0.11	97.10±0.09	97.85±0.09
FC	Adam	MNIST	96.02±0.13	96.09±0.12	98.22±0.05	97.82±0.02	98.14±0.07
VCONV	SGD	CIFAR-10	58.92±0.62	58.83±0.27	63.21±0.07	63.06±0.73	67.02±0.43
VCONV	Adam	CIFAR-10	64.86±1.18	64.68±0.84	69.45±0.76	71.40±0.47	72.43±0.54
GCONV	SGD	CIFAR-10	67.36±0.56	66.86±0.44	74.57±0.43	78.52±0.39	78.90±0.37
GCONV	Adam	CIFAR-10	67.09±0.58	67.08±0.27	77.12±0.19	79.68±0.32	80.32±0.35

Continuous NPF Learning

[Arora et al. 2019](https://arxiv.org/pdf/1904.11955.pdf) propse a pure kernel method named CNTK which achieves 77.43% test accuracy on cifar10 and is within 5-6% of the finite deep net architecture performace. We trained Fixed NPF(FLNPF) network by copying gates from a standared ReLU network at the various stage of training. We found that as gates are learned, the performance of FLNPF improves and subsequently surpasses the CNTK.

Citation

Please cite the paper if it helps you:

@inproceedings{chandra2020npf,
    title={Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning},
    author={Lakshminarayanan, Chandrashekar and Singh, Amit Vikram},
    booktitle={Advances in Neural Information Processing Systems(NeurIPS)},
    year={2020}
}