You need an account to view media

Sign in to view media

Don't have an account? Please contact us to request an account.

IJCNN Regular Sessions
Oral
Deep neural networks

Improving Discrete Latent Representations With Differentiable Approximation Bridges

Jason Ramapuram

Date & Time

Mon, July 20, 2020

5:45 pm – 7:45 pm

Location

On-Demand

Abstract

Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow non- differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable approximation bridge (DAB) neural network which approximates the non-differentiable forward function and provides gradient updates during backpropagation. We present empirical results from more than 600 experiments in four different domains: unsupervised (image) representation learning, variational (image) density estimation, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that training with DAB aided discrete non- differentiable functions improves image reconstruction quality and posterior linear separability by 10% against the Gumbel-Softmax relaxed estimator as well as providing a 9% improvement in the test variational lower bound in comparison to the state of the art RELAX discrete estimator. We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator in an image classification setting. The DAB network is not used for inference and expands the class of functions that are usable in neural networks. Our code is available at https://github.com/apple/ml-dab .


Presenter

Jason Ramapuram

University of Geneva
Sign in to join the conversationDon't have an account? Please contact us to request an account.
Sign in to view documentsDon't have an account? Please contact us to request an account.

Session Chairs

Baozhou Zhu

Delft University of Technology

Zaid Al-Ars

Delft University of Technology