This paper dispels the myth of bad local minima in high dimension and shows that the loss surfaces of neural networks have remarkable properties. This seminal paper has helped renew interest for understanding non-convex optimization as well as the study of neural nets using methods from statistical physics.

Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. This work proposes a remarkably simple learning principle called mixup to alleviate these issues by encouraging the network to be linear between examples.

The prevalent approach to sequence to sequence problems such as translation was recurrent neural networks. In this work, we introduced a state-of-the-art architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training to better exploit the GPU hardware.