1-Lipschitz Neural Networks are more expressive with N-Activations

Bernd Prach, Christoph H. Lampert
11/10/2023
cs.LG

Official code: berndprach/NActivation (4 stars, MIT, last commit 2023-11-13).

Abstract

A crucial property for achieving secure, trustworthy and interpretable deep learning systems is their robustness: small changes to a system's inputs should not result in large changes to its outputs. Mathematically, this means one strives for networks with a small Lipschitz constant. Several recent works have focused on how to construct such Lipschitz networks, typically by imposing constraints on the weight matrices. In this work, we study an orthogonal aspect, namely the role of the activation function. We show that commonly used activation functions, such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily restrict the class of representable functions, even in the simplest one-dimensional setting. We furthermore introduce the new N-activation function that is provably more expressive than currently popular activation functions. We provide code at https://github.com/berndprach/NActivation.

View on arXivView PDF

Code Implementations(1)

berndprach/NActivationOfficial
MIT
40PythonNov 3, 20232 years agoMIT

Similar Papers

ExLipBaB: Exact Lipschitz Constant Computation for Piecewise Linear Neural Networks

84% match1 repo2026

A Unified Algebraic Perspective on Lipschitz Neural Networks

84% match1 repo102023

Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations

83% match2024

Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

82% match2024

Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

81% match1 repo42023

Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

81% match1 repo42026

Cite this paper

@article{prach2023lipschitz,
  title  = {1-Lipschitz Neural Networks are more expressive with N-Activations},
  author = {Bernd Prach and Christoph H. Lampert},
  year   = {2023},
  eprint = {2311.06103},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.06103v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.06103.svg)](https://codeofpaper.com/paper/2311.06103)

Discussion