Related papers: Configurable p-Neurons Using Modular p-Bits

We propose a probabilistic framework for dynamic quantization of neural networks that allows for a computationally efficient input-adaptive rescaling of the quantization parameters. Our framework applies a probabilistic model to the…

In the neuroevolution literature, research has primarily focused on evolving the number of nodes, connections, and weights in artificial neural networks. Few attempts have been made to evolve activation functions. Research in evolving…

Neural networks are universal function approximators which are known to generalize well despite being dramatically overparameterized. We study this phenomenon from the point of view of the spectral bias of neural networks. Our contributions…

This article summarizes principles and ideas from the emerging area of applying \textit{conditional computation} methods to the design of neural networks. In particular, we focus on neural networks that can dynamically activate or…

To enhance the nonlinearity of neural networks and increase their mapping abilities between the inputs and response variables, activation functions play a crucial role to model more complex relationships and patterns in the data. In this…

Successive linear transforms followed by nonlinear "activation" functions can approximate nonlinear functions to arbitrary precision given sufficient layers. The number of necessary layers is dependent on, in part, by the nature of the…

A pivotal aspect in the design of neural networks lies in selecting activation functions, crucial for introducing nonlinear structures that capture intricate input-output patterns. While the effectiveness of adaptive or trainable activation…

Instantaneous and on demand accuracy-efficiency trade-off has been recently explored in the context of neural networks slimming. In this paper, we propose a flexible quantization strategy, termed Switchable Precision neural Networks…

We propose a novel activation function that implements piece-wise orthogonal non-linear mappings based on permutations. It is straightforward to implement, and very computationally efficient, also it has little memory requirements. We…

We propose a data-driven approach to represent neuronal network dynamics as a Probabilistic Graphical Model (PGM). Our approach learns the PGM structure by employing dimension reduction to network response dynamics evoked by stimuli applied…

Biological systems use neural circuits to integrate input information and produce outputs. Synaptic convergence, where multiple neurons converge their inputs onto a single downstream neuron, is common in natural neural circuits. However,…

Neuromorphic systems that densely integrate CMOS spiking neurons and nano-scale memristor synapses open a new avenue of brain-inspired computing. Existing silicon neurons have molded neural biophysical dynamics but are incompatible with…

We propose a stochastic dynamical model of noisy neural networks with complex architectures and discuss activation of neural networks by a stimulus, pacemakers and spontaneous activity. This model has a complex phase diagram with…

Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical…

An activation function has crucial role in a deep neural network. A simple rectified linear unit (ReLU) are widely used for the activation function. In this paper, a weighted sigmoid gate unit (WiG) is proposed as the activation function.…

The human brain can self-organize rich and diverse sparse neural pathways to incrementally master hundreds of cognitive tasks. However, most existing continual learning algorithms for deep artificial and spiking neural networks are unable…

Photonic neural networks benefit from both the high channel capacity- and the wave nature of light acting as an effective weighting mechanism through linear optics. The neuron's activation function, however, requires nonlinearity which can…

This paper proposes a novel spiking artificial neuron design based on a combined spin valve/magnetic tunnel junction (SV/MTJ). Traditional hardware used in artificial intelligence and machine learning faces significant challenges related to…

Activation functions are widely used in neural networks to decide the activation value of the neural unit based upon linear combinations of the weighted inputs. The effective implementation of activation function is highly important, as…

We introduce the percolation with plasticity (PWP) systems that exhibit neuromorphic functionalities including multi-valued memory, random number generation, matrix-vector multiplication, and associative learning. PWP systems have multiple…