Related papers: Power efficient ReLU design for neuromorphic compu…

Deep learning is currently extensively employed across a range of research domains. The continuous advancements in deep learning techniques contribute to solving intricate challenges. Activation functions (AF) are fundamental components…

Spin Hall effect converts charge current to spin current, which can exert spin-torque to switch the magnetization of a nanomagnet. Recently, it is shown that the ratio of spin current to charge current using spin Hall effect can be made…

Activation functions are essential to deep learning networks. Popular and versatile activation functions are mostly monotonic functions, some non-monotonic activation functions are being explored and show promising performance. But by…

This paper proposes a spintronic neuron structure composed of a heterostructure of magnets and a piezoelectric with a magnetic tunnel junction (MTJ). The operation of the device is simulated using SPICE models. Simulation results illustrate…

We consider the injectivity property of the ReLU networks layers. Determining the ReLU injectivity capacity (ratio of the number of layer's inputs and outputs) is established as isomorphic to determining the capacity of the so-called…

Recent research has indicated that introducing impurities that increase the resistivity of Pt can enhance the efficiency of the spin Hall torque it generates. Here we directly demonstrate the usefulness of this strategy by fabricating…

Reinforcement learning (RL) is a foundation of learning in biological systems and provides a framework to address numerous challenges with real-world artificial intelligence applications. Efficient implementations of RL techniques could…

Magnetization switching by spin-orbit torque (SOT) via spin Hall effect represents as a competitive alternative to that by spin-transfer torque (STT) used for magnetoresistive random access memory (MRAM), as it does not require high-density…

Implicit Neural Representation (INR), leveraging a neural network to transform coordinate input into corresponding attributes, has recently driven significant advances in several vision-related domains. However, the performance of INR is…

Non-volatile Neuromorphic Computing (NC) elements utilizing Spin Orbit Torque (SOT) provide a viable solution to alleviate the memory wall bottleneck in contemporary computing systems. However, the two challenges, low SOT efficiency and the…

We present a novel methodology for repairing neural networks that use ReLU activation functions. Unlike existing methods that rely on modifying the weights of a neural network which can induce a global change in the function space, our…

An artificial neuron is modelled as a weighted summation followed by an activation function which determines its output. A wide variety of activation functions such as rectified linear units (ReLU), leaky-ReLU, Swish, MISH, etc. have been…

We introduce the "inverse square root linear unit" (ISRLU) to speed up learning in deep neural networks. ISRLU has better performance than ELU but has many of the same benefits. ISRLU and ELU have similar curves and characteristics. Both…

We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds…

The spin Hall effect (SHE) can generate a pure spin current by an electric current, which is promisingly used to electrically control magnetization. To reduce power consumption of this control, a giant spin Hall angle (SHA) in the SHE is…

Spin Hall effect (SHE), a mechanism by which materials convert a \textit{charge} current into a \textit{spin} current, invokes interesting physics and promises to empower transformative, energy-efficient memory technology. However,…

Recent years have witnessed growing interest in the use of Artificial Neural Networks (ANNs) for vision, classification, and inference problems. An artificial neuron sums N weighted inputs and passes the result through a non-linear transfer…

The choice of activation function plays a critical role in neural networks, yet most architectures still rely on fixed, uniform activation functions across all neurons. We introduce SmartMixed, a two-phase training strategy that allows…

We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, we choose between SILU or RELU depending on a Bernoulli draw.…

Always-on AI applications, from environmental sensors to biomedical implants, require ultra-low power consumption. Analog circuits offer a path to sub-microwatt inference, yet existing analog implementations are limited to feedforward…