Related papers: Selective pruning and neuronal death generate heav…
Deep neural networks are strongly over-parameterized, often containing far more weights than required for their task. Although such redundancy can aid optimization, it leads to inefficient deployment and high computational cost, motivating…
Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods…
Ever since the Barab\'{a}si-Albert (BA) scale-free network has been proposed, network modeling has been studied intensively in light of the network growth and the preferential attachment (PA). However, numerous real systems are featured…
Neuron pruning is widely used to reduce the computational cost and parameter footprint of large language models, yet it remains unclear whether neurons in task-specific models contribute uniformly to task performance. In this work, we…
We show that not only preferential attachment but also preferential depletion leads to scale-free networks. The resulting degree distribution exponents is typically less than two (5/3) as opposed to the case of the growth models studied…
We introduce a pruning algorithm that provably sparsifies the parameters of a trained model in a way that approximately preserves the model's predictive accuracy. Our algorithm uses a small batch of input points to construct a data-informed…
Network compression is crucial to making the deep networks to be more efficient, faster, and generalizable to low-end hardware. Current network compression methods have two open problems: first, there lacks a theoretical framework to…
Many networks in the brain are sparsely connected, and the brain eliminates synapses during development and learning. How could the brain decide which synapses to prune? In a recurrent network, determining the importance of a synapse…
Advances in neural recording methods enable sampling from populations of thousands of neurons during the performance of behavioral tasks, raising the question of how recorded activity relates to the theoretical models of computations…
Attributing the output of a neural network to the contribution of given input elements is a way of shedding light on the black-box nature of neural networks. Due to the complexity of current network architectures, current gradient-based…
We propose a model to create synthetic networks that may also serve as a narrative of a certain kind of infrastructure network evolution. It consists of an initialization phase with the network extending tree-like for minimum cost and a…
In contrast to conventional artificial neural networks, which are structurally static, we present two approaches for evolving small networks into larger ones during training. The first method employs an auxiliary weight that directly…
Growing synthetic networks that follow power law distributions of a node's degree often involves adding one node at a time. Each node is added to the network with a fixed amount of edges and those edges are frozen for all future time steps.…
Many complex systems--from social and communication networks to biological networks and the Internet--are thought to exhibit scale-free structure. However, prevailing explanations rely on the constant addition of new nodes, an assumption…
Stochastic Gradient Descent (SGD) is the main approach to optimizing neural networks. Several generalization properties of deep networks, such as convergence to a flatter minima, are believed to arise from SGD. This article explores the…
Spiking Neural Networks (SNNs) are more biologically plausible and computationally efficient. Therefore, SNNs have the natural advantage of drawing the sparse structural plasticity of brain development to alleviate the energy problems of…
In distributed training of deep neural networks, people usually run Stochastic Gradient Descent (SGD) or its variants on each machine and communicate with other machines periodically. However, SGD might converge slowly in training some deep…
The unmatched ability of Deep Neural Networks in capturing complex patterns in large and noisy datasets is often associated with their large hypothesis space, and consequently to the vast amount of parameters that characterize model…
During the first part of life, the brain develops while it learns through a process called synaptogenesis. The neurons, growing and interacting with each other, create synapses. However, eventually the brain prunes those synapses. While…
The recursive removal of leaves (dead end vertices) and their neighbors from an undirected network results, when this pruning algorithm stops, in a so-called core of the network. This specific subgraph should be distinguished from…