Category: Uncategorized

In the previous blog I mentioned that I noticed that the layers reduce at different rates when they are pruned using the MAAV method. One way I thought of to illustrate this was to train some models with 1,000 activations in each layer. All of the models that I work with start with the biggest…

I have pruned models to change behaviour, and to try and make them faster and more efficient. One problem is that removing parameters does not speed up a matrix multiplication. The 0 values still need to be processed. To make a fully connected model faster, I need to remove whole rows and columns from the…

The models that I use have a linear final layer, and are trained using a quadratic cost function. For this reason I call them linear quadratic(LQ) models, to differentiate them from softmax models trained with a cross entropy cost function(SCE). Some of my first experiments were on adversarial attacks, tricking models with images corrupted with…

I have built a python library for training artificial neural networks(NNs) called Sparana. The name is short for sparse parameter analysis. I started with a library using tensorflow that made it easier for me to look at parameters, and prune them. I found tensorflow did not support optimization over sparse weights, which seemed like an…

I need to start by explaining why I have only pruned the middle 2 layers of this model. For the initial layer it is quite obvious, we want all of the input information. As a simple example, using MNIST a 7 is drawn with a horizontal line starting in the left part of a square,…

I do experiments on sparse neural networks, and I believe that working with sparsity is one way to progress machine learning. Sparse neural networks are just neural networks that have some of their weight values set to 0. I have a novel method of pruning neural networks. I have written a couple of blogs about…

Pruning is setting weight values to 0. Simple as that. If you set enough of them to 0 then it might be worthwhile converting the weights to sparse matrices, my software can do that. This is probably best shown in a notebook. Mostly I don’t convert to sparse neural networks, there are a lot of…

For this first experiment I am going to start with a simple goal, getting a neural network to forget one output class, while maintaining as much performance with the other classes as possible. The model is a fully connected neural network trained on MNIST. The method I am going to use is called mean activated…

Forgetting is removing parameters from a neural network to change how they behave. This blog is not really an introduction to the forgetting, it is an intro to the blog series that I am going to be working on while I develop the idea and methods. My last series of blogs has taken too long…

My software library is called Sparana, or sparse parameter analysis. The sparse part is because I wanted to use sparse data structures. The parameter analysis is because I wanted to look at parameter and what they were doing. Well if I am going to call my software library that, i should probably have a new…