Project Learning | Week 5
Week 5 of the learning Machine Learning through a Project based approach
A little late (2 days) to record the previous week. Nonetheless, we are here. Previous week I finished up with non Deep Learning approach in the Digit Recognizer competition and also started with Custom Neural Networks and slightly tweaking them to vary their performance. Let’s see what we got!
Final Non Deep Learning Model
After trying out different dimensionality reduction techniques(3 to be precise), with Support Vector Machines. I also tried t-SNE with K-Nearest Neighbors algorithm. This algorithm is really simple compared to other machine learning algorithms anyone can come across. It just compares the distance of the K (hyperparameter) nearest neighbors and assigns a test data point according to a criteria like voting. The StatQuest video explains the same. Relative to the previous approaches I tried, this combination gave a good score of 0.96982, after which we moved onto the Deep Learning approach.
(1) StatQuest: K-nearest neighbors, Clearly Explained — YouTube
Exploratory Data Analysis
Before moving on, I mentioned in my previous post that there are a very small number(0 maybe; not sure) dedicated EDA notebooks on Digit Recognizer dataset. So, I worked on one, it includes the following:
- Basic Data Exploration (Number of samples, distribution of samples and checking for non null values)
- Data Exploration (View the images, view images of a certain label)
- Exploration in Lower Dimensions (PCA and t-SNE visualizations)
ExploratoryDataAnalysis-DigitRecognizer | Kaggle
Deep Learning | Custom Neural Network
Part of why I started with this Project Learning approach are Neural Networks. From the outset, these algorithms seem really interesting and exciting. Given a certain distribution of data, the networks automatically learn the features presented to them and then make predictions. The Neural Network animation shown in one of the 3Blue1Brown video would really spark anyone’s interest in Neural Networks.
(1) But what is a Neural Network? | Deep learning, chapter 1 — YouTube
Enough of the motivation, the first approach I tried was following this notebook. This notebook has got the highest number of upvotes in the Digit Recognizer competition and is really great for a beginner starting out with Neural Networks.
An Image Data Generator augmenting images by shifting, rotating and zooming is used. The Neural Network architecture consists of 2 Convolutional Filters of size 32 followed by Max Pooling Layer and 2 Convolutional Filters of size 64 followed by Max Pooling Layer. This configuration is then attached to a Neural Network, with layers of size 256 and 10(10 digits). RMSProp is used as optimizer of the network.
Using this architecture I was able to achieve a new high score of 0.98957 and a ranking of 1584(Top 36%).
Customization of Custom Neural Network
The most customizable part I found with the above Neural Network was the Image Preprocessing part. After checking the incorrect predictions of the Neural Network, I saw that there were mainly two types of digits the network couldn’t understand. First were numbers that were written in skew or incorrect angle and second were numbers that were written in a very similar manner to other digits. The second type of digits don’t seem to be recognizable even after corrections, however, the first part could be improved in some way and that could potentially be through configuration of the Data Generator.
Image Data Generator of Keras library has a lot of adjustable parameters which were left untouched in the first version of the notebook. Some parameters are feature-wise normalization, sample-wise normalization, zca whitening and shear transformations. I tried various combinations of these changes but for reasons uncovered yet, I was not able to get the same performance as was demonstrated before. This is something I would like to study more about maybe in the next week.
The performance was so poor, that I even hesitated to submit these predictions. Hopefully, after some improvement I would do so.
Future Work
Since, we have finally started with Deep Learning techniques on the Digit Recognition dataset, we can also start with using Premade Neural Networks(like LeNet, Inception Network or VGG). Along with implementing our for sure score increasing technique the K Fold Cross Validation. Let’s see what we can get done in the next week.
