Image Classification on Small Datasets with Keras

Training a convnet with a small dataset

Having to coach an image-classification mannequin utilizing little or no information is a typical scenario, which you’ll possible encounter in follow in the event you ever do laptop imaginative and prescient in an expert context. A “few” samples can imply wherever from a couple of hundred to a couple tens of hundreds of pictures. As a sensible instance, we’ll concentrate on classifying pictures as canines or cats, in a dataset containing 4,000 photos of cats and canines (2,000 cats, 2,000 canines). We’ll use 2,000 photos for coaching – 1,000 for validation, and 1,000 for testing.

In Chapter 5 of the Deep Learning with R e-book we evaluation three methods for tackling this drawback. The first of those is coaching a small mannequin from scratch on what little information you have got (which achieves an accuracy of 82%). Subsequently we use function extraction with a pretrained community (leading to an accuracy of 90%) and fine-tuning a pretrained community (with a ultimate accuracy of 97%). In this publish we’ll cowl solely the second and third methods.

The relevance of deep studying for small-data issues

You’ll generally hear that deep studying solely works when plenty of information is on the market. This is legitimate partly: one basic attribute of deep studying is that it will possibly discover attention-grabbing options within the coaching information by itself, with none want for guide function engineering, and this could solely be achieved when plenty of coaching examples can be found. This is particularly true for issues the place the enter samples are very high-dimensional, like pictures.

But what constitutes plenty of samples is relative – relative to the dimensions and depth of the community you’re attempting to coach, for starters. It isn’t attainable to coach a convnet to unravel a posh drawback with only a few tens of samples, however a couple of hundred can doubtlessly suffice if the mannequin is small and effectively regularized and the duty is easy. Because convnets be taught native, translation-invariant options, they’re extremely information environment friendly on perceptual issues. Training a convnet from scratch on a really small picture dataset will nonetheless yield affordable outcomes regardless of a relative lack of knowledge, with out the necessity for any customized function engineering. You’ll see this in motion on this part.

What’s extra, deep-learning fashions are by nature extremely repurposable: you possibly can take, say, an image-classification or speech-to-text mannequin skilled on a large-scale dataset and reuse it on a considerably totally different drawback with solely minor adjustments. Specifically, within the case of laptop imaginative and prescient, many pretrained fashions (normally skilled on the ImageInternet dataset) at the moment are publicly obtainable for obtain and can be utilized to bootstrap highly effective imaginative and prescient fashions out of little or no information. That’s what you’ll do within the subsequent part. Let’s begin by getting your arms on the information.

Downloading the information

The Dogs vs. Cats dataset that you just’ll use isn’t packaged with Keras. It was made obtainable by Kaggle as a part of a computer-vision competitors in late 2013, again when convnets weren’t mainstream. You can obtain the unique dataset from https://www.kaggle.com/c/dogs-vs-cats/data (you’ll have to create a Kaggle account in the event you don’t have already got one – don’t fear, the method is painless).

The photos are medium-resolution shade JPEGs. Here are some examples:

Unsurprisingly, the dogs-versus-cats Kaggle competitors in 2013 was received by entrants who used convnets. The greatest entries achieved as much as 95% accuracy. Below you’ll find yourself with a 97% accuracy, though you’ll practice your fashions on lower than 10% of the information that was obtainable to the rivals.

This dataset incorporates 25,000 pictures of canines and cats (12,500 from every class) and is 543 MB (compressed). After downloading and uncompressing it, you’ll create a brand new dataset containing three subsets: a coaching set with 1,000 samples of every class, a validation set with 500 samples of every class, and a check set with 500 samples of every class.

Following is the code to do that:

original_dataset_dir <- "~/Downloads/kaggle_original_data"

base_dir <- "~/Downloads/cats_and_dogs_small"
dir.create(base_dir)

train_dir <- file.path(base_dir, "practice")
dir.create(train_dir)
validation_dir <- file.path(base_dir, "validation")
dir.create(validation_dir)
test_dir <- file.path(base_dir, "check")
dir.create(test_dir)

train_cats_dir <- file.path(train_dir, "cats")
dir.create(train_cats_dir)

train_dogs_dir <- file.path(train_dir, "canines")
dir.create(train_dogs_dir)

validation_cats_dir <- file.path(validation_dir, "cats")
dir.create(validation_cats_dir)

validation_dogs_dir <- file.path(validation_dir, "canines")
dir.create(validation_dogs_dir)

test_cats_dir <- file.path(test_dir, "cats")
dir.create(test_cats_dir)

test_dogs_dir <- file.path(test_dir, "canines")
dir.create(test_dogs_dir)

fnames <- paste0("cat.", 1:1000, ".jpg")
file.copy(file.path(original_dataset_dir, fnames), 
          file.path(train_cats_dir)) 

fnames <- paste0("cat.", 1001:1500, ".jpg")
file.copy(file.path(original_dataset_dir, fnames), 
          file.path(validation_cats_dir))

fnames <- paste0("cat.", 1501:2000, ".jpg")
file.copy(file.path(original_dataset_dir, fnames),
          file.path(test_cats_dir))

fnames <- paste0("canine.", 1:1000, ".jpg")
file.copy(file.path(original_dataset_dir, fnames),
          file.path(train_dogs_dir))

fnames <- paste0("canine.", 1001:1500, ".jpg")
file.copy(file.path(original_dataset_dir, fnames),
          file.path(validation_dogs_dir)) 

fnames <- paste0("canine.", 1501:2000, ".jpg")
file.copy(file.path(original_dataset_dir, fnames),
          file.path(test_dogs_dir))

Using a pretrained convnet

A standard and extremely efficient strategy to deep studying on small picture datasets is to make use of a pretrained community. A pretrained community is a saved community that was beforehand skilled on a big dataset, sometimes on a large-scale image-classification activity. If this unique dataset is giant sufficient and common sufficient, then the spatial hierarchy of options realized by the pretrained community can successfully act as a generic mannequin of the visible world, and therefore its options can show helpful for a lot of totally different computer-vision issues, though these new issues might contain fully totally different lessons than these of the unique activity. For occasion, you may practice a community on ImageInternet (the place lessons are largely animals and on a regular basis objects) after which repurpose this skilled community for one thing as distant as figuring out furnishings objects in pictures. Such portability of realized options throughout totally different issues is a key benefit of deep studying in comparison with many older, shallow-learning approaches, and it makes deep studying very efficient for small-data issues.

In this case, let’s think about a big convnet skilled on the ImageInternet dataset (1.4 million labeled pictures and 1,000 totally different lessons). ImageInternet incorporates many animal lessons, together with totally different species of cats and canines, and you may thus anticipate to carry out effectively on the dogs-versus-cats classification drawback.

You’ll use the VGG16 structure, developed by Karen Simonyan and Andrew Zisserman in 2014; it’s a easy and broadly used convnet structure for ImageInternet. Although it’s an older mannequin, removed from the present cutting-edge and considerably heavier than many different current fashions, I selected it as a result of its structure is just like what you’re already conversant in and is simple to know with out introducing any new ideas. This could also be your first encounter with considered one of these cutesy mannequin names – VGG, ResNet, Inception, Inception-ResNet, Xception, and so forth; you’ll get used to them, as a result of they are going to come up often in the event you preserve doing deep studying for laptop imaginative and prescient.

There are two methods to make use of a pretrained community: function extraction and fine-tuning. We’ll cowl each of them. Let’s begin with function extraction.

Feature extraction consists of utilizing the representations realized by a earlier community to extract attention-grabbing options from new samples. These options are then run via a brand new classifier, which is skilled from scratch.

As you noticed beforehand, convnets used for picture classification comprise two elements: they begin with a collection of pooling and convolution layers, they usually finish with a densely linked classifier. The first half known as the convolutional base of the mannequin. In the case of convnets, function extraction consists of taking the convolutional base of a beforehand skilled community, working the brand new information via it, and coaching a brand new classifier on prime of the output.

Why solely reuse the convolutional base? Could you reuse the densely linked classifier as effectively? In common, doing so must be prevented. The cause is that the representations realized by the convolutional base are prone to be extra generic and subsequently extra reusable: the function maps of a convnet are presence maps of generic ideas over an image, which is prone to be helpful whatever the computer-vision drawback at hand. But the representations realized by the classifier will essentially be particular to the set of lessons on which the mannequin was skilled – they are going to solely comprise details about the presence likelihood of this or that class in the whole image. Additionally, representations present in densely linked layers now not comprise any details about the place objects are positioned within the enter picture: these layers do away with the notion of area, whereas the item location continues to be described by convolutional function maps. For issues the place object location issues, densely linked options are largely ineffective.

Note that the extent of generality (and subsequently reusability) of the representations extracted by particular convolution layers relies on the depth of the layer within the mannequin. Layers that come earlier within the mannequin extract native, extremely generic function maps (resembling visible edges, colours, and textures), whereas layers which are increased up extract more-abstract ideas (resembling “cat ear” or “dog eye”). So in case your new dataset differs so much from the dataset on which the unique mannequin was skilled, chances are you’ll be higher off utilizing solely the primary few layers of the mannequin to do function extraction, somewhat than utilizing the whole convolutional base.

In this case, as a result of the ImageInternet class set incorporates a number of canine and cat lessons, it’s prone to be helpful to reuse the knowledge contained within the densely linked layers of the unique mannequin. But we’ll select to not, in an effort to cowl the extra common case the place the category set of the brand new drawback doesn’t overlap the category set of the unique mannequin.

Let’s put this in follow by utilizing the convolutional base of the VGG16 community, skilled on ImageInternet, to extract attention-grabbing options from cat and canine pictures, after which practice a dogs-versus-cats classifier on prime of those options.

The VGG16 mannequin, amongst others, comes prepackaged with Keras. Here’s the listing of image-classification fashions (all pretrained on the ImageInternet dataset) which are obtainable as a part of Keras:

• Xception
• Inception V3
• ResNet50
• VGG16
• VGG19
• MobileNet

Let’s instantiate the VGG16 mannequin.

library(keras)

conv_base <- application_vgg16(
  weights = "imagenet",
  include_top = FALSE,
  input_shape = c(150, 150, 3)
)

You go three arguments to the perform:

• weights specifies the load checkpoint from which to initialize the mannequin.
• include_top refers to together with (or not) the densely linked classifier on prime of the community. By default, this densely linked classifier corresponds to the 1,000 lessons from ImageInternet. Because you plan to make use of your individual densely linked classifier (with solely two lessons: cat and canine), you don’t want to incorporate it.
• input_shape is the form of the picture tensors that you just’ll feed to the community. This argument is only non-obligatory: in the event you don’t go it, the community will have the ability to course of inputs of any dimension.

Here’s the element of the structure of the VGG16 convolutional base. It’s just like the straightforward convnets you’re already conversant in:

Layer (kind)                     Output Shape          Param #  
================================================================
input_1 (InputLayer)             (None, 150, 150, 3)   0       
________________________________________________________________
block1_conv1 (Convolution2D)     (None, 150, 150, 64)  1792     
________________________________________________________________
block1_conv2 (Convolution2D)     (None, 150, 150, 64)  36928    
________________________________________________________________
block1_pool (MaxPooling2D)       (None, 75, 75, 64)    0        
________________________________________________________________
block2_conv1 (Convolution2D)     (None, 75, 75, 128)   73856    
________________________________________________________________
block2_conv2 (Convolution2D)     (None, 75, 75, 128)   147584   
________________________________________________________________
block2_pool (MaxPooling2D)       (None, 37, 37, 128)   0        
________________________________________________________________
block3_conv1 (Convolution2D)     (None, 37, 37, 256)   295168   
________________________________________________________________
block3_conv2 (Convolution2D)     (None, 37, 37, 256)   590080   
________________________________________________________________
block3_conv3 (Convolution2D)     (None, 37, 37, 256)   590080   
________________________________________________________________
block3_pool (MaxPooling2D)       (None, 18, 18, 256)   0        
________________________________________________________________
block4_conv1 (Convolution2D)     (None, 18, 18, 512)   1180160  
________________________________________________________________
block4_conv2 (Convolution2D)     (None, 18, 18, 512)   2359808  
________________________________________________________________
block4_conv3 (Convolution2D)     (None, 18, 18, 512)   2359808  
________________________________________________________________
block4_pool (MaxPooling2D)       (None, 9, 9, 512)     0        
________________________________________________________________
block5_conv1 (Convolution2D)     (None, 9, 9, 512)     2359808  
________________________________________________________________
block5_conv2 (Convolution2D)     (None, 9, 9, 512)     2359808  
________________________________________________________________
block5_conv3 (Convolution2D)     (None, 9, 9, 512)     2359808  
________________________________________________________________
block5_pool (MaxPooling2D)       (None, 4, 4, 512)     0        
================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0

The ultimate function map has form (4, 4, 512). That’s the function on prime of which you’ll stick a densely linked classifier.

At this level, there are two methods you would proceed:

• Running the convolutional base over your dataset, recording its output to an array on disk, after which utilizing this information as enter to a standalone, densely linked classifier just like these you noticed partly 1 of this e-book. This resolution is quick and low cost to run, as a result of it solely requires working the convolutional base as soon as for each enter picture, and the convolutional base is by far the most costly a part of the pipeline. But for a similar cause, this system received’t permit you to use information augmentation.

• Extending the mannequin you have got (conv_base) by including dense layers on prime, and working the entire thing finish to finish on the enter information. This will permit you to use information augmentation, as a result of each enter picture goes via the convolutional base each time it’s seen by the mannequin. But for a similar cause, this system is way dearer than the primary.

In this publish we’ll cowl the second method intimately (within the e-book we cowl each). Note that this system is so costly that you need to solely try it in case you have entry to a GPU – it’s completely intractable on a CPU.

Because fashions behave identical to layers, you possibly can add a mannequin (like conv_base) to a sequential mannequin identical to you’d add a layer.

mannequin <- keras_model_sequential() %>% 
  conv_base %>% 
  layer_flatten() %>% 
  layer_dense(models = 256, activation = "relu") %>% 
  layer_dense(models = 1, activation = "sigmoid")

This is what the mannequin appears to be like like now:

Layer (kind)                     Output Shape          Param #  
================================================================
vgg16 (Model)                    (None, 4, 4, 512)     14714688                                     
________________________________________________________________
flatten_1 (Flatten)              (None, 8192)          0        
________________________________________________________________
dense_1 (Dense)                  (None, 256)           2097408  
________________________________________________________________
dense_2 (Dense)                  (None, 1)             257      
================================================================
Total params: 16,812,353
Trainable params: 16,812,353
Non-trainable params: 0

As you possibly can see, the convolutional base of VGG16 has 14,714,688 parameters, which may be very giant. The classifier you’re including on prime has 2 million parameters.

Before you compile and practice the mannequin, it’s crucial to freeze the convolutional base. Freezing a layer or set of layers means stopping their weights from being up to date throughout coaching. If you don’t do that, then the representations that had been beforehand realized by the convolutional base can be modified throughout coaching. Because the dense layers on prime are randomly initialized, very giant weight updates could be propagated via the community, successfully destroying the representations beforehand realized.

In Keras, you freeze a community utilizing the freeze_weights() perform:

size(mannequin$trainable_weights)

[1] 30

freeze_weights(conv_base)
size(mannequin$trainable_weights)

[1] 4

With this setup, solely the weights from the 2 dense layers that you just added can be skilled. That’s a complete of 4 weight tensors: two per layer (the primary weight matrix and the bias vector). Note that to ensure that these adjustments to take impact, it’s essential to first compile the mannequin. If you ever modify weight trainability after compilation, you need to then recompile the mannequin, or these adjustments can be ignored.

Using information augmentation

Overfitting is brought on by having too few samples to be taught from, rendering you unable to coach a mannequin that may generalize to new information. Given infinite information, your mannequin could be uncovered to each attainable side of the information distribution at hand: you’d by no means overfit. Data augmentation takes the strategy of producing extra coaching information from current coaching samples, by augmenting the samples by way of various random transformations that yield believable-looking pictures. The aim is that at coaching time, your mannequin won’t ever see the very same image twice. This helps expose the mannequin to extra facets of the information and generalize higher.

In Keras, this may be carried out by configuring various random transformations to be carried out on the photographs learn by an image_data_generator(). For instance:

train_datagen = image_data_generator(
  rescale = 1/255,
  rotation_range = 40,
  width_shift_range = 0.2,
  height_shift_range = 0.2,
  shear_range = 0.2,
  zoom_range = 0.2,
  horizontal_flip = TRUE,
  fill_mode = "nearest"
)

These are only a few of the choices obtainable (for extra, see the Keras documentation). Let’s rapidly go over this code:

• rotation_range is a worth in levels (0–180), a variety inside which to randomly rotate photos.
• width_shift and height_shift are ranges (as a fraction of complete width or top) inside which to randomly translate photos vertically or horizontally.
• shear_range is for randomly making use of shearing transformations.
• zoom_range is for randomly zooming inside photos.
• horizontal_flip is for randomly flipping half the photographs horizontally – related when there are not any assumptions of horizontal asymmetry (for instance, real-world photos).
• fill_mode is the technique used for filling in newly created pixels, which might seem after a rotation or a width/top shift.

Now we are able to practice our mannequin utilizing the picture information generator:

# Note that the validation information should not be augmented!
test_datagen <- image_data_generator(rescale = 1/255)  

train_generator <- flow_images_from_directory(
  train_dir,                  # Target listing  
  train_datagen,              # Data generator
  target_size = c(150, 150),  # Resizes all pictures to 150 × 150
  batch_size = 20,
  class_mode = "binary"       # binary_crossentropy loss for binary labels
)

validation_generator <- flow_images_from_directory(
  validation_dir,
  test_datagen,
  target_size = c(150, 150),
  batch_size = 20,
  class_mode = "binary"
)

mannequin %>% compile(
  loss = "binary_crossentropy",
  optimizer = optimizer_rmsprop(lr = 2e-5),
  metrics = c("accuracy")
)

historical past <- mannequin %>% fit_generator(
  train_generator,
  steps_per_epoch = 100,
  epochs = 30,
  validation_data = validation_generator,
  validation_steps = 50
)

Let’s plot the outcomes. As you possibly can see, you attain a validation accuracy of about 90%.

Fine-tuning

Another broadly used method for mannequin reuse, complementary to function extraction, is fine-tuning
Fine-tuning consists of unfreezing a couple of of the highest layers of a frozen mannequin base used for function extraction, and collectively coaching each the newly added a part of the mannequin (on this case, the totally linked classifier) and these prime layers. This known as fine-tuning as a result of it barely adjusts the extra summary
representations of the mannequin being reused, in an effort to make them extra related for the issue at hand.

I said earlier that it’s essential to freeze the convolution base of VGG16 so as to have the ability to practice a randomly initialized classifier on prime. For the identical cause, it’s solely attainable to fine-tune the highest layers of the convolutional base as soon as the classifier on prime has already been skilled. If the classifier isn’t already skilled, then the error sign propagating via the community throughout coaching can be too giant, and the representations beforehand realized by the layers being fine-tuned can be destroyed. Thus the steps for fine-tuning a community are as follows:

• Add your customized community on prime of an already-trained base community.
• Freeze the bottom community.
• Train the half you added.
• Unfreeze some layers within the base community.
• Jointly practice each these layers and the half you added.

You already accomplished the primary three steps when doing function extraction. Let’s proceed with step 4: you’ll unfreeze your conv_base after which freeze particular person layers inside it.

As a reminder, that is what your convolutional base appears to be like like:

Layer (kind)                     Output Shape          Param #  
================================================================
input_1 (InputLayer)             (None, 150, 150, 3)   0        
________________________________________________________________
block1_conv1 (Convolution2D)     (None, 150, 150, 64)  1792     
________________________________________________________________
block1_conv2 (Convolution2D)     (None, 150, 150, 64)  36928    
________________________________________________________________
block1_pool (MaxPooling2D)       (None, 75, 75, 64)    0        
________________________________________________________________
block2_conv1 (Convolution2D)     (None, 75, 75, 128)   73856    
________________________________________________________________
block2_conv2 (Convolution2D)     (None, 75, 75, 128)   147584   
________________________________________________________________
block2_pool (MaxPooling2D)       (None, 37, 37, 128)   0        
________________________________________________________________
block3_conv1 (Convolution2D)     (None, 37, 37, 256)   295168   
________________________________________________________________
block3_conv2 (Convolution2D)     (None, 37, 37, 256)   590080   
________________________________________________________________
block3_conv3 (Convolution2D)     (None, 37, 37, 256)   590080   
________________________________________________________________
block3_pool (MaxPooling2D)       (None, 18, 18, 256)   0        
________________________________________________________________
block4_conv1 (Convolution2D)     (None, 18, 18, 512)   1180160  
________________________________________________________________
block4_conv2 (Convolution2D)     (None, 18, 18, 512)   2359808  
________________________________________________________________
block4_conv3 (Convolution2D)     (None, 18, 18, 512)   2359808  
________________________________________________________________
block4_pool (MaxPooling2D)       (None, 9, 9, 512)     0        
________________________________________________________________
block5_conv1 (Convolution2D)     (None, 9, 9, 512)     2359808  
________________________________________________________________
block5_conv2 (Convolution2D)     (None, 9, 9, 512)     2359808  
________________________________________________________________
block5_conv3 (Convolution2D)     (None, 9, 9, 512)     2359808  
________________________________________________________________
block5_pool (MaxPooling2D)       (None, 4, 4, 512)     0        
================================================================
Total params: 14714688

You’ll fine-tune the entire layers from block3_conv1 and on. Why not fine-tune the whole convolutional base? You might. But you have to think about the next:

• Earlier layers within the convolutional base encode more-generic, reusable options, whereas layers increased up encode more-specialized options. It’s extra helpful to fine-tune the extra specialised options, as a result of these are those that have to be repurposed in your new drawback. There could be fast-decreasing returns in fine-tuning decrease layers.
• The extra parameters you’re coaching, the extra you’re susceptible to overfitting. The convolutional base has 15 million parameters, so it could be dangerous to try to coach it in your small dataset.

Thus, on this scenario, it’s technique to fine-tune solely a few of the layers within the convolutional base. Let’s set this up, ranging from the place you left off within the earlier instance.

unfreeze_weights(conv_base, from = "block3_conv1")

Now you possibly can start fine-tuning the community. You’ll do that with the RMSProp optimizer, utilizing a really low studying price. The cause for utilizing a low studying price is that you just wish to restrict the magnitude of the modifications you make to the representations of the three layers you’re fine-tuning. Updates which are too giant might hurt these representations.

mannequin %>% compile(
  loss = "binary_crossentropy",
  optimizer = optimizer_rmsprop(lr = 1e-5),
  metrics = c("accuracy")
)

historical past <- mannequin %>% fit_generator(
  train_generator,
  steps_per_epoch = 100,
  epochs = 100,
  validation_data = validation_generator,
  validation_steps = 50
)

Let’s plot our outcomes:

You’re seeing a pleasant 6% absolute enchancment in accuracy, from about 90% to above 96%.

Note that the loss curve doesn’t present any actual enchancment (the truth is, it’s deteriorating). You might surprise, how might accuracy keep secure or enhance if the loss isn’t reducing? The reply is easy: what you show is a mean of pointwise loss values; however what issues for accuracy is the distribution of the loss values, not their common, as a result of accuracy is the results of a binary thresholding of the category likelihood predicted by the mannequin. The mannequin should still be bettering even when this isn’t mirrored within the common loss.

You can now lastly consider this mannequin on the check information:

test_generator <- flow_images_from_directory(
  test_dir,
  test_datagen,
  target_size = c(150, 150),
  batch_size = 20,
  class_mode = "binary"
)

mannequin %>% evaluate_generator(test_generator, steps = 50)

$loss
[1] 0.2158171

$acc
[1] 0.965

Here you get a check accuracy of 96.5%. In the unique Kaggle competitors round this dataset, this could have been one of many prime outcomes. But utilizing fashionable deep-learning methods, you managed to succeed in this consequence utilizing solely a small fraction of the coaching information obtainable (about 10%). There is a big distinction between with the ability to practice on 20,000 samples in comparison with 2,000 samples!

Take-aways: utilizing convnets with small datasets

Here’s what you need to take away from the workouts prior to now two sections:

• Convnets are the very best kind of machine-learning fashions for computer-vision duties. It’s attainable to coach one from scratch even on a really small dataset, with first rate outcomes.
• On a small dataset, overfitting would be the most important concern. Data augmentation is a strong solution to struggle overfitting if you’re working with picture information.
• It’s straightforward to reuse an current convnet on a brand new dataset by way of function extraction. This is a invaluable method for working with small picture datasets.
• As a complement to function extraction, you need to use fine-tuning, which adapts to a brand new drawback a few of the representations beforehand realized by an current mannequin. This pushes efficiency a bit additional.

Now you have got a stable set of instruments for coping with image-classification issues – specifically with small datasets.