In [1]:
from fastai.vision.all import *
set_seed(0)

Train the model¶

Note: this is an intentionally poor model; yours will almost certainly be better because your data is better.

In [2]:
train_dataset_path = untar_data('https://students.cs.calvin.edu/~ka37/example_letter_images.zip')
train_images = get_image_files(train_dataset_path)
train_labels = [img.parent.name for img in train_images]
103.26% [114688/111066 00:00<00:00]
In [ ]:
Counter(train_labels)

TODO: Cleanly separate out the Homework 1 part from the Homework 2 part; see instructions posted on Ed.

In [3]:
dataloaders = ImageDataLoaders.from_lists(
    path = train_dataset_path, fnames=train_images, labels=train_labels,
    valid_pct=0.2,
    seed=42,
    bs=2,
    item_tfms=RandomResizedCrop(224),
    # Use data augmentation
    batch_tfms=aug_transforms(size=224)
)
dataloaders.show_batch()
In [4]:
dataloaders.train.n
Out[4]:
21
In [5]:
sizes_of_images_in_batch = [image_batch.shape[0] for image_batch, label_batch in dataloaders.train]
print(f'{len(train_images)} total images: {dataloaders.train.n} in train set, {dataloaders.valid.n} in valid set')
print(f'Training data loader gave us {len(sizes_of_images_in_batch)} batches in an epoch')
print(f'Each batch had {set(sizes_of_images_in_batch)} images.')
print(f'So the learner will get trained on a total of {sum(sizes_of_images_in_batch)} images.')

26 total images: 21 in train set, 5 in valid set
Training data loader gave us 10 batches in an epoch
Each batch had {2} images.
So the learner will get trained on a total of 20 images.
In [6]:
learn = vision_learner(
    dls = dataloaders,
    arch=resnet18,
    metrics=[accuracy]
)

Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
In [7]:
saved_clf_filename = 'classifier.pth'
try:
    learn.load(saved_clf_filename)
    print("Loaded saved learner")
except FileNotFoundError:
    # Note that save/load is odd; it actually saves and loads from (learn.path/learn.model_dir/filename).
    # Maybe we should be using learn.export()?
    print("Running fine-tuning")
    learn.fine_tune(epochs=10)
    learn.recorder.plot_loss()
    learn.save(saved_clf_filename)

Running fine-tuning
epoch train_loss valid_loss accuracy time
0 1.427542 1.204675 0.400000 00:05
epoch train_loss valid_loss accuracy time
0 1.140317 0.691164 0.600000 00:01
1 1.484382 0.387477 0.800000 00:00
2 1.547534 0.572340 0.600000 00:00
3 1.446050 0.660998 0.800000 00:01
4 1.446995 0.613119 0.800000 00:00
5 1.395874 0.627853 0.800000 00:00
6 1.380330 0.485403 0.800000 00:00
7 1.356561 0.420344 0.800000 00:00
8 1.269402 0.440142 0.800000 00:00
9 1.277924 0.311430 0.800000 00:00

Load the test set¶

Here's the code for getting the test set loaded. The basic approach is the same as constructing the DataLoaders above, but instead of making brand new dataloaders, we make a new dataloader that's like the original one but using a new set of images. (This is different than creating a new dataloaders because we don't want to split the new data into a training set and validation set; we want to use the whole thing.)

We're giving it the labels so that we can diagnose the results. In a Kaggle-style competition format, you won't get the true labels, so you'll just use dataloaders.test_dls(test_images, with_labels=False).

In [8]:
test_url = 'https://students.cs.calvin.edu/~ka37/letter_images_dataset_v0.zip'
test_dataset_path = untar_data(test_url)
100.01% [52699136/52695272 00:05<00:00]
In [9]:
test_images = get_image_files(test_dataset_path)
test_labels = [img.parent.name.upper() for img in test_images]
In [10]:
test_images
Out[10]:
(#944) [Path('/root/.fastai/data/letter_images_dataset_v0/c/group_5_c3.jpg'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_9_lower_c_1.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_4_C_0.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_0_C_30.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_0_C_7.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_10_c20.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_1_C_34.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_8_c13.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_1_C_4.png'),Path('/root/.fastai/data/letter_images_dataset_v0/c/group_3_C_0.png')...]
In [11]:
# Note: we need to "zip" the filenames together with the corresponding filenames.
# To see how `zip` works, try looking at the output of `list(zip(test_images[:5], test_labels))`.
test_dl = dataloaders.test_dl(list(zip(test_images, test_labels)), with_labels=True)
In [12]:
test_dl.show_batch()

Evaluate our learner on the testing data¶

Once we have the test DataLoader, we can use it for ClassificationInterpretation. By default, the validation set is used, but we can tell it to use test_dl instead:

In [13]:
interp = ClassificationInterpretation.from_learner(learn, dl=test_dl)

Here are three basic reports that you might want to look at. (See Chapter 2 of the book.)

In [14]:
interp.print_classification_report()

              precision    recall  f1-score   support

           A       0.54      0.48      0.51       332
           B       0.63      0.12      0.21       346
           C       0.40      0.88      0.55       266

    accuracy                           0.46       944
   macro avg       0.52      0.49      0.42       944
weighted avg       0.53      0.46      0.41       944
In [15]:
interp.plot_confusion_matrix()
In [16]:
interp.plot_top_losses(25)

Aside: here's the code for getting the predictions manually. docs

This works by treating a correct prediction as a 1 and an incorrect prediction as 0. (I call this the sum-as-count pattern) This code also demonstrates a clean way to spread out a Python expression over multiple lines: put everything in parentheses.

In [17]:
predicted_probs, targets = learn.get_preds(dl=test_dl, with_preds=True, with_targs=True)
# Use the class with the highest predicted probability as the predicted class.
predicted_classes = predicted_probs.argmax(axis=1)
In [18]:
(
    # Make a Tensor of Trues and Falses, True if the classifier got the corresponding image right
    (
        # the predictions that the model made on the test set
        predicted_classes
        # compare with the target labels provided by the DataLoader (we got these from get_preds above)
        == targets
    )
    # convert True to 1.0, False to 0.0
    .to(float)
    # Compute the fraction of True's.
    .mean()
)
Out[18]:
TensorBase(0.4608, dtype=torch.float64)

This also works:

In [19]:
accuracy(predicted_probs, targets)
Out[19]:
TensorBase(0.4608)