All research

Applied machine learning · 2024

Deep learning for classifying waste in urban street images

A two-stage classifier that first finds street images containing garbage and then names the types present, evaluated on 25,000 images from car-mounted cameras.

Qatar University and University of Doha for Science and Technology · Qatar National Research Fund grant UREP29-205-2-056

Workflow diagram. Street images labelled not-garbage, or with several garbage types, are resized to 224 by 224 pixels and normalised. A single-label GCDN-Net separates not-garbage from garbage images. Garbage images then pass to a multi-label GCDN-Net, which predicts labels such as bulky waste, cardboard, garbage bag and litter for each image.
Figure 1 from Hossen et al. (2024). The two-stage method: images are resized and normalised, a single-label classifier separates garbage from non-garbage images, and a multi-label classifier names the garbage types in each garbage image.View full-size figureSource article© 2023 Elsevier Ltd · author reuse

Approach

Can a network that fuses two pretrained classifiers find garbage in street images and name the types present, even when several appear together?

Growing cities produce more waste, and about a third of global waste is disposed of without proper sorting. Automated sorting and street cleaning need models that recognise waste in real scenes, where several types often lie together against busy backgrounds. This study proposed GCDN-Net, which combines DenseNet201 and Inception-v3. A first stage separates street images with and without garbage, and a second stage names the garbage types present.

  1. Clean and prepare street images
  2. Detect garbage in each image
  3. Name the waste types present

Cleaning and preparing the street images

Waste Management · 2024

The study used the public GIGO dataset of 25,000 images taken by two car-mounted cameras on city streets: 9,352 with garbage and 15,648 without. Garbage images carry labels for four types, and one image can show several: bulky waste such as sofas and fridges, garbage bags, cardboard and litter.

Faces and number plates in the dataset are hidden by grey or yellow boxes, which a model could mistake for garbage. About 3,000 affected images were removed, leaving 7,410 garbage images, and 8,760 images without garbage were sampled to balance the classes. Each dataset was split 70%, 10% and 20% for training, validation and testing, and every model was evaluated with five-fold cross-validation.

Horizontal flips and rotations of up to 30 degrees raised the training images to about 5,000 per class. Images were resized to 224 × 224 pixels and normalised with the dataset’s mean and standard deviation.

Cleaned images used
16,170
Garbage types labelled
4
Cross-validation
5-fold
Images per class for training, validation and testing in one fold, before augmentation. The first two rows form the single-label dataset; the last four are labels in the multi-label dataset. From Table 1 of the paper.
TrainingValidationTest
Garbage5,1877411,482
Not garbage6,1328461,752
Garbage bag1,307182392
Cardboard2,853414832
Bulky waste3,249466933
Litter2,447343686

Fusing two pretrained networks to detect garbage

Single-label classification

Four ImageNet-pretrained networks were compared first: ResNet50, MobileNetV2, DenseNet201 and Inception-v3. DenseNet201 and Inception-v3 performed best, so GCDN-Net combines them. The features from the last layers of both networks are joined before the final classifier.

Three auxiliary outputs add deep supervision. One joins features from the middle of both networks, and the others come from the end of each network. The auxiliary losses get smaller weights the shallower they are, and the model was trained with the Adam optimiser at a learning rate of 0.0001.

GCDN-Net separated garbage from non-garbage images with 95.77% accuracy and 95.54% specificity. Joining the two networks raised accuracy from 91.60% for Inception-v3 alone to 93.15%, one auxiliary output raised it to 94.72%, and three auxiliary outputs gave the final result.

Accuracy
95.77%
Specificity
95.54%
Accuracy gain over Inception-v3 alone
4.17 points
Single-label classification, combined over five folds (%). From Table 2 of the paper.
AccuracyPrecisionSpecificityF1 score
MobileNetV287.6187.6387.5687.62
ResNet5090.3890.3990.0590.37
DenseNet20190.6990.6990.3490.67
Inception-v391.6091.6091.4291.60
GCDN-Net, no auxiliary output93.1593.1793.8593.14
GCDN-Net, one auxiliary output94.7294.7394.5394.73
GCDN-Net, three auxiliary outputs95.7795.7895.5495.77
Three block diagrams. (a) DenseNet201 with four dense blocks of 6, 12, 48 and 32 layers. (b) Inception-v3 with Inception-A, B and C modules separated by grid-size reductions. (c) GCDN-Net running both pretrained networks on the same input, with auxiliary output 1 joining features after dense block 2 and the first Inception modules, auxiliary outputs 2 and 3 at the end of each network, and the final output joining the last features of both.
Figure 3 from Hossen et al. (2024). Architectures of (a) DenseNet201 and (b) Inception-v3, and (c) GCDN-Net, which joins both networks and adds three auxiliary outputs.View full-size figureSource article© 2023 Elsevier Ltd · author reuse

Naming the types of waste in each image

Multi-label classification

The 7,410 garbage images then went to a multi-label GCDN-Net. A sigmoid output with binary cross-entropy loss treats each of the four types as a separate yes-or-no decision, so one image can receive several labels.

Across the four types, GCDN-Net reached a mean average precision of 0.69 and an F1 score of 75.01%. Bulky waste was recognised best, with a mean average precision of 0.73 and an F1 score of 80.23%. Garbage bags were hardest: they made up only 13.33% of the labels and are often small in the frame. An earlier study on the same dataset reported a mean average precision of 0.47 with a vision transformer.

Score-CAM heatmaps showed which parts of each image drove the predictions. For images with one, two or more waste types, the heatmaps concentrated on the garbage rather than on surrounding buildings, vehicles or trees.

Mean average precision
0.69
F1 score
75.01%
F1 score for bulky waste
80.23%
Multi-label classification with GCDN-Net, combined over five folds. Precision, recall and F1 score in %. From Table 3 of the paper.
Mean average precisionPrecisionRecallF1 score
Bulky waste0.7373.9787.6980.23
Cardboard0.7373.9681.9177.69
Litter0.6669.7472.5070.96
Garbage bag0.5671.0557.0563.12
Overall0.6972.5678.3275.01
Grid of nine street scenes, each shown above its Score-CAM heatmap. Labels range from a single type, such as cardboard, litter or a garbage bag, to two types, such as bulky waste and litter, and up to all four types. In each heatmap, the warm red and yellow region sits over the piles of waste on the pavement or kerb, while buildings, trees and vehicles stay blue.
Figure 4 from Hossen et al. (2024). Test images with one label (I, IV, VII), two labels (II, V, VIII) and three or four labels (III, VI, IX), each above its Score-CAM heatmap.View full-size figureSource article© 2023 Elsevier Ltd · author reuse

My contribution

I helped develop the framework and contributed to the evaluation, experimental analysis, interpretation and manuscript review.

Study scope

All images came from one dataset of urban street scenes, and about 3,000 were removed because privacy boxes could be mistaken for garbage. Without bounding boxes, the model reports which waste types are present but cannot locate or count individual items, and busy backgrounds made multi-label classification harder.