All research

Applied machine learning · 2021

Satellite image segmentation for climate and geopolitical risk

A multiclass segmentation pipeline that maps buildings, vegetation, water and other classes from satellite images, built for a team project on risks to water and energy financing in West and Central Africa.

Omdena AI challenge · Project partner Finz · Completed team project

Four panels for one Sentinel-2 test scene: the input band in false colour, with a no-data strip along the right edge; the model’s predicted pixel values in greyscale; the mask after thresholding, with white marking pixels assigned to the class; and the polygons traced from that mask.
One Sentinel-2 test scene processed by the trained model: the input band, the predicted pixel values, the mask after thresholding and the polygons traced from it. From the Task 4 report.View full-size figureTask 4 report, Omdena–Finz challenge (2021)Contains modified Copernicus Sentinel data

Approach

Can deep segmentation models map buildings, vegetation and water from satellite images well enough to add local detail to climate and geopolitical risk scores?

Finz helps governments, banks and foundations finance small water and energy units in urban and rural areas. In this eight-week Omdena challenge, an international team studied how climate hazards and geopolitical events threaten such assets in Togo, northern Côte d’Ivoire and Cameroon. The team built a flood risk map for Togo and a geopolitical risk score from historical conflict events. My task added a satellite-image pipeline to map buildings, vegetation and water around a site.

  1. Select and prepare satellite data
  2. Train and compare U-Net models
  3. Test on Sentinel-2 images

Choosing a labelled satellite dataset

Task lead · July 2021

The pipeline needed satellite images labelled with several classes at once, so that one model could map buildings, roads, vegetation and water around a site. About 30 public aerial and satellite datasets were compared by size, platform, number and resolution of images, label type and classes. Many covered a single class, such as buildings, ships or trees, or came from drones.

Five datasets were shortlisted, and the Dstl Satellite Imagery Feature Detection dataset fitted best. Its WorldView-3 scenes each cover 1 km × 1 km and come as RGB images and as 16-band images, with eight multispectral bands from 400 to 1,040 nm and eight short-wave infrared bands from 1,195 to 2,365 nm. Ten classes are labelled as polygons.

Datasets compared
About 30
Labelled classes
10
Spectral bands per image
16
Classes labelled in the Dstl dataset, grouped by type.
Classes and what they include
Buildings and structuresResidential, non-residential and large buildings, fuel storage and fortified buildings, plus other man-made structures
Transport routesRoads, and tracks such as dirt tracks and footpaths
VegetationTrees, from woodland to single trees, and crops, including grain and row crops
WaterWaterways and standing water
VehiclesLarge vehicles such as lorries and buses, and small vehicles such as cars and motorbikes

Building an end-to-end segmentation pipeline

Task lead · August 2021

The pipeline runs from raw images and polygon labels to predicted masks and polygons in one notebook. Polygon labels were converted into a mask for each class. Tests showed that some multispectral bands outside red, green and blue picked out vegetation more clearly, so the multispectral images were used for training.

Memory on Google Colab limited the input size. Each image was resized to 512 × 512 pixels and split into 16 patches of 128 × 128, which kept enough detail in each patch without creating too many patches per image. About a quarter of the training data was held out for validation. Test images went through the same steps, and their predicted patches were stitched back together.

Predicted masks were turned into binary masks with a threshold set for each class, for example about 0.1 to 0.15 for buildings and 0.4 to 0.5 for crops. The masks were then traced into polygons in the same format as the labels.

Pixels per training patch
128 × 128
Patches per resized image
16
Training data held out for validation
About 25%

Comparing U-Net variants

Task lead · August–September 2021

Five architectures from the U-Net family were built from scratch in TensorFlow-Keras, with configurable depth, width and kernel size: U-Net, an ensembled U-Net, U-Net+, U-Net++ and MultiResUNet. Each used four levels, 128 filters in the first layer and 3 × 3 kernels, a balance between model size and patch size found by experiment. A second version of each model replaced upsampling in the decoder with transposed convolution.

The models were scored with the Jaccard index, or intersection over union, between predicted and labelled masks. The original U-Net scored highest overall, at 0.727. The denser models did not improve on it, probably because they overfitted the available data.

Best overall intersection over union (U-Net)
0.727
Architectures
5
Model versions compared
10
Overall intersection over union for each architecture and decoder type, from the Task 4 report.
Upsampling decoderTransposed-convolution decoder
U-Net0.7270.724
Ensembled U-Net0.6920.706
U-Net+0.6920.696
U-Net++0.7060.711
MultiResUNet0.7060.723

Testing on Sentinel-2 images and linking to risk scores

Final report · 2021

Finz planned to work with Sentinel-2 images, so the trained models were also run on Sentinel-2 scenes, where they marked trees, crops and water bodies. A separate task team wrote scripts to fetch and store Sentinel-2 tiles for Togo, Côte d’Ivoire and Cameroon.

To grow the training data later, the pipeline can also label new images. A trained model proposes polygons, a person refines them, and the new labels are added to the database for retraining.

Within the wider project, segmentation was meant to add local detail to the team’s risk models. The share of a scene covered by a class follows from its pixel count: in a 512 × 512 image of 1 km², 100,000 water pixels would cover about 0.38 km², or 38% of the area. More water nearby, for example, could raise the weight given to flood risk. Combining these shares with the flood and geopolitical scores was left for follow-on work.

Main outputs of the challenge and what each provides.
What it provides
Flood risk map for TogoFive risk levels from a hazard map (rainfall, slope, drainage density and soil) and a vulnerability map (land cover, population density and distance to rivers)
Geopolitical risk scoreMean and maximum risk for a location in Côte d’Ivoire from conflict events recorded since 1997, weighted by time, distance and event type
Segmentation pipelineShare of a 1 km² area covered by buildings, vegetation, water and other classes
Resources library210 datasets, articles, videos and tutorials gathered by the team
Flowchart of a labelling loop. A new image goes into the trained model, which produces new labels. The labels are refined by hand and saved as polygon labels for each class, which are added to the database. The database is then used to retrain the model.
Labelling loop for growing the training set: a trained model labels a new image, the labels are refined by hand and added to the database for retraining. From the Task 4 report.View full-size figureTask 4 report, Omdena–Finz challenge (2021)

My contribution

As Junior Machine Learning Engineer and leader of Task 4, I selected the dataset, built the TensorFlow segmentation pipeline, trained and compared the U-Net models, tested them on Sentinel-2 images and wrote the task report. I also worked with the other task teams on the overall model architecture.

Study scope

The models were trained on one labelled dataset from a single region, and the Sentinel-2 check was visual because those images had no labels. Within the eight weeks, the pipeline remained a separate tool and was not merged into the team’s risk scores.