The SpaceNet 7 Multi-Temporal Urban Development Challenge: Announcing the Winners

Dec. 04, 2020

Adam Van Etten Chief Data Scientist, IQT Labs

Preface: SpaceNet LLC is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e., building footprint & road network detection). SpaceNet is run in collaboration by co-founder and managing partner CosmiQ Works, co-founder and co-chair Maxar Technologies, and our partners including Amazon Web Services (AWS), Capella Space, Topcoder, IEEE GRSS, the National Geospatial-Intelligence Agency and Planet.

The 7th iteration of the SpaceNet Challenge Series is officially in the books, and our top 5 winners have been determined. The Multi-temporal Urban development Challenge staked out an ambitious goal: identify and track building footprint change over a deep temporal stack. We are happy to report that this challenging task was met with a high degree of ingenuity by the top competitors. Our public baseline model achieved a score of 17 (out of 100) on the final test set, yet the winning submission more than doubled this score, with a score of 41. In the following sections we detail the winners of the challenge, and provide a prelude of some of the insights gleaned from this challenge (further analysis will have to wait for subsequent blogs).

1. Background

This is the first SpaceNet challenge to explicitly include the time dimension, and accordingly used a new medium resolution satellite imagery dataset from Planet that encompasses over 100 locations around the globe and 24 consecutive monthly observations that cover multiple seasons and significant construction in most locations. See here for more details about the dataset.

As noted in previous blogs [e.g. 1] the task of identifying buildings and tracking unique building identifiers (i.e. addresses) over time has numerous humanitarian, urban planning, and disaster response applications. We constructed a new metric, SCOT, to measure performance in tracking buildings, and released a baseline algorithm demonstrating that the ambitious goal of SpaceNet7 was indeed achievable.

2. Winners

Over the course of the previous six SpaceNet challenges, we have observed significant improvements in algorithmic performance. Much of these improvements stemmed from careful curation of training datasets and adoption of state-of-the-art neural network architectures. Less effort has been spent on algorithm design; for example in SpaceNet 5 all competitors used the CRESI algorithmic framework, but swapped out segmentation models to achieve small performance gains. SpaceNet 7, however, yielded significant variation in approaches, and creative approaches to post-processing that did not simply rely upon the tried-and-true (though uninteresting) fallback of adding yet another model to an ensemble. In fact, the winning model did not use an ensemble of neural network architectures at all, and managed an impressive score with only a single, rapid model. Table 1 details the top-5 prize winning competitors of the 300+ participants in SpaceNet 7.

We see from Table 1 that ensembles of models are not a panacea, and in fact post-processing techniques have a far greater impact on performance than the individual architecture chosen. We will explore the novel post-processing techniques used in a later blog. For now, we note that the winning algorithm is a clear leader when it comes to the combination of performance and speed, as illustrated in Figure 1. Figures 2 and 3 show predictions from the winning algorithm.

*Figure 1. Performance vs speed for the winning algorithms. Up and to the right is best. Note that 1st place algorithm is many times faster than the runner-up submissions.*

*Figure 2. Initial and subsequent predictions by lxastro0 over a test region. True positives are in white, while blue denotes false negatives, and yellow false positives.*

*Figure 3. Initial and subsequent predictions by lxastro0 over another test region. True positives are in white, while blue denotes false negatives, and yellow false positives.*

3. Preliminary Analysis of Approaches

As noted above, post-processing techniques are really where the winning submissions differentiated themselves (and will be covered in depth later), but there are a few trends in the initial deep learning segmentation approach worth noting.

3A. Upsampling Improved Performance

The moderate resolution of imagery poses a significant challenge when extracting small footprints, so multiple competitors upsampled the imagery 3–4x and noted improved performance (see Figure 4).

*Figure 4. Predictions using multiple models for lxastro0. The 3x upscaled imagery yields slightly better results.*

3B. 3-channel Training Masks

The small pixel sizes of many buildings results in very dense clustering in some locations, complicating the process of footprint extraction. Accordingly, multiple competitors found utility in a multi-channel training mask for deep learning segmentation that incorporates building spacing, see Figure 5.

Figure 5. Input 3-channel training mask used by cannab showing building volume (blue), building borders (magenta), and separation area (green). The border and separation channels help differentiate closely spaced buildings.

3C. Ensembles Remain the Norm

While the winning algorithm eschewed multi-model ensembles (to great speed benefits), the remainder of the top-4 competitors used an ensemble of segmentation models. A typical approach is illustrated below.

Figure 6. Detection pipeline used by motokimura.

4. Conclusions

The seventh iteration of the SpaceNet Challenge is in the books, with rookie competitor lxastro0 notching the win. While beyond the scope of this blog to dive in too deeply, we look forward to reviewing the creative algorithms used to solve the difficult problem of tracking building footprints in medium resolution imagery. At the end of the day, the winning score of SCOT = 0.41 is extremely impressive, and subsequent blogs will dive further into algorithmic details, failure modes, and success stories.

Also recall that SpaceNet 7 is featured as a NeurIPS competition, so feel free to check out the upcoming session.

This blog was originally published on the DownLinQ here.