EXPERT AND ALGORITHM: TWO INSEPARABLE SIDES OF OUR AI SOLUTION!

The team at Whale Seeker developed Möbius, an algorithm developed in-house to identify whales in aerial images. Our clients send us the images they wish to analyze, and we send them back annotated data ready to be consulted for decision making. But why don’t we sell access to Möbius so that individuals and organizations can use it themselves on their data?

We could consider this process if we were working with a constant, standardized input, e.g. if we were to analyze identical manufactured parts that are always presented to the camera from the same angle and in the same light. However, we deal with ecological data issued directly from the natural world, which is an irregular and constantly changing material. Therefore, there is no single solution that can give reliable results under all circumstances. Möbius is not an algorithm that can be left to fend for itself in the wild! 

The first characteristic that determines the performance of a model that has to deal with heterogeneous data is its power of generalization, i.e. its capacity to apply what it has learned during its training to data that it has never seen before. For example, if we trained a hypothetical model with perfect generalization ability with images of seals lying on sand, it would be just as good at recognizing a seal on snow, in rocks, in water, or in a multi-colored ball pit. Although this is a very simple task for a human, and considerable progress in artificial intelligence is made almost every day in this direction, even the best current models are far from reaching this degree of generalization. As a result, it is preferable to always train the model with images that have the same general characteristics as those to be analyzed. And we can’t expect our clients to retrain the model themselves on their data before using it!

Now let's imagine that we develop a model that generalizes very well and gives excellent results with all the data we can show it today: it is always possible that its performance will suffer from data drift. This has nothing to do with any of our equipment drifting at sea! Data drift refers to the loss of generalizability of a model due to changes over time in the data it is designed to process. In our case, such drift could be due to two main factors:

Changes in technology and sampling protocols: Advances in the photographic equipment used, increases in resolution or modifications in the pre-processing of the images, and the consequent changes in the altitude at which the survey aircrafts fly, could produce changes almost invisible to the human eye but very difficult for a computer vision algorithm to handle.

Biological changes: With climate change, we can expect the unexpected in marine conditions and species distribution. For example, some species of whales might extend their ranges in new environments where the model has not been trained to look for them. Alternatively, with fluctuations in water temperature, disruptions in algal or phytoplankton communities could change the appearance and color of the sea surface.

In short, this means that a model that is excellent today, if left as it is without being updated, could potentially begin to yield results that are a bit worrisome next year, and frankly mediocre in five, ten or fifteen years. How can Whale Seeker deal with the limits of the generalization power of our algorithm, and minimize its vulnerability to data drift?

The model alone cannot provide optimal performance, so we use a human-in-the-loop approach. Each time a new dataset is submitted to Möbius, we re-train it with updated, relevant, and correctly labeled images, and an expert reviews any cases where the algorithm is uncertain to confirm or refute its decisions. The algorithm then incorporates these new annotations to make better predictions in the future, so it is constantly improving and updating. The expertise of the professionals at Whale Seeker can't be sold, learned in a weekend, or passed on in a few clicks. And that's what makes our AI solution so valuable: by running a loop that connects an experienced human to the algorithm, we can consistently deliver fast, accurate, and reliable results!

This blog was originally written in French.