AI Enables Who's Who Of Brown Bears In Alaska

Being able to distinguish individual animals - including their unique history, movement patterns and habits - can help scientists better understand how their species function, and therefore better manage habitats and study population dynamics. Today, most computer vision systems for tracking animals are effective on species with patterns and markings, such as zebras, leopards and giraffes. The task is much more complicated for unmarked species where individual differences are harder to spot. Distinguishing a particular brown bear from its peers in a non-invasive way requires an incredible eye for detail and years of viewing the same bears over time. What's more, these bears emerge from hibernation in the spring with shaggy fur and having lost quite a bit of weight and then substantially increase their body weight feasting on salmon, as well as fully shedding their winter coat - that's enough to throw off experts as well as AI algorithms. A team of scientists from EPFL and Alaska Pacific University has developed an AI program that can recognize individual brown bears over time in photos, despite changes in the bears' appearance and the difficulties associated with image capture for these elusive and far-ranging animals.

Our biological intuition was that head features combined with pose would be more reliable than body shape alone, which changes dramatically with weight gain. The data proved us right - PoseSwin significantly outperformed models that used body images or ignored pose information

Machine learning based on head and posture

The McNeil River State Game Sanctuary in Alaska is home to the world's largest seasonal population of brown bears. Every summer, nearly 150 of these animals move through this area undisturbed over 500 km² of pristine land. They gather on high-protein sedge meadows, and at large, low-grade waterfalls to catch salmon, providing an opportunity for the few humans allowed in the sanctuary to observe them. "The latter are strictly supervised; this is bear territory!" smiles Alexander Mathis, a professor at EPFL's Brain Mind Institute and Neuro-X Institute. This remote area is also home to Beth Rosenberg, a researcher at the Fisheries, Aquatic Science, and Technology Laboratory at Alaska Pacific University, for four months of the year. She has built up an extraordinary database of brown-bear images: between 2017 and 2022, she took over 72,000 photos of 109 different brown bears under all sorts of conditions - in the rain, in varying times of day, and with bears in every available behavior and posture (or angle) - in order to fully depict the bears in their natural habitat.

To develop their AI program, called PoseSwin, the scientists drew on their biological expertise to focus on four characteristics of bears that change surprisingly little over time: the shape of the muzzle (which has minimal fatty tissue), the brow bone angle, and the placement of the ears. Crucially, they incorporated pose information - analyzing photos of bears from various angles including frontal, profile, and tilted views. "This pose-aware approach enabled us to use as many pictures as possible, even those that do not clearly show the bear's face perfectly," says Mathis. "Our biological intuition was that head features combined with pose would be more reliable than body shape alone, which changes dramatically with weight gain. The data proved us right - PoseSwin significantly outperformed models that used body images or ignored pose information. "

Capturing a bear's true identity

The architecture behind the scientists' program is based on transformers - the same fundamental technology that powers large language models like ChatGPT - but adapted specifically for image analysis. "We used a technique called metric learning to train a transformer to understand the relationships between different parts of the images," says Mathis. That means the algorithm learned not only to recognize individual bears based on the characteristics mentioned earlier, but also to compare two images of bears. The team exposed the algorithm to groups of three photos: two of the same bear taken at different times and one of another bear. The algorithm projected the images onto a multidimensional mathematical space, placing the photos of the same bear near each other and pushing those of the other bears further away. "It is a real game of attraction and repulsion, a digital tug-of-war where images shuffle around until they form coherent groups," says Mathis. "Each bear ended up being represented as a unique constellation of points, which suggests the AI program was able to capture something fundamental - not just a bear's appearance but something closer to its identity." PoseSwin can also flag bears that it has never seen before, which is a major advantage for studies in unenclosed areas where new individuals can appear regularly.

The next step was to apply the program in a new environment. For that, the scientists turned to citizen science: they collected photos taken by visitors to Katmai National Park and Preserve, located just over 60 km from McNeil River, and fed them into the PoseSwin algorithm. The program clearly recognized several of the bears, indicating specifically where the animals move seasonally in search of food. "This is a concrete example of the PoseSwin model's potential," says Beth Rosenberg. "The technology could eventually be used to analyze the thousands of pictures that visitors take every year and help to build a map of how brown bears use this expansive area. This helps us to understand what they need, how their population dynamics work, and many other important ecological questions."

"A bear is a complicated version of a mouse"

Thanks to photos of the bears and some virtual measurements of their morphology, scientists are now able to track Sloth, Rocky, That Bear and around 100 of their peers without interfering with them physically. "The better we can distinguish individual bears, the better we can understand them and their behaviors at the species level," says Rosenberg. "Bears are at the top of the food chain and ensure the proper functioning of their ecosystem. They are critical to maintaining healthy systems."

PoseSwin will make field work more broadly applicable for the scientists involved in the study, as well as for other scientists working in other contexts. It also achieved excellent accuracy on benchmark datasets of macaques, suggesting its broad applicability beyond bears. "Bears are perhaps the hardest species to recognize individually," says Mathis. "We focused on them first with the idea that our program could be adapted to other species from mice to chimps, which seem to exhibit much less visual variation." The team has provided open-source access to their algorithm and the data used to develop it so that other researchers can use and adapt it as needed.

The scientists plan to continue developing PoseSwin for Alaskan brown bears. Because the program is scalable, they are already able to add data collected in other seasons and from other locations. Their goal is to automate much of the system so that it can help monitor wild animal populations over the long term.