Combining two user-friendly machine learning tools increases species detection from acoustic recordings

The technological advances in recent decades revolutionized the way we currently monitor habitats and species. Among the emerging techniques for biomonitoring, several automated and non-invasive methods have rapidly become standard tools in ecology, such as camera trapping, remote sensing, and passive acoustic monitoring (Lahoz-Monfort and Magrath 2021). These automated and non-invasive techniques offer researchers the ability to expand the spatial and temporal scales of their studies and contribute to collect large amounts of data. However, datasets obtained through automated techniques often pose issues for investigators because manual processing of automated data is time-consuming, tedious, and subject to human bias. To overcome these issues, machine learning algorithms can effectively process such large datasets (e.g., Priyadarshani et al. 2018; Stowell 2022; Xie et al. 2022).

Passive acoustic monitoring is increasingly being used to detect different groups such as anurans, bats, birds, or insects (Sugai et al. 2019; Hoefer et al. 2023). Surveys relying on passive acoustic monitoring easily generate substantial volumes of recordings, making it impossible to visually inspect or listen to all files (e.g., Pérez-Granados and Schuchmann 2020). The development of machine learning algorithms has become crucial to deal with these large numbers of files (Stowell 2022; Xie et al. 2022). Unfortunately, implementing some of the state-of-the-art machine learning models can be complex and intimidating for ecologists and managers without an engineering or computing background. Indeed, the difficulty in using sound detection tools is a limiting factor for passive acoustic monitoring surveys (Wood et al. 2023a). However, a new generation of user-friendly and ready-to-use machine learning tools have recently emerged and may further improve the effectiveness of automated audio recognition, such as BirdNET and Kaleidoscope Pro (e.g., Manzano-Rubio et al. 2022; Bota et al. 2023; Wood et al. 2023a).

BirdNET is an automated sound classifier that is free, open source, and based on a convolutional neural network architecture for automated identification of over 6000 wildlife species, including birds, anurans, and mammals (Kahl et al. 2021; Pérez-Granados 2023; Wood et al. 2023a, 2023b). For each 3 s fragment of an audio file, BirdNET provides a species identification accompanied by a confidence score, allowing researchers to filter the output according to a desired confidence level. Although BirdNET is a promising tool, its effectiveness for wildlife monitoring has yet to be extensively assessed (reviewed by Pérez-Granados 2023), with only a single case study testing its capabilities for anuran monitoring (Wood et al. 2023a). BirdNET can be run through a user-friendly interface or the command line, and requires no expertise in machine learning (Wood et al. 2023a). Another ready-to-use and user-friendly machine learning tool for audio recognition is Kaleidoscope Pro (Manzano-Rubio et al. 2022), which requires a paid subscription for an annual license. Unlike BirdNET, the Kaleidoscope Pro workflow relies on the automated detection of candidate sounds based on the input of signal parameters and their classification, through unsupervised machine learning (hidden Markov models), into clusters of species vocalizations (Pérez-Granados and Schuchmann 2020). Both BirdNET and Kaleidoscope Pro can easily be trained to develop species-specific algorithms without the need for technical expertise. Despite its potential, current knowledge on the ability of automated classification of anuran vocalization in large acoustic datasets remains limited (e.g., Huang et al. 2014; Wood et al. 2023a).

In this paper, we aim to evaluate two user-friendly machine learning tools, BirdNET and Kaleidoscope Pro, to detect the American toad (Anaxyrus americanus (Holbrook, 1836)) in recordings. Specifically, we evaluated the efficacy of each approach in detecting the species compared to a human and then evaluated the efficacy of both methods combined. Additionally, we measured the computing time required to scan the validation acoustic dataset (n = 371 3 min recordings) and the amount of human time needed to verify the output. Then, we evaluated the effectiveness and speed of a two-step approach to detect the presence of the species using a large field acoustic dataset collected in northern Canada (n = 6194 3 min recordings). By sharing our assessments and insights, we hope to provide valuable guidance to apply automated detection and machine learning approaches in wildlife monitoring. For clarity, we use the term “detections” to denote potential multiple instances of predictions made by BirdNET or Kaleidoscope Pro in a given recording, whereas we use the term presence to indicate that the species was confirmed at least once by a human in a 3 min recording.

The inherent challenges of automated acoustic recognition of target species in large acoustic datasets limited the widespread use of passive acoustic monitoring (Wood et al. 2023a). However, recent advancements of algorithms (e.g., Kahl et al. 2021; Wood et al. 2023a) together with the development of user-friendly software for automated recognition have opened up new avenues for automated wildlife recognition from sound recordings (Manzano-Rubio et al. 2022; Bota et al. 2023). Here, we demonstrated the ability of two user-friendly machine learning approaches (Kaleidoscope Pro and BirdNET) for detecting the presence of the American toad.

Kaleidoscope Pro was faster and detected the target species in a larger number of recordings than BirdNET. Our findings align with a recent study that compared both Kaleidoscope Pro and BirdNET in detecting the presence of a threatened bird species (Botaurus stellaris (Linnaeus, 1758), see Manzano-Rubio et al. 2022). The target species considered by Manzano-Rubio et al. (2022) and the American toad have simple vocalizations (Fig. 1), which may partly explain the high ability of Kaleidoscope Pro for detecting both species. However, convolutional neural networks, such as BirdNET, may outperform Kaleidoscope Pro at detecting more complex vocalizations. Further research should investigate into the ability of Kaleidoscope Pro and BirdNET for detecting a wider range of species. In our dataset, Kaleidoscope Pro produced a higher number of false positives than BirdNET, but the computing time and the time required to verify the output of Kaleidoscope Pro by a human was considerably reduced compared to BirdNET. Such speed can be attributed to the ease in verifying the output directly within Kaleidoscope Pro and to the workflow we employed. Users can easily navigate through Kaleidoscope Pro output while checking the sonograms, confirm the classification by pressing a key, and move on to the next candidate sound. In contrast, verifying the BirdNET output required to individually open the file associated with each recording and locate the 3 s spectrogram annotated. However, the most recent version of BirdNET (version 2.4, released in June 2023) now allows users to extract all the segments (in .wav file) where the target species were predicted. Although our study was conducted before this new addition to BirdNET, we expect that this new feature will substantially accelerate output verification. It is worth of highlight that we used an Intel(R) Core(TM) i7 (8th Gen, CPU 1.80 GHz, 1.99 GH, 8 GB RAM) with the acoustic recordings stored and analysed from an external hard drive. Therefore, the speed of data transfer from the hard drive to the laptop was the main limiting factor of computing time (i.e., computing time would be lower if processed from the internal memory of the laptop). However, it is important to consider that many monitoring programmes typically store their acoustic data in hard drives. Therefore, the timing values provided in our study can be useful for managers and researchers as an estimate of the time required for automated detection using Kaleidoscope Pro or BirdNET.

The workflow we applied in our study greatly expedited the review process. Initially, we leveraged the unsupervised clustering performed by Kaleidoscope Pro, focusing our attention on candidate sounds from clusters with a high probability of containing the target species. This approach reduced the number of candidate sounds to be verified by up to 80% from the original output (see also Pérez-Granados and Schuchmann 2020). We acknowledge the possibility of a few missed American toad vocalizations using this approach. Nonetheless, the number of missed presences (false negatives) at the recording level is expected to be minimal. To expedite the reviewing process, we focused solely on confirming the presence of the species in each recording. Once the American toad was confirmed within a recording, we did not check subsequent candidate sounds within that recording.

BirdNET was originally developed for automated bird song recognition (>6000 bird species included), but it has expanded to include tens of anurans and a few mammals (e.g., Wood et al. 2023a, 2023b). Our assessment, to the best of our knowledge, is the second study evaluating the performance of BirdNET for monitoring anurans (see Wood et al. 2023a). We hope that our results will encourage investigators seeking automated anuran detection to consider BirdNET. The ability of BirdNET to detect the American toad was low compared to Kaleidoscope Pro. Nonetheless, BirdNET was still able to detect the species using the default values in over half of the recordings, despite their short duration of 3 min. We encourage researchers wishing to use BirdNET to further assess the influence of the sensitivity, overlap, and confidence score parameters on wildlife detection. Moreover, further research assessing BirdNET’s effectiveness in detecting a wider range of anurans would be particularly valuable, as our current knowledge is limited to three species from North America (Wood et al. 2023a and our study).

Surprisingly, BirdNET detected a large number of presences undetected by Kaleidoscope Pro (8% in the validation dataset and 10% in the field acoustic dataset). Upon visual inspection and received sound level, it became evident that most of BirdNET’s detections corresponded to American toad vocalizations emitted far from the recorder. This pattern suggests that BirdNET is more sensitive than Kaleidoscope Pro in detecting the species when it vocalized at larger distances. However, the reason why BirdNET did not detect the species more frequently than Kaleidoscope Pro remains unclear. Differences in the training dataset used for creating the American toad algorithm in BirdNET, such as the inclusion of weak vocalizations, may have contributed to these discrepancies. Unfortunately, because the contents of the BirdNET dataset are not publicly released, it is not possible to determine the number, type and quality of the vocalizations used for training the algorithm. Further research should investigate the underlying reasons for the differences observed between the two machine learning approaches we compared, although their distinct structures (e.g., convolutional neural network architecture on BirdNET and hidden Markov models in Kaleidoscope Pro) may complicate direct comparisons. Moreover, the unsupervised nature of Kaleidoscope Pro machine learning makes the classification process more obscure than with BirdNET.

Our two-step approach consisted in using the speed of Kaleidoscope Pro in scanning and verifying the whole field acoustic dataset and then running BirdNET exclusively on the recordings where Kaleidoscope Pro had not detected the species. This sequential workflow significantly reduced the overall time required for data scanning and output verification in BirdNET. The two-step approach improved the detection probability of the American toad by approximately 10%, with only 45 additional minutes dedicated to BirdNET output verification. Importantly, this approach enabled us to remove all false positives from the final dataset, amounting to a mere 0.5% of the total recording time. Removing false positives is needed for efficient use of acoustic algorithms in large acoustic datasets and to provide reliable results for further analyses highly sensitive to false positives such as occupancy analyses (Guillera‐Arroita et al. 2017; Wood et al. 2023a).

Our acoustic monitoring protocol was set to record from 7 pm to 11 pm, aiming to monitor the assemblage of pond-breeding anuran species in the study area (see Feldman et al. 2023). However, the maximum vocal activity of the American toad occurs between 10 pm and 2 am (Taylor 2006). Therefore, our protocol does not align with the peak of calling activity of the American toad and may have resulted in a potential underestimation of species occurrence. Although it does not have an impact on our conclusions, further research aiming to acoustically monitor the American toad should extend the recording schedule to cover the hours of maximum calling activity of the species.

In this study, we demonstrated the ability of two user-friendly machine learning approaches as efficient tools for automated acoustic recognition of an anuran at the northern portion of its distribution. Our assessment opens a new door for managers and researchers with minimal technical expertise to leverage the power of automated recognition software. Moreover, we developed a streamlined two-step approach that increases the probability of detecting the species while reducing the time of acoustic processing and manipulation by humans, showing that both machine learning approaches can be easily combined. The manual verification approach that we used allowed us to remove every false positive from the database while expediting the process (i.e., once the presence was verified, no more detections for the same 3 min recording were checked). We hope that our assessments and recommendations will encourage managers and researchers to embrace passive acoustic monitoring as a viable technique for wildlife monitoring. Our results indicate that automated detection may no longer be a limiting factor for passive acoustic monitoring surveys.

We thank M. Brigham and two anonymous reviewers whose comments improved the manuscript. CP acknowledges the support of the Ministerio of Educación y Formación Profesional through the Beatriz Galindo Fellowship (Beatriz Galindo—Convocatoria 2020). MJF gratefully acknowledges the field assistance of R. Chevallier in setting up the acoustic equipment, and L. Imbeau and N. Fenton for their contributions to the conception and design of the study. The Natural Sciences and Engineering Research Council of Canada—UQAT Industrial Research Chair on Nordic Biodiversity in a Mining Context funded the field work.