1. Western EcoSystems Technology (WEST), Inc.
Comparison of Full-Spectrum and Zero-Crossing Automated Bat Call Classifiers
Donald Solick, Matthew Clement, Kevin Murray, Christopher Nations, and Jeffery Gruver
Western EcoSystems Technology (WEST), Inc.
Good day!

2. Identification of bat echolocation calls to species is becoming increasingly important, particularly for assessing the risk posed to bats by wind-energy development and for monitoring the spread of White-Nose Syndrome. However, acoustic monitoring typically yields large amounts of data, and manually classifying calls can be subjective and time-consuming. To make these analyses more objective and manageable, several software programs have been developed in recent years that perform automatic classification of calls to species.

3. Full-Spectrum (FS) Zero-Crossing (ZC)
Time and Frequency
Amplitude
Multiple frequency content
-harmonics, multiple bats, calls against background noise
Zero-Crossing (ZC)
Time and Frequency
Dominant frequency content
-loudest sound gets recorded
Acoustic data for bat studies come in two main flavors. Full-spectrum data—which you might record on a Pettersson detector--are a digitized representation of the full acoustic soundscape, and provide information on time, frequency, and amplitude of a bat call, as well as multiple frequency content. That is, all competing sounds such as harmonics, multiple bats, or background noise are fully rendered. In contrast, zero-crossing data that you might record on an Anabat provide just time and frequency information, and only render the dominant frequency –or loudest sound—so that harmonics, other bats, or the calls themselves may be incomplete or obscured.

4. More information = better species discrimination
Assumption
More information = better species discrimination ?
Objective
To determine which type of classifier is better at species discrimination given the same set of known calls
The fact that full-spectrum data contain more information has led to a prevailing notion that these data are better suited for discriminating species than are zero-cross data. However, this has never been tested. Therefore, our objective was to determine—given the same set of known calls—whether full-spectrum automated classifiers are better at species discrimination than existing zero-cross methods.

5. FS Reference Calls Perimyotis subflavus PESU 97 Lasiurus borealis LABO
Species
Abbreviation
Number of files
Perimyotis subflavus
PESU 97
Lasiurus borealis
LABO 40
Eptesicus fuscus
EPFU
115
Lasionycteris noctivagans
LANO 23
Lasiurus cinereus
LACI 18
Myotis leibii
MYLE
Myotis septentrionalis
MYSE 46
Myotis lucifugus
MYLU
263
Myotis sodalis
MYSO 26
We addressed this objective by obtaining 652 full-spectrum reference calls for nine eastern US bat species. We used only high quality files by hand-released bats that contained a minimum of 5 calls and were free of interlopers or other extraneous noise.

6. SonoBat 3.04 Northeast
Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree
We first analyzed calls using SonoBat Northeast, which is currently the only software available for automated analysis of full-spectrum data in the US. Sonobat reports three types of classification output:

7. SonoBat 3.04 Northeast
Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree
x = Lano
Mean Classification makes decisions based on the average parameter values for all calls within a file;
Lano
0.9991

8. SonoBat 3.04 Northeast
Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree
By Vote looks at each call and determines which species had the majority of calls within the file;
Lano
4 of 5
Lano
Lano
Epfu
Lano
Lano

9. SonoBat 3.04 Northeast
Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree
Lano
4 of 5
and Consensus classifies a file as a species when both the Mean Classification and By Vote decisions agree.
Lano
Lano
0.9991

10. ZC Classifiers
Bat Classification and Identification (BCID) East v2.4mAC
EchoClass 64 v1
Discriminant Function Analysis for New York
Developed by Eric Britzke for use by NY Dept.of Environmental Conservation
The zero-cross classifiers we tested were BCID, EchoClass, and a Discriminant Function Analysis developed for use by the New York Department of Environmental Conservation.

11. Converting FS to ZC
Anabat Converter 0.8 (
AnalookW 3.8e
Applied filter and extracted parameters
Except EchoClass
We converted the full-spectrum files to zero-cross data for use in the zero-cross classifiers using Anabat Converter Using AnalookW, we then applied a general filter to files and extracted call parameters for use in BCID and the NY DFA. This step was skipped for EchoClass, as it operates directly on the zero-cross files themselves.

12. Overall Classification Rates (%)
SonoBat
BCID
Echo Class
NY DFA
Class
Vote
Consensus
% Correct 49 56 45 53 43
% Incorrect 6 7 4 38 36
% Unknown 14 2 15
And here are some results. This table shows the overall classification rates—expressed as percentages--for each of the methods. For example, for the SonoBat Mean Classification output, 49% of the files were correctly classified to species, 6% were misclassified, and 43% were not classified at all. I want to point out a few things with this table. First, none of the classifiers did a great job overall, correctly classifying between 43 and 60% of all calls.

13. Overall Classification Rates (%)
SonoBat
BCID
Echo Class
NY DFA
Class
Vote
Consensus
% Correct 49 56 45 53 43
% Incorrect 6 7 4 38 36
% Unknown 14 2 15
Second, the incorrect classification rates—or misclassified calls—are much higher for the zero-cross methods than for Sonobat. These methods are much less likely to classify files as unknown, and do not instill a lot of confidence in the data, as essentially any given file classification has a 40-50% chance of being wrong. In contrast, Sonobat is more conservative, and the By Vote output looks to provide the best trade-off in terms of correct, incorrect, and unknown classifications.

14. Correct Classification Rates (% Correct)
Species
SonoBat
BCID
Echo Class
NY DFA
Class
Vote
Consensus
PESU 96 98 77 95
LABO 43 70 30 60 65 63
EPFU 88 93 86 51 57 78
LANO 74 83 52 26
LACI 53 68 58
MYLE 17 9 4
MYSE 33 46
MYLU 31 24 56 25
MYSO 38 35
This table shows the correct classification rates for each species, again in percentages. Overall, the classifiers performed the best on tricolored bats, correctly classifying between 77 and 98 percent of calls, and performed the worst on eastern small-footed bats, only classifying 0 to 26 percent of calls correctly.

15. Correct Classification Rates (% Correct)
Species
SonoBat
BCID
Echo Class
NY DFA
Class
Vote
Consensus
PESU 96 98 77 95
LABO 43 70 30 60 65 63
EPFU 88 93 86 51 57 78
LANO 74 83 52 26
LACI 53 68 58
MYLE 17 9 4
MYSE 33 46
MYLU 31 24 56 25
MYSO 38 35
Here, the highlighted cells indicate which classifier performed best for each species. For example, both the Sonobat By Vote and BCID performed best for tricolored bats, correctly classifying 98 percent of calls. By Vote performed the best for the eastern red bat, and so on.
In general, Sonobat performed best for non-Myotis species. Sonobat was particularly successful at discriminating big brown and silver-haired bats, which are notoriously difficult to separate, correctly classifying 93 and 83 percent of calls, respectively. With the exception of eastern small-footed bats, the zero-cross classifiers performed best for the Myotis species. That said, classification rates still only ranged between 31 and 56 percent.

16. Correct Classification Rates (% Correct)
Species
SonoBat
BCID
Echo Class
NY DFA
Class
Vote
Consensus
PESU 96 98 77 95
LABO 43 70 30 60 65 63
EPFU 88 93 86 51 57 78
LANO 74 83 52 26
LACI 53 68 58
MYLE 17 9 4
MYSE 33 46
MYLU 31 24 56 25
MYSO 38 35
Of particular interest is that the zero-cross methods only classified 31-38% of Indiana bat calls correctly, and Sonobat did not identify any. This suggests that automated methods may not be very reliable for monitoring endangered Indiana bats, and that Sonobat in particular may be too conservative for this task.

17. Summary None of the classifiers performed well overall
SonoBat better for non-Myotis, ZC better for Myotis
SonoBat more conservative
Caution when using automated classification for Indiana bat surveys
To summarize, none of the classifiers performed well overall. This is a bit surprising, as we only used high-quality reference calls in this study. We would expect performance to be worse for calls that were passively collected in the field. Performance varied by species, with Sonobat performing better for non-Myotis species in general and zero-cross classifiers performing better for Myotis species. This difference may be because discriminating non-Myotis calls relies more on harmonics and amplitude, while Myotis classification relies more on slope and duration, but this remains unclear. Sonobat is also more conservative than zero-cross methods, which is perhaps its greatest strength. However, Sonobat may be too conservative when it comes to Indiana bats. Zero-cross methods also did not perform very well for classifying Indiana bats. This is of concern, as the U.S. Fish and Wildlife Service has proposed a protocol to determine the presence of Indiana bats solely from acoustic data. Our results suggest that acoustic data and automated classifiers are not yet sufficient for this task.

18. Caveats Low sample size for some species Lost in translation?
Different Analook filters could improve or worsen ZC classifier performance
Focus on trends, not absolute comparisons
That said, there are some caveats to mention. First, we had a low sample size for several species which is not ideal, but highlights the lack of full-spectrum reference libraries available in the US and Canada. It is also possible that some information was lost in translation when converting to zero-cross and may have affected species ID. However, the fact that zero-cross methods outperformed Sonobat for some species suggests this may not have been much of an issue. The filter we used for extracting parameters in Analook, however, likely did affect performance. Matthew Clement will be presenting research later today that demonstrates how different filters can lead to very different classifications. A filter was not used on the data for EchoClass, which might explain why it did not perform well relative to the other classifiers. That said, I also want to emphasize that the point of this study was not to see which classifier was best, but to see how one type of classifiers compared to another. We are certain that the performance of each of these classifiers will vary with different filters, different species, and different datasets, but we are also confident that the trends revealed by our study represent real differences between full-spectrum and zero-cross automated classifiers.

19. Conclusion
Illustrates limitations of automated classification of bat acoustic data
Species presence/probable absence should be based on multiple lines of evidence
In conclusion, our results illustrate some of the limitations of automated classification for bats, and suggest that species presence and probable absence should be determined from multiple lines of evidence rather than any single data source.

20. Thank You! Ryan Allen, Bat Call Identification, Inc.
Eric Britzke, US Army Engineer Research & Development Center
John Chenger, Bat Conservation and Management
Carl Herzog, New York Dept. of Environmental Conservation
Amie Shovlain, Montana Natural Heritage Program
Craig Stihler, West Virginia Dept. of Natural Resources
Joe Szewczak, Humboldt State University
Thank you!