Raes, N. and H. ter Steege (2007). “A null-model for significance testing of presence-only species distribution models.” Ecography 30(5): 727-736.
Validation of SDMs is both important and potentially difficult and problematic, especially in the case of presence-only data. Using pseudo-absence (background) points to calculate the AUC of trained models can create a number of problems, particularly that the maximum AUC is no longer 1 but rather 1-a/2 where a is the fraction of the landscape of interest genuinely covered by the species’ true distribution. As a result the interpretation of AUC values becomes muddied and typically “good” AUC values may not be what they seem. The authors propose a null model based approach to significance testing using “collection” localities, randomly drawn from the background, in equal number to the actual number of collection localities. This method relies, however, on the often unmet assumption that researchers have sampled all localities equally well. They propose to address this issue by drawing the random “collection” localities exclusively from the set of known collection localities. The authors illustrate this method using occurrences of the genus Shorea on the island Borneo and the SDM MaxEnt. They build both background based random null-models and null-models based on 1837 known collection localities for all plants across Borneo. Both of these types of null-models were used to construct a 95% confidence interval for AUC values such that AUCs outside of the interval implied that the model performed significantly differently from the random null model. 91% of species had a higher AUC than the random null-model CI, while only 61% had an AUC higher than the bias adjusted CI. In so doing, they show the importance of correcting for bias in your presence points when building null models, while recognizing that often researchers will not have an accessible set of nearly 2000 potential collection points to use for correcting their null-model. They suggest using distance to features such as roads, rivers, cities, and nature reserves, as a proxy for this data. They found that the AUCs of their true models and both null-models decreased with increasing number of presence points (and growing predicted range size) which they interpret to mean that as the unknown true distribution of the species increases only the maximum AUC decreases and not the actual predictive ability of models. Though the concept of null-model based hypothesis testing has been used previously in SDM and elsewhere of course as well, the addition of the bias-corrected null model seems valuable for effective evaluation. It does ignore, however, the fact that the actual predictive accuracy of a given model will be affected by bias in the presence data. This method, though, may make our AUC evaluations more fair and accurate and so seems worth further study.
