ATM Challenge evaluation: background information & discussion

ATM Challenge 1 | ATM Challenge 2 | ATM Challenge 3 | Model evaluation for ATM Challenges |

ATM Challenge 1 (2015)

Paul W. Eslinger, Ted W. Bowyer, Pascal Achim, Tianfeng Chai, Benoit Deconninck, Katie Freemane, Sylvia Generoso, Philip Hayes, Verena Heidmanng, Ian Hoffman, Yuichi Kijima, Monika Krysta, Alain Malok, Christian Maurer, Fantine Ngan, Peter Robins, J. Ole Ross, Olivier Saunier, Clemens Schlosser, Michael Schöppner, Brian T. Schrom, Petra Seibert, Ariel F. Stein, Kurt Ungar, Jing Yih (2016): International challenge to predict the impact of radioxenon releases from medical isotope production on a comprehensive nuclear test ban treaty sampling station, J. Environ. Radioactivity, 157, 41-51. doi:10.1016/j.jenvrad.2016.03.001

Source: Fleurus Institute for Radioelements (IRE), Belgium
Receptor: DEX33 Schauinsland (SIL)

ATM Challenge 2 (2016/17)

Source: ANSTO nuclear medicine facility, Australia
Receptors: Stations across the Pacific and in the Southern Atlantic: AUX04, AUX09, FRX27, NZX46, BRX11, GBX68

Maurer, C., J. Bare, J. Kusmierczyk-Michulec, A. Crawford, P. W. Eslinger, P. Seibert, B. Orr, A. Philipp, O. Ross, S. Generoso, P. Achim, M. Schoeppner, A. Malo, A. Ringbom, O. Saunier, D. Quelo, A. Mathieu, Y. Kijima, A. Stein, T. Chai, F. Ngan, S. J. Leadbetter, P. D. Meutter, A. Delcloo, R. Britton, A. Davies, L. G. Glascoe, D. D. Lucas, M. D. Simpson, P. Vogt, M. Kalinowski, and T. W. Bowyer, 2018, International challenge to model the long-range transport of radioxenon released from medical isotope production to six Comprehensive Nuclear-Test-Ban Treaty monitoring stations. J. Environ. Radioactivity 192, 667-686. URL: https://doi.org/10.1016/j.jenvrad.2018.01.030.

ATM Challenge 3

Blind phase to close on 30 June 2020.

CTBTO SnT Conference Presentation: Scenario and Practical Guidelines (PDF)

Preliminary evaluation by C. Maurer (ZAMG), Wosmip remote PDF

Click image for larger version, or download high-quality, scalable PDF-version. All images under the CC BY 4.0 licence. Produced with Generic Mapping Tools (GMT).

Overview

Four target stations: USX75 (Charlottesville), DEX33 (Schauinsland), SEX63 (Stockholm) and CAX17 (St. John's) [green circles]
Main sources: Daily releases from the MIPFs [orange stars] Chalk River Laboratories (CRL) and L'Institut National des Radioelements (IRE)
Minor sources (1): Daily releases from all NPP [yellow circles] and major research reactors [pink squares] in the Northern Hemisphere
Minor sources (2): Annual mean releases from releases from seven smaller MIPF [orange triangles]: NIIAR (Dimitrovgrad, Russia), and Karpov Institute (Russia), Mallinckrodt (The Netherlands), HFETR (China), PINSTECH PARR-1 (Pakistan), Lantheus Medical Imaging (Canada), Nordion (Canada)

Publication of results:

Maurer, S. Galmarini, E. Solazzo, J. Kuśmierczyk-Michulec, J. Baré, M. Kalinowski, M. Schoeppner, P. Bourgouin, A. Crawford, A. Stein, T. Chai, F. Ngan, A. Malo, P . Seibert, A. Axelsson, A. Ringbom, R. Britton, A. Davies, M. Goodwin, P.W. Eslinger, T.W. Bowyer, L.G. Glascoe, D.D. Lucas, S. Cicchi, P. Vogt, Y. Kijima, A. Furuno, P.K. Long, B. Orr, A. Wain, K. Park, K .-S. Suh, A. Quérel, O. Saunier, D. Quélo (2022). Third international challenge to model the medium- to long-range transport of radioxenon to four Comprehensive Nuclear-Test-Ban Treaty monitoring stations, Journal of Environmental Radioactivity, 255, https://doi.org/10.1016/j.jenvrad.2022.106968 (open access).

Difficulties in model evaluation for the CTBTO ATM Challenges

data span several magnitudes
data tend to be log-normally distributed, but include zero (like precipitation)
rare events
observations below MDC - only upper boundary known, can be zero or not
observation uncertainty is available, how to consider?
near miss is not a total miss - geographically, in time, and with respect to concentration above/below MDC
total size of data set is small
emission strength can vary by orders of magnitude in short time - correlation with meteorological events or not can cause bifurcation / nonlinearities

Links related to model evaluation

Note that all these pages have relevant further links, e.g. to literature!

WWRP/WGNE Joint Working Group on Forecast Verification Research

their detailed page with all scores etc.

AMS Committee on Probability and Statistics
Model evaluation kit of the Harmo initiative

If you want your comments added here, please email me!