Evaluation and diagnostics

Scientific Working Group on Evaluation and Diagnostics

The Scientific Working Group on Evaluation and Diagnostics exists to foster activities that help describe, document and understand the performance of integrated assessment models so as to improve model performance and reliability and to make recommendations in the areas of priorities for community activities and standards of best practice.

Integrated assessment modelers have and will continue to be asked by potential users and critics why they should have confidence in the results of IAMs, or what level of confidence they should have in the results. Like all models, IAMs describe a critical set of relationships between assumed values for key input assumptions, which lie outside the model, and variables of interest that the model predicts, albeit a contingent prediction. For IAMs exogenous assumptions typically include demographic characteristics, economic growth, technology characteristics and availability, and the policy environment. IAMs produce contingent predictions of such variables as energy supply, demand, transformation, trade and prices, agricultural supply, demand, land use, land cover, and prices, carbon and other greenhouse gas prices, and the economic cost of policy interventions.

Determining the usefulness of a given model for a given purpose is both art and science. The purpose of the model is critical to determining usefulness. It is also critical to determining the right set of test to apply in the evaluation and diagnosis of a given model.

A variety of techniques have been applied to evaluate and diagnose models. While the SWG will not undertake model evaluation, diagnostics or validation, the SWG can undertake activities that help shape and facilitate such activities.

In recent years, a number of community activities have undertaken model documentation, diagnostics and evaluation activities. Workshops on model evaluation and diagnostics have been conducted by the US DOE-funded PIAMDDI and PCHES (2010-present) and the EU FP7 funded AMPERE projects (2011-2014) that helped to enhance our understanding about the various ways contributing to model evaluation and the analogies to the approaches in other communities such as the operations research and climate modelling communities. Both projects developed diagnostic methods and indicators to better understand underlying reasons for similarities and differences in model behavior. The EU FP7 funded ADVANCE project (2013-2016) developed standards for model documentation (see common IAM documentation wiki) and continued work on IAM diagnostics and evaluation (see Wilson et al. in Climatic Change volume 166, Article number: 3 (2021)). In the framework of the EU H2020 NAVIGATE project (2019-2023), an extension of AMPERE project, a selected set of well-defined indicators as a community standard, to systematically and routinely assess IAM behaviour, similar to metrics used for other modeling communities such as climate models has been built (see Harmsen et al. in Environmental Research Letters, Vol 16 No. 15, 2021 “Integrated assessment model diagnostics: key indicators and model evolution“.

The Evaluation and Diagnostic SWG has a wide scope of potential activities. While all of the areas listed below come within the SWG perview, work has focused primarily on Informaiton Exchange.

1. Information Exchange: Identify IAM evaluation and diagnostic community activities that are underway and of relevance for IAMC members.
2. Framing: Facilitating a discussion about adequate conceptual frameworks for IAM evaluation, including experts from outside the IAM community. Part of this could be the articulation of important model outputs that IAMs produce and the questions this community is trying to answer.
3. Model Validation Against Historical Data: Model validation experiments using historical data (sometimes referred to as “hindcast” exercises) have been conducted by a number of teams to test IAM behavior against historical developments. There are a host of questions and challenges to resolve, such as what prescribed historical period to choose, what common input assumptions and what historical developments should be compared to model outputs? What are the data needs on historical and prognostic (model output) data for such validation activities? The SWG could facilitate the discussion of these challenging questions among interested teams and projects.
4. Model Evaluation Against Historical Patterns: The SWG facilitates the systematic exploration of robust historical socio-economic patterns of energy and land use and economic activity that may continue to hold in the future, at least under baseline conditions. (This approach is sometimes referred to as “stylized facts”.) Such patterns may be of particular relevance for model evaluation, and the SWG supports activities that try to employ historical patterns for behavior testing of models (complementary to Historical Data Validation, above).
5. Model Diagnostics: The SWG supports the development and use of a repository of model diagnostic runs that have been developed in model comparison exercises and elsewhere and that can be used as a source of information for researchers attempting to understand model performance and develop relevant indicators.
6. Developing standards: The SWG recommends best practice for evaluation and diagnostic activities.
7. Model documentation (in cooperation with the SWG on Data Protocols and Management): The Evaluation and Diagnostics SWG in cooperation with the SWG on Data Protocols and Management, recommends standards of good practice, for model documentation, model data documentation, and model code availability.

As part of its activities, in August 2019 the SWG started surveying the members of the IAMC with regard to teams’ approaches to model validation. Given the diversity of models and modeling approaches, the approaches to model validation are expected to be similarly varied. The SWG would like to begin to get a better idea of how models approach the problem with an eye toward sharing approaches. For this purpose, IAMC Members were requested to put together a short summary in the form of a PowerPoint document with one to two slides summarizing individual teams approaches. Slides should be sent ot the IAMC Secretariat at iamc@iamconsortium.org by November 1st, 2019. The SWG intends to hold a discussion on methods and approaches at the next IAMC Annual Meeting. Responses will be used to develop a summary of approaches. However, responses will not be shared without explicit permission from the submitting modeling team. A copy of three examples, from the AIM, GCAM and REMIND modeling teams is available here.

John WEYANT, Stanford Energy Modeling Forum (EMF)

Matthew Binsted (PNNL), Shinichiro Fujimori (NIES and U Kyoto),Celine Guivarch (CIRED), Barry Hughes (University of Denver), Ilkka Keppo (UCL), Volker Krey (IIASA), Giacomo Marangoni (CMCC), Evangelos Panos (PSI),  Steven Smith (PNNL), Evelina Trutnevyte (University of Geneve), Mariesse Van Sluisveld (LTH, Lund University and PBL), Detlef van Vuuren (PBL), Taoyuan Wei (CICERO), Charlie Wilson (Tyndall Center), Benham Zakeri (IIASA).