4.1.1. ClimateGPT: a family of LLMs and an LLM-based system for climate change question-answering

Models of the ClimateGPT family have been developed and made available through a collaboration of several research institutions, including the Endowment for Climate Change Intelligence (ECI)Footnote ¹⁹ and ErasmusAI (Thulke et al., Reference Thulke, Gao, Pelser, Brune, Jalota, Fok, Ramos, van Wyk, Nasir and Goldstein2024).Footnote ²⁰ The former is a decentralized foundation that delivers AI solutions for climate change, while the latter is a platform that leverages LLMs and generative AI to “address complex global challenges, such as climate change” (Erasmus.AI, 2024). The intended use of the five LLMs pretrained and instruction-tuned on climate-relevant data is question-answering: they are to serve as “personal climate experts” that are “breaking down questions and concepts to the level of expertise of the user” (Thulke et al., Reference Thulke, Gao, Pelser, Brune, Jalota, Fok, Ramos, van Wyk, Nasir and Goldstein2024, p.12), while the intended audience includes decision-makers, scientists, policymakers, and journalists involved in climate discussions. The ClimateGPT model family comprises the models: ClimateGPT-FSG-7B, ClimateGPT-FSC-7B, ClimateGPT-70B, ClimateGPT-13B, and ClimateGPT-7B.Footnote ²¹ In addition to individual models, an LM-based system is available on the website of ECI as two versions: ClimateGPT,Footnote ²² available to researchers contingent upon their application for access through ECI’s website, and ClimateGPT+,Footnote ²³ an enterprise version aimed at businesses and organizations that would like to use the models with their own data.

Model architecture: Models of the ClimateGPT family are decoder-only Transformer models. Such models are also known as generative pretrained models (GPTs). They are trained with the objective of predicting the next word—token—in a sequence and they are mainly used for text completion tasks (Raschka, Reference Raschka2024). In the design, Thulke et al. (Reference Thulke, Gao, Pelser, Brune, Jalota, Fok, Ramos, van Wyk, Nasir and Goldstein2024) closely follow the architecture of Meta’s Llama-2 (Touvron et al., Reference Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava and Bhosale2023).Footnote ²⁴ Details about the model components, including normalization techniques for stable and efficient model learning, positional embeddings, and the activation function, are available in the model’s extensive technical report (Thulke et al., Reference Thulke, Gao, Pelser, Brune, Jalota, Fok, Ramos, van Wyk, Nasir and Goldstein2024).

Training and data: The models are pretrained with the next-token prediction objective on a large climate-relevant corpus, and are instruction fine-tuned for the downstream task of question answering. Two domain-specific pretraining methods are applied: from-scratch pretraining (FSPT) and continued pretraining (CPT). In FSPT, model training starts from randomly initiated weights, and the model relies on domain-specific data only. In CPT, training starts from a general-purpose foundation model, which already holds an existing knowledge base from having been trained on general data, and is then adapted to a specific domain using domain-relevant information.Footnote ²⁵

Pretraining is done with two corpora: a corpus of 300 billion (B) tokens curated for content on the topics of climate, humanitarian issues, and science, and a 4.2B token corpus of hand-picked climate-change data. The 300B corpus contains news, various publications, modern books, patents, Wikipedia data, policy and finance data, and science. The 4.2B corpus contains news on extreme weather events, over 500 pages of technical documentation on game-changing breakthroughs in the areas of energy, CC, food security, health etc., a breakdown of the United Nations’ (UN) 17 Sustainable Development Goals, CC news, CC specific corpora including documents published by the World Bank, IPCC, the United States Government, and other international development organizations, treaty organizations, non-governmental organizations, and national state governments, as well as academic research in climate. The 300B corpus is used for the from-scratch pretraining of ClimateGPT-FSG-7B. The 300B and the 4.2B corpus are used for the from-scratch pretraining of ClimateGPT-FSC-7B. Finally, the 4.2B corpus is used for the continued pretraining of ClimateGPT-7B, ClimateGPT-13B, and ClimateGPT-70B, which are the models built on top of Llama-2.

Instruction fine-tuning (IFT) is done with 271,525 IFT training samples (TSs); of these, 106,269 TSs are from general-domain datasets, and 165,256 TSs are from climate-specific datasets. The general-domain TS set comprises the datasets Databricks DollyFootnote ²⁶ (Conover et al., Reference Conover, Hayes, Mathur, Xie, Wan, Shah, Ghodsi, Wendell, Zaharia and Xin2023), OpenAssistant Conversations 1 (OASST-1)Footnote ²⁷ (Köpf et al., Reference Köpf, Kilcher, von Rütte, Anagnostidis, Tam, Stevens, Barhoum, Duc, Stanley, Nagyfi, ES, Suri, Glushkov, Dantuluri, Maguire, Schuhmann, Nguyen and Mattick2023), as well as FLAN v2 and CoT (as described in (Wang et al., Reference Wang, Ivison, Dasigi, Hessel, Khot, Chandu, Wadden, MacMillan, Smith, Beltagy and Hajishirzi2023), an internal set of prompt-completion pairs, referred to as AppTek General, and 9,846 TSs formulated as question-and-answer pairs from the StackExchange communities for earth science, sustainability, and economics.

The climate-specific TS set contains demonstrations,Footnote ²⁸ grounded expert demonstrations, and grounded non-expert demonstrations. The 1,332 TSs dubbed “demonstrations” are created by interviewing a small team of senior climate experts on foundational concepts in an expert’s field of expertise, current CC trends, expected developments, pivotal findings and research papers, key arguments, and scenarios in which an LLM would be of use to an expert. Another 7,254 TSs are collected from grounded expert demonstrations and 146,871 TSs from grounded non-expert demonstrations. The grounded expert demonstrations include CC questions and structured answers provided by nine climate scientists at graduate or PhD level, as well as synthetically generated question-answer pairs corrected by the same group of experts; while non-expert demonstrations contain ideas for prompt and completion pairs, where the completion contains in-text citations to the relevant sources, collected from external annotators.Footnote ²⁹

Protocols for safe text generation are observed by generating completions for each prompt in the dataset using a safe model, namely Llama-2-Chat-70B (Touvron et al., Reference Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava and Bhosale2023). Many of the responses are then manually checked, appended to the Do-Not-Answer Dataset, which is a collection of instructions that responsible models should not follow and the appropriate responses (Wang et al., Reference Wang, Li, Han, Nakov, Baldwin, Graham and Purver2024),Footnote ³⁰ and this augmented resource is then added to the IFT dataset.

Improving retrieval: Retrieval augmented generation (RAG) is used to counter model errors and overcome limitations posed by the knowledge cut-off date, that is, enable models to access documents published after their training had been completed. RAG is not part of a model’s architecture and is a component that can be added to an LLM-powered text generation system, where a user’s prompt triggers a retrieval system to query a knowledge base of documents, retrieve the ones that have the highest similarity to the query, and generate a response using the retrieved documents. Documents that are stored in the database used in RAG include IPCC reports, the Potsdam Papers, documents from the Earth4All process, and 73 other non-specified open-access documents.Footnote ³¹

Evaluation and results: Two types of evaluation are implemented: automatic and human. For automatic evaluation the authors use both domain-specific and general-purpose datasets. The climate-specific evaluation datasets include ClimaBench (Spokoyny et al., Reference Spokoyny, Laud, Corringham and Berg-Kirkpatrick2023),Footnote ³² a collection of open-source climate-related datasets allowing systematic evaluation of model performance across various classification tasks. ClimaBench includes the following datasets: ClimateStanceFootnote ³³ and ClimateEngFootnote ³⁴ (Vaid et al., Reference Vaid, Pant, Shrivastava, Louvan, Madotto and Madureira2022), Climate-FEVERFootnote ³⁵ (Diggelmann et al., Reference Diggelmann, Boyd-Graber, Bulian, Ciaramita and Leippold2020), ClimaTextFootnote ³⁶ (Varini et al., Reference Varini, Boyd-Graber, Ciaramita and Leippold2020), and CDP-QAFootnote ³⁷ (Spokoyny et al., Reference Spokoyny, Laud, Corringham and Berg-Kirkpatrick2023). In addition to the evaluation datasets of the ClimaBench collection, the authors use the datasets Pira 2.0 MCQFootnote ³⁸ (Pirozelli et al., Reference Pirozelli, José, Silveira, Nakasato, Peres, Brandão, Costa and Cozman2024) and Exeter MisinformationFootnote ³⁹ (Coan et al., Reference Coan, Boussalis, Cook and Nanko2021). The general-domain evaluation datasets include HellaSwagFootnote ⁴⁰ (Zellers et al., Reference Zellers, Holtzman, Bisk, Farhadi, Choi, Korhonen, Traum and Màrquez2019), PIQAFootnote ⁴¹ (Bisk et al., Reference Bisk, Zellers, Gao and Choi2020), OpenBookQAFootnote ⁴² (Mihaylov et al., Reference Mihaylov, Clark, Khot, Sabharwal, Riloff, Chiang, Hockenmaier and Tsujii2018), WinoGrandeFootnote ⁴³ (Sakaguchi et al., Reference Sakaguchi, Bras, Bhagavatula and Choi2021), and the MMLU datasetFootnote ⁴⁴ (Hendrycks et al., Reference Hendrycks, Burns, Basart, Zou, Mazeika, Song and Steinhardt2020).

The performance of ClimateGPT models is compared against the performance of same-size models of the Llama-2-Chat family, as well as against 7 other general-purpose foundation models in the 3-13B size range, namely: Stability-3B, Pythia-6.9B, Falcon-7B, Mistral-7B, Llama-2-7B, Jais-13B, Jais-13B-Chat.Footnote ⁴⁵ On the climate benchmarks, the three CPT models of the ClimateGPT family outperform the 7B, 13B, and 70B models of the Llama-2-Chat family, as well as the 7 other general-purpose foundation models. ClimateGPT FSPT models perform worse than the general-purpose foundation models, the same-size Llama models, and the Llama-based ClimateGPT models. On the general-domain benchmarks, the ClimateGPT CPT models outperform their respective counterparts (in terms of parameters) from the Llama-2-Chat family; however, both ClimateGPT-13B and ClimateGPT-7B lag behind Mistral-7B (Jiang et al., Reference Jiang, Sablayrolles, Mensch, Bamford, Chaplot, Casas, Bressand, Lengyel, Lample and Saulnier2023). The results of the automatic evaluation, given as weighted averages, are summarized in Table 1.

Table 1. Performance comparison of different ClimateGPT models on climate-specific and general benchmarks (weighted averages of accuracy)

The authors also conduct human evaluation, where seven climate change students at master, PhD, and post-doc level provide feedback on the output of ClimateGPT-70B, ClimateGPT-7B, and ClimateGPT-FSC-7B by ranking them against each other on a series of items, including qualifying the goodness of each answer on a scale in the range of -2 to +2, where -2 is the lowest, 0 is average, and 2 is the highest score. ClimateGPT-70B receives an average rank of 1.0 and the lowest number of hallucinationsFootnote ⁴⁶ among the models (2 instances of hallucination; the highest number is 5). ClimateGPT-70B and ClimateGPT-7B, both CPT models built on top of Llama-2, perform better than the FSPT model, ClimateGPT-FSC-7B.

Access, transparency, and engagement: The five models are published on the HFH.Footnote ⁴⁷ In addition, a QA chat system powered by ClimateGPT models can be accessed through two websites: ECI’sFootnote ⁴⁸ and Erasmus.AI’s.Footnote ⁴⁹ Both require registration. On ECI’s website, the chatbot is available in two versions: ClimateGPT and ClimateGPT+. The latter is an enterprise version aimed at businesses and organizations that would like to use the models with their own data. According to ECI, a portion of the revenue generated from ClimateGPT+ will be reinvested into ECI, thereby honouring their commitment to open-access climate AI and funding future endeavours in this field. ClimateGPT is made available in 21 other languagesFootnote ⁵⁰ using a cascaded machine translation approach.Footnote ⁵¹

Transparency in terms of listing the data used to train and evaluate the models is preserved. Not all items comprising the pretraining data, the fine-tuning data, and the instruction fine-tuning data are publicly available; this means that information about the data cannot be obtained beyond the comprehensive description provided in the paper. In terms of evaluation, the authors make available the relevant Python script that, at the time of writing, plugs in all datasets for automatic evaluation, except for Climate-FEVER.Footnote ⁵² In addition, a model card and a sustainability scorecard are provided, with information about the hardware and software used in the training process, as well as the models’ carbon footprint.

In terms of engagement, at the time of reviewing,Footnote ⁵³ ClimateGPT-7B had the highest number of all-time downloads (18151), followed by ClimateGPT-70B (3760), ClimateGPT-13B (670), ClimateGPT-7B-FSG (468), and ClimateGPT-7B-FSC (291).

4.2.1. BART-based model for summarizing climate-related political press releases

The publicly available model, whose HFH identifier is z-dickson/bart-large-cnn-climate-change-summarization, has the intended use of serving as a text summarization model that takes as input a political text in the domain of climate change, environment, or energy, and generates as output a summary of it. The model should detect the primary issue in the text and include it in the generated summary, alongside the position of the political party giving the press release, and a general summary of one to two sentences. The intended audience is not specified, but can be inferred as researchers interested in political parties’ stance on climate change policies (Dickson and Hobolt, Reference Dickson and Hobolt2024).

Model architecture, training, and data: The model is a fine-tuned version of Facebook/bart-large-cnn (Lewis et al., Reference Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov, Zettlemoyer, Jurafsky, Chai, Schluter and Tetreault2020), which has been trained on the summarization dataset CNN/Daily Mail (Nallapati et al., Reference Nallapati, Zhou, dos Santos, Gulçehre, Xiang, Riezler and Goldberg2016). The dataset is publicly availableFootnote ⁵⁴ and contains multi-sentence summaries of approximately 300,000 news articles published by CNN and the Daily Mail. The underlying model BART (which stands for Bidirectional Auto-Regressive Transformer), is an encoder-decoder Transformer model. As per Dickson (Reference Dickson2023), the model Facebook/bart-large-cnn was fine-tuned on 7,000 press release/summary pairs from 66 political parties in 12 countries.Footnote ⁵⁵ In Dickson and Hobolt (Reference Dickson and Hobolt2024), the number of press release/summary pairs is reported at 6,000. As per Dickson and Hobolt (Reference Dickson and Hobolt2024), the fine-tuning data was generated by prompting GPT-3.5 for automatic summaries. However, the associated Hugging Face Model Card specifies the use of a more recent model, GPT-4, for summary generation. The generated summaries are then qualitatively examined and slightly modified as necessary.

Evaluation and results: The model is not evaluated against a baseline and an F1 score is not reported. It is mentioned by Dickson and Hobolt (Reference Dickson and Hobolt2024) that the model output has been qualitatively checked. Dickson (Reference Dickson2023) points out that while the model is capable of identifying the primary issue in the political text, it does not include the name of the political party in the response. It identifies the author of the text as “the party” and summarizes the position as such (Dickson, Reference Dickson2023). In terms of access, transparency, and engagement, the model is publicly available; the fine-tuning dataset is not. Its total number of downloads at the time of reviewing is 7456.

4.3.1. ClimateQA

The intended audience and use of ClimateQA’s (Luccioni et al., Reference Luccioni, Baylor and Duchene2020) are analysts combing through financial reports for climate change-related risks and liabilities. The model classifies paragraphs of a report as potential answers to a pre-determined set of questions. Users of ClimateQA are not expected to have substantial technical knowledge, as the model is integrated in a web application hosted on the Microsoft Azure cloud solution; it is intended for them to interact with the model by uploading PDF files for analysis and downloading an output file, in which passages pertaining to a set of 14 questions proposed by the Task Force on Climate-Related Financial Disclosures (TCFD) are highlighted. Processing time is between 5 and 15 minutes per report.

Model architecture, training, and data: ClimateQA is based on the architecture of RoBERTa-base, a foundation model with 125M parameters.Footnote ⁵⁶ RoBERTa stands for “robustly optimized BERT approach”; it is a language model seen as an improvement of the original Bidirectional Encoder Representations from Transformers (BERT) model (Devlin et al., Reference Devlin, Chang, Lee, Toutanova, Burstein, Doran and Solorio2019), mostly in terms of being able to handle more data and use better optimization strategies during training.

The foundation model is both pretrained on unlabelled data and fine-tuned on labelled data. For the former, the authors scrape 2249 publicly available financial reports from the databases of the Electronic Data Gathering, Analysis, and Retrieval system (EDGAR)Footnote ⁵⁷ and the Global Reporting Initiative (GRI).Footnote ⁵⁸ EDGAR has been created by the US Securities and Exchange Commission to enable corporate filings by entities who are required to submit such forms under US legislation, while GRI is an international, independent organization that “helps businesses, companies, and other organizations understand and communicate their impact on issues such as climate change, human rights, and corruption” (Global Reporting Initiative, 2024). The collected reports span a timeline of 10 years and are from publicly-traded companies in the sectors: agriculture, food, and forests; energy; banks; transportation; insurance; and materials and buildings.

The classification task is grounded by the set of 14 TCFD questions. Passages seen as appropriate answers to the TCFD questions are labelled by a team of sustainability analysts. This results in a dataset of question-answer pairs (QA dataset), with the question being one of the TCFD questions, and the answer the portion of the text labelled as an answer by the human expert. Negative examples are generated by pairing the remaining sentences with the questions. The labelled dataset has a train set of 15,000 negative and 1,500 positive examples, a development set of 7,500 negative and 750 positive examples, and a test set of 1,200 negative and 400 positive examples.

Evaluation and results: The model is evaluated by comparing the difference in performance between the F1 scores ClimateQA achieves on the validation and the test QA data splits. They find that on average, the F1 score achieved on the test data split is 9.7% lower than the F1 score achieved on the validation data split. There are substantial differences in the model’s performance on data from different sectors, with the best performance being observed in the sector Energy (F1 score on validation dataset higher by 4.4% relative to that on test dataset), and the worst in the sector Materials and Buildings (24.2% validation-test F1 score difference). The type of question also affects performance, with less-frequent questions, and questions whose answers could be more diverse proving to pose the biggest challenge to the model. The difference between the validation and the test score for each question is also provided.

Access, transparency, and engagement: The model is not publicly available, and at the time of writing, it is also not accessible through the dedicated web application. The labelled dataset generated for this study is not publicly available. While an official model card is not provided, the authors do elaborate on the choice of a foundation model vis-à-vis training time and energy efficiency. It is indicated that the model RoBERTa-base is selected over RoBERTa-largeFootnote ⁵⁹ because in the training experiments, the former’s training time was less than 5 hours on a 12GB GPU, as opposed to the latter, which needs almost 12 hours. The large model showed minor improvements in the F1 scores achieved on the validation and test datasets. It is pointed out that longer training times translate into higher energy costs, which is another factor in favour of choosing a smaller model.

4.3.2. ClimateBERT: a family of LMs for climate-relevant text classification

What started as four models pretrained on CC data (Webersinke et al., Reference Webersinke, Kraus, Bingler and Leippold2022) grew into a Hugging Face repository which, at the time of writing, hosts four foundation models and nine fine-tuned classification models for tasks involving CC texts, as well as 7 fine-tuning datasets. Information about models belonging to this family is also available on a designated website.Footnote ⁶⁰ Unlike ClimateGPT, this family of models is based on a PLM and mainly focuses on text classification tasks.Footnote ⁶¹

Intended use and audience: Models of the ClimateBERT family are intended for researchers using NLP models to process CC texts, and are geared towards climate-related classification tasks on paragraphs and sentences. Models supporting paragraph-level classification have been developed to classify paragraphs as: (1) climate-related or not, (2) expressing a sentiment of opportunity, risk, or a neutral one, (3) being specific or not specific about climate-related actions and intentions, (4) being about climate commitments and actions or not, and (5) being related to climate-relevant recommendation categories in the context of financial disclosure or not. The sentence-level classification models can detect (1) whether a sentence is an environmental claim or not, (2) whether it contains content on transition and physical climate risks or not, (3) whether it is about renewable energy or not, and (4) whether it is connected to net zero and reduction targets or not.

Model architecture, training, and data: The initial ClimateBERT models, ClimateBERT _F, ClimateBERT _S, ClimateBERT _D, and ClimateBERT _D+S,Footnote ⁶² were developed by conducting domain-adaptive pretraining of the model DistilRoBERTa-base (Sanh et al., Reference Sanh, Debut, Chaumond and Wolf2020), a distilled version of the model RoBERTa (Liu et al., Reference Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov2019).Footnote ⁶³ Knowledge distillation is a method where a smaller and simpler model is trained to mimic the behaviour of a larger, more complex model. The goal is to develop a faster, more efficient model that possesses similar functionalities to the larger model.

A large corpus of texts considered representative of general- and domain-specific climate-related language, containing a total of 2,046,523 paragraphs, is compiled and used for domain-adaptive pretraining. Of the total number of paragraphs, 1,025,412 (50.1% of all paragraphs) are news articles on the topics of climate politics, climate actions, flood, and droughts retrieved from Refinitiv Workspace and climate-related news articles crawled from the web, 530,819 (25.9%) are abstracts of scientific papers published mainly between 2000 and 2019 and retrieved from the Web of Science, and 490,292 (23.9%) are texts from corporate climate and sustainability reports of over 600 companies published between 2015 and 2020, retrieved from Refinitiv Workspace and from the respective companies’ websites. The authors report the average length of paragraphs in each category: news-related paragraphs have a mean length of 56 words, abstracts 218 words, and corporate report paragraphs 65 words. As the size of the pretraining corpus is not expressed in number of word-based tokens, using the average paragraph length for each category and the number of paragraphs, I calculate an estimation of the training corpus size expressed as number of words: the news corpus would have ca. 57 million words, the abstracts corpus would have ca. 116 million words, and corporate reports would account for ca. 32 million words; in total, the domain-specific pretraining corpus size is estimated at ca. 205 million words.

The complete corpus is used in domain-adaptive pretraining of the model ClimateBERT _F. The models ClimateBERT _S, ClimateBERT _D, and ClimateBERT _D+S are trained on different portions of the corpus: ClimateBERT _S is trained on 70% of corpus samples most similar to samples of the planned text classification task, ClimateBERT _D is trained on 70% of corpus samples most diverse to the samples of the planned text classification task, and ClimateBERT _D+S is trained on 70% of samples that have the highest composite score, calculated by using the similarity and diversity metrics applied to the previous two cases and summing over the scaled values. In addition, the vocabulary of DistilRoBERTa-base is extended by adding the 235 most common tokens from the pretraining corpus to the model’s tokenizer, thereby allowing the model to learn representations of frequently occurring terms in climate-related texts, such as CO ₂ , emissions, temperature, environmental, soil etc. The four foundation models are available for download from the HFH.Footnote ⁶⁴ The models will be referred to as ClimateBERT-DAPT (domain-adaptive pretrained) models hereinafter.

Finally, the four DAPT models are fine-tuned on three tasks: (1) text classification, where hand-selected paragraphs from companies’ annual or sustainability reports are labelled as climate-related or not, (2) sentiment analysis, where climate-related paragraphs are labelled as expressing a negative sentiment of risk, positive sentiment of opportunity, or a neutral sentiment (sentiment analysis), and (3) fact-checking, for which the Climate-FEVER dataset is used (Diggelmann et al., Reference Diggelmann, Boyd-Graber, Bulian, Ciaramita and Leippold2020).

Later works make use of, update, or create new fine-tuned versions of ClimateBERT _F for paragraph-level classification tasks (Bingler et al., Reference Bingler, Kraus, Leippold and Webersinke2022b, Reference Bingler, Kraus, Leippold and Webersinke2024) and sentence-level classification tasks (Stammbach et al., Reference Stammbach, Webersinke, Bingler, Kraus and Leippold2022; Deng et al., Reference Deng, Leippold, Wagner and Wang2023; Schimanski et al., Reference Schimanski, Bingler, Kraus, Hyslop, Leippold, Bouamor, Pino and Bali2023a), resulting in five and four new task-specific models, respectively. The tasks, models, and the datasets used to train the model on paragraph- and sentence-level classification tasks are given below. All datasets have a train and a test split, and some have a development split, too. Train and development splits are used during model training and optimization, and the test split is used for model evaluation.

Paragraph-level text classification. Details about the dataset creation and annotation procedure for the training and testing data mentioned in items 1 to 5 are available in Webersinke et al. (Reference Webersinke, Kraus, Bingler and Leippold2022) (1 and 2), Bingler et al. (Reference Bingler, Kraus, Leippold and Webersinke2024) (1, 2, 3, and 4), and Bingler et al. (Reference Bingler, Kraus, Leippold and Webersinke2022b, Reference Bingler, Kraus, Leippold and Webersinke2024) (5). At the time of writing, all models and datasets for paragraph-level text classification are available for download.

1. 1. Classification of paragraphs as climate-related or not: climatebert/distilroberta-base-climate-detector. The dataset consists of hand-selected paragraphs from companies’ annual or sustainability reports and annotated with yes if they relate to climate, or no if they do not.Footnote ⁶⁵

2. 2. Classification of climate-related paragraphs into the classes OPPORTUNITY, NEUTRAL, or RISK: climatebert/distilroberta-base-climate-sentiment. The sentiment analysis dataset was developed by annotating paragraphs annotated with yes in dataset (1) as expressing a neutral sentiment, a sentiment of opportunity or a sentiment of risk. Footnote ⁶⁶

3. 3. Classification of paragraphs as either SPECIFIC or NON-SPECIFIC. A paragraph is specific if it contains details about climate-related performance, action, or tangible and verifiable targets, and non-specific if these features are missing: climatebert/distilroberta-base-climate-specificity. The manually annotated paragraphs have been extracted from corporate reports.Footnote ⁶⁷

4. 4. Classification of paragraphs as being or not being about climate commitments and actions: climatebert/distilroberta-base-climate-commitment. The paragraphs are extracted from companies’ annual reports.Footnote ⁶⁸

5. 5. Classification of climate-related paragraphs into four CC-related categories as defined by the Task Force on Climate-Related Financial Disclosures (TCFD),Footnote ⁶⁹ namely: governance, strategy, risk management, and metrics and targets: climatebert/distilroberta-base-climate-tcfd. The dataset consists of paragraphs extracted from reports issued by companies supporting TCFD reporting guidelines and annotated as related to one of the four categories or not.Footnote ⁷⁰

Sentence-level text classification. Details about the annotation and dataset design are available in Stammbach et al. (Reference Stammbach, Webersinke, Bingler, Kraus and Leippold2022) (item 1), Deng et al. (Reference Deng, Leippold, Wagner and Wang2023) (2 and 3), and Schimanski et al. (Reference Schimanski, Bingler, Kraus, Hyslop, Leippold, Bouamor, Pino and Bali2023a) (4). The datasets of items 1 and 4 are publicly available at the time of writing.

1. 1. Classification of sentences as environmental claims or not. An environmental claim refers to the practice of suggesting or creating an impression that a service or a product is either environmentally friendly or not as damaging to the environment as competing goods:Footnote ⁷¹ climatebert/environmental-claims. The dataset contains sentences extracted from companies’ sustainability reports, earning calls, and annual reports.Footnote ⁷²

2. 2. Classification of sentences as being related to transition or to physical climate risks or not: climatebert/transition-physical. The dataset contains sentences extracted from earnings conference call transcripts and manually annotated as related to the topics of transition and physical climate exposure or not.

3. 3. Classification of sentences as being related to renewable energy or not: climatebert/renewable. The dataset contains the sentences of dataset (2) marked as related to transition, which are further annotated as relating to renewable energy or not.

4. 4. Classification of sentences as either being connected to emission net-zero or reduction targets or not: climatebert/netzero-reduction. The sentences have been collected and annotated in collaboration with the Net Zero Tracker project (Lang et al., Reference Lang, Hyslop, Yeo, Black, Hale, Chalkley, Hans, Hay, Höhne, Hsu, Kuramochi, Mooldijk and Smith2023), which collects data from companies and governments and attempts to measure how serious they are about cutting their net emissions to zero. Sentences from previous climate-related projects are also added.Footnote ⁷³

Models of the ClimateBERT-DAPT family have been fine-tuned in other contexts - for example, Garrido-Merchán et al. (Reference Garrido-Merchán, González-Barthe and Vaca2023) fine-tune a model of this family to predict whether a sentence is climate-related or not using the dataset ClimaText (see Section 4.1.1 for dataset details). While the authors do show that a ClimateBERT-DAPT model that has been additionally fine-tuned on climate-relevant sentences outperforms the baseline ClimateBERT-DAPT model, it is not clear which model of the ClimateBERT-DAPT group has been fine-tuned and the resulting fine-tuned model has not been published.

Evaluation and results: The foundation models ClimateBERT _F, ClimateBERT _S, ClimateBERT _D, and ClimateBERT _D+S are compared against DistilRoBERTa on the downstream tasks of (1) text classification (classify paragraphs as climate-related or not), (2) sentiment analysis (classify climate-related paragraphs as expressing risk, opportunity, or being neutral), and (3) fact-checking. All models outperform the baseline in terms of F1 score, with the highest F1 scores achieved by ClimateBERT _S in task (1), ClimateBERT _F in task (2), and ClimateBERT _D+S in task (3). The results are originally reported in Webersinke et al. (Reference Webersinke, Kraus, Bingler and Leippold2022) and aggregated in Table 2.

Table 2. ClimateBERT baselines and performance (loss / F1). Validation means that loss has been calculated for the validation dataset

The performance on paragraph-level classification tasks is reported in the context of ClimateBERT _CTI, an NLP methodology for calculating a “cheap talk index”. The index aims to measure the level of superficiality in companies’ climate commitments (Bingler et al., Reference Bingler, Kraus, Leippold and Webersinke2024). There are four baseline scores obtained from four machine learning and deep learning models: a Least Absolute Shrinkage and Selection Operator (LASSO), a Naïve Bayes classifier, a Support Vector Machine (SVM) with Bag-of-Words (BoW), and an SVM with Embeddings from Language Model (ELMo). Based on the reported F1 scores, ClimateBERT _CTI outperforms all baseline models on the four tasks. Detailed evaluation information is available in Table 3 and is based on the results reported in Bingler et al. (Reference Bingler, Kraus, Leippold and Webersinke2022a, Reference Bingler, Kraus, Leippold and Webersinke2024).

Table 3. Paragraph classification tasks: baseline models and F1 scores versus ClimateBERT _F

Note: Details on the baseline models’ training can be found in Bingler et al. (Reference Bingler, Kraus, Leippold and Webersinke2022a, Reference Bingler, Kraus, Leippold and Webersinke2024).

^a LASSO stands for Least Absolute Shrinkage and Selection Operator, a type of machine learning model.

In terms of the fine-tuned models for sentence-level classification, climatebert/environmental-claims is compared against SVM-based models and three Transformer models. On the train dataset, climatebert/environmental-claims is outperformed by both RoBERTa-base and RoBERTa-large; on the development dataset, it either outperforms or is on par with other Transformer-based models, and on the test dataset, it is outperformed only by RoBERTa-large. For climatebert/renewable and climatebert/transition-physical, Deng et al. (Reference Deng, Leippold, Wagner and Wang2023) report an F1 score of 0.96 and 0.97, respectively. Finally, Schimanski et al. (Reference Schimanski, Bingler, Kraus, Hyslop, Leippold, Bouamor, Pino and Bali2023a) report an F1 score of 0.962 for climatebert/netzero-reduction, which is higher than the F1 scores of DistilRoBERTa and GPT-3.5-turbo, and very close to the score of RoBERTa-base, 0.958. These results are summarized in Table 4.

Table 4. Sentence classification tasks: baseline models and F1 scores versus fine-tuned ClimateBERT _F, evaluated on the test data split

Note: There were many baseline models as points of comparison for climatebert/environmental-claims; this table only includes those with comparable performance. For more details see Stammbach et al. (Reference Stammbach, Webersinke, Bingler, Kraus and Leippold2022).

Access, transparency, and engagement: The four ClimateBERT-DAPT and the nine fine-tuned models, as well as most of the datasets used in fine-tuning the models on downstream tasks, are available on HFH at the time of writing.Footnote ⁷⁴ Transparency in terms of data collection, usage, and evaluation approach is also preserved. Webersinke et al. (Reference Webersinke, Kraus, Bingler and Leippold2022) include information about the carbon footprint of developing the ClimateBERT-DAPT models, accompanied by a climate performance model card.

In terms of engagement, of the ClimateBERT-DAPT models, ClimateBERT _F has the highest number of downloads (over 200 thousand); of the models for paragraph-level text classification, the model climatebert/distilroberta-base-climate-detector has the highest number of downloads (nearly 1.4 million), while climatebert/environmental-claims is the most popular LM among the sentence-level classification models (over 58 thousand downloads). Fine-tuned models for paragraph classification seem to be more popular than sentence classification models. Table 5 gives information about the all-time downloads of all models of the ClimateBERT family.

Table 5. Number of all-time downloads for 13 models of the ClimateBERT family, in descending order for each LM type

Note: The count of downloads was retrieved in August 2025.

4.3.3. DARE

The BERT-based distillation and reinforcement ensemble model (DARE) designed by Xiang and Fujii (Reference Xiang and Fujii2023) should show how a lighter, more efficient version of purely BERT-based models can be developed for text classification tasks in a low-data setting. The model’s intended audience and use are stakeholders interested in using a language model to analyse ambiguities of CC-related information for sentiment analysis (risk, opportunity, neutral) and fact-checking, and do so with limited access to compute power.

Model architecture, training, and data: In their choice of architecture, the DARE developers attempt to address two concerns in model engineering and usage: (1) training and deployment of large language models are resource-intensive in terms of compute power, and (2) developing fine-tuned models is a data-intensive process. Concern (1) is addressed by combining knowledge distillation and domain adaptation. As explained in Section 4.3.2, knowledge distillation is a method where a smaller and simpler model is trained to mimic the behavior of a larger, more complex model. For DARE, the “teacher” model is an encoder-only 12-layer BERTbase model, while the “student” model is a Bi-LSTM-Attention model.Footnote ⁷⁵ Domain adaptation is the use of domain-specific text to adapt an existing model to a new target domain. To address domain-specific data scarcity, Xiang and Fujii (Reference Xiang and Fujii2023) propose a new data augmentation strategy, where a component titled Generator-Reinforcer Selector collaboration network is used to replace nouns, verbs, adjectives, and adverbs in a sentence with suitable candidates. The Generator proposes sentences with replaced words, while the Reinforced Selector chooses samples that truly augment the data, rather than add noise to it.

The data used for domain-adaptive pretraining comprises scientific literature related to climate change and health (Berrang-Ford et al., Reference Berrang-Ford, Sietsma, Callaghan, Minx, Scheelbeek, Haddaway, Haines and Dangour2021), published between 1 January 2013 and 9 April 2020, obtained from the Web of Science Core Collection and Scopus. While Xiang and Fujii (Reference Xiang and Fujii2023) do not provide a pointer to the data, it seems that part of it is publicly available as supplementary materials to work done by Berrang-Ford et al. (Reference Berrang-Ford, Sietsma, Callaghan, Minx, Scheelbeek, Haddaway, Haines and Dangour2021).Footnote ⁷⁶ For sentiment analysis, the authors create a dataset of 1220 hand-selected paragraphs extracted from the Web of Science records and another 1000 paragraphs from Scopus records, and, similarly to the sentiment analysis dataset used in fine-tuning climatebert/distilroberta-base-climate-sentiment, annotate the paragraphs as OPPORTUNITY (positive sentiment), RISK (negative sentiment), and NEUTRAL, using the annotation software Prodigy.Footnote ⁷⁷ Finally, these datasets are augmented with the Generator-Reinforcer Selector data augmentation strategy described above, which doubles the number of records from 40,671 to 80,750. The latter is split in a train set of 60,560 records and a development set of 20,190 records. The authors do not clarify what the term “records” refers to and what the average number of tokens per record is; therefore, it is not possible to estimate the number of tokens in the training corpus. Neither the sentiment-annotated data nor the augmented data are publicly available.

Evaluation and results: Evaluation is conducted on two downstream tasks: sentiment analysis, using the sentiment analysis dataset created as part of the study, and fact-checking, using the Climate-FEVER dataset. The model’s performance is compared against eight other models: BERTbase and domain-pretrained BERTbase (Devlin et al., Reference Devlin, Chang, Lee, Toutanova, Burstein, Doran and Solorio2019), TinyBERT (Jiao et al., Reference Jiao, Yin, Shang, Jiang, Chen, Li, Wang, Liu, Cohn, He and Liu2020), DistilBERT (Sanh et al., Reference Sanh, Debut, Chaumond and Wolf2020), ClimateBERT (Webersinke et al., Reference Webersinke, Kraus, Bingler and Leippold2022), a coordinated Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) attention + Max Pooling model (Zhang et al., Reference Zhang, Zheng, Jiang, Huang and Chen2019), a Bi-LSTM-Attention model, and a POS-Bi-LSTM-Attention model. The latter two incorporate the same type of model as the student model used in the knowledge distillation process described above (a Bi-LSTM-Attention model); in the latter one, POS stands for part-of-speech, as the authors incorporate POS vectors, which are believed to strengthen the sentiment connection and features in the sentiment analysis task.

On the sentiment analysis task, DARE’s accuracy and F1 score are lower than those of BERTbase, the domain-pretrained BERTbase model, and DistilBERT. DARE comes second best in the fact-checking task, being outperformed by BERTbase. The authors point out that although DARE is outperformed by BERTbase and DistilBERT, the model still performs comparatively well, with a 50.65x and 12.66x inference speed-up respectively, thereby offering substantial acceleration. Finally, the authors conduct an ablation study and show that all building blocks of the DARE model positively contribute to its performance. Table 6 provides an overview of DARE’s performance relative to other language models.

Table 6. Performance comparison of different models on sentiment analysis (F1) and fact-checking (macro F1)

Note: The type of F1 score for the sentiment analysis task is not specified. This is a simplified representation of the results; for a detailed overview of various experimental setups, see Xiang and Fujii (Reference Xiang and Fujii2023).

Access, transparency, and engagement: The model is not available either as a public or a proprietary model. The sentiment analysis dataset and the augmented data have not been published. The authors give a detailed report on the model parameter settings and mention that all experiments are conducted on a single Nvidia 16 GB V100 GPU. The speed-up times are also recorded, which is of relevance given that one of the study’s central goals is to provide a light-weight model.

4.4.1. ClimateGPT-2

The model’s intended audience is policymakers, researchers, and climate activists, and its intended use is to help them analyse large and complex climate-change-related collections of texts, by making use of the model’s two main functionalities: claim (text) generation and fact-checking.

Model architecture, training, and data: Vaghefi et al. (Reference Vaghefi, Muccione, Huggel, Khashehchi and Leippold2022) initialize this model with weights from GPT-2 (Radford et al., Reference Radford, Wu, Child, Luan, Amodei and Sutskever2019). GPT-2 belongs to a family of generative pretrained Transformer-based decoder-only models trained on general-domain corpora. The family contains models with 117M, 355M, and 1.5B parameters. While Vaghefi et al. (Reference Vaghefi, Muccione, Huggel, Khashehchi and Leippold2022) mention that GPT-2 comes in different sizes, the size of the GPT-2 model that has been used in the study is not specified.

The dataset for domain-adaptive pretraining consists of 360,233 abstracts from papers on climate change published by well-known scientists in the CC domain, whose names have been retrieved from a list of CC scientists curated by Reuters.Footnote ⁷⁸ Once domain-adaptive pretraining is completed, the model is fine-tuned on two tasks: (1) fact-checking, using the Climate-FEVER dataset (Diggelmann et al., Reference Diggelmann, Boyd-Graber, Bulian, Ciaramita and Leippold2020) (see Section 4.1 for dataset information) and (2) text generation, where the model is tasked with generating CC texts prompted with (a) a title only and (b) a title accompanied by a list of keywords.

Evaluation and results: The model’s performance is evaluated against that of GPT-2. In the fact-checking task, where the model should correctly “decide” if an evidence sentence supports or refutes a claim, an F1 score of 0.72 is reported, which is higher than GPT-2’s baseline score of 0.67. For text generation, Vaghefi et al. (Reference Vaghefi, Muccione, Huggel, Khashehchi and Leippold2022) report improved performance, reflected in a lower validation loss. The authors also report that the model generates semantically coherent sentences and that the initial three sentences of the model’s output are related to the title and/or the keywords.

Access, transparency, and engagement: The model is available in a public GitHub repository.Footnote ⁷⁹ The collection of abstracts used to conduct the domain-specific pretraining is not available. In terms of model evaluation, the authors do not elaborate what metrics for semantic coherence and relatendess of the generated sentences to the keywords and title are being applied. The authors mention the GPU infrastructure and time required for model training, but do not provide a model scorecard.

5.1.1. ChatClimate

Built on top of GPT-3.5 Turbo and GPT-4, ChatClimate is intended to answer climate questions by drawing on factual information (Vaghefi et al., Reference Vaghefi, Stammbach, Muccione, Bingler, Ni, Kraus, Allen, Colesanti-Senni, Wekhof and Schimanski2023). The system’s intended audience includes decision-makers and anyone interested in obtaining trustworthy information on climate change.

System architecture and data: The decoder-only models GPT-3.5 Turbo and GPT-4, which power the application, remain unchanged. To improve the quality of the generated answers, the authors rely on (1) external long-term memory and (2) prompts. External long-term memory is built by parsing PDF files of IPCC reports of the sixth assessment cycle into JSON files.Footnote ⁸⁰ The extracted text is then chunked into smaller sizes using LangChain,Footnote ⁸¹ a framework for building LLM-based applications, and embedded with text-embedding-ada-002, an embedding model developed by OpenAI.Footnote ⁸² The vectors are then stored into a vector database from which they can be retrieved.

Using prompts, the QA tool is guided whether to make use of: (1) GPT-4 only, (2) IPCC AR6 reports only, an approach dubbed ChatClimate, which is configured to exclude the LLM’s in-house knowledge, and (3) IPCC AR6 reports and in-house GPT-4 knowledge, an approach dubbed Hybrid ChatClimate.

Evaluation and results: A set of 13 questions with five levels of difficulty is used to evaluate the system. The difficulty levels and the number of questions per level are: very low (level 1)/1 question, low (2)/1 question, medium (3)/4 questions, high (4)/4 questions, and very high (5)/3 questions.Footnote ⁸³ The system’s answers are evaluated by experts, who give a score of 1 to 5 to the answers generated by each augmented retrieval method, with 1 being the lowest and 5 the highest score. It is observed that ChatClimate, which is instructed not to take into account the in-house knowledge of the model, tends to hallucinateFootnote ⁸⁴ less than GPT-4 and Hybrid ChatClimate. Each system version’s answer to questions Q3 to Q13 is available in the supplementary materials to the paper.Footnote ⁸⁵ To facilitate comparability between the three models, I summarized the contents of the evaluation report in Table 8. The authors acknowledge that a more comprehensive evaluation is needed to evaluate the system’s responses, alongside a fact-checking step.

Table 8. Average response accuracy score for each system on a total of 11 questions (responses to questions 1 and 2 are not available)

Access and transparency: ChatClimate is available as a web service,Footnote ⁸⁶ where users can choose between GPT-3.5 Turbo and GPT-4, used in two modes: stand-alone and hybrid. In terms of transparency, at the time of writing, there is no official disclosure about the data that powered the training of OpenAI’s GPT-3.5 Turbo and GPT-4 models. The documents used to curb the models’ hallucination and to overcome the limitation of outdated data are publicly available as PDF files; the parsed data that has been used for system development is not accessible. The code to reproduce the ChatClimate system is published on GitHub.Footnote ⁸⁷

5.1.2. LLM CC agent: a prototype

LLM agents are systems that are built on top of an LLM, which acts as an “agent” performing various tasks, such as data analysis or web browsing. Kraus et al. (Reference Kraus, Bingler, Leippold, Schimanski, Senni, Stammbach, Vaghefi and Webersinke2023) present a prototype system whose intended use is to serve as a question-answering system that has access to recent and precise information on the topic of climate change, and the intended audience is perceived to be organizations, institutions, and companies interested in obtaining such information.

System architecture and data: The backbone of the LLM agent is OpenAI’s model text-davinci-003, which is a decoder-only model. The system is built in the LangChain environment and utilizes a ReAct framework (Yao et al., Reference Yao, Zhao, Yu, Du, Shafran, Narasimhan and Cao2022).Footnote ⁸⁸ The resulting LLM agent should be able to interact with structured data, such as data stored in tables (pandas dataframes in Python), and, if necessary, conduct a simple Google search and retrieve additional information.

The system is guided with the help of a prompt to (1) read the question asked by the user, (2) think about what it should do, (3) describe what type of action the LLM agent should undertake, (4) specify the input to the action, and (5) present the result of the action. The five sections in the prompt are called: Question, Thought, Action, Action Input, and Observation. Finally, the system is requested to perform a Google search only if it cannot find relevant information in the data from Climate Watch, an online platform aggregating various climate-change-related datasets.Footnote ⁸⁹

Evaluation and results: The LLM agent is asked two questions (Q1 & Q2): Q1, What is the average emission of Italy between 2010 and 2015?, whose answer requires the use of one data source, and Q2, Which European country has the most ambitious net zero plans? How did the emissions of this country develop over the last 10 years? Remember only to include single countries., whose answer requires the use of combined data sources. The LLM agent is reported to have successfully retrieved the relevant data and calculated the emissions of CO₂ in Italy between 2010 and 2015; for Q2, the LLM agent is reported to have successfully retrieved the additional data by performing a Google search. No additional evaluation steps are reported.

Access and transparency: The LLM that serves as a backbone to the tool is OpenAI’s text-davinci-003, which at the moment of writing is a deprecated model. Data used as a source for the system to generate an answer to the user question is provided by Climate Watch and is described as publicly available. It is not clear whether the tool that allows the system to browse through the data obtained from Climate Watch is open-source. The other library used to augment the system’s functionalities, Google Serper,Footnote ⁹⁰ is a proprietary API which offers new users a limited number of free queries. The prototype presented in the paper is not publicly available. The authors express awareness of the carbon footprint of LLMs, both in the context of training and inference, and point out that further research must ensure that the benefits of using LLMs are not outweighed by the environmental cost.

5.1.3. My Climate Advisor

Nguyen et al. (Reference Nguyen, Karimi, Hallgren, Harkin, Prakash, Stammbach, Ni, Schimanski, Dutia, Singh, Bingler, Christiaen, Kushwaha, Muccione, Vaghefi and Leippold2024) develop a RAG-based question-answering system whose intended use and audience is to assist farmers and farm advisors gain access to information from scientific papers, grey literature, and climate projection data. The focus is on Australia as a geographical region, and the goal is to help farmers improve their resilience to the effects of climate change.

System architecture and data: The RAG database is built by drawing on three sources of information. The first addresses general-purpose agriculture questions. It is a corpus of 1.36 million articles obtained from the Semantic Scholar Open Research Corpus (S2ORC) using the labels “Agricultural and Food Sciences” and “Environmental Science”. The second source addresses climate adaptation issues, and the dataset for it is a 126,000-article corpus obtained from the top 100 highest-impact agriculture journals and from the scientific publisher Elsevier. Finally, the third source targets climate issues specific to Australia and relies on an expert-curated corpus of climate risk information, books, and industry reports, with a total of 28 documents. The corpus size is reported in gigabytes, number of documents, and number of chunks, where each chunk is approximately 400 tokens long. As per this information, the corpus might have approximately 12.3B tokens.Footnote ⁹¹ The RAG database is used with the model Llama-3-8B as a backbone.

Evaluation and results: The output of the system is compared against 11 models across seven criteria: context, structure, language, specificity, comprehensiveness, accuracy, and citation. The models against which the system is compared are: GPT-4 Turbo, Llama-3-70B, Claude 3 Opus, Gemini 1.5 Pro, Llama-3-8B, Claude 3 Haiku, Mistral-7B, Gemini 1.0 Pro, GPT-3.5 Turbo, Llama-3-70B+RAG, and Mistral-7B+RAG. The evaluation is performed by two human experts, a climate scientist and an agronomist, who score the systems’ answers to a set of 15 questions about the Australian climate change impacts and adaptation. The questions had been developed in consultation with climate risk and adaptation experts. The scores range between 0 and 4, and an average score is calculated from the scores on the seven criteria. The results reveal that GPT-4 Turbo has the highest score on 6 of 8 categories, and that Llama-3-8B + RAG scores higher only on the criterion citation. However, the inter-annotator agreement is rather low, and preference for GPT-4 Turbo and Llama-3-70B is observed.

Access and transparency: The dataset used for the database is not publicly available at the time of writing. The developed system is also not publicly available at the time of writing, but the authors promise to publish it in the future.

5.1.4. Climate policy radar: responsible question-answering

Juhasz et al. (Reference Juhasz, Dutia, Franks, Delahunty, Mills and Pim2024) present a RAG-supported system, ultimately to be hosted by Climate Policy Radar (CPR), whose intended use is to improve accessibility to information contained in documents on climate law and policy. The intended audience includes policymakers, analysts, academics, and any professionals who read climate policy and law documents as part of their job.

System architecture and data: The authors emphasize the importance of evaluation and user experience (UX) considerations in the system design. The system developed in this study has four components, of which the first and fourth components serve as guardrails against malicious content entered either by the user or generated by the LM. Sandwiched between them are an information retrieval component and an answer synthesis component, both of which are thoroughly evaluated. Models used as a backbone in the generation experiment include Llama-3.1-70B Instruct, Llama-3.1-8B Instruct, Gemini 1.5 Flash, and ClimateGPT-7B.

The dataset comprises a sample of 550 documents sourced from CPR’s database of national laws and policies and from submissions from the United Nations Framework Convention on Climate Change (UNFCCC). The sample of texts is distributed equally across World Bank Regions. For the evaluation portion that focuses on adherence to CPR’s policy standards for generation of content, the sample is complemented with documents published by the International Energy Agency (IEA), the International Atomic Energy Agency (IAEA), the Organization for Security and Co-operation in Europe (OSCE), and the World Meteorological Organization (WMO).

Evaluation and results: This study focuses on creating an evaluation pipeline that integrates human annotations and LLMs-as-a-judge. The comprehensive evaluation is done along two main tracks: (1) retrieval of the most relevant passages from the source documents, and (2) generation of answers that meet a set of criteria, which include: alignment with the CPR generation policy, faithfulness,Footnote ⁹² formatting, and system-response.

For the evaluation under (1), Juhasz et al. (Reference Juhasz, Dutia, Franks, Delahunty, Mills and Pim2024) have human annotators mark passages as relevant or not to 194 synthetic questions and use the annotations to deploy an automated solution using LLM-as-a-judge with GPT-4o. The human evaluators pinpoint several problems when assessing retrieved passages as relevant, namely: a passage that “signalled” the possibility of a relevant passage nearby would be marked as relevant; imprecise language in the passage makes it difficult to assess its level of usefulness; and document metadata is at times necessary to respond to a query. A separate human-annotated dataset is created to measure the degree to which the models can generate an answer in line with CPR’s generation policy. To this end, 16 domain experts from several national governments and international organizations, including the United Nations (UN), the International Renewable Energy Agency (IRENA) and WMO, participate in a 3-week annotation sprint to annotate generated data related to 800 documents.

The system’s generation is also assessed across three prompt templates: a basic task explanation, a prompt steering the system towards an “educative” response, and a Chain-of-Thought (CoT) prompt. These are populated with non-adversarial queries, which are sourced from user interviews, and adversarial queries, whose purpose is to “nudge” the system towards generating an answer that violates the guardrails or the prompt instructions.

The aggregated results across the prompt types and evaluation levels show that a basic prompt seems to work best for faithfulness and adherence to CPR policy, while a prompt for an educative response results in the system observing the formatting requirements. The models seem to successfully identify adversarial queries, as 6.4% to 15% of the no-response cases are related to this type of query. It is found that violations of the CPR generation policy are concerning, especially given that the end-use of such systems is a user-facing scenario. In addition, during the annotation sprint, it is found that policy violations correlate with violations of faithfulness and that violations of formatting, which include missing or non-existing citations, coincide with hallucinations.

In terms of access and transparency, the prompts, the methodology, and the evaluation datasets are publicly available.Footnote ⁹³

5.1.5. ChatNetZero

Hsu et al. (Reference Hsu, Laney, Zhang, Manya, Farczadi, Stammbach, Ni, Schimanski, Dutia, Singh, Bingler, Christiaen, Kushwaha, Muccione, Vaghefi and Leippold2024) acknowledge that LLM-powered chatbots, such as Google’s Gemini (Team et al., Reference Team, Anil, Borgeaud, Alayrac, Yu, Soricut, Schalkwyk, Dai, Hauth and Millican2023) or OpenAI’s ChatGPT (OpenAI, 2022), have become the first point of contact for many when conducting initial research on a topic. While the intended audience of ChatNetZero is not narrowly defined, the intended use of the system is to serve as a question-answering platform for climate policy-specific information, with a special focus on net-zero texts. The authors collaborate with experts on two occasions: (1) to develop the dataset that serves as a basis for the RAG component, and (2) to evaluate ChatNetZero’s output.

System architecture and data: The database is built from the following documents: a report titled “Integrity Matters: Net Zero Commitments by Businesses, Financial Institutions, Cities and Regions” by the United Nations High-Level Expert Group (HLEG) (HLEG, 2022), the Net Zero Tracker databaseFootnote ⁹⁴ and the Net Zero Stocktake reports (Net Zero Tracker, 2022, 2023), and the Corporate Climate Responsibility Monitor Reports published by the NewClimate InstituteFootnote ⁹⁵ (New Climate Institute, 2022, 2023). In addition to the database, ChatNetZero has a module for query processing, as well as anti-hallucination, reference, and enhanced analytical capabilities modules. Query processing involves a pre-processing step for identification of all actors Footnote ⁹⁶ in the query, and ensuring that the retrieved passages from the database refer to the identified actor. The anti-hallucination module is used to post-process the LLM+RAG output and verify that each generated sentence can be traced to the original passage from the dataset.Footnote ⁹⁷ The reference module adds a reference based on the text passage’s ID. Finally, the enhanced analytical capabilities module restructures tabular source data into natural sentences for easier retrieval. The authors also include the system prompt sent to GPT-4 Turbo, which seems to be the backbone LLM.

Evaluation is performed (1) by checking the factuality of generated answers and (2) by having ten climate scientists and policy experts score answers to 12 questions on a scale of 1 to 5 across three dimensions: quality, factual accuracy, and relevance. To be considered factually accurate in the sense of (1), the system’s response would have to contain the exact factual information from the reference material, including figures. On (1), the system outperforms ChatClimate (Vaghefi et al., Reference Vaghefi, Stammbach, Muccione, Bingler, Ni, Kraus, Allen, Colesanti-Senni, Wekhof and Schimanski2023), GPT-4 Turbo, Gemini 1.0 Ultra, and Coral with Web Search (Cohere, 2023). However, in the evaluation by experts, ChatNetZero receives the lowest ranking among the LLMs, which is thought to be a consequence of its answers being shorter in length compared to those of the other models.

Access and transparency: The system is available online at the time of writing.Footnote ⁹⁸ The generated answers that were used to conduct the evaluation are also available online.Footnote ⁹⁹

5.2.1. CHATREPORT

Ni et al. (Reference Ni, Bingler, Colesanti-Senni, Kraus, Gostlow, Schimanski, Stammbach, Ashraf Vaghefi, Wang, Webersinke, Wekhof, Yu, Leippold, Feng and Lefever2023) develop a system whose intended use is assisting the analysis of sustainability reports by calculating (1) a report’s conformity score (on a scale from 0 to 100) to the reporting guidelines developed by the Task Force on Climate-Related Financial Disclosures (TCFD) and (2) an option for user-defined analysis with question-answering. Its intended audience includes policymakers, investors, and the general public.

Ni et al. (Reference Ni, Bingler, Colesanti-Senni, Kraus, Gostlow, Schimanski, Stammbach, Ashraf Vaghefi, Wang, Webersinke, Wekhof, Yu, Leippold, Feng and Lefever2023) use OpenAI’s model text-embedding-ada-002 to retrieve text embeddings, and ChatGPT and GPT-4 as backbone models for summarization and question-answering tasks. CHATREPORT’s architecture includes the following elements: chunking and embedding a target report, generating a summary of the target report grounded in TCFD questions, calculating a TCFD adherence score, and a question-answering module. To combat hallucinations, the authors make the model’s answer traceable (similarly to Thulke et al. (Reference Thulke, Gao, Pelser, Brune, Jalota, Fok, Ramos, van Wyk, Nasir and Goldstein2024) and Hsu et al. (Reference Hsu, Laney, Zhang, Manya, Farczadi, Stammbach, Ni, Schimanski, Dutia, Singh, Bingler, Christiaen, Kushwaha, Muccione, Vaghefi and Leippold2024)) by assigning numbers to the source texts. Domain experts are utilized in an iterative process to craft prompts for summarization and question-answering: when a model generates an output based on a prompt, an expert provides feedback on the output. This feedback is then integrated into the prompt.

The system’s success in retrieving the correct text passage for an answer and not hallucinating in the generated text is evaluated by (1) sampling 10 sustainability reports with 110 question-answer pairs, and (2) having two different annotators label the system’s answers as containing hallucinations or not. Hallucination is defined in terms of content, where all generated content needs to be traceable to the source data, and source, where the model needs to retrieve the correct references (dubbed “honesty”).Footnote ¹⁰⁰ ChatGPT outperforms GPT-4 on this task with an “honesty” rate of 86.63%, as opposed to 51.5%. However, the inter-annotator agreement, measured as Cohen’s Kappa score, between the two annotators on this task is 0.54 for ChatGPT and 0.21 for GPT-4, indicating that identifying hallucination is not an easy feat. The authors believe that the higher inter-annotator agreement for ChatGPT might indicate that it is easier to recognize this phenomenon in ChatGPT’s answers.

In terms of access and transparency, the annotated data and code are published on GitHub,Footnote ¹⁰¹ while the system itself is available in a web interface. The usage of the system is also explained in a video available on YouTube.Footnote ¹⁰² The collection of corporate sustainability reports has not been published, and there is no information about the total cost of the API calls for this experiment.