2.1 Issues in NLP datasets

Recent works have shown that NLP datasets have a number of quality problems, for example, label mistakes (Wang et al. Reference Wang, Pruksachatkun, Nangia, Singh, Michael, Hill and Bowman2019), entity missingFootnote a (Tejaswin et al. Reference Tejaswin, Naik and Liu2021), and unwanted biases resulting from the annotation process (Kaushik and Lipton Reference Kaushik and Lipton2018; Nadeem et al. Reference Nadeem, Bethke and Reddy2021). For instance, Wang et al. (Reference Wang, Pruksachatkun, Nangia, Singh, Michael, Hill and Bowman2019) identified a notable 5.38 percent rate of label mistakes in the CoNLL03 NER dataset, a concerning figure for a widely used benchmark in NLP research. Tejaswin et al. (Reference Tejaswin, Naik and Liu2021) manually checked 600 randomly selected instances from three sources: CNN/DailyMail (Hermann et al. Reference Hermann, Kocisky, Grefenstette, Espeholt, Kay, Suleyman and Blunsom2015; Nallapati et al. Reference Nallapati, Zhou, dos Santos, Gulçehre and Xiang2016), Gigaword (Rush et al. Reference Rush, Chopra and Weston2015), and XSum (Narayan et al. Reference Narayan, Cohen and Lapata2018), which are datasets commonly used for text summarization tasks. Their analysis revealed a significant proportion of instances with issues of Entity Missing and Evidence MissingFootnote b in these datasets. This indicates that the target summaries often contained entities or concepts absent from the source texts, raising questions about the accuracy of these datasets.

Furthermore, studies by Sugawara et al. (Reference Sugawara, Stenetorp, Inui and Aizawa2020) and Gururangan et al. (Reference Gururangan, Swayamdipta, Levy, Schwartz, Bowman and Smith2022) suggest that performance metrics on certain machine reading comprehension and natural language inference datasets might be artificially inflated. This is attributed to models exploiting spurious correlations rather than truly understanding the underlying language structures, resulting in poor generalization when applied to real-world scenarios.

An equally significant concern in NLP dataset construction is data leakage, particularly test-train overlap, which poses substantial risks to model evaluation. Studies like Lewis et al. (Reference Lewis, Stenetorp and Riedel2021) reveal that a considerable portion of test data may mirror the training set, risking models’ overfitting to the data rather than generalizing, thus inflating performance scores. Larson et al. (Reference Larson, Lim and Leach2023) echoes this sentiment, highlighting similar concerns in document classification realms. These studies collectively call for improved dataset division methods and robust validation techniques to mitigate data leakage and truly measure a model’s generalization capabilities on unseen data.

However, most works focus on a specific issue of the datasets, and most issues are highly related to the model training process. Inspired by CTT, we built our dataset quality evaluation framework from reliability, difficulty, and validity dimensions. And we developed metrics for assessing the quality of datasets under the above three dimensions in conjunction with NER task characteristics and experimentally validated the effectiveness of our metrics in dataset evaluation.

2.2 Data-centric AI

In the contemporary landscape of NLP and machine learning, the pivotal role of datasets has increasingly been acknowledged. The seminal work by Ng et al. (Reference Ng, Laird and He2021) has galvanized the shift toward a data-centric AI paradigm, underscoring the potential of enhancing data quality to achieve superior model performance over merely refining algorithms. This approach dovetails with the initiatives like the NHS’s Data Quality Maturity Index Methodology,Footnote c which provides a structured framework to assess and improve the quality of data in healthcare, a sector that greatly benefits from NLP technologies.

Simultaneously, the introduction of DataCLUE by Xu et al. (Reference Xu, Liu, Pan, Lu and Hou2021) marks a significant stride in this domain, offering the first benchmark specifically tailored for evaluating data-centric approaches in NLP. This benchmark aligns with tools such as the Data Quality for AI Tool (Jariwala et al. Reference Jariwala, Chaudhari, Bhatt and Le2022) provided by IBM, which facilitates exploratory data analysis through its API, thereby enabling a more rigorous and systematic enhancement of datasets.

Moreover, a comprehensive review (Zha et al. Reference Zha, Bhat, Lai, Yang, Jiang, Zhong and Hu2023) offers a detailed exploration of the need for data-centric AI, addressing the methodological pivot from a model-centric to a data-centric perspective in AI research. This survey highlights the indispensable need for high-quality data to train robust machine learning models, especially in domains where data are prone to noise, sparsity, and bias.

In light of these developments, our proposed evaluation metrics aim to contribute to the ongoing efforts of dataset quality improvement. These metrics are designed to facilitate both automatic and semi-automatic enhancements of datasets, ensuring that the data used to train NLP models are of the highest fidelity and thus capable of driving the performance of these models to new heights. The systematic application of such metrics can significantly streamline the process of data quality assurance, making it more tractable for researchers and practitioners to achieve data excellence in AI systems.

The cumulative effect of these methodologies and tools signifies a transformative movement in AI research, where data are no longer a passive element but a dynamic and critical component of the AI development lifecycle. As this data-centric ethos permeates the field, it is anticipated that future advancements in NLP will be increasingly driven by innovations in data quality management, thereby catalyzing a new era of AI systems that are both powerful and reliable.

4.1 Metrics under reliability

The metrics under reliability aim to evaluate how accurate and trustworthy a dataset is, including Redundancy, Accuracy, and Leakage Ratio. Reliability metrics—Redundancy, Accuracy, and Leakage Ratio—are key elements in assessing a dataset’s trustworthiness. The evaluation of Redundancy aims to uncover duplicate information within the dataset, which is crucial for ensuring consistency in results as it aids in securing an unbiased representation of data. By manually verifying Accuracy, we can assess the dataset’s capability in accurately reflecting real-world information, a fundamental basis for reliable outcomes. The detection of the Leakage Ratio prevents the spillover of knowledge from test data to training data, essential for measuring the model’s true performance. Together, these metrics form the cornerstone of dataset reliability, ensuring the effectiveness of NLP modeling.

Redundancy measures the proportion of duplicate instances in a dataset $D$ . A lower Redundancy value is better as it indicates fewer duplicates and, therefore, a higher diversity in the data. It is calculated by dividing the number of instances appearing more than once by the dataset’s total number of instances:

(1) \begin{equation} \text{Red}(D) = \frac{\sum _{i=1}^{n} \sum _{j=i+1}^{n} \left[\left(x^{(i)}, y^{(i)}\right) = \left(x^{(j)}, y^{(j)}\right)\right]}{n} \end{equation}

In the case of Accuracy, a higher value is preferred because it reflects the proportion of correctly annotated instances, suggesting a more reliable dataset. Accuracy aims to evaluate the annotation correctness of the dataset and can be calculated as follows:

(2a) \begin{align} \delta \left(x^{(i)}, y^{(i)}\right) &= \begin{cases} 1, & \text{if } y^{(i)} \text{ is accurate for } x^{(i)}, \\[4pt] 0, & \text{else} \end{cases} \end{align}

(2b) \begin{align} \text{Acc}(D) &= \frac{\sum _{i=1}^{n} \delta (x^{(i)}, y^{(i)})}{n} \end{align}

We recommend selecting 100 instances from each dataset split and inviting at least three professional linguists to annotate the Accuracy. To evaluate inter-rater reliability, we compute the Cohen Kappa coefficient (Cohen Reference Cohen1960) pairwise among three annotators, subsequently averaging these values. A mean Kappa exceeding $0.75$ indicates substantial rater agreement, ensuring annotation reliability.

Leakage Ratio is a critical metric used to assess the extent of data leakage between different dataset partitions, specifically how many instances in the test set ( $Te$ ) have incorrectly appeared in the training set ( $Tr$ ) or development set ( $De$ ). A lower Leakage Ratio is indicative of better dataset partitioning as it suggests that there is minimal to no overlap between the sets, which is essential for preventing models from merely memorizing specific instances instead of learning to generalize. The Leakage Ratio is defined as:

(3) \begin{equation} \text{LeakR}(D) = \frac{\sum _{i=1}^{|Te|} \left[\left(Te^{(i)} \in Tr\right) \text{ or } \left(Te^{(i)} \in De\right)\right]}{|Te|} \end{equation}

4.2 Metrics under difficulty

We propose four metrics under difficulty to assess how challenging the datasets are, including three intrinsic metrics (Unseen Entity Ratio, Entity Ambiguity Degree, and Text Complexity) and one extrinsic metric (Model Differentiation). These difficulty metrics assess a dataset’s challenge level for NLP models. Unseen Entity Ratio tests generalization by measuring novel entities, pushing models beyond their training. Entity Ambiguity Degree and Text Complexity challenge models with varied entity types and dense entity arrangements, requiring nuanced interpretation. Model Differentiation shows a dataset’s power to separate model performances, testing robustness. Together, they define the dataset’s challenge in terms of generalization, ambiguity, density, and differentiation, fitting the difficulty dimension.

The Unseen Entity Ratio quantifies the proportion of new entities in the test set labels that are not present in the training set, promoting the model’s ability to generalize. A higher Unseen Entity Ratio is desirable as it indicates a greater challenge for the model to recognize entities it has not encountered during training. The calculation is as follows:

(4) \begin{equation} \text{UnSeenEnR}(D) = \frac{|e\left (\boldsymbol{y_{Te}}\right) \setminus{e(\boldsymbol{y_{Tr}})}|}{|e(\boldsymbol{y_{Te}})|} \end{equation}

Entity Ambiguity Degree is mainly used to measure how many entities are labeled with more than one kind of entities types. For example, if “apple” is labeled as “Fruit” in one instance and labeled as “Company” in another instance, then there is a conflict in $D$ . A higher Entity Ambiguity Degree represents a more challenging dataset because it indicates more instances where an entity is labeled with different types, thereby confusing NER models. We introduce $e^*(D)$ to represent the number of conflict entities in dataset $D$ and obtain the Entity Ambiguity Degree by:

(5) \begin{equation} \text{EnAmb}(D) = 1 -\frac{e^*(D)}{n} \end{equation}

Text Complexity measures the average Entity Density in sentences within the dataset. Higher Text Complexity signals a more difficult dataset because it implies that sentences are densely packed with entities, requiring more nuanced understanding and recognition by the model. It is formulated as:

(6) \begin{align} \text{EnDen}(D) = \sum _{i=1}^n \frac{|e\left (y^{(i)}\right)|}{n m^{(i)}} \end{align}

Model Differentiation evaluates the dataset’s ability to distinguish the performance of different models. A higher Model Differentiation value is better as it indicates that the dataset can effectively reveal differences in model performances, making it a useful tool for benchmarking. It is determined using the standard deviation of the scores of $k$ different models:

(7) \begin{align} \text{ModDiff}(D) = \text{Std}(\theta _1,\theta _2,\cdots,\theta _k) \end{align}

We recommend using the top five model scores on the datasetFootnote f for ModDiff calculation.

4.3 Metrics under validity

The metrics under validity, for example, Entity Imbalance Degree and Entity-Null Rate for NER datasets, are mainly proposed to evaluate the effectiveness of the dataset in evaluating the model’s ability on the specific task. Validity metrics like Entity Imbalance Degree and Entity-Null Rate assess if a dataset can effectively evaluate a model’s task-specific abilities. Entity Imbalance Degree checks for equal entity representation, ensuring models learn without bias—a key for valid evaluations. Entity-Null Rate measures how rich the dataset is in entity examples, vital for testing model learning depth. Both metrics directly contribute to assessing a dataset’s ability to provide a fair and thorough evaluation of model performance, embodying the essence of validity.

Entity Imbalance Degree mainly measures the unevenness of the distribution of different entities in $D$ . A lower Entity Imbalance Degree is better as it indicates a more balanced distribution of entity types, which is desirable for ensuring that the model is equally exposed to all categories and does not develop a bias toward the more frequent ones. Specifically, we use standard deviation to quantify the degree of dispersion of the distribution of all the different types of entities $\mathscr{C}$ in the dataset:Footnote g

(8) \begin{equation} \begin{split} \text{EnImBaD}(\mathscr{D}) \!= \!Std\left (P_{y_D}(c_1),P_{y_D}(c_2),\cdots \!,P_{y_D}(c_v)\right) \end{split} \end{equation}

Entity-Null Rate evaluates the proportion of instances in the dataset that do not contain any entity. A lower Entity-Null Rate is preferred because it suggests that the dataset contains a richer set of examples for the model to learn from, with more instances that include entity information. The Entity-Null Rate is defined as:

(9a) \begin{align} \zeta (y^{(i)}) &= \begin{cases} 1, & \text{if } y^{(i)} \text{ has no entity}, \\ 0, & \text{else} \end{cases} \end{align}

(9b) \begin{align} \text{EnNullR}(D) &= \frac{\sum _{i=1}^{n} \zeta (y^{(i)})}{n} \end{align}

5.1 Datasets

We provide the basic information about the datasets in Table 2.

Table 2. Standard named entity recognition dataset statistics.

Zh and En mean Chinese and English, respectively. It is important to note that OntoNotes 4 has four common tags in the Chinese dataset, although OntoNotes 4 has a total of eighteen tags (for the English dataset).

Figure 2. Evaluation results of WNUT16, CoNLL03, Resume, and MSRA under different dimensions and metrics. The abbreviations and corresponding full names of the metrics are presented in Sec. 4.

English NER datasets include the following: CoNLL03 NER (Sang and De Meulder Reference Sang and De Meulder2003) is a classical NER evaluation dataset consisting of 1,393 English news articles. WNUT16 NER (Strauss et al. Reference Strauss, Toma, Ritter, De Marneffe and Xu2016) is provided by the second shared task at WNUT-2016 and consists of social media data from Twitter. OntoNotes5 (Weischedel et al. Reference Weischedel, Palmer, Mitchell, Hovy, Pradhan, Ramshaw and Houston2013) is a multi-genre NER dataset collected from broadcast news, broadcast conversation, weblogs, and magazine genre, which is a widely cited English NER dataset.

Chinese NER datasets consist of the following: CLUENER (Xu et al. Reference Xu, Dong, Liao, Yu, Tian, Liu and Zhang2020), a well-defined NER dataset, includes finer-grained entity types beyond standard ones (person, organization, and location), such as Company, Game, and Book. OntoNotes4 (Weischedel et al. Reference Weischedel, Pradhan, Ramshaw, Palmer, Xue, Marcus and Houston2011) is copyrighted by Linguistic Data ConsortiumFootnote h (LDC), a large manual annotated database containing various fields with structural information and shallow semantics. MSRA (Levow Reference Levow2006) is a large NER dataset in the field of news, containing distinctive text structure characteristics. PeopleDaily NER Footnote i is a very classic benchmark dataset to evaluate different NER models. Resume NER (Zhang and Yang Reference Zhang and Yang2018) features resumes of senior executives from Chinese stock market companies, with a high annotator agreement of 97.1 percent. It includes 1027 randomly selected summaries annotated for 8 entity types using the YEDDA system (Yang et al. Reference Yang, Zhang, Li and Li2018). They randomly select 1027 resume summaries and manually annotate 8 types of named entities with YEDDA system (Yang et al. Reference Yang, Zhang, Li and Li2018). The inter-annotator agreement is 97.1 percent. Weibo NER (Peng and Dredze Reference Peng and Dredze2015; He and Sun Reference He and Sun2017) is sourced from the Sina Weibo social media platform. WikiAnn (Pan et al. Reference Pan, Zhang, May, Nothman, Knight and Ji2017) is a Chinese part of a multilingual NER dataset from Wikipedia articles.

5.2 Settings

According to the metrics we proposed in Sec. 4, we calculate the statistical scores for each dataset. Specifically, we average the scores of the training, the development, and the test split of the datasets for Redundancy, Accuracy, Entity Ambiguity Degree, Entity Density, Entity Imbalance Degree, and Entity-Null Rate, respectively. For Leakage Ratio, Unseen Entity Ratio, and Model Differentiation, we only calculate the scores on the specific splits involved according to Sec. 4.1 and Sec. 4.2.

Table 3. Results of metrics (except Leakage Ratio) under the reliability dimension of the NER datasets.

Red and Acc denote Redundancy and Accuracy, respectively. Zh and En mean Chinese and English, respectively.

5.3 Dataset reliability

5.3.3 Overall reliability

Combining several metrics under the reliability dimension in Table 1, we can conclude that Resume and MSRA maintain high reliability.

In specific, there is no data redundancy in Resume and MSRA. That is to say, the instances of each part of the dataset are unique and non-repeating. Additionally, they achieve the highest Accuracy scores and hardly show data leakage problems, with a Leakage Ratio of 0.01 (1 percent) and 0.00 (0 percent), respectively.

Figure 3. Leakage Ratio values of WikiAnn and Weibo. It is observed that 0.13 (13 percent) and 0.17 (17 percent) of the instances in the test set of WikiAnn and Weibo, respectively, appear in their corresponding training or development sets.

5.4 Dataset difficulty

5.5 Dataset validity

6.1 Models

For experiments on Chinese NER datasets, we use three models: 1) Lattice-LSTM (Zhang and Yang Reference Zhang and Yang2018), based on LSTM networks (Chiu and Nichols Reference Chiu and Nichols2016), which automatically identifies key words from the context; 2) Flat-Lattice (Li et al. Reference Li, Yan, Qiu and Huang2020), which converts the lattice structure into a flat structure; and 3) Roberta (Liu et al. Reference Liu, Ott, Goyal, Du, Joshi, Chen and Stoyanov2019), a transformer-based pretrained model which removes the next sentence predict task in BERT.

For the English datasets, we also take three models, including: 1) LSTM CRF (Lample et al. Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016), a traditional model based on the bidirectional LSTM with conditional random fields (CRF); 2) LUKE (Yamada et al. Reference Yamada, Asai, Shindo, Takeda and Matsumoto2020), which provides new pretrained contextualized representations of words and entities by predicting masked words and entities in entity-annotated corpus based on the bidirectional transformer (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez and Polosukhin2017); and 3) W2NER (Li et al. Reference Li, Yan, Qiu and Huang2020), which converts NER to word–word relationship classification and models the neighboring relations between entity words with Next-Neighboring-Word (NNW) and Tail-Head-Word (THW) relations.

6.2 Experiment settings

All the experiments are done on the NVIDIA RTX 2080 GPU and 3090 GPU and evaluated by seqeval.Footnote k Specifically, we utilize Micro F1 scores to measure the performance of the NER model. For the experiment with Train-Dev Dataset Adjustment (Sec. 6.4), we report the averaged results and variances over three random seeds.

6.3 Model replication results

We replicated six NER models in accordance with the experimental setup, and the results of the model replication are presented in Table 4 and 5.

Table 4. Chinese NER model replication results.

repro. denotes reproduction. - denotes that the authors of the literature we cited did not experiment on that dataset. And ori. denotes original paper results.

Table 5. English NER model replication results.

repro. denotes reproduction. - denotes that the authors of the literature we cited did not experiment on that dataset. And ori. denotes original paper results.

6.4 Controlled dataset adjustment

To investigate how statistical properties affect model performance, we conducted controlled dataset adjustments: 1) we modified the test set to create two new sets (of the same size) with distinct statistical values for specific metrics (i.e., Test Dataset Adjustment). 2) Similarly, we adjusted the training and development sets to form new sets (of the same size) with distinguishable metrics values (i.e., Train-Dev Dataset Adjustment).

6.4.2 Train-Dev Dataset Adjustment

Initially, we chose datasets with a high Entity-Null Rate (WNUT16, OntoNotes5 for English; Weibo, OntoNotes4 for Chinese). We then filtered the training and development sets to adjust the Entity-Null Rate to 0.20 (20 percent) and 0.80 (80 percent), ensuring equal numbers of instances in these subsets. Finally, we trained the data with various models before testing and comparing the results with the same test set.

Figure 4. Model performance on NER datasets when the proportion of unseen entities (UnSeenEn) in the test set is 0.80 (UnSeenEn_8) and 0.20 (UnSeenEn_2), respectively.

Figure 5. Model performance on NER datasets when the proportion of ambiguous entities in the test set is 0.80 (EnAmb_8) and 0.20 (EnAmb_2), respectively.

6.5 Experiment results and analysis

• • Datasets with high Unseen Entity Ratio are more difficult for NER models: Intuitively, those entities that were seen during training are less challenging for NER models compared to those that did not appear in the training set. Figure 4 supports our intuition. Models perform better on datasets with a lower proportion of unseen entities than on datasets with a relatively high proportion of unseen entities.

• • Entities with strong Entity Ambiguity Degree are indeed more likely to confuse the model: We can infer from Figure 5 that datasets with a high Entity Ambiguity Degree are more challenging for the model. As for models tested on Chinese datasets, their average performance is 6.42 (F1) points higher on datasets with low entity ambiguity rates than on datasets with high entity ambiguity rates. The English NER model is more likely to be confused by entities with a high entity ambiguity rate and make wrong decisions.

• • The models exhibit improved performance with increased test set leakage, highlighting the necessity for enhanced generalization in NER models: As shown in Table 6, three models (i.e., Lattice-LSTM, Flat-Lattice, and Roberta) consistently achieve better performance when the leakage rate of the test set is 0.80 (80 percent) than when it is 0.20 (20 percent). In particular, we found that the performance of Flat-Lattice on the Weibo test set with a Leakage Ratio of 0.80 (80 percent) outperformed the 0.20 (20 percent) by a large margin, that is, 25.69 percent. We speculate that because the model has seen the leaked data in the test set during training, it performs better on the test set with a relatively high data leakage rate. Looking at the experimental results from another perspective, researchers need to pay more attention to improving the NER model’s generalization ability.

• • Entity-Null Rate plays a small difference: As shown in Tables 7 and 8, the F1 score of the training set and development set with EnNullR of 0.20 (20 percent) is better than 0.80 (80 percent). Therefore, we conclude that the contribution of instances without entities to the model is less than the instances with entities during training. However, are instances without any entities completely useless for model training? We delete all these instances and show the results in Tables 7 and 8. The performance of models trained on such datasets decreases, which indicates that the instances without entity are necessary, as they keep the distribution of the test set and training set relatively consistent.

Table 6. Model performance when the proportion of leaked samples in the test set is 80 percent and 20 percent, respectively.

LSTM represents Lattice-LSTM.

Table 7. Model performance on English datasets when the proportion of samples without entities in the training set and development set is 0.80 (80 percent), 0.20 (20 percent), 0.00 (0 percent), and original, respectively.