2.1. Key variables and measurement

We do not rely on LMArena’s citation variable construction, which we found did not adequately distinguish between search logs and citations for our use case. We instead reconstruct them from scratch using the model logs in their dataset.

Ecosystem exploitation (attribution gap). Our primary dependent variable measures the difference between unique relevant pages visited by the LLM during search and unique pages actually cited to the user in the API response. This captures the extent to which a model consumes relevant web content without providing appropriate attribution.

Citations. We define citations as any in-text URL reference that is grounded in the model’s search results log, including both explicit URLs and numbered references that link to specific sources. We exclude “hallucinated citations” (numbered citations, such as ^[13], that link to an empty source) and “ungrounded citations” (citation of URLs that exist but are not in the search log). Unique URLs (websites) for citations or attribution are those seen by the model that do not link to the same web page when stripped of parameters (e.g., example.com/?tracking_id=23222 is turned into example.com). This amounts to transforming URLs into their base URL and then checking for duplicates.

Relevant sites visited (consumed). We define relevant sites visited by the LLM as those listed in their search log, regardless of whether they were cited in-text or not. We treat every URL that appears in the search log as relevant, on the understanding that the provider has already filtered out nonrelevant visits. Note, however, that this log may itself be incomplete: some vendors—OpenAI, in particular—seem to pretrim their traces, returning only a subset of the relevant pages the model actually visited.

Conversation classification. We categorize all conversations into 12 topic areas using o4-mini with high reasoning to enable analysis of topic-specific attribution patterns. This classification covers areas from technical queries (software engineering, data science) to consumer-oriented topics (shopping, health, finance). We provided the 12 categories to GPT, drawing on an unsupervised visualization of clusters in the query data on Nomic Atlas (see https://atlas.nomic.ai/data/srulyrosenblat/ai-model-search-comparison-dataset/map).

Data cleaning. After removing conversations with classification failures or misaligned search traces, our final sample contains 13,929 observations, all successfully classified and with clean attribution data.

Prefiltering of logs. Most models appear to filter their logs for relevant websites visited only—rather than all sites visited. But OpenAI appears to filter them more strictly. Hallucinated citations (numbered citations provided in-text that do not exist in the search log) appear for Gemini and to a lesser extent Sonar (they are by definition zero for GPT-4o models since OpenAI’s models do not use numbers in square brackets for their citations, which is what we track for this metric). Moreover, in terms of citations provided that do not appear in the search logs (ungrounded citations): 7% of citations by GPT-4o models were not found in their search logs, indicating either an overly restrictive prefiltering of the search logs or simply the model citing from pretrained knowledge.

4.1. Hurdle model specification

The dependent variable $ {Y}_i\;\left(i=1,\dots, n\right) $ is the attribution gap: the number of relevant websites a model visits but fails to cite, for query $ i $ , where $ i $ runs from $ 1 $ to $ n=\mathrm{13,929} $ . Because most answers are perfectly attributed ( $ {Y}_i=0 $ ), while the rest show a skewed count distribution, we use a hurdle model. It breaks up $ E\left[ Attribution\hskip0.24em Gap\right]=P\left( Gap>0\right)\times E\left[ Gap\;|\; Gap>0\right] $ into two sequential components:

1. (a) Hurdle stage: What is the probability that the gap is exactly zero $ \to $ $ P\left( Gap>0\right) $ ? This is a Bernoulli outcome with probability $ {\pi}_i $ .

2. (b) Count stage: If an attribution gap exists ( $ {Y}_i>0 $ ), how large is the gap: $ E\left[ Gap\;|\; Gap>0\right] $ ? We model positive counts with a zero-truncated negative binomial distribution, which has mean $ {\lambda}_i $ and over-dispersion parameter $ \alpha $ (if $ \alpha \to 0 $ , the model reduces to a Poisson hurdle; see Supplementary Material for the full log-likelihood specification and model diagnostics).

Putting the two parts together gives the probability mass function:

(1) $$ \Pr \left({Y}_i=y|{\mathbf{x}}_i\right)=\left\{\begin{array}{ll}{\pi}_i,& y=0,\\ {}\left(1-{\pi}_i\right)\hskip0.1em \frac{f_{\mathrm{nb}}\left(y;{\lambda}_i,\alpha \right)}{1-{f}_{\mathrm{nb}}\left(0;{\lambda}_i,\alpha \right)},& y\ge 1,\end{array}\right. $$

where

(2) $$ {f}_{\mathrm{nb}}\left(y;\lambda, \alpha \right)=\frac{\Gamma \left(y+{\alpha}^{-1}\right)}{\Gamma \left({\alpha}^{-1}\right)\hskip0.1em y!}\hskip0.1em {\left(\alpha \lambda \right)}^y\hskip0.1em {\left(1+\alpha \lambda \right)}^{-\left(y+{\alpha}^{-1}\right)},\hskip0.4em \alpha >0, $$

where $ {f}_{\mathrm{nb}}\left(y;\lambda, \alpha \right) $ is the mean-parameterized negative binomial probability mass function.

Linear predictors. Each of the two regression stages has its own regression equation with predictors, consisting of: model family, the query category or “classification” (e.g., “Data Science”), the log of LLM output character count, and the number of unique search results. This leads to the following two equations (we note that the number of conversation “turns” had poor predictive power in our model and was therefore excluded):

(3) $$ \mathrm{logit}\left({\pi}_i\right)={\gamma}_0+{\boldsymbol{\gamma}}_{\mathrm{fam}}^{\top}\hskip0.1em \mathbf{1}\left\{\mathrm{model}\_{\mathrm{fam}\mathrm{ily}}_i\right\}+{\boldsymbol{\gamma}}_{\mathrm{cls}}^{\top}\hskip0.1em \mathbf{1}\left\{{\mathrm{classification}}_i\right\}+{\gamma}_{\mathrm{\ell}}\hskip0.1em \log \left(\mathrm{response}\_\mathrm{character}\_{\mathrm{count}}_i\right), $$

₍₄₎ $$ {\displaystyle \begin{array}{l}\log {\lambda}_i={\beta}_0+{\boldsymbol{\beta}}_{\mathrm{fam}}^{\top}\hskip0.1em \mathbf{1}\left\{\mathrm{model}\_{\mathrm{fam}\mathrm{ily}}_i\right\}+{\boldsymbol{\beta}}_{\mathrm{cls}}^{\top}\hskip0.1em \mathbf{1}\left\{{\mathrm{classification}}_i\right\}\\ {}+{\beta}_{\mathrm{sr}}\hskip0.1em \mathrm{unique}\_\mathrm{search}\_\mathrm{results}\_{\mathrm{count}}_i+{\beta}_{\mathrm{\ell}}\hskip0.1em \log \left(\mathrm{response}\_\mathrm{character}\_{\mathrm{count}}_i\right)\\ {}+{\left(\mathbf{1}\left\{\mathrm{model}\_{\mathrm{fam}\mathrm{ily}}_i\right\}\otimes \mathbf{1}\left\{{\mathrm{classification}}_i\right\}\right)}^{\top }{\boldsymbol{\beta}}_{\mathrm{fam}\times \mathrm{cls}}\\ {}+{\left(\mathbf{1}\left\{\mathrm{model}\_{\mathrm{fam}\mathrm{ily}}_i\right\}\otimes \mathrm{unique}\_\mathrm{search}\_\mathrm{results}\_{\mathrm{count}}_i\right)}^{\top }{\boldsymbol{\beta}}_{\mathrm{fam}\times \mathrm{sr}}.\end{array}} $$

Equation (3) is the logit probability, where $ \exp \left({\boldsymbol{\gamma}}_{\mathrm{fam}}\right) $ represents the odds ratio for producing any attribution gap by model family, $ \exp \left({\boldsymbol{\gamma}}_{\mathrm{cls}}\right) $ represents the odds ratio by classification topic, and $ \exp \left({\gamma}_{\mathrm{\ell}}\right) $ is the odds ratio for answer length. The equation contains only main effects (no interactions): the odds of any gap depend separately on model family, topic, and answer length. For example, $ \exp \left({\gamma}_{\mathrm{\ell}}\right)=1.4 $ would mean a one-unit increase in log(response length) multiplies the odds of a gap by 1.4.

Equation (4) is the negative binomial count component, whereby $ \exp \left({\boldsymbol{\beta}}_{\mathrm{fam}}\right) $ represents the multiplicative changes in the expected number of uncited website visits by model family, given that a gap exists. $ \exp \left({\boldsymbol{\beta}}_{\mathrm{cls}}\right) $ represents the effects by classification topic, and $ \exp \left({\beta}_{\mathrm{sr}}\right) $ is the multiplicative effect of additional search results. A value of 1.30 implies a 30% larger gap, for example. Equation (4) adds interaction terms $ {\boldsymbol{\beta}}_{\mathrm{fam}\times \mathrm{cls}} $ , so the effect of a model family can differ by query topic once a gap exists, and $ {\boldsymbol{\beta}}_{\mathrm{fam}\times \mathrm{sr}} $ , to account for the varying impact of searching (number of URLs) on the attribution gap by model family.

4.2. Head-to-head model comparison regression

To isolate the number of URLs each LLM cites from one additional website search visit, while controlling for question types, we run a second regression model. This exploits the LMArena head-to-head design, whereby every question is answered by exactly two model systems. This means that any unobservable query-specific factors driving citation behavior cancel out. This design also ensures that $ {\beta}_{1m} $ is not biased by systematically “easy” or “hard” opponents. We collapse each model pair that answers the same query into a single observation (focal model $ - $ opponent) and run a separate OLS regression for every focal model $ m $ , where $ i $ now runs from $ 1 $ to $ n=\mathrm{6,951} $ , and $ m $ contains 11 focal models:

(6) $$ {d}_{im}={\beta}_{0m}+{\beta}_{1m}\hskip0.1em \Delta {s}_{im}+{\beta}_{2m}\hskip0.1em \Delta {\mathrm{\ell}}_{im}+\sum \limits_k{\gamma}_k^{(m)}\hskip0.1em {D}_{ik}+\sum \limits_j{\delta}_j^{(m)}\hskip0.1em {O}_{ij}+{\varepsilon}_{im}. $$

In Equation (6), $ {d}_{im} $ denotes the citation-gap advantage—the difference in unique citations produced by the focal model compared to its “opponent,” $ {\mathrm{citations}}_m-{\mathrm{citations}}_{\mathrm{opp}} $ . The term $ \Delta {s}_{im} $ captures the search retrieval difference, defined as the gap in unique URLs visited by the two systems in its logs, while $ \Delta {\mathrm{\ell}}_{im} $ measures the length difference in characters between their answers. The vector $ {D}_{ik} $ comprises topic dummies controlling for the classification of question $ i $ (reference category: “Current Affairs and Factual Information”), and $ {O}_{ij} $ contains dummies identifying which rival model the focal system faces (baseline = the first alphabetic opponent). The slope $ {\beta}_{1m} $ measures citation efficiency: the expected additional citations that model $ m $ produces per additional URL it opens, conditional on topic, verbosity, and opponent. Thus, $ {\beta}_{1m}=0.40 $ implies that 10 extra retrievals yield roughly four extra citations. The coefficient $ {\beta}_{2m} $ captures the effect of answer length net of retrieval, while topic dummies ( $ {\gamma}_k^{(m)} $ ) and opponent dummies ( $ {\delta}_j^{(m)} $ ) control for domain-specific and match-up effects, respectively.

Each model’s $ {\beta}_{1m} $ is our key quantity of interest and summarizes model $ m $ ’s retrieval-to-citation efficiency or yield: the impact of a marginal website visit on citations.

5.1. Attribution gap regression model

The regression results, transformed into probabilities, show notable differences in attribution reliability across AI model families (Table 2 and Figure 1). To obtain policy-interpretable results, we used the fitted model to make predictions for a standardized query covering Current Affairs and Factual Information with median characteristics (5 search results, 2,089 character responses). The hurdle component estimates the probability of having any attribution gap, while the count component estimates the expected gap size when gaps occur; multiplying these gives the unconditional expectation. The “expected attribution gap” is the total prediction from our model: $ E\left[ Attribution\hskip0.24em Gap\right]=P\left( Gap>0\right)\times E\left[ Gap\;|\; Gap>0\right] $ , whereas the first row (“Probability of a Gap”) is only the first term of this equation, $ P\left( Gap>0\right) $ .

Table 2.

Expected total attribution gap by model family

Note: Results from negative binomial hurdle model for Current Affairs and Factual Information queries with median characteristics (5 search results, 2,089 characters, per query), with bootstrapped mean. Confidence intervals from parametric bootstrap (n = 1,000) for total expected attribution gap. Attribution gap is the missing web page (URL) citations per search query relative to relevant websites consumed, measured as a web page gap. This pattern holds consistently across all topic classifications (see Supplementary Table A3).

Figure 1.

Expected attribution gaps, predicted (by model family). Note: Predicted values for number of citations missing relative to web pages consumed. Based on negative binomial hurdle model regression coefficients. Bars show the model’s expected citation gap (websites visited in the logs minus websites cited in the output), estimated at the median conversation length and median website visits, without interaction effects included. Showing 95% confidence intervals calculated with the emmeans package in $ \mathrm{\mathbb{R}} $ .

The GPT-4o models appear to show less exploitative attribution behavior, but this is most likely due to them being more circumspect in their disclosures by not showing the full extent of their logs (using OpenAI search in the user interface shows far more website visits based on limited testing). OpenAI’s models have only a 10.5% probability of having any citation gaps and an expected 0.18 missing citations (relative to relevant websites consumed) per query (Table 2). Sonar exhibits citation gaps in 99.3% of queries with 3.12 expected missing citations per query in total. Gemini falls somewhere between these extremes with citation gaps in 70.3% of queries and 3.04 expected missing citations in total per query.

Sonar’s near-universal citation gap (99.3% of queries) makes it particularly concerning for applications requiring source transparency. Gemini is arguably relying too heavily on internal knowledge given that a large portion of its zero attribution gap is due to it having zero (disclosed) website searches, at 34% of queries.

Finally, in terms of other factors driving how large the gap is once it appears (the count component), we can look at the incidence rate ratio (IRR), which tells us the multiplicative change in the expected count when the predictor increases by one unit, holding everything else constant. Longer answers actually shrink the gap size: When logged response character count doubles (from, e.g., 100 to 200 characters), the expected number of missing citations decreases by about 11% (IRR 0.89 or so). Reading more web pages from search inflates the gap: every extra page the LLM system reads raises the expected uncited count by 13% (IRR ~1.13).

5.2. Head-to-head model results: citation efficiency

In this section, we focus on the $ {\beta}_{1m} $ coefficient from our head-to-head regressions (Equation (6)). This coefficient tells us how many more citations the focal model generates for each additional search result compared to its opponent. A smaller coefficient signals poorer performance: the focal model converts each extra relevant page it opens into fewer citations than its comparator does. For example, if Sonar’s coefficient is 0.4, this means that when it undertakes five additional relevant web page visits than GPT-4o on a question, it is expected to have $ 5\times 0.4=2 $ more citations than GPT-4o. 264 coefficients are estimated in total, running a separate regression for each of the 11 models with 24 parameters each: 1 intercept, 11 topic dummies, 1 focal-search-diff, 10 opponent-model dummies, and 1 focal-length-diff.

We plot $ {\beta}_{1m} $ in Figure 2 (see Supplementary Table A4). Almost all coefficients from the regression are positive and highly significant: the focal_search_diff coefficient is significant and positive for every model, indicating that extra retrieval consistently translates into more citations, though the payoff varies. Answer length matters for the basic GPT and Sonar variants but not for “high” search variants or Gemini. Topic and opponent effects are sparse and model specific.

Figure 2.

Focal model: citation difference per extra URL visited. Note: Extra citations gained for each additional URL the focal model opens (differences between models). This holds match-up effects constant, isolating technology effects. Regression coefficient $ {\beta}_{1m} $ shown, predicting differences in citations for model pairs, for a given query. See equation before.

Figure 2 shows that best-in-class variants (Sonar-reasoning-pro-high, Gemini-flash-grounding, GPT-4o-search-high-loc) yield $ \sim 0.43 $ citations per extra URL, whereas baseline variants (ppl-sonar-pro-high) return only $ \sim 0.19 $ . This implies that the best model converts every extra retrieved URL into ~0.45 additional citations (compared to its competitor models), whereas the weakest model variant converts that same extra URL into ~0.19 citations. The span is therefore about $ 0.45-0.19\approx 0.26 $ citations per URL, showing that RAG implementation choices can more than double the payoff from each additional page the model visits. This illustrates just how wide the performance window is, and thus how much room developers (or regulators) have to raise low performers up to the current best practice.

More specifically, our results show that RAG implementation type (technology) is more important than model family (e.g., OpenAI vs. Gemini):

• • An ANOVA test shows that within-family variation (the variance) of search coefficients $ {\beta}_{1m} $ is almost 8 times larger than between-family variation ( $ {\mathrm{MS}}_{\mathrm{within}}=0.0116 $ vs. $ {\mathrm{MS}}_{\mathrm{between}}=0.0015 $ ; $ {F}_{\left(2,8\right)}\approx 0.13 $ , $ p=0.88 $ ). This means that differences in model behavior are far greater within model families than between them: implementation choices explain roughly 8 times more of the spread in citation efficiency than the provider family label itself.

• • Within Sonar alone, upgrading to the reasoning tier more than doubles efficiency (from 0.19–0.20 $ \to $ 0.44–0.45), with Sonar-reasoning (0.44) versus Sonar-pro-high (0.19) = 0.25 difference. For GPT models: GPT-4o-search-high-loc (0.42) versus GPT-4o-mini (0.25) = 0.17 difference.

• • Location signals matter too. Adding a country code to GPT-4o raises search-citation coefficient efficiency by roughly 10 % (0.39 $ \to $ 0.43), confirming that retrieval relevance translates directly into better attribution (though it is unclear why this is the case). Practically, this effect size is moderate since for a model getting 10 extra search results, location signals would generate ~0.37 additional citations. But we know that local search operates differently, both in traditional search and increasingly with LLMs (Rollison, Reference Rollison2025).

Finally, the regression results show that more basic models (GPT-mini, GPT-search, and basic Sonar models) compensate for lower search efficiency through verbosity—they need longer answers to achieve similar citation performance. Advanced (“high” variant and Gemini) models achieve higher citation rates through superior search utilization, making verbosity unnecessary to achieve improved citation rates.