3.1 Overview

In the present chapter we provide a concise overview of the historical antecedents of the current debate on linguistic data/evidence. First, we will summarise those tenets which shaped the methodological foundations of mainstream linguistics in the twentieth century and call them ‘the standard view of the analytical philosophy of science’. Second, our analysis will reveal that the two predominant camps in linguistics from the middle of the 1950s to the end of the 1990s, corpus linguists and generative linguists, heavily rely on methodological assumptions which are in perfect accord with the standard view of the analytical philosophy of science. We will call the set of these assumptions ‘the standard view of linguistic data and evidence’. It is this view which has been dominant over the past decades, right up to the turn of the last millennium, and which new approaches to linguistic data and evidence seriously question.

3.2 The problem of evidence in the philosophy of science

In order to solve (P)_I, we have to refer to the well-known circumstance that the standard view of the analytical philosophy of science played a decisive role in the emergence of the norms against which scientific theories have been evaluated for several decades in the twentieth century. Basically, the standard view of the analytical philosophy of science involved two approaches: logical empiricism initiated by the Vienna Circle and Popper's critical rationalism.¹ The reason these approaches have been subsumed under the standard view is that despite their differences they shared similar assumptions on a number of important issues.

First of all, both of them accepted Reichenbach's (Reference Reichenbach1938) distinction between the ‘context of discovery’ and the ‘context of justification’. The ‘context of discovery’ covers the creative, cognitive, social, historical etc. aspects of scientific discovery, theory formation and problem solving, while ‘the context of justification’ involves the logical reconstruction and the evaluation of the results of the discovery process, i.e. scientific theories. Both trends focused solely on the justification of scientific theories, excluding the process of discovery, theory formation and problem solving from their field of interest.

Another central idea of the standard view of the analytical philosophy of science was that empirical theories must consist of statements which can be tested on the basis of a special subset of data, namely, evidence.² According to this idea – to put it in a very simplified manner, as a first approximation – the intuition underlying the notion of evidence was the following:

1. (E) Evidence

   1. (a) is objective;

   2. (b) serves as a neutral arbiter among rival hypotheses/theories;

   3. (c) is expected to justify (verify, falsify, confirm) hypotheses/theories;

   4. (d) is immediately given;

   5. (e) is primary to the theory;

   6. (f) is reliable.

The properties mentioned in (E) motivate the central role of evidence; they can be regarded as the common intuitive core of the different approaches to evidence within the standard view of the analytical philosophy of science.³ Nevertheless, the interpretation of these properties is highly problematic. Therefore, in twentieth-century philosophy of science the discussions concerning the nature of evidence aimed at their explication, and also attempted to decide which of them were really relevant. In the course of the controversies, each of (E)(a)--(f) has been seriously questioned and each of them has been explicated in a variety of ways often incompatible with each other.⁴ In certain cases, however, suggestions were put forward that seem to have been widely accepted.⁵ During the debates quandaries and paradoxes were continuously raised (see primarily Goodman Reference Goodman1983 [1955]; for a detailed discussion of Goodmans' and others' paradoxes see Stegmüller Reference Stegmüller1970).

According to inductivists such as Carnap, the function of evidence is the verification of the hypotheses. Later, still within inductivism, it was also explicated in terms of confirmation, according to which evidence strengthens the hypothesis, or makes it more probable, without, however, proving its truth.⁶ Thus, confirmation includes verification as an extreme case. Popper's hypothetico-deductive view highlighted the falsificatory function of evidence in the sense that, although it cannot verify a hypothesis, it is capable of falsifying (i.e., refuting) it.⁷ Although these three explications – each of which has several versions – were developed in different periods of the analytical philosophy of science and were the focus of heated debates, they are logically related:

In close connection with the distinction of the context of discovery and the context of justification, the norms of the standard view of the analytical philosophy of science require that the structure of scientific theories be deductive. It has to be emphasised that inductivists impose this requirement on theories, too. From this it follows that with rational reconstruction, the relevance of empirical evidence does not stem from the role it plays in the discovery process, in the generation of hypotheses, or in the creation of the theory but solely from its role as a means of justification. Therefore, as we have seen, the function of evidence is the verification/confirmation, or, according to falsificationism, the potential falsification, of hypotheses/theories which have been obtained by whatever methods.

For the sake of later reference, we summarise the basic tenets of the standard view of the analytical philosophy of science from the point of view of the present book as follows:

1. (SVAPS)

   1. (a) Only theories conforming to the norms of rationality can be regarded as scientific.

   2. (b) Philosophy of science aims at the justification of scientific theories with the help of the rational reconstruction of theories.

   3. (c) There is a clear distinction between the ‘context of discovery’ and the ‘context of justification’. This distinction is motivated by the consideration that neither the creative, intuitive, cognitive processes present during scientific theorising, nor the historical and sociological factors that may influence the researcher can be revealed and controlled. Therefore, rational reconstruction deems them irrelevant. It is solely certain characteristics of the final product of the discovery process (i.e. the logical structure and the empirical foundation of closed scientific theories) that significance can be assigned to during the justification of theories.

   4. (d) The analytical philosophy of science involves two dominant trends, namely inductivism (verificationism/confirmationism) and hypothetico‐deductivism.

   5. (e) Evidence, as characterised in (E), is attributed special significance in the justification of hypotheses and theories. That is, hypotheses of empirical theories have to be verified/confirmed by evidence, or they have to be falsifiable by evidence.⁸

   6. (f) Empirical scientific theories are deductive (axiomatic) systems.⁹

(SVAPS) is the metatheoretical background against which the standpoints in the debate on data/evidence will be evaluated in Part I of the book.

3.3 Debates on empiricalness in theoretical linguistics and the standard view of linguistic data (SVLD)

Dominant linguistic theories in the twentieth century – although in different ways and to a different extent – adapted their methodological norms from the standard view of the analytical philosophy of science (SVAPS).¹⁰ The emergence of generative linguistics gave rise to a series of closely related and inextricably intertwined methodological debates. Each of these debates centred around particular aspects of the problem of the empiricalness of linguistic theories. However, from debate to debate, this problem was tackled from different perspectives, explicated differently, and reduced to different subproblems. In order to motivate the problem the present book will focus on, we need to refer briefly to four of them.

As it is well known, the methodology of American structuralism was predominantly taxonomic and inductive. Chomsky launched a fierce attack against this in the 1950s–60s, contrasting it with the hypothetico-deductive method of the standard view of the analytical philosophy of science (SVAPS). With the application of the latter he intended to follow the norms prescribed for theories in the natural sciences (Chomsky Reference Chomsky1975 [1957], Reference Chomsky1965).¹¹ Thus, in the 1950s–60s a debate evolved around the central methodological question which was – to put it in a simplified way – whether linguistics as an empirical discipline should rely primarily on inductive generalisations or on the deductive testing of hypotheses (Allan Reference Allan2003, Reference Allan2007).

In the 1970s, new themes emerged and became the object of fierce dis-cussions in the debate on the problem of the empiricalness of theoretical linguistics. Instead of the confrontation of the two main trends within (SVAPS), the debate focused on the antagonism between (SVAPS) and hermeneutics. Linguists following the hypothetico-deductive branch of the standard view of the analytical philosophy of science (SVAPS) regarded generative linguistics as an explanatory empirical theory similar to theories in natural sciences,¹² while hermeneutics saw it as a non-empirical, normative, interpretative enterprise (see e.g. Allan Reference Allan2003, Reference Allan2007; Penke & Rosenbach Reference Penke and Rosenbach2004a; Wunderlich Reference Wunderlich1976; Perry Reference Perry1980 etc.). By the late 1970s the debate had become fruitless, since the participants defended their views with reference to rigidly and mechanically applied abstract methodological principles. At the same time, they disregarded aspects relevant for the practising linguist as well as the arguments of the opposition camp (see Kertész Reference Kertész1991, Reference Kertész2004a; Kepser & Reis Reference Kepser, Reis, Kepser and Reis2005a; Penke & Rosenbach Reference Penke and Rosenbach2004a etc.).

From the middle of the 1970s – simultaneously with the gradual dying down of this controversy – a new phase of the debate on empiricalness unfolded as a consequence of several independent but converging developments in linguistics. The question by then was not whether hermeneutics or the hypothetico-deductive approach within the standard view of the analytical philosophy of science was more appropriate for capturing the structure of linguistic theories. Rather, once again, the conflict between inductive and deductive methodology already seen in the 1950s and 1960s moved to the centre of attention, but this time from another point of view (for an overview see e.g. McEnery & Wilson 1996: 1ff.; Lemnitzer & Zinsmeister Reference Lemnitzer and Zinsmeister2006: 7ff.). One¹³ – but in no way the only – development was the appearance and rapid spread of electronic data-processing. This led to the emergence of a tendency which turned against the use of introspective data preferred by generative linguists and contributed significantly to the development of corpus linguistics. The Brown Corpus of Standard American English, put forward by W. N. Francis and H. Kučera in the early 1960s, was a seminal turning point. Of similar importance was the Survey of English Usage associated with the name of Quirk, as well as the neo-Firthian COBUILD-project launched at the University of Birmingham by J. Sinclair and others. Generative linguists, however, considered the collection and analysis of great amounts of data stemming from language use trivial and irrelevant. Chomsky sharply questioned the relevance of induction at every stage of theory formation. Neither in the ‘discovery’ of theories (that is as a technique of data collection) nor in the ‘justification’ of theories did he attribute special significance to induction.¹⁴ As opposed to this, researchers in favour of corpus data argued for the indispensability of induction and the strict methodological rules concerning the handling of inductive data. They did not consider data based on the linguist's introspection as empirical.¹⁵

This means that with respect to the treatment of the data/evidence problem, basically two camps emerged: on the one hand, the advocates of corpus data and on the other hand generativists in favour of introspective data.¹⁶ Despite the considerable differences, however, the views share a series of assumptions which we call the standard view of linguistic data (SVLD):

1. (SVLD)

   1. (a) Not all linguistic theories can be considered as empirical but only those which rely on the appropriate type of data. Only certain data types may be regarded as relevant and legitimate: in the generativist view it is introspective data, while according to corpus linguists it is primarily corpus data.¹⁷

   2. (b) It is solely the origin of the data that decides what counts as the appropriate type of data.

   3. (c) Both camps regard their own data-handling procedure as unproblematic. They believe that a few general methodological rules are sufficient for controlling the reliability of the data.

   4. (d) They deem the relationship between the data and the hypotheses of the theory unidirectional, and determined by general, strict methodological rules. They explicate it either as induction from data to hypotheses, or as deduction from hypotheses to data.¹⁸

   5. (e) Evidence is considered as an empirical datum playing a pivotal role. Its significance lies in the features captured in (E). The most important of these is that it is assumed to constitute a directly controllable special subset of data, that is, it is qualified as reliable without reference to any theory. Thus, it is treated as an unquestionable fact suitable for the justification (verification, confirmation or falsification) of hypotheses.

The theses summarised in (SVLD)(a)–(e) witness that during the period we have overviewed, linguists took for granted the following background assumptions:

1. (a) The structure of data is simple; their origin determines what data types are legitimate. The treatment of data is unproblematic; a few general methodological rules are sufficient. The use of data in linguistic theories is unproblematic as well; as certainly true evidence, they allow for the justification (verification, confirmation or falsification) of theories.

2. (b) It is the standard view of the analytical philosophy of science that serves as a methodological background.

In connection with these assumptions, we highlight four points. First, they had been regarded for decades as the firm methodological foundation of linguistic theorising. Second, they reveal that the standard view of linguistic data and evidence (SVLD) is motivated basically not by linguistic but by general metascientific (philosophical) considerations. Third, they also show that (SVLD) is applied in some cases in an unreflected and unsystematic manner. For example, induction is often (also) regarded as a tool of hypothesis generation. Fourth, they constitute the point of departure for our solution to (P)_I to be summarised in Chapter 6.

In the 1980s and 1990s, the two camps’ position was still strong. Both of the two opposing views disregarded criticism by the other party. They seemed to coexist side by side without cooperation and without any chance of rapprochement. This could have resulted in the infertility, or even the termination of the empiricalness debate. However, the exact opposite of this happened. From the second half of the 1990s the methodological debates on data and evidence were gradually extended, and were enriched with new elements. Today they affect several different research fields and branches of linguistics. They concern not only the origin of linguistic data but questions regarding their function, structure, and the criteria for their acceptability, too. Previously, such questions had been neglected and had remained unreflected. In particular, at the turn of the millennium new works were published that changed the focus of the problem once again. As we have mentioned, in the 1970s the central problem was the confrontation between hermeneutics and the hypothetico-deductive branch of the analytical philosophy of science. In the 1980s and 1990s, this opposition was reduced to the dichotomy of introspective and corpus data. Today, however, the following questions are focused on: what types of data may be used, what data may count as evidence, and what function can be attributed to them in different fields of linguistic theorising?

This is the central problem of the fourth debate which has been conducted currently. It is this problem that we raised as (CP) in Chapter 1.¹⁹ The latest stage of the debate on data/evidence – in which each element of the standard view of linguistic data and evidence (SVLD), along with the two background assumptions just discussed were questioned – unfolded, in the first place, in the following publications:

1. (a) Schütze (Reference Schütze1996) is the seminal work which motivated the reappraisal of the problem of the structure and function of linguistic data in theoretical linguistics in the years immediately before the turn of the millennium.

2. (b) Studies in Language devoted a thematic issue to the problem of data/evidence based on the materials of the congress of the Deutsche Gesellschaft für Sprachwissenschaft in 2003. The papers in this issue provide us with a systematic overview of the main aspects of the data/evidence problem in several different fields of linguistics by confronting alternative positions (Penke & Rosenbach Reference Penke and Rosenbach2004b).²⁰

3. (c) The journal Lingua published a similar thematic issue on the problem of data in theoretical linguistics (Borsley Reference Borsley2005b).

4. (d) At the University of Tübingen there has been a research project in progress since 1999, investigating the problem of linguistic data (Reis et al. Reference Reis1998). The main results were presented at the conference on ‘Linguistic Evidence, Empirical, Theoretical, and Computational Perspectives’ in 2004 and published in Kepser & Reis (Reference Kepser, Reis, Kepser and Reis2005b).

5. (e) The continuation of this enterprise was published as Featherston & Winkler (Reference Featherston2009) and Winkler & Featherston (Reference Winkler and Featherston2009). The two collections of papers contain the proceedings of the international conference ‘Linguistic Evidence 2008’.

6. (f) The Linguistic Review devoted a thematic issue to the methodological problems in linguistics. One of its sections addressed the problem of data. The most relevant papers of this volume are those of Lehmann (Reference Lehmann2004), Mereu (Reference Mereu2004) and Simone (Reference Simone2004).

7. (g) In 2005, the Institut für Deutsche Sprache Mannheim also devoted a conference to the problem; the conference lectures were published in Kallmeyer & Zifonun (Reference Kallmeyer and Zifonun2007).

8. (h) Kristiansen et al. (Reference Kristiansen, Achard, Dirven and de Mendoza Ibáñez2006) discusses the problem of the empirical base of cognitive linguistics, highlighting the problem of data.

9. (i) There are also two further sub-branches of the data/evidence debate in linguistics. First, in 2007 Corpus Linguistics and Linguistic Theory published a special issue entitled Grammar without grammaticality (see Stefanowitsch & Gries Reference Stefanowitsch and Gries2007). Second, Theoretical Linguistics published a special issue entitled Data in generative grammar (see Sternefeld Reference Sternefeld2007). Since we discussed them separately in Kertész & Rákosi (Reference Kertész, Rákosi, Kertész and Rákosi2008b, c), we will refer to them in this book only sporadically.

10. (j) In the Forum Section of Zeitschrift für Sprachwissenschaft 28 (2009) discussion papers were published on the topic ‘Linguistic data: Elicitation – Evaluation – Theoretical implications’.

3.4 Conclusions

The latest literature listed above expressly calls attention to the importance of raising the problem of data and evidence in linguistics.²¹ It also emphasises that the current situation is not simply the automatic continuation of former debates, but rather an attempt to address new and more relevant issues.²² The problem of linguistic data and evidence is not restricted to a single field of linguistics, but has been acknowledged as relevant by a series of very different approaches and research areas in contemporary linguistic research.

At the same time, however, it is inseparably intertwined with metascientific background assumptions without the consideration of which it cannot be solved. Lehmann, for example, formulates this point as follows:

Accordingly, if we want to draw a realistic picture of the current stage of the debate on data/evidence, then there is no avoiding the examination of the role metascientific considerations play in the solutions that have been proposed to the problems raised. Thus, we will try to find an answer not only to (P)_I(a) but to (P)_I(b) as well.

Nevertheless, before presenting our solution to (P)_I(a) and (b), we will give a concise overview of some tendencies which illustrate how practising linguists gradually depart from the standard view of linguistic data and evidence (SVLD) while pursuing their research activities – without, however, declaring the abandonment of these views.

4.1 Overview

The aim of this chapter is to reveal a series of tendencies which suggest that the practice of linguistic research – that is, object-scientific inquiry – has clearly departed from the standard view of linguistic data (SVLD). The first tendency we mention is that novel methods and data types entered the scene which differ from those applied so far. Second, in order to support their hypotheses, some generative linguists make use of research results which, instead of being restricted to introspection as the sole data source, also take into consideration corpora or experiments. Third, an increasing number of factors have been identified that may influence the reliability of the data, thus witnessing that even the preferred data types are regarded as problematic. Fourth, not infrequently, the methodological rules which are assumed to govern the relation between data and theory by one of the versions of (SVLD), are violated. Finally, there is also a tendency towards the temporary rejection of counterexamples, which means that the way evidence is handled is not in accord with (SVLD).

From these new developments in object-scientific research, we will conclude that present-day linguistic practice implicitly tends to abandon the tenets of the standard view of linguistic data and evidence.

4.2 New tendencies in object-scientific research

The barren methodological debates of the 1980s and 1990s did not reflect on several tendencies related to the problem of data and evidence undoubtedly present in everyday linguistic research. These converging tendencies suggest that there is a wide gap between the practice of linguistic theorising and the metascientific principles summarised in the standard view of linguistic data and evidence (SVLD). Relying on the literature mentioned, below we give a brief overview of the tendencies which bear witness to the development that the practice of linguistic research tries to follow the methodological stances described in (SVLD) less and less rigidly, and that there is an increasing demand on the differentiation of the data to be made use of in linguistic research.

(i) Ad (SVLD)(a). New methods and data types appeared that question the categorisation described in (SVLD)(a). Approaches using probabilistic methods, for example, allowed a new interpretation of grammatical competence (see Penke & Rosenbach Reference Penke and Rosenbach2004a: 495ff.). Computer modelling also constitutes a new method which in principle should be rejected both by generative linguists accepting linguistic intuition as data and by linguists working with corpora consisting of linguistic utterances. The use of such data, however, may be fruitful, since they have been applied successfully in testing learning models which are not directly testable at the present state of neurology (see Penke & Rosenbach Reference Penke and Rosenbach2004a: 491).

Schütze (Reference Schütze2009) characterises the World Wide Web as a corpus which is, from several points of view, the best available. It is not only the largest one but it grows continuously and always remains up-to-date. It considerably increases the chances of finding examples which are relevant to the current research but which would not otherwise come to the linguist's mind. Despite this, the reliability of data originating from this source is often called into question and is usually checked against the linguistic intuition of the researcher or of native speakers. That is, data from the internet are rather a mixture of corpus and intuitive data.

Various kinds of experimental data constitute a data type which cannot be placed within a dichotomy of corpus data vs introspective data. As Schlesewsky (Reference Schlesewsky2009: 170) emphasises, questionnaires, priming experiments, eye movement studies or experiments conducting neuro-anatomic measurements put participants into an artificial situation and try to make them produce ‘interpretable’ reactions to a complex cognitive stimulus by eliminating as many external factors which might influence the results as possible.¹ From this we may conclude that experimental data based on participants’ responses to the stimuli are substantially different from both corpus data and introspective data: they are neither results of spontaneous language use like the former, nor are they regarded as representative of a homogeneous speech community as is the case with introspective data. Moreover, experimental data are in most cases more complex and involve the results of measurements related to manifestations of linguistic behaviour.

(ii) Ad (SVLD)(b). In order to support their hypotheses, some generative linguists – disregarding (SVLD)(b) – make use of research results which are not solely based on the analysis of introspective data, but arise from some kind of corpus or experiment. Linguists aiming at the identification of the principles of universal grammar, for example, found the results of typological, historical linguistic, neuro- and psycholinguistic research useful, and therefore integrated these in their investigations (see Penke & Rosenbach Reference Penke and Rosenbach2004a: 494f.). As Schlesewsky (Reference Schlesewsky2009: 174f.) remarks in connection with experimental data, in contrast to earlier practice, these new data types are more and more frequently regarded as equivalent or even superior to the traditionally accepted introspective data insofar as they no longer serve as mere illustrations of one's theoretical point of view but they may function as counterexamples whose explanation leads to the revision of the theory. At the same time, those working with corpus data rely on their linguistic intuition as well. First, the comparison of the meanings of constructions containing the linguistic structure at issue is an indispensable component of data analysis (see Borsley Reference Borsley2005a: 1477). Second, when processing corpora we have to distinguish sentences containing slips of the tongue and other defective utterances from grammatical sentences. To make such decisions, linguists fall back on their own linguistic intuition (see Schütze Reference Schütze1996: 2; Lehmann Reference Lehmann2004: 200, 207).

(iii) Ad (SVLD)(c). A further important development resulting from the deepening of linguistic research is that the researchers – breaking with (SVLD)(c) – no longer regard even the types of data they themselves prefer as unproblematic, since they identify more and more factors that may influence the reliability of data. The complexity of the data used and the number of the factors to be controlled are constantly increasing, while researchers often find themselves unable to control all factors involved. However, they do not try to overcome such problems with the help of general methodological principles, but strive to reveal those specific factors that influence the reliability of certain data types as accurately as possible. Let us mention a few examples.

As for corpora, for example, both ‘positive’ and ‘negative evidence’ raise serious problems that are either unsolvable with the methodologies at the researchers’ disposal or their solution is fraught with difficulties. Although ‘positive evidence’ may support a hypothesis, it is often difficult to decide how to differentiate relevant data from insignificant, isolated, unreliable, dubious information. Controlling the reliability of data inevitably leads to further searches for data and to the increased complexity of the theory – but there are obvious practical limitations to this process. ‘Negative evidence’ (that is, the absence of a certain form or construction) may provide one's hypotheses only with weaker supportive force. Moreover, it does not necessarily reveal whether the absence of a certain form or construction can be regarded as systematic, or is only accidental resulting from the contingency of certain circumstances (e.g. one did not examine the necessary amount of data, or one failed to notice the existence of a rare but existing and recordable form or construction). With ‘quantitative data’ the question is not simply whether a given phenomenon occurs or not, but the extent to which it does (i.e. whether its occurrence is statistically significant; see Penke & Rosenbach Reference Penke and Rosenbach2004a: 486; see also point (iv)). It has also been realised that the choice of the statistical method applied in corpus linguistic research is a factor which has a significant impact on the range and reliability of the results that can be achieved. For example, Arppe (Reference Arppe2009: 1ff.) states that linguistic phenomena whose complexity requires the use of a wide range of variables from different levels of linguistic analysis cannot be accounted for properly by focusing on only one level or one single feature. He suggests that for a satisfactory explanation of all relevant features as well as their interactions multivariate (multicausal) models are needed which might be capable of incorporating all of the variables into the analysis at the same time.

Questionnaires and interviews also raise a number of serious methodological problems which are well known from sociology. An example is the ‘observer's paradox’ which means that linguists usually want to find out how speakers naturally behave when they are not being observed, but this can only be discovered if the linguist observes them. One typical manifestation of this paradox is that informants often judge non-standard forms as bad, and distance themselves from them during the experiment, while, as a rule, they still use such forms actively (Penke & Rosenbach Reference Penke and Rosenbach2004a: 490; Schütze Reference Schütze1996: 61).

Experimental settings often involve a similar problem as well. So as to control the parameters that potentially influence the result, experiments may construct artificial situations which therefore yield less natural data. There is a relatively new insight among linguists working with experimental data that measurements – in contrast to the established view – do not mirror linguistic stimuli directly (see Schlesewsky Reference Schlesewsky2009: 170). Instead, the output of the stimulus processing includes – besides linguistic manipulation – several external factors such as the task environment, the capacity of memory or of the attention etc. whose impact on the results is not clear and which cannot yet be accounted for by any cognitive model. Moreover, the standards which could govern the experimental procedure and the evaluation of the results are missing. While in the natural sciences experimental data are controlled by the replication of the experiment, this is relatively rarely practised by linguists.

Schütze (Reference Schütze1996: 3f.) refers to similar problems discussed in the literature in connection with judgement data. First, besides the artificial nature of the elicitation situation, ‘the subject is being asked for a sort of behavior that, at least on the face of it, is entirely different from everyday conversation’. Metalinguistic tasks often relate to ‘areas of linguistic knowledge that rarely occur in regular speech’ (Schütze Reference Schütze1996: 91). This raises the question of whether and to what extent results obtained from research based on grammaticality/acceptability judgements² as data can be generalised to the whole of linguistic behaviour. Second, grammaticality judgements are no longer regarded as directly mirroring grammaticalness and linguistic competence but as products of metalinguistic performance. As such, they are interpreted as complex phenomena produced by a series of different cognitive skills. Thus, linguistic competence is only one of the contributing factors which are not directly accessible and whose role cannot be clarified without revealing the structure of the whole process of producing acceptability judgements, that is, without having any idea about the range of skills which contribute to this process and about the peculiarities of the contribution they make.

Schütze (Reference Schütze1996: chapters 4–5) provides an overview of the subject-related and task-related factors which, according to the literature, may distort the reliability of grammaticality judgements. Within the first group, field dependence (fusing aspects of the world and experiencing it globally vs differentiating information and experiences into components), handedness (which is supposed to be a good indicator of linguistic structures in the brain), age, sex and general cognitive endowment belong to the organismic related factors, while linguistic training, literacy and education, amount of exposure to a language, accumulated world knowledge etc. belong to the experiential factors. The second group consists of two subgroups, too. Procedural factors involve the instructions (that is, how the procedure of judging is explained to the subjects), the order of presentation of the sentences presented for judgement, repetition (the effect of repeated exposure to sentences on judgements of their grammaticality, and the interaction of repetition with other manipulations), mental state (the effect of objective vs subjective self-awareness), judgement strategy (focusing on syntactic and semantic structure vs looking at pragmatic use), modality (spoken vs written presentation of the stimuli) and register (clarity, awkwardness, slanginess, floweriness), and speed of judgement. Stimulus factors are, for example, context, meaning, parsability, frequency, lexical content, morphology and spelling, and rhetorical structure.

The problems related to judgement data also highlighted the need to change the data-handling techniques of generative linguistics and motivated the search for new, more reliable and useable data types. Schütze (Reference Schütze1996: 55ff.) mentions several proposals such as asking subjects to supplement their judgement with an explanation; asking subjects under what conditions, if any, the sentence at issue could be grammatical; requesting subjects to create rank orderings of sentences by grammaticality; asking for comparison of the types of violation in bad sentences; eliciting paraphrases for ambiguous sentences; using rating scales, or measurements of event-related brain potentials. Featherston (Reference Featherston2009b) compares the advantages and disadvantages and problematic points of three methods for gathering judgements, including the standard seven-point scale, magnitude estimation and his own method, which he calls ‘thermometer judgements’.

(iv) Ad (SVLD)(d). Breaking with (SVLD)(d), representatives of different approaches violate the methodological rules concerning the relation of data and theory required by one of the versions of the standard view of the analytical philosophy of science (SVAPS). First, among the linguists dealing with corpus data there are some who actually accept Popperian falsificationism, though they still assign induction an important – but not exclusive – role (see e.g. Geeraerts Reference Geeraerts2006: 24). Second, more and more researchers have realised that the Popperian model is too strict.

One of the strategies applied by linguists working with corpus data was the weakening of the criterion of falsifiability. Beside the ‘strong’ version of the principle (according to which a single counterexample is sufficient to falsify a hypothesis), in research practice there emerged a ‘weak’ version as well, stating that rules of language have to be interpreted not as strict prescriptions but as statistical tendencies. Thus, statistically rare occurrences are not sufficient to falsify a hypothesis. As Penke & Rosenbach (Reference Penke and Rosenbach2004a: 483) remark, it is not clear, however, how to distinguish exactly such rare occurrences from counterexamples that do falsify the hypothesis.

Another strategy, employed primarily by generativists, is the temporary ignoring of counterexamples. According to Chomsky (Reference Chomsky2002: 98ff.), this may be a rational decision in certain cases, because at a later stage of the development of the theory the tools might be available that allow for the resolution of the given inconsistency (Penke & Rosenbach Reference Penke and Rosenbach2004a: 484).³ In such cases, it would have been premature to reject the theory/hypothesis in question right at the outset.

(v) Ad (SVLD)(e). The temporary ignoring of counterexamples means a tentative departure from (SVLD)(e) as well, since one does not consider the counterexample as falsifying evidence, and the inconsistency is resolved through the – temporary – rejection of the given datum, and not that of the hypothesis at hand.

The insight mentioned in (iv), according to which both induction and deduction play a role in the process of theory formation also indicates the departure from (SVLD)(e) and at the same time from the rigid distinction between the ‘context of discovery’ and the ‘context of justification’ (see Geeraerts Reference Geeraerts2006: 24f.).

4.3 Conclusions

We may draw the conclusion that in the ongoing changes in research practice the intention to break with the views described in (SVLD) is clearly present. The question, therefore, is whether the changes in research practice are accompanied by changes in metascientific reflection, and if so, in what way, and to what extent.

In the next chapter, we will summarise the views on linguistic data presented in the current literature cited in Section 3.3. Although – as we will see – there are considerable differences between these views, they share some relevant factors. The latter make it possible to present their proposals related to (CP) on the basis of uniform criteria by highlighting the convergences and pointing out the divergences, too.

5.1 Overview

We tackle the issue raised at the end of Chapter 4, namely, whether the changes in research practice are accompanied by changes in metascientific reflection, and if so, in what way, and to what extent. By analysing current approaches to linguistic data and evidence from seven different points of view, we obtain the answer that they clearly intend to break with the standard view of linguistic data and evidence, but, corresponding to the seven points of view mentioned, they leave open the following questions: (OQ1) Under what conditions is the diversity of data legitimate and how can data belonging to different data types be combined? (OQ2) What role do sources play in the judgement of the reliability of data? (OQ3) How can the complexity of data be accounted for? (OQ4) How can the uncertainty of data be treated? (OQ5) What is the relationship between data and theories? (OQ6) On the basis of what criteria can it be decided whether a contradiction may be tolerated, and what strategies should we adopt for inconsistency resolution? (OQ7) What data can serve as evidence in linguistic theories, if, on the one hand, evidence has to be capable of supporting or weakening hypotheses, and, on the other, data cannot be regarded as true with certainty?

The state-of-the-art analysis of the literature presented in Sections 5.2–5.8 unveils the intrinsic double-facedness of the views surveyed. On the one hand, they still contain several components of the standard view of the analytical philosophy of science (SVAPS) and the standard view of linguistic data and evidence (SVLD). On the other hand, however, they also include ideas which are not compatible with (SVAPS) and (SVLD) and which clearly indicate the need to depart from the latter. Moreover, the literature goes at several points clearly beyond the old methodology by raising sophisticated, insightful and progressive suggestions.

5.2 The diversity and the combinability of data

Metatheoretical reflection on the nature of linguistic data – along the lines of the tendencies described in Section 4.2 (i) and (ii) – first of all has to supersede the dichotomy of introspective data vs corpus data that dominated the previous stage of the debate on empiricalness. It should take into account the development that a number of new data types and new methods of data-handling have appeared, mainly due to the deepening of linguistic research and the cooperation between linguistics and certain fields of cognitive science, sociology and computer science. Kepser & Reis (Reference Kepser, Reis, Kepser and Reis2005a: 1) mention, for example, the following types of data: introspective data, corpus data, data from (psycho)linguistic experiments, synchronic vs diachronic data, typological data, neurolinguistic data, data from first and second language acquisition, data from language disorders etc. Kepser & Reis (Reference Kepser, Reis, Kepser and Reis2005a), Penke & Rosenbach (Reference Penke and Rosenbach2004a), Borsley (Reference Borsley2005a) and Schütze (Reference Schütze1996) consider all kinds of information in connection with some aspect of linguistic behaviour as potential data.¹

Haspelmath (Reference Haspelmath2009) argues that although all data types are deficient and problematic, they are relevant and usable in different fields of linguistic research. His typology covers physiological data, acceptability judgements, judgements of meaning, translation, psycholinguistic tests, stimulated narratives, ordinary spontaneous corpora, and non-verbal data.

Lehmann's view, however, is different. Though he suggests a typology which allows us to grasp the diversity of linguistic data, his approach does not constitute a complete break with the standard view of linguistic data and evidence (SVLD). While he attempts to delineate a much more refined typology than the mere contrasting of introspective vs corpus data, he rejects the legitimacy of the former in the spirit of (SVLD):

Geeraerts (Reference Geeraerts2006: 27) shares a similar view, because he regards only experimental methods applied in psychology, methods of data collection used in sociology, and corpus analysis as legitimate and rejects the use of introspective data.

Summary. As we have seen, the standpoints mentioned go significantly beyond the dichotomy of corpus vs introspective data. Some representatives of current literature seem to consider all data types listed above (at least under certain conditions) as legitimate. Nevertheless, one of the otherwise most refined approaches – which undoubtedly elaborates a far more flexible taxonomy of data than most other attempts – fiercely rejects introspective data and in this respect remains close to the view described in (SVLD)(a) and (b). The following question remains open:

(OQ1)

Under what conditions is the diversity of data legitimate and how can data belonging to different data types be combined?

5.3 The role of the sources in the reliability of data

According to the literature, in exploring the properties of the individual data types, among other relevant factors, one has to consider those stemming from the diversity of their sources. Penke & Rosenbach (Reference Penke and Rosenbach2004a: 488ff.) distinguish two main groups of data on the basis of their sources. Spontaneous speech data are naturally occurring linguistic phenomena which, recorded in corpora, may serve as the subject of linguistic analysis. Corpora may be collected by the linguist herself with the specific aim of studying a certain linguistic structure, but she may also use public collections of data with no specific focus on a certain linguistic construction. Elicited data are gained by experiment or observation in a situation created and controlled by the linguist. Penke and Rosenbach are of the opinion that the two types of data have different advantages and disadvantages. The quality of spontaneous data is obviously influenced by the representativeness of the corpus and the systematicity of the way it has been collected. The reliability of elicited data crucially depends on the method of data recording, and the experimental technique used.

Lehmann (Reference Lehmann2004: 196ff.) applies a similar typology when – after rejecting introspection – he distinguishes two ‘serious ways’ of data collection: data discovered and data produced by the researcher. However, he clearly prefers the former group. He claims that at least in two respects data discovered better meet the very requirement of empiricalness which declares, among other things, that the properties of the objects under investigation shall be independent of the researcher (see Lehmann Reference Lehmann2004: 192). First, since in the case of research based on corpus data there is less possibility for manipulation of the data, it ensures a higher objectivity than research relying on other types of data. Second, he believes that corpora not produced by the linguist are varied, full of surprises, and therefore they serve as sources of ingenious insights undermining preliminary expectations (Lehmann Reference Lehmann2004: 201). Among the data produced by the researcher, Lehmann considers only data obtained from linguistic behaviour elicited by non-linguistic stimuli or controlled by the combination of different techniques as legitimate. He regards, for example, translation from one language to another as well as work with data elicited by metalinguistic procedures as a non-empirical, hermeneutic method (see Lehmann Reference Lehmann2004: 201ff.), because he considers the factors influencing the choice of relevant data, the reliability of the data and the evaluation of the results to be uncontrollable. Thus, he assumes, such data are acceptable only when they are used together with other data types. However, while he exposes the weaknesses even of data discovered with an extraordinarily keen eye and accuracy, in judging them he is more permissive than in the case of data produced, although there is close similarity between the problems in the two cases (see e.g. Lehmann Reference Lehmann2004: 200).²

Summary. As regards the role which data sources play, there has been a considerable change in attitude compared to the standard view of linguistic data and evidence (SVLD). Whereas (SVLD)(b) saw it as a crucial factor, in the literature we have surveyed it is simply one of the relevant factors. Further, while representatives of the standard view of linguistic data and evidence (SVLD) considered only one data source as legitimate, now a wide variety of sources is acknowledged. Nevertheless, in the papers examined here, this significant and progressive change is not accompanied by systematic investigations into the nature of data sources but happens in a rather inconsequential way. The following question is still unanswered:

1. (OQ2) What role do sources play in the judgement of the reliability of data?

5.4 The complexity of linguistic data

The views summarised in the standard view of linguistic data and evidence (SVLD) are regarded as insufficient by the authors mentioned not only because they narrow down the range of data, but – as (SVLD)(c) clearly demonstrates – they also neglect another factor of crucial importance: the complexity of data.

Kepser and Reis trace this complexity back to the fact that the object of linguistic inquiry is not directly accessible, but has to be reconstructed from the directly observable manifestations of linguistic behaviour (Kepser & Reis Reference Kepser, Reis, Kepser and Reis2005a: 1).

In accordance with this, Penke & Rosenbach (Reference Penke and Rosenbach2004a: 487ff.), too, distinguish between data reflecting knowledge of language directly and indirectly. They believe that corpus and intuitive data reflect aspects of knowledge of language in a more direct way than results from experiments which yield information, for example, about the properties of a certain linguistic construction only indirectly. The authors provide us with a detailed overview of the problems resulting from the complexity of data types described in Section 4.2 (iii); however, they do not attempt to systematise and generalise them.

Lehmann (Reference Lehmann2004) identifies considerably more aspects of the structure of linguistic data, proposing a sophisticated and comprehensive system of criteria:

1. In his view linguistic data are semiotic representations, that is, representations of particular aspects of speech events. This means that data are not given at the outset, but are – at least to a certain extent – produced by the researcher. Although they refer to some entity independent of the researcher, the identification of even the most elementary linguistic datum is based on abstraction and semiotic processes.

2. Primary data are representations of speech events with spatiotemporal coordinates. Secondary data do not possess historical identity, that is, cannot be linked to a certain place and time, and are therefore more abstract. Secondary data include types of primary data that, in turn, may be regarded as tokens. Metalinguistic statements also constitute secondary data, which describe different properties of speech events or capture generalisations; for example, sentences preceded by an asterisk marking ungrammaticality, or ‘negative data’ stating that a certain linguistic construction does not exist. Such data, Lehmann claims, are semiotic objects of a higher order in the sense that they contain the combination of meta- and object-language.

3. He also differentiates between raw data (non-symbolic representations) and processed data (symbolic representations). Raw data are iconic representations of speech events such as a tape recording. Linguists, however, almost never work with raw data, but process them by converting them into symbolic representations that highlight special properties of the speech event, and downplay features deemed irrelevant in the given inquiry. This, however, inevitably leads to the reduction and distortion of the speech event. Transcripts of linguistic utterances or phrase structure trees in syntax are good examples of symbolic representations.

4. Original recordings are based directly on the speech event, while derived representations are based on other representations, that is, on other data. For example, a recording of a conversation, for linguist A constitutes the data on the basis of which he carries out acoustic phonetic investigations. Linguist B, being a segmental phonologist, will consider the symbolic representation of the recording transcribed with the help of the IPA as data. Linguist C – who is a post-generative phonologist – complements this with symbols marking suprasegmental relations. Linguist D is a conversation analyst who uses one of the transcription systems of conversation analysis to produce a certain transcription of the conversation recorded that includes, among others, turn-taking. Linguist E is a grammarian who considers the latter as data which may help him elaborate a fragment of the grammar of the spoken version of a Swabian dialect near Tübingen. Linguist F considers the hypotheses of this grammar as data which he relies on in order to point out the differences between written and spoken language. The list of examples could be continued, and this perfectly illustrates at least two important issues. First, that something that constitutes the result of data analysis for one linguist, may serve as data for the other. Second, that defining the domain of the data does not depend on their inherent characteristics, but rather, on the particular interest, purpose, tools, conceptual framework and problems of the research at issue.

Summary. The authors seem to agree that (SVLD)(c) has to be given up, because it crucially hinders the efficient handling of the problems described in Section 4.2 (iii). Therefore, they are of the opinion that one of the inevitable tasks of the metatheoretical reflection on data consists in revealing their structure. At the same time, the attempts to capture the complexity of data make use of very different categorisations which are of different depths. Hence, the following question is still open:

1. (OQ3) How can the complexity of data be accounted for?

5.5 The treatment of the uncertainty of data

Current literature – breaking with the views summarised in (SVLD)(a)–(c), and more or less in accordance with the state of the art outlined in Section 4.2 (iii) – seems to hold that the acceptability of data can be judged on the basis of a complex process which has to consider the source and the structure of data along with certain properties of the theory at issue as well.

According to Kepser and Reis, the decision regarding the acceptability of the data is not to be restricted to the examination of a certain datum or data type. It has to consider all available data/data types and in addition further, partly non-linguistic factors as well. In this respect the authors significantly go beyond the monolithic character of (SVLD)(b):

Penke and Rosenbach are content to notice that data collection must be systematic, although the examples they mention clearly indicate that this criterion is not sufficient.

Borsley raises the problem of the reliability of data, too, but he leaves it open.³ Nevertheless, it is noteworthy that he – referring to Chomsky – does not deem, for example, grammaticality judgements incontestable. He assumes that the reliability of grammaticality judgements is influenced by several factors which, nevertheless, cannot be controlled completely (see Borsley Reference Borsley2005a: 1476f.).

Geeraerts thinks that the pluralism of methods and, accordingly, of data types is useful because they can supplement each other and thus provide more reliable results than when they are applied in isolation.⁴ However, he also maintains that it is necessary to establish a common, strict methodology which the methods have to be based on. One of the most important tasks linguistics has to solve is therefore the development of a theory-neutral empirical basis. He assumes that the hypotheses of linguistic theories should be formulated in such a way that they can be confronted with ‘observations’ (see Geeraerts Reference Geeraerts2006: 23ff.). In order to achieve this, a theory-independent observation language is needed in his view in which the data can be described and into which the hypotheses of the theory can be translated. However, with this claim he falls back on the acceptance of the standard view of the analytical philosophy of science (SVAPS).⁵

On the basis of the typology we summarised in Section 5.4, Lehmann (Reference Lehmann2004: 204ff.) formulates the generalisation that linguistic data are normally not speech events taking place in space and time, but rather, are representations located on different levels of abstraction and supplemented by different kinds of information. From this claim he concludes that data inevitably exhibit problem- and theory-dependent characteristics. First, they represent only certain aspects of the object of investigation, and second – since they are constructed through a series of intermediate stages – they simplify and distort the original. Nevertheless, he believes that the methodology which allows us to gain reliable data is available. In this respect he seems to agree with Geeraerts:

Therefore, he presumes that with the help of raw data the correctness of the representation can be checked.⁶ This assumption does not go beyond the views summarised in the standard view of linguistic data and evidence (SVLD). In this respect, Lehmann seems to advocate the standard view of the analytical philosophy of science (SVAPS) and assume that general methodological rules are capable of guaranteeing that data can be treated as firm facts. However, here one level of metareflection comes into conflict with another. As for the analysis of the internal structure of data, several of Lehmann's claims indicate that data – since they are representations – cannot be considered as ‘facts’ which are ‘true’ with certainty and are given at the outset.⁷ Rather, they are merely points of departure which need further refinement or which may turn out to be insufficient, not reliable enough or even useless (see e.g. Lehmann Reference Lehmann2004: 207). But, referring to the operational procedures as mentioned in the last quotation points in the opposite direction. The obtaining of perceptible and thus maximally reliable data is required not only as a norm but is supposed to be an objective which can easily be achieved.

Schütze (Reference Schütze1996: 1) does not believe that the methodological principles prevalent in the natural sciences, in sociology or psychology are infallible, although he, too, rejects theories obtained on the basis of uncontrolled data that have not been supported by experiments (see Schütze Reference Schütze1996: 4f.). He points out that the evaluation of the reliability of data is a highly complex task which cannot be separated from the process of theory formation. The factors influencing the reliability of data can be controlled or even eliminated only if the properties of their sources are systematically revealed – that is if hypotheses are put forward concerning the structure and functioning of data sources. Accordingly, the hypotheses concerning the reliability of data are part of the theory at issue.⁸ Schütze's strategy is fruitful since it encourages one to look actively for the unquestioned background assumptions which may turn out to be wrong and can decrease the reliability of data. This also means that the detection of the factors influencing the reliability of data, the formulation of hypotheses and their confrontation with the data cannot be separated.

While there seems to be a consensus about the necessity of developing new standards of data collection and data evaluation, there is disagreement among researchers about the content of the criteria to be applied. While Schütze (Reference Schütze1996) identifies, as we have seen in Section 4.2, a series of subject-related and task-related factors which might distort the results of experiments which elicit introspective judgements from the participants, Featherston (Reference Featherston2009a) expresses the opinion that the distorting effect of subject-related factors such as social, educational, professional, dialect background, age, sex, or speed of response required is negligible, and it is only some task-related factors in connection with the content, length, complexity and comprehensibility of the stimulus material which have to be controlled carefully.⁹ Fanselow (Reference Fanselow2009: 133ff.) takes an extreme position when he states that the practice of using uncontrolled introspective data in generative linguistics is fully legitimate and the research results based on them are not burdened with the effects of the application of ‘marginally acceptable’ or ‘unclear facts’. Moreover, he deems experimental data or corpus data in most cases useless in connection with questions regarding linguistic competence and proposes to restrict their application to ‘specific problematic cases’. He suggests that the use of acceptability experiments is not only superfluous but impossible, too, because of the practical limits of data collection.¹⁰ Acceptability judgements, in contrast, he labels as ‘self-psychical facts’ which can only be distorted by the influence of the theory preferred by the linguist. Fanselow takes the view that their use requires only the application of a few well-known maxims of caution such as the systematic variation of lexical material or care in the construction and evaluation of minimal pairs.

Summary. As we have seen, most authors recognise the significance of the problems related to the acceptability of data and try to find appropriate criteria. This clearly signals the refinement of metascientific reflection in current linguistic literature. However, there are obvious differences between the criteria proposed. None of the authors outlines a coherent solution to the problem of the acceptability of linguistic data which would really be able to capture the difficulties mentioned. Above all, the means which could capture the uncertainties the complexity of data inevitably yields are missing. None of the standpoints is able to resolve the tension between two poles. One is the recognition that the factors influencing the reliability of data cannot be controlled completely. The other is the intention inherited from the standard view of the analytical philosophy of science (SVAPS) that data should provide a firm basis for testing the hypotheses of the theory (see also Sections 5.6 and 5.8). Therefore, the following question has to be raised:

1. (OQ4) How can the uncertainty of data be treated?

5.6 The relationship between data and theory

In the metatheoretical literature of the fourth stage of the data/evidence debate in linguistics, the views concerning the relation between data and theory are still dominated by models based on induction and deduction.¹¹ Nevertheless – in accordance with the tendencies mentioned in Section 4.2 (iv) – the authors also formulate ideas which go beyond (SVLD)(d). First, it is claimed that during scientific theorising, the relation between data and theory cannot be reduced simply to the linear application of the deductive or inductive method, but is, rather, cyclic in nature:

Second, some authors assume that the selection of data is problem- and theory-dependent.¹² Featherston, for instance, suggests a close connection between the complexity of data and the complexity of their application in the process of linguistic theorising:

Haspelmath (Reference Haspelmath2009) is of the opinion that in principle, all data types which have been applied in linguistics so far are relevant for linguistic inquiry. Nevertheless, peculiarities of the problem under investigation impose strict limits on their application. In possession of the currently available linguistic data it is possible, for example, to raise questions in relation to the rules of language use of a given language and to the rule types of the languages in the world. The study of questions pertaining to the mental grammar of a speaker, to the biological basis of mental grammar, or to the origin of the limitations which can be revealed in the rules of the world's languages, however, leads only to highly speculative answers on the basis of the data currently available. Therefore, answers to these questions would clearly require the application of new data types.

This theory- and problem-dependence is, however, also incompatible with the linear relation between data and theory. Namely, if the selection of data is significantly influenced by the properties of the particular problem to be solved¹³ as well as by the conceptual and methodological background of the theory,¹⁴ then the relationship between data and theory cannot be reduced to the confrontation of the hypotheses with experience – that is, to the justification of the hypotheses by data independent of the theory.

Featherston places great emphasis on the equal importance of hypothesis generation and hypothesis testing and assigns data a vital role in both of them, without, however, raising the question of how data which serve as the starting point of hypothesis generation can be interpreted without a theoretical framework:

Lehmann (Reference Lehmann2004: 191) raises the issue of the relation between theory and data on the basis of more refined guidelines than other authors. He attributes three functions to data. First: during theory formation (this corresponds to the ‘context of discovery’ in the sense of the analytical philosophy of science), data work as the basis of induction in the process of finding the hypotheses. Second: they are the means of testing hypotheses obtained by deduction (this seems to correspond to the ‘context of justification’). Finally, with respect to the presentation of the theory, they play the role of evidence serving the persuasion of the reader in the course of argumentation. However, he does not clarify the relation of these functions to each other. This leads, of course, to considerable difficulties.

One problem arises from the circumstance that, as we have just mentioned, due to the role they play in the discovery of hypotheses, he assumes data to be inevitably theory- and problem-dependent:

In spite of this, as we have seen in Section 5.5, Lehmann also assumes that data can fulfil their second function, that is, the testing of hypotheses, only if the effects of the relativistic elements associated with them are sorted out and the return to the least theory- and problem-dependent kind of data, namely, raw data, is secured.¹⁵

However, he expressly asserts that although the recursion to raw data may make the data more reliable, it cannot guarantee their certainty.¹⁶ On the basis of this, Lehman puts forward the desideratum that data should be – at least to a certain extent – independent of the researcher.¹⁷ This is in contrast to the hypothesis of the theory- and problem-dependence of data. He does not ask the question of why and to what extent this partial theory-independence could be sufficient for the fulfilment of the control function of data.

The fact that the relation between the three functions of data has not been clarified raises a second problem as well. Namely, Lehmann associates a particular criterion with the argumentative function of data which is at variance with the criteria connected to the previous two functions:

This formulation characterises data as the unquestionable basis of theory formation. However, this unquestionability is not traced back to some kind of objective criterion but seems, rather, to be interpreted relativistically by being associated with the argumentative role data may play in the theory at issue. If this is so, then it should follow that data are theory-dependent – and this leads to a contradiction (see also Section 5.8).

Summary. The views we have surveyed witness the difficulties resulting from the incompatibility of the theory- and problem-dependence, the complexity and the uncertainty of the data with the norms of the standard view of the analytical philosophy of science (SVAPS). The latter prescribes the rigid distinction between the ‘context of discovery’ and the ‘context of justification’, and requires, therefore, a unidirectional relation between the data and the theory. The views we have discussed, however, assume this relation – as a result of the acceptance and legitimisation of object-scientific practice – to be much more complicated. Nevertheless, since they have no alternative metatheoretical framework at their disposal, these views remain incomplete and problematic in several respects. The relationship between data and linguistic theories, that is, the function of data in linguistic theories, remains unclarified. Therefore, the following question remains open:

1. (OQ5) What is the relationship between data and theories?

5.7 Data and the treatment of inconsistencies

Another relevant property of linguistic data is that they may be the source of contradictions. The most trivial manifestation of this is that linguists are continuously confronted with ‘exceptions’ or ‘counterexamples’ whenever they put forward hypotheses.

Referring to Chomsky (Reference Chomsky2002: 98ff.), Penke & Rosenbach (Reference Penke and Rosenbach2004a: 484) emphasise that in linguistic theorising there are situations in which the immediate resolution of a contradiction triggered by data does not appear to be necessary. The fact that there are data contradicting the hypotheses does not necessarily have to lead to the rejection of the theory. It may be reasonable to ignore them till, in a later stage of the research, the tools become available that allow for the theory to explain their behaviour:

However, they contrast this strategy without comment with the standpoint of corpus linguists and others opposing generative grammar who reject this way of ignoring data which contradict the hypotheses.¹⁸ Moreover, Penke and Rosenbach do not attempt to clarify the relation between the above strategy, on the one hand, and the ‘strong’ and ‘weak’ versions of falsification we mentioned in Section 4.2 (iv), on the other.

In another context, Kepser & Reis (Reference Kepser, Reis, Kepser and Reis2005a: 3), too, emphasise that contradictions resulting from the diversity of data may be fruitful.¹⁹ Different data types may lead to contradictory hypotheses, but these contradictions may be resolved and thus play a progressive role in the inquiry.²⁰

Nevertheless, the substantial problem is that, in spite of these important diagnoses, the authors mentioned do not attempt to reveal the strategies of inconsistency resolution related to data.

As regards the latter, Schütze (Reference Schütze1996: 4) notes that in linguistics the techniques of inconsistency resolution which have been applied in the natural sciences are absent, but he neither clarifies what techniques he has in mind nor supports this claim with references.

With respect to corpora, Geeraerts assumes that it is statistical methods that might provide the means of inconsistency resolution.²¹ His suggestion, just like the previous ones, touches on important aspects of the treatment of contradictions, but does not break radically with the views summarised in the standard view of linguistic data and evidence (SVLD). Rather, it seeks more refined solutions within the boundaries of the standard view of the analytical philosophy of science (SVAPS). In a series of cases he accepts Popperian falsificationism without critique, although he questions its applicability in certain situations. In particular, if one finds a small number of occurrences of a certain linguistic phenomenon in a corpus, then it might be the case that these are marginal data which cannot be considered as falsifying or confirming evidence. In such cases, possible alternatives should be examined by comparing them with the help of statistical methods. But this procedure leaves a series of questions unanswered. First, it does not resolve, for instance, those situations in which there is a conflict between two hypotheses supported by different types of data approximately to the same degree. Second, it cannot be applied to every type of data. Third, in contrast to Schütze, who mentions only the criteria for selecting the ‘more reliable’ hypothesis (see Schütze Reference Schütze1996: 4), Geeraerts seems to assume that statistical methods are entirely objective and yield unambiguous results. However, it is well known that the choice of the test applied and the interpretation of its results depends – at least to a certain extent – on the researcher's decision; another test may lead to another result, thus generating inconsistency. It is also often the case that the repetition of the experiment with the same or with similar materials leads to the opposite outcome. Statistical methods do not yield ultimate results, either; they may only increase the reliability of one's conclusions gained from the examination of data.²² With respect to experiments, Geeraerts refers to experimentum crucis²³ – but of course he can do so only in the spirit of the analytical philosophy of science.

Summary. It is an important and progressive insight of the literature analysed that inconsistencies arising from the use of different types of data are natural parts of linguistic theorising. This insight signals the departure from the standard view of the analytical philosophy of science (SVAPS) and the convergence with new tendencies in the philosophy of science. Inconsistency in science is one of the most important topics of contemporary philosophy of science (see Meheus Reference Meheus2002). Therefore, the development that the issue of inconsistency is raised in the works we examined is a very progressive step towards making the methodology of linguistics more up-to-date. Nevertheless, there are only a few works which systematically discuss the role inconsistency plays in linguistic theorising (Moravcsik Reference Moravcsik, Eid and Iverson1993, Reference Moravcsik2006, Reference Moravcsik2010; Kertész Reference Kertész2004a, b; Kertész & Rákosi Reference Kertész and Rákosi2006, Reference Kertész, Rákosi, van Heusden and Wildgen2009a, c). Nevertheless, there is no agreement on the answer to the following questions: what are the principles which one should apply to decide when a certain contradiction can be tolerated and when it cannot? How can inconsistencies be resolved? The views of the authors cited are incomplete. Thus, they are not capable of deciding between the strategies outlined in Section 4.2 (iv) or of putting forward new methods. This means that the treatment of inconsistency must also be considered an open question:

1. (OQ6) On the basis of what criteria can it be decided whether a contradiction may be tolerated, and what strategies should we adopt for inconsistency resolution?

5.8 Linguistic evidence

The latest literature does not explicate the notion of ‘linguistic evidence’. The publications mentioned in Section 3.3 clearly indicate the confusion characterising the use of this notion as well as a tendency which allows us to highlight certain of its relevant aspects. From the circumstance that none of the authors define the notion of evidence, it need not, of course, necessarily follow that there is confusion around its interpretation. It would also be possible that its meaning is clear and familiar to everyone. However, if we carefully examine the way the authors use this term, it will turn out that it is applied in different senses and incoherently.

In Penke & Rosenbach (Reference Penke and Rosenbach2004a) the terms ‘data’ and ‘evidence’ occur synonymously at several places. For example, when data types are discussed, on page 485 we find the expression ‘qualitative vs quantitative data’ whereas on the next page ‘qualitative vs quantitative evidence’ (emphasis added). They classify direct and indirect evidence as data types, too, but – interpreting and relating the two categories in an entirely different way – they also claim that one of the functions of data is to serve as evidence for hypotheses.²⁴ In this way – following the intentions of the standard view of the analytical philosophy of science (SVAPS) – they interpret evidence as a special, designated and secure subset of data whose task is the testing of hypotheses. However, they think that there are no general methodological criteria on the basis of which it can be decided which data are capable of serving this function, because theory- and problem-dependent aspects have to be considered, too. Thus, in this respect, they go beyond (SVAPS).²⁵ They also refer to potential conflicts between the problem- and the theory-dependence of data. Namely, it may happen that at a certain point of research particular types of data seem to be relevant which the methodology of the theory either does not acknowledge or even forbids. They think that in such cases it is the ‘open-mindedness’ of the linguist that may further the research. By this they mean that, in order to find an effective solution to the problem at issue, linguists are often ready to consider and to treat as evidence such data which their theory does not permit. However, the methodological norms on the basis of which the control of the data's reliability could be associated with a certain phase within the process of theorising remain unclear. They do not reveal the relations which hold between the hypotheses of the theory, its methodological background assumptions, the criteria of the acceptability of data, and the process of object-theoretical activity. Although Penke and Rosenbach touch on the problem that the confrontation of the hypotheses with the data may lead to the revision of methodological principles – for example, to the modification of the range of the acknowledged data types – they do not integrate the mutual influence of data and the hypotheses of the theory into the process of theorising.

The standpoint of Kepser and Reis is similar. They hold that the selection of evidence from among the set of data is a complex and far from trivial task. This task cannot be carried out solely on the basis of general methodological norms, independently of concrete linguistic research (Kepser & Reis Reference Kepser, Reis, Kepser and Reis2005a: 1). The question of what counts as evidence can be answered only with respect to the particularities of the theory at issue and those of a concrete problem. Namely, the latter factors decide whether certain data or a certain data type can be regarded as adequate and relevant, that is, if it is capable of supporting or refuting some hypothesis of the theory:²⁶

Since they think that the adequacy of the data can be decided only on an individual level, with respect to the particular problem at issue, they do not go into the problem of adequacy. At the same time, they emphasise that certain data can serve as ‘true evidence’ if they are reliable and reproducible:

In this way they assume that an empirical basis can be established which makes it possible to treat certain data as ‘true evidence’. This means, however, that Kepser and Reis’ view is double-faced in a similar way to Penke and Rosenbach's. They admit the uncertainty and fallibility of data rooted in their complexity, but at the same time, in accordance with the standard view of the analytical philosophy of science (SVAPS), they try to select those data which are expected to secure a firm and theory-independent base for testing the theory. Kepser and Reis seem to assume that theory formation has to be preceded by the selection of reliable and reproducible evidence and that the process of theory formation is to be regarded as linear. Nevertheless, this procedure is not able to provide such a firm empirical basis for theory formation. In particular, so as to select the components of the basis – that is to control the linguistic and non-linguistic factors that may influence the reliability of the data – one or more further empirical theories would be needed; these further theories, however, rely on data whose own reliability can be controlled only with the help of further theories, and so on ad infinitum.

Moreover, as Lehmann shows with the help of several examples, unlike, for example, in physics or other natural sciences, in certain kinds of linguistic research data are not reproducible, but rather, individual and unique:

As regards Lehmann's view, first, as we have seen in Section 5.6, he maintains that one function of data is that they play ‘the role of unquestionable evidence in the argumentation’ (Lehmann Reference Lehmann2004: 181). In this way, he reverses the relation between data and evidence: it is not the case that evidence has to be selected from the set of data, but rather, being evidence is the precondition of becoming a datum. It is worth noting that in the above quotation Lehman relates the question of whether something is evidence to the argumentation at issue. This implies that it is not the abstract logical structure of the theory that is focused on, but its argumentation structure. This claim might have led to a radical turn in the reconstruction of linguistic theories. However, Lehmann did not draw this conclusion, as the following quotation witnesses:

Second, it is easy to see that the above quotation is incompatible with the second quotation by Lehmann in Section 5.6 (p. 33 above). This claims that if the representation of a certain speech event plays the role of unquestionable evidence in the argumentation, then it functions as a datum in the research at issue. In contrast, in the former quotation evidence is merely a rhetorical tool serving to persuade the reader.

Third, in accordance with the standard view of the analytical philosophy of science (SVAPS), Lehmann seems to assume that with respect to theories it is their logical structure that is relevant, whereas argumentation structure may play a role only in the presentation of the theory. However, this stance is not compatible with the clause in the first sentence of the last quotation which says that the linguist takes the data for granted – we might also say that they are considered as ‘unquestionable evidence’. This means that Lehmann raises but leaves unexplicated the relation between the logical structure of linguistic theories and linguistic argumentation. In this respect, he still holds a position akin to the standard view of linguistic data and evidence (SVLD) in that – similarly to Geeraerts and Kepser & Reis – he considers evidence as a firm empirical basis independent of theories.

Summary. We may conclude that in characterising the notion of evidence the literature is far from unanimous. On the one hand, the view summarised in (SVLD)(e) still prevails, as well as a series of methodological principles borrowed from the standard view of the analytical philosophy of science (SVAPS) – above all, the assumption that a certain group of data can be selected that works as a firm basis during the testing of the hypotheses. On the other hand, all the different standpoints we discussed clearly include components which are no longer compatible with (SVAPS). For example, the theory- and problem-dependence of data has been realised, as well as the insight that the relation between the hypotheses and data which may be more or less uncertain is much more sophisticated than (SVAPS) assumes: namely, that the data do not confirm or falsify the hypotheses once and for all. However, the methodological principles which could provide the metatheoretical explication of the new insights are completely missing. Thus, the following question remains to be answered:

1. (OQ7) What data can serve as evidence in linguistic theories, if, on the one hand, evidence has to be capable of supporting or weakening hypotheses, and, on the other hand, data cannot be regarded as true with certainty?

5.9 Conclusions

The above state-of-the-art analysis of the literature has revealed two facts, among others. On the one hand, in the views we have surveyed, components of the standard view of the analytical philosophy of science (SVAPS) and the standard view of linguistic data and evidence (SVLD) are still present. On the other hand, the same approaches include sophisticated, insightful and progressive suggestions as well, ideas which are not compatible with (SVAPS) and (SVLD) and which clearly indicate the need to depart from the latter. In their proposals for (CP), they differentiate, relativise and weaken the requirements linguistic data and evidence are expected to meet.

Differentiation means that the authors do not seek to define the notion of ‘linguistic datum’ on the basis of the properties of just one data type. Rather, they acknowledge the pluralism of linguistic data in that it is not just one type of data that may be legitimate. They strive to put forward a rich system of criteria which allows us to reveal the internal structure of different data types and to elaborate the typology of data as applied in different linguistic theories. They consider linguistic data to be complex entities whose reliability is influenced by the peculiarities of their structure and source.

Relativisation means that the evaluation of the applicability of linguistic data is not assumed to be epistemologically independent of the linguistic theory at issue, but rather, data and evidence are regarded as problem- and theory-dependent.

By weakening we mean that these approaches treat not only statements which capture directly given linguistic utterances with spatiotemporal coordinates as empirical data, but abstractions from utterances at different levels, too, which therefore unavoidably idealise and reduce the speech events in certain ways. Moreover, they permit different versions of falsifiability, and they also touch on the temporary tolerability of inconsistencies arising from the application of different data types.

Consequently, as a general conclusion, the only claim we are allowed to make is that the demand for a departure from the standard view of linguistic data and evidence (SVLD) and the standard view of the analytical philosophy of science (SVAPS) can be identified in these views, but there is no dominating tendency fundamentally overruling them. (SVLD) is not replaced by a radically new perspective, but rather, by a double-faced tendency. This tendency comprises a series of different views which have in common, first, that they are still close to (SVLD) and (SVAPS) in several respects, and second, that they try to go beyond the latter by putting forward relevant new suggestions, although in several respects they do this inconsistently.