1.1 Writing the Article

A central goal of scientific writing is to convey what was actually done so that the methods and procedures can be replicated. Concrete, specific, operational, objective, and precise are some of the characteristics that describe the writing style. The effort to describe research in concrete and specific ways is critically important. However, the task of the author goes well beyond description.

Preparation of the report for publication involves three interrelated tasks that I refer to as description, explanation, and contextualization. Failure to appreciate or to accomplish these tasks serves as a main source of frustration for authors, as their papers traverse the process of manuscript review toward journal publication. Description is the most straightforward task and includes providing details of the study. Even though this is an obvious requirement of the report, basic details often are omitted in published articles (e.g., sex, socioeconomic status, and race of the participants; means and standard deviations) (e.g., Reference Case and SmithCase & Smith, 2000; Reference Gerber, Arceneaux, Boudreau, Dowling, Hillygus, Palfrey, Biggers and HendryGerber et al., 2014; Reference Tate, Perdices, Rosenkoetter, McDonald, Togher, Shadish, Horner, Kratochwill, Barlow, Kazdin, Sampson, Shamseer and SampsonTate et al., 2016). Omission of basic details can hamper scientific progress. If a later study fails to replicate the findings, it could be because the sample is very different along some dimension or characteristic. Yet, we cannot surmise that without knowing at least basic details of the sample in both studies. If a study does repeat the findings, that is important, but is the new finding an extension to a new type of sample? Again, we need basic information in the studies to allow such comparisons.

Explanation is more demanding insofar as this refers to presenting the rationale of several facets of the study. The justification, decision-making process, and the connections between the decisions and the goals of the study move well beyond description. Here the reader of the manuscript has access to the author decision points. There are numerous decision points pertaining to such matters as selecting the sample, choosing among many options of how to test the idea, selecting the measures, and including various control and comparison groups. The author is obliged to explain why the specific options elected are well suited to the hypotheses or the goals of the study. There is a persuasion feature that operates here. The author of the manuscript is persuaded that the decisions are reasonable ways to address the overriding research question. Now the author must convey that to persuade the reader. In other words, explanation conveys why the procedures, measures, and so on were selected, but that explanation ought to be cogent, persuasive, and above all explicit. We do not want the reader to think, “This is an important research question, but why study it that way?” And in many cases, the related prior question of the same ilk emerges, why do we even need this study or why is study important? For the many decision points beginning with selection of the research question, these are very reasonable questions that the author ought to anticipate and pre-empt.

Finally, contextualization moves one step further away from description and addresses how the study fits in the context of other studies and in the knowledge base more generally. This latter facet of the article preparation reflects such lofty notions as scholarship and perspective, because the author places the descriptive and explanatory material into a broader context. Essentially, the author is making the case for the study based on the knowledge base. Relatively vacuous claims (e.g., this is the first study of this or the first study to include this or that control condition or measure) are rarely a strong basis for the study and often means or is interpreted as meaning that the author could not come up with something better. Without context, any “first” is not very important by itself. Indeed, it is easy to be first for a topic that is not very important and has been purposely neglected. We need a more compelling rationale.

For example, if this study is done on why people commit suicide we need the context of why this specific study ought to be done and where in the puzzle of understanding this piece fits. Perhaps prior research omitted some critical control procedure; perhaps there is a special group that has a novel characteristic that reduces (or increases) the likelihood of suicide that would inform the field in unique ways; or perhaps some new twist on a theory or intervention will have clear implications for reducing suicide attempts. These and other such comments convey three points that are wise to address: (1) there is a gap in knowledge, (2) that gap is important, and (3) that gap will be filled in whole or in part by this study.

1.2 General Comments

The three components I identified vary in difficulty. When individuals write their first project for publication, they focus heavily on the descriptive part to make sure all the material and sections are included. And this part is fundamental. Explanation and contextualization are much more difficult. Explanation requires having considered options and conveying to the reader why the one selection was a good choice. Yet one’s first study is often with or from an advisor who has made these decisions and the bases of these decisions might be buried in one of the advisor’s other articles but otherwise is tacit. As authors we need to be prepared for other scientists looking at our paper and doing their job by asking, “Why on earth did we [use: that population, measure, control condition, means of data evaluation, and so on]”. These are not only legitimate questions but are central to science.

Contextualization is even more difficult. Contextualization benefits from experience, scholarship, time, and knowledge of as many related areas of work as one can bring to bear. How is the study connected to the literature or topic, how does it relate to theory, to other disciplines, to a critical problem we ought to care about or that is now facing society? The puzzle analogy might help. A given study is one puzzle piece and merely showing that piece to someone is not inherently interesting. It may be inherently boring. Yet, the piece becomes more interesting as all the other pieces are shown (e.g., the outside box with a full photo of the puzzle) and even more interesting, fascinating actually, if one can paint a verbal picture of the whole puzzle and show how one or two pieces are needed and this study is that part! Explanation gives the rationales for decisions; contextualization determines whether the study is compelling or not. Authors often complain that the reviewers did not understand, “get it,” appreciate the importance of their study. The authors are usually completely right, but guess whose responsibility that is?

The extent to which description, explanation, and contextualization are accomplished increases the likelihood that the report will be viewed as a publishable article and facilitates integration of the report into the knowledge base. Guidelines are provided later in the chapter to convey these tasks more concretely in the preparation and evaluation of research reports. The guidelines focus on the logic of the study, the interrelations of the different sections, the rationale for specific procedures and analyses, the strengths and limitations, and where the study fits in the knowledge base. Consider main sections of the manuscript that are prepared for journal publication and how these components can be addressed.Footnote ¹

2.1 Title

The title of an article includes the key variables, focus, and population with an economy of words. The special features of the study are included to convey the focus immediately to potential readers. It is critical here to be direct, clear, and concise (e.g., “Memory loss and gains associated with aging” or “Predictors of drug use and abuse among adolescents”). These examples are especially concise. Ordinarily an author is encouraged to fit the title within 10–12 words. The words ought to be selected carefully. Titles occasionally are used to index articles in large databases. Words that are not needed or that say little (e.g., “preliminary findings,” “implications,” “new findings”) might be more judiciously replaced by substantive or content words (e.g., among preschool children, the elderly; consequences for sleep and stress) that permit the article to be indexed more broadly than it otherwise would have been.

Occasionally, comments about the method are included in the title or more commonly in the subtitle. Terms like “a pilot study” or “preliminary report” may have many different meanings, such as the fact that this is an initial or interim report of a larger research program. These words could also be gently preparing readers for some methodological surprises and even tell us not to expect too much from the design. These qualifying terms might be accurate, but they implicitly apologize or ask for mercy as well. Better to give a strong title and in the write up give the explanation (decision making) and contexts to convey why this study was done, where it fits in the scheme of this literature, and why this was important. No apologies needed; just let the reader know your thinking on the matter. Although I am reluctant to boast, my dissertation won a prize for the best qualifying terms in a title. (In the subtitle of my dissertation, I conveyed this as: “A pre-preliminary, tentative, exploratory pilot study©.”)

In some cases, terms are added to the study such as, “A Controlled Investigation,” which moves our expectation in the other direction, namely, that the present study is somehow well conducted and controlled, and perhaps by implication stands in contrast to other studies in the field (or in the author’s repertoire). Usually words noting that the investigation is controlled are not needed unless this is truly a novel feature of research on the topic. Some words when added can be important because they are novel. An example would be the subheading, “A replication.” That is important because replications are of interest and not too often published. They have taken on even increased importance given the concerns in science that many studies produce findings that are not replicable. Another word to add as a subtitle might be: A review or A meta-analysis. These are important to convey that the article is not an individual investigation but an evaluation of a broad literature.

Occasionally authors are wont to use titles with simple questions, “Is depression really a detriment to health?” or “Is childhood bullying among boys a predictor of domestic violence in adulthood?” In general, it is advisable to avoid “yes, no” questions in the title. Science and findings are often nuanced and findings are likely to be both yes and no but under very different circumstances or for some subgroups of people but not for others. As an example, consider a hypothetical yes–no question for the title of a study as, “Is cigarette smoking bad for one’s health?” For anyone on the planet, the answer might be a resounding yes. Yet, the yes–no nature of the question makes this a poor choice of title because the answer is likely to depend on either how smoking is defined (e.g., how much smoking – a cigarette a year, a pack after each meal) and how health is defined (e.g., mental, physical, what diseases, disorders). Very familiar is how horrible smoking is for one’s physical health in so many domains (e.g., heart disease, cancer, chronic respiratory disease), but the question in the title can be answered both yes and no. Less familiar is the fact that cigarette smoking and exposure to cigarette smoke (among nonsmokers) reduce the risk for Parkinson’s disease and there are reasonable explanations for that based on brain chemistry and neurotransmitters (Reference Ma, Liu, Neumann and GaoMa et al., 2017; Reference Miller and DasMiller & Das, 2007). Clearly, the hypothetical title is plainly simplistic and not very helpful or informative because we can show many circumstances in which yes and no are correct answers. I am not arguing in favor of cigarette smoking (although I used to be a chain smoker until I switched to cigarettes). I am advising against titles of empirical articles that have a yes–no question given that most answers involve essays. Few phenomena allow the simplistic thinking the question can reflect.

2.2 Abstract

The Abstract is likely to be read by many more people than is the full article. The Abstract will be entered into various databases and be accessible through Internet and online searches. Many journals list the tables of contents for their issues and provide free access on the Web to abstracts of the articles but charge for the full article. Consequently, the Abstract is the only information that most readers will have about the study. For reviewers of the manuscript and readers of the journal article, the Abstract conveys what the author studied and found. Ambiguity, illogic, and fuzziness here are ominous. Thus, the Title and Abstract are sometimes the only impression or first impression one may have about the study.

Obviously, the purpose of the Abstract is to provide a relatively brief but comprehensive statement of goals, methods, findings, and conclusions of the study. Critical methodological descriptors pertain to the participants and their characteristics, experimental and control groups or conditions, design, and major findings. Often space is quite limited; indeed a word limit (e.g., 150–250 words maximum) may be placed on the abstract. It is useful to deploy the words to make substantive statements about the characteristics of the study and the findings, rather than to provide general and minimally informative comments. For example, vacuous statements (“Implications of the results were discussed” or “Future directions for research were suggested”) ought to be replaced with more specific comments of what one or two implications and research directions are (e.g., “The findings suggest that the family and peers might be mobilized to prevent drug abuse among adolescents and that cultural influences play a major role.”). Also, the more specific comments can convey the study’s relevance and interest value beyond what is suggested by the manuscript title or the opening comments of the Abstract. As a reader, I am not going to read very eagerly an article with the vacuous “implications” or “future directions” sentences, but if I am interested in the specific topics mentioned as implications (brain activity, the immune system, family, peers, culture), this article is a must for me to read. As authors, we often lament the word restrictions placed on us in the Abstract, but the first task is to make sure we are using the existing allotment with maximum information.

2.3 Introduction

The Introduction is designed to convey the overall rationale and objectives. The task of the author is to convey in a crisp and concise fashion why this study is needed and the current questions or deficiencies the study is designed to address. The section should not review the literature in a study-by-study fashion, but rather convey issues and evaluative comments that set the stage for the study. Placing the study in the context of what is and is not known (contextualization) and the essential next step in research in the field requires mastery of the pertinent literatures, apart from reasonable communication skills. Ironically, mastery of the literature is needed so the author knows precisely what to omit from the Introduction. A vast amount of material one has mastered and that is very interesting will need to be omitted because it does not set the stage or convey the precise context for this study.

Saying that the study is important (without systematically establishing the context) or noting that no one else has studied this phenomenon (measure or sample) usually are feeble attempts to short-circuit the contextualization of the study. In a manuscript I reviewed, the author mentioned four times in the Introduction (and three more in the Discussion) that this was the first time this study has been done. This was not amusing. Someone had not advised or helped the author very much and a very poor case was made for the study. Among the tasks of the Introduction is to lead the reader to the conclusion that the study is important and worthwhile. Telling the reader that the study is important is an argument from authority and that is not how science works. This might even strongly suggest that the author has not done his or her contextualization homework.

It may be relevant to consider limitations of previous work and how those limitations can be overcome. These statements build the critical transition from an existing literature to the present study and the rationale for design improvements or additions in relation to those studies. It is important to emphasize that “fixing limitations” of prior work is not necessarily a strong basis for publishing a study. The author must convey that the limitations of prior work are central to a key building block in theory or the knowledge base. Convey that because of that specific limitation, we really do not know what we thought we did or that there is a new ambiguity that is important but hidden in prior studies considering what was studied and by what means. Alternatively, the study may build along new dimensions to extend the theory and constructs to a broader range of domains of performance, samples, and settings. The rationale for the specific study must be very clearly established. Theory and previous research usually are the proper springboard to convey the importance of the current study.

In general, the Introduction will move from the very general to the specific. The very general refers to the opening of the Introduction that conveys the area, general topic, and significance of a problem. For example, in studies of diagnosis, assessment, treatment, or prevention of clinical dysfunction, the Introduction invariably includes a paragraph to orient the reader about the seriousness, prevalence or incidence, and economic and social costs of the disorder. Reviewers of the manuscript are likely to be specialists in the area of the study and hence know the context very well. Yet, many potential readers would profit from a statement that conveys the significance, interest, and value of the main focus of the study.

After the initial material, the Introduction moves to the issues that underlie this specific study. Here the context that frames the specific hypotheses of the study are provided and reflect theory and research that are the impetus for the investigation. There is an introduction syllogism, as it were, a logic that will lead the reader from previous theory and research to the present study with a direct path. Extended paragraphs that are background without close connections to the hypotheses of the study serve as a common weakness of manuscripts rejected for publication.

The Introduction does not usually permit us to convey all the information we wish to present. In fact, the limit is usually 2–5 manuscript pages. A reasonable use of this space is in brief paragraphs or implicit sections that describe the nature of the problem, the current status of the literature, the extension to theory and research this study is designed to provide, and how the methods to be used are warranted. The penultimate or final paragraph of the Introduction usually includes a statement of the purpose of the study and the specific hypotheses and predictions. By the time the reader reaches this paragraph or set of paragraphs, it should be very clear that these hypotheses make sense, are important, and address a critical issue or need in the knowledge base. In short, the Introduction must establish that the study addresses a central issue. To the extent that the author conveys a grasp of the issues in the area and can identify the lacunae that the study is designed to fill greatly improves the quality of the report and the chances of acceptance for journal publication. By the time the readers arrive at the purpose of the study or hypotheses paragraph, they should be nodding enthusiastically and saying to themselves, “This study is really needed, it should have been done years ago, I am so glad this is being done now.” As authors we often believe a description of the study is all that is needed. The identical study (description of what was done) can be viewed as a weak and just another study or strong, compelling, and sorely needed. All this can be decided by how the Introduction is cast.

2.4 Method

This section of the paper encompasses several points related to who was studied, why, and how. The section not only describes critical procedures, but also provides the rationale for methodological decisions. Subject selection, recruitment, screening, and other features ought to be covered in detail. Initially, the subjects or clients are described. Why was this sample included and how is this appropriate to the substantive area and question of interest? In some cases, the sample is obviously relevant because participants have the characteristic of interest (e.g., parents accused of child abuse, siblings of children with autism spectrum disorder) or are in a setting of interest (e.g., daycare center, wilderness camp). In other cases, samples are included merely because they are available. Such samples, referred to as samples of convenience, may include college students or a clinic population recruited for some other purpose than to test the hypotheses of this study. The rationale for the sample should be provided to convey why this sample provides a good – or if not good, a reasonable – test of the hypotheses and whether any special features may be relevant to the conclusions.

The design is likely to include two or more groups that are treated in a specific fashion. The precise purpose of each group and the procedures to which they are exposed should be clarified. Control groups should not merely be labeled as such with the idea that the name is informative. The author should convey precisely what the group(s) is designed to control. The author is advised to identify the critical methodological concerns and to convey how these are controlled in the design. Plausible threats to experimental validity that are uncontrolled deserve explicit comment to arrest the reasonable concerns of the reviewers (see Reference KazdinKazdin, 2017).

Several measures are usually included in the study. Why the constructs were selected for study should have been clarified in the Introduction. Then the specific measures and why they were selected to operationalize the constructs should be presented in the Method section. Information about the psychometric characteristics of the measures is often highlighted. This information relates directly to the credibility of the results. Apart from individual assessment devices, the rationale for including or omitting areas that might be regarded as crucial (e.g., multiple measures, informants, settings) deserves comment.

Occasionally, ambiguous statements may enter into descriptions of measures. For example, measures may be referred to as “reliable” or “valid” in previous research, as part of the rationale for their use. There are, of course, many different types of reliability and validity. It is important to identify those characteristics of the measure found in prior research that are relevant to the present research. For example, high internal consistency (reliability) in a prior study may not be a strong argument for use of the measure in a longitudinal design where the author cares more about test–retest reliability. Even previous data on test–retest reliability (e.g., over 2 weeks) may not provide a sound basis for repeated testing over annual intervals. The author ought to present information to convey the suitability of the measures for the study.

It often appears that reliability and validity of assessment are not routinely taught, at least if one looks at Method sections of articles in clinical psychology. These are important concepts because they can determine what is measured by a given instrument and how well. One sees more routinely that authors report Cronbach’s alpha for a measure and then move on. Alpha is one measure of reliability (internal consistency) and can be very useful to know. However, this has little to do with validity of the measure and by itself is not a justification for using a specific measure without much more explanation. Perhaps add a couple of sentences in this section to comment specifically on reliability and validity and what types have been supported in prior research. This is not merely to convince a reader but also ourselves on the wisdom of electing this measure. It is unreasonable to expect the measures to have the ideal reliability and validity data that the investigator would like to make a flawless case for use of these measures. Yet, make the case from what psychometric data there are. If data are not available, include some analyses in the study to suggest the measure(s) behave in ways that suggest pertinent forms of reliability or validity (Reference KazdinKazdin, 2017).

2.5 Results

It is important to convey why specific statistical tests were selected and how these serve the goals of the study. A useful exercise is for the investigator to read that paragraph about hypotheses and predictions from the Introduction and then immediately start reading the Results section, i.e., for the moment completely bypass the Methods. The results ought to speak directly to and flow from that narrative statement in the Introduction.

Analyses often are reported in a rote fashion in which, for example, the main effects are presented and then interactions for each measure. The author presents the analyses in very much the same way as the software output. Similarly, if several dependent measures are available, a set of analyses is automatically run (e.g., omnibus tests of multivariate analyses of variance followed by univariate analyses of variance for individual measures). The tests may not relate to the hypotheses, predictions, or expectations outlined at the beginning of the paper. It is important that the statistical tests be seen and presented as tools to answer questions or enlighten features of those questions and to convey this to the reader. The reader should not be able to legitimately ask, “Why was that statistical test done?” Knowledge of statistics is critical for selecting the analyses to address the hypotheses and conditions met by the data. Yet, as important in the presentation is to convey precisely why a given statistical test or procedure is suitable to test the hypotheses and then again what the results of that test reveal in relation to those hypotheses.

It is often useful to begin the Results by presenting basic descriptors of the data (e.g., means, standard deviations for each group or condition), so the reader has access to the numbers themselves. The main body of the Results is to test the hypotheses or to evaluate the predictions. Organization of the Results (subheadings) or brief statements of hypotheses before the analyses are often helpful to prompt the author to clarify how the statistical test relates to the substantive questions and to draw connections for the reader.

Several additional or ancillary analyses may be presented to elaborate the primary hypotheses. For example, one might be able to reduce the plausibility that certain biases may have accounted for group differences based on supplementary or ancillary data analyses. Ancillary analyses may be more exploratory and diffuse than tests of primary hypotheses. Manifold variables can be selected for these analyses (e.g., sex, race, height differences) that are not necessarily conceptually interesting in relation to the goals of the study. The author may wish to present data, data analyses, and findings that were unexpected, were not of initial interest, and were not the focus of the study. The rationale for these excursions and the limitations of interpretation are worth noting. From the standpoint of the reviewer and reader, the results should make clear what the main hypotheses were, how the analyses provide appropriate and pointed tests, and what conclusions can be reached as a result. As in other portions of the manuscript, how the author has reached a decision (what analysis) and why are very important.

2.6 Discussion

The Introduction began with a statement of the need for this study and issues or lacunae in theory or research the study was designed to address. The Discussion continues the storyline by noting what we know now and how the findings address or fulfill the points noted previously. With the present findings, what puzzle piece has been added to the knowledge base, what new questions or ambiguities were raised, what other substantive areas might be relevant for this line of research, and what new studies are needed? From the standpoint of contextualization, the new studies referred to here are not merely those that overcome methodological limitations of the present study, but rather focus on the substantive next steps for research. Also, this is not the place for vacuous suggestions such as, “This study needs to be replicated with people who are …”. If one is suggesting an extension of the study to different subjects, settings, or other dimensions, specify exactly why this specific extension would be of special interest.

The Discussion usually includes paragraphs to provide an overview of the major findings, integration or relation of these findings to theory and prior research, limitations and ambiguities and their implications for interpretation, and future directions. These are implicit rather than formally delineated sections and the author ought to consider the balance of attention to each topic. Usually, the Discussion is completed within 3–5 manuscript pages.

Description and interpretation of the findings can raise a tension between what the author wishes to say about the findings and their meaning versus what can be said in light of how the study was designed and evaluated. It is in the Discussion that one can see the interplay of the Introduction, Methods, and Results sections. For example, the author might draw conclusions that are not quite appropriate given the method and findings. The Discussion may convey flaws, problems or questionable methodological decisions within the design that were not previously evident. That is, the reader of the manuscript can now state that if these are the statements the author wishes to make, the present study (design, measures, or sample) is not well suited. The slight mismatch of interpretative statements in the Discussion and Methods is a common, albeit tacit basis for not considering a study as well conceived and executed. A slightly different study may be required to support the specific statements the author makes in the Discussion. It is important to be precise about what can and cannot be asserted in light of the design and findings.

It is usually to the author’s credit to examine potential limitations or sources of ambiguity of the study. A candid, non-defensive appraisal of the study is very helpful. Here too, contextualization may be helpful because limitations of a study also are related to the body of prior research, what other studies have and have not accomplished, and whether a finding is robust across different methods of investigation. Although it is to the author’s credit to acknowledge the limitations of the study, there are limits on the extent to which reviewers grant a pardon for true confessions. At some point, the flaw is sufficient to preclude publication, whether or not the author acknowledges it. For example, the authors of the study might note, “A significant limitation of the present study is the absence of a suitable control group. We are aware that this might limit the strength of the conclusions.” Awareness here does not strengthen the demonstration itself. A huge limitation in the study is sufficiently damaging to preclude drawing valid inferences.

In noting the limitations of the study, there is a useful structure for the presentation. First note the limitation. Then discuss to the extent reasonable that this limitation is not likely to influence the conclusion (if this is the case). If the role of the limitation cannot be diminished or dismissed by sound reasoning or related data, note that addressing this issue is a logical if not important next step for research. All studies have limitations by their very nature so reasoning about their likely and unlikely impact on the findings is invariably relevant.

At other points, acknowledging potential limitations conveys critical understanding of the issues and guides future work. For example, in explaining the findings, the author may note that although the dependent measures are valid, there are many specific facets of the construct of interest that are not covered. Thus, the results may not extend to different facets of the construct as measured in different ways. Here too it is useful to be specific and to note precisely why other constructs and their measure might show different results. In short, be specific as to why a limitation or point might really make a difference. This latter use of acknowledgment augments the contribution of the study and suggests concrete lines of research.

4.1 Impetus for Reporting Guidelines

There have been a long-series of guidelines on how to conduct and report research and these are directly related to preparation of a study for publication. The history includes special emphasis on ethical treatment of participants. Regulations followed in response to atrocities of the Nazi regime during World War II and the resulting development of the Nuremburg Code (1940). Since then, many other codes have developed (e.g., Declaration of Helsinki of the World Medical Association, Belmont Report), beyond the scope of the present discussion (see Reference KazdinKazdin, 2017). Protection of participant rights remain as important as ever and even of greater concern considering new opportunities to obtain and combine data sources (“big data”), often with information that is public in some way (e.g., medical records, social media, tracking locations and purchases). Individuals may not be aware of the collection and use of the information. Even when participants may be anonymous, in fact often groups (e.g., by ethnic, culture or setting) can be readily singled out and identified in ways that can reflect quite negatively on them (e.g., Reference Metcalf and CrawfordMetcalf & Crawford, 2016; Reference ZimmerZimmer, 2010).

The need for guidelines for reporting research has emerged from multiple additional concerns. First, collaborative research currently is more the rule rather than the exception in science. Collaborations often involve scores of authors, from multiple disciplines, and from many different countries. There is interest across nations in reaching common standards in relation to the openness of research, access to information, the merit-review process, and ethical issues (e.g., Reference SureshSuresh, 2011). That has provided a critical context for providing guidelines for research and the reporting of research that span multiple disciplines, countries, and journals.

Second, lapses in what is reported in research have been well documented. For example, information often is omitted such as exactly who the participants are (e.g., subject and demographic variables) and how they were recruited, who administered treatment or experimental procedures, the extent of their training, whether the integrity or execution of treatment was assessed, fundamental characteristics of the data evaluation, and more, as reflected in citations noted previously.

Third, selective reporting of results and data analyses has been raised as a critical issue that introduces biases in individual studies and entire literatures. The selective reporting of results of some data analyses or some of the dependent measures can increase the likelihood of more chance findings in the literature (Reference Simmons, Nelson and SimonsohnSimmons et al., 2011). For example, in identifying evidence-based treatment, often there is a clear bias in how authors report the data by not presenting the full range of measures, some of which would not support conclusions about the impact of treatment (Reference De Los Reyes and KazdinDe Los Reyes & Kazdin, 2008). Guidelines are intended to foster consistency and clarity in how the study will be reported to minimize the biases in reporting that emerge.

Fourth, more flagrant than “mere” omission of information and selective reporting has been fraud and fabrication of data in science. Fraud is not new in science. However, both the visibility of fraud to the public, including the circulation through social media, and direct and disastrous implications from fraudulent studies have been more evident than ever before (see Reference Levelt and CommitteesLevelt, Noort, & Drenth Committees, 2012; Reference Watanabe and AokiWatanabe & Aoki, 2014).Footnote ²

Finally, there has been renewed concern about the replicability of research. Replication as a general tenet, if not practice, has always been the backbone of science. Given many of the points I have already mentioned (publication biases, statistical analyses, and selective reporting), various authors have reached the dramatic conclusion, occasionally supplemented with mathematical proofs and simulations, that that many and even most published research findings are not correct, i.e., are false (see Reference FrancisFrancis, 2012; Reference IoannidisIoannidis, 2005; Reference Moonesinghe, Khoury and JanssensMoonesinghe et al., 2007). Several calls for increased replication have been made. Psychology has taken the lead in calling for and supporting replications and underscoring the importance of transparency of procedures (Center for Open Science, https://cos.io/). Central to replication, of course, is making the procedures explicit and the materials and results available. Guidelines for conducting research to increase the likelihood that a study can be replicated are obviously important. Many journals, national and international, require providing information about a study and the data so they are freely available to others to facilitate re-evaluation of the data and replication of the entire study.

Overall, science has come under increased scrutiny both from within the sciences, government, and the public at large. Even though the assorted problems I highlighted are seemingly infrequent, the circulation of information (e.g., Web, news and social media) is more extensive than ever before and retractions (when authors and journals make some effort to “take back” and renounce what was published) are more visible and available as well. And news media more routinely comment on scientific findings and reflect skepticism about replication and replicability of effects (e.g., Reference LehrerLehrer, 2010). The points I have raised have served in part as the impetus for improving research, especially focusing on transparency and accountability of investigators. Guidelines have been helpful in fostering greater consistency in reporting of the research and in the process sensitizing researchers of what to attend to in advance of a study.

4.2 Sample Guidelines Briefly Noted

Several organizations and groups have developed standards for reporting research and in the process convey the need to address many of the issues I have highlighted previously. The scope of guidelines that are available is enormous. An international umbrella organization that collects, oversees, and promotes the use of research guidelines is the Equator Network. The network maintains a comprehensive database of guidelines, numbering over 400, as of this writing (www.equator-network.org/reporting-guidelines/). The network is nicely organized by type of paper (guidelines for empirical studies, literature reviews, meta-analyses) and by different methodologies (e.g., randomized trials, observational studies), and so on. With so many guidelines, with enormous overlap in what they cover, one can see this is not a minor movement to improve research.

Examples of such standards are the:

• Consolidated Standards of Reporting Trials (CONSORT; Reference Moher, Schulz and AltmanMoher et al., 2001);

• International Clinical Trials Registry Platform (ICTRP; World Health Organization, www.who.int/ictrp/en/);

• Transparent Reporting of Evaluations with Nonexperimental Designs (TREND; Reference Des Jarlais, Lyles and CrepazDes Jarlais et al., 2004);

• Strengthening for Reporting of Observational Studies in Epidemiology (STROBE; Reference Von Elm, Altman, Egger, Pocock, Gøtzsche and VandenbrouckeVon Elm et al., 2007);

• Journal Article Reporting Standards for Research in Psychology (JARS; Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and RaoAppelbaum et al., 2018; Reference Levitt, Bamberg, Creswell, Frost, Josselson and Suárez-OrozcoLevitt et al., 2018);

• Standards for Reporting on Empirical Social Science Research in American Educational Research Association (AERA) publications (AERA, 2006);

• Transparency and Openness Promotion (TOP) Guidelines (Reference Aalbersberg, Appleyard, Brookhart, Carpenter, Clarke, Curry, Dahl, DeHaven, Eich, Franko, Freedman, Graf, Grant, Hanson, Joseph, Kiermer, Kramer, Kraut, Karn and VazireAalbersberg et al., 2018);

• Materials Design Analysis Reporting (MDAR; Reference HawkinsHawkins, 2019); and

• Meta-Analytic Reporting Standards (MARS; Reference Kepes, McDaniel, Brannick and BanksKepes et al., 2013).

Most of the guidelines include some combination of checklists, flow charts, and narrative explanations of what specific items are to be included in a report and what the information is designed to accomplish. I mention two briefly.

First, the CONSORT standards, mentioned above, are arguably the most familiar set of guidelines. They have been adopted by hundreds of professional journals encompassing many disciplines and countries (see www.consort-statement.org/). The CONSORT guidelines have been devised primarily for clinical trials in medical research but have extended well beyond that and are routinely used in clinical trials of psychosocial interventions. As noted in the most recent version, clinical trials have a history of omitting significant information such as description of who was included in the study; sample size calculation (e.g., why a specific size was included in relation to statistical power issues), descriptions of procedures; and presentation of procedures (e.g., randomization) that were not really invoked; and so on with other lapses (Reference Moher, Schulz and AltmanMoher et al., 2001). Beginning in the early 1990s, efforts began to make recommendations for reporting of studies and from that the CONSORT guidelines emerged.

The guidelines consist of a checklist of essential items that ought to be included in any randomized controlled trial of treatment. The checklist displays what is needed, but along with the checklist is a detailed explanation of the items and their rationale for inclusion (Reference Moher, Schulz and AltmanMoher et al., 2001). In addition, the website provides educational material and a database of materials related to reporting of randomized controlled trials (e.g., examples from real trials). In preparation of journal articles, the CONSORT criteria include a list of what to cover and how. This is more concrete than my general statements of ensuring there is a logical flow and underlying theme to the journal article. Yet, the details are basic and critical and hence these guidelines are valuable and widely adopted by many journals.

Second, ClinicalTrials.Gov (https://clinicaltrials.gov/) provides another model to guide research. This consists of preregistration of a study that requires authors to convey their plan for conducting research and analyzing the data. Preregistration allows for the range of participants in research (investigators, peer reviewers, journal editors, funding agencies, policy makers, the public at large) to determine whether the research, when completed, has deviated from the pre-registered plan. Pre-registration of research is now common across many funding agencies and journals (Reference Nosek, Ebersole, DeHaven and MellorNosek et al., 2018). ClinicalTrials.Gov is a large database that includes privately and publicly funded studies of investigations throughout the world. Indeed, this is the largest clinical trials database and as of this writing over 327,000 studies are registered and include studies from all 50 states in the United States and 209 countries (as of January 2020). When clinical trials are comparing interventions or an intervention against a control group, funding agencies (e.g., National Institutes of Health), organizations (e.g., World Health Organization), and a consortium of journal editors (the International Committee of Medical Journal Editors) require investigators to register their clinical trials in advance of the study.

In registration of one’s study, information covers diverse facets of the project. Indeed, there is a multi-page template that includes identification of the investigators of the study, the design, what the interventions are, what will be the outcome criteria (e.g., primary and secondary), the number of anticipated subjects, criteria for inclusion of the subjects, status of procedures to protect clients, and much more. Merely mentioning some of the domains that are included in the register does not underscore their significance in the conduct and reporting of the study. Consider three examples to convey the point.

First, the guidelines require specification of the outcome criteria and which outcomes or measure will be primary and secondary. This is a pre-commitment of the investigator to be clear about the outcome. This does not mean that investigators cannot look at all outcome measures or derive new ones based on interesting findings, pre-specified or not, as the study is completed. However, pre-specification can reduce the tendency in written reports to underscore, emphasize, and consider as primary, those measures that “come out,” i.e., support the hypotheses.

Second, in many studies there are multiple investigators whose roles vary in the design, execution, analysis and other facets of the study and these investigators are likely to be listed as authors. Specification of the investigators and their roles clarifies accountability for the final manuscript. Also, this requires that people in fact have a role in the study before being included. All the expected dynamics of human interactions (e.g., who does and does not get to be an author, where they are placed in the list of authors) and human emotions (e.g., indignation, disappointment, rage, helplessness) surround authorship. War stories here could fill volumes. The guidelines can help a little. At the end of a study, there is accountability of who is in charge, who had what role, and who was involved. If there is fraud or faked data or questionable practices or a manuscript retraction (once there is a question about practices or a finding), the team involved in the study and their roles can be delineated. This can enhance the integrity of the research progress by making clear that one is accountable for the study and its conduct. Third, the guidelines specify whether, how, and where the data will be stored and whether other materials critical to the study will be available. Occasionally journals or funding agencies require that the data are deposited and made available. This practice fostered by the guidelines increases the transparency of the research but also helps replication efforts.

A few comments in passing. To begin, pre-registration does not fix the research so that no further changes can be made. In fact, one can update changes that occur in the course of a study. “Pre”-registrations can be updated after participant enrollment or even after data collection has begun to document any changes that occur in the course of a study (Reference DeHavenDeHaven, 2017; Reference Nosek, Ebersole, DeHaven and MellorNosek et al., 2018). All that is required is to make sure the changes are clear, transparent, and explained. The registration still thwarts post-hoc decision making based on how the data come out or switching some measures and ignoring others, some of the sins of research. An additional point, the many guidelines are designed to improve reporting of research. However, so many facets need to be considered ahead of time in these guidelines that they necessarily influence and guide the design of a study. This article is about preparing a manuscript for journal publication. Consulting and following many widely adopted guidelines underscores the point that key issues about the publication of a journal article emerge before the first subject is run in the study.

6.1 Overview of the Journal Review Process

Alas, through careful deliberation and 30 minutes with your coauthor at a Ouija board, you select a journal and are ready to submit your manuscript for publication. Before you do, consult the Instructions to Authors written in the journal to make sure you submit the manuscript correctly. Usually manuscripts are submitted through a journal portal, i.e., electronically, in which the manuscript file and a letter of submission are uploaded to the journal website. In some cases, you may be required to include sentences or paragraphs in the letter you submit that say this study is not being considered elsewhere in another journal, has not been published before, has met ethical guidelines specified by university or institutional policy and various laws, and that you will give the copyright to the publisher if the manuscript is accepted. Processing of the manuscript could be delayed if your letter does not meet the guidelines provided in the journal.

Once the manuscript is submitted, the journal editor usually sends the electronic file to two or more reviewers who are selected because of their knowledge and special expertise in the area of the study or because of familiarity with selected features of the study (e.g., novel methods of data analyses). Reviewers may be selected from the names of authors whose articles you included in your Introduction. Some reviewers are consulting editors who review often for the journal and presumably have a perspective of the type and quality of papers the journal typically publishes; other reviewers are ad-hoc reviewers and are selected less regularly than consulting editors. Reviewers are asked to evaluate the manuscript critically and to examine whether or the extent to which:

• The question(s) is important for the field;

• The design and methodology are appropriate to the question;

• The results are suitably analyzed;

• The interpretations follow from the design and findings; and

• The knowledge yield contributes in an incremental way to what is known already.

(You may note that these bulleted points encompass the explanation and contextualization features I noted in relation to manuscript preparation. Each point is one that can be readily addressed by the author in preparing the manuscript.) Typically, reviewers are asked to give a summary recommendation (e.g., reject or accept the manuscript). All recommendations to an editor are advisory and not binding in any way. At the same time, the editor sought experts and usually follows their recommendations. Yet reviewers too must make the case for their comments.

Once the paper is reviewed, the editor evaluates the manuscript and the comments of the reviewers. In some cases, the editor may provide his or her own independent review of the paper; in other cases he or she may not review the paper at all but defer to the comments and recommendations of the reviewers. The editor writes the author and notes the editorial decision. Usually, one of three decisions is reached: the manuscript is accepted pending a number of revisions that address points of concern in the reviewers’ comments; the manuscript is rejected and will not be considered further by the journal; or the manuscript is rejected but the author is invited to resubmit an extensively revised version of the paper for reconsideration.

The accept decision usually means that the overall study was judged to provide important information and was well done. However, reviewers and the editor may have identified several points for further clarification and analysis. The author is asked to revise the paper to address these points. The revised paper would be accepted for publication.

The reject decision means that the reviewers and/or editor considered the paper to include flaws in conception, design, or execution or that the research problem, focus, and question did not address a very important issue. For journals with high rejection rates, papers are usually not rejected because they are flagrantly flawed in design. Rather, the importance of the study, the suitability of the methods for the questions, and specific methodological and design decisions conspire to serve as the basis for the decision.

The reject–resubmit decision may be used if several issues emerged that raise questions about the research and the design. In a sense, the study may be viewed as basically sound and important but many significant questions preclude definitive evaluation. The author may be invited to prepare an extensively revised version that includes further procedural details, additional data analyses, and clarification of many decision points pivotal to the findings and conclusions. The revised manuscript may be re-entered into the review process and evaluated again as if it were new. On other occasions, the manuscript may be resent to reviewers familiar with the prior version. Less often the editor may make an executive decision and accept or reject the manuscript without outside input.

Of the three letters, clearly a rejection letter is the most commonly received. Authors, and perhaps new authors in particular, are insufficiently prepared for this feature of the journal publication business.Footnote ⁴ Journals often publish their rejection rates, i.e., proportion of submitted manuscripts that are rejected, and this figure can be quite high (e.g., 70–90 percent). Often the prestige value of the journal is in part based on the high rejection rate. Yet, the rate is ambiguous at best because of self-screening among potential authors. For example, for very prestigious publication outlets (e.g., Psychological Review, Science) where psychological papers are published, the rejection rates cannot consider the fact that most authors are not likely to even try that outlet if they have a contribution that falls within the topic and format domain. Rejection rates across journals are not directly comparable. Even so, the rates give the would-be author the approximate odds if one enters the fray.

Although beyond our purpose, the review process deserves passing comment. The entire process of manuscript submission, review, and publication has been heavily lamented, debated, and criticized. The peer-review process has a long history as an effort of quality control over the content and standards of what is published (Reference SpierSpier, 2002). The topic is central to science broadly and continues to be assessed, commented on, and evaluated with efforts to alter or improve the processes (e.g., Reference Elson, Huff and UtzElson et al., 2020; Reference Kirman, Simon and HaysKirman et al., 2019). The alternatives to peer review (e.g., no review, judgment by one person such as the editor) have their own liabilities. Many journals invoke procedures where the identity of the authors and the reviewers is masked, i.e., names are not included on the manuscript sent to reviewers or the reviews sent to authors. The goal is to try to limit some of the human factors that can operate about responses to a person, name, or other facet and to allow reviewers to be candid in their evaluations without worrying about facing the colleague who will never speak to them again. The peer-review system is far from perfect. The imperfections and biases of peer review, the lack of agreement between reviewers of a given paper, the influence of variables (e.g., prestige value of the author’s institution, number of citations of one’s prior work within the manuscript) on decisions of reviewers, and the control that reviewers and editors exert over authors have been endlessly vigorously discussed (e.g., Reference Bailar and PattersonBailar & Patterson, 1985; Reference Benos, Bashari, Chaves, Gaggar, Kapoor, LaFrance, Mans, Mayhew, McGowan, Polter, Qadri, Sarfare, Schultz, Splittgerber, Stephenson, Tower, Walton and ZotovBenos et al., 2007; Reference CicchettiCicchetti, 1991; Reference SmithSmith, 2006; Reference Stahel and MooreStahel & Moore, 2014).

Understanding the review process can be aided by underscoring the one salient characteristic that authors, reviewers, and editors share, to wit, they are all human. This means that they (we) vary widely in skills, expertise, perspectives, sensitivities, motives, and abilities to communicate. Science is an enterprise of people and hence cannot be divorced from subjectivity and judgment. In noting subjectivity in the manuscript review and evaluation process, there is a false implication of arbitrariness and fiat. Quality research often rises to the top and opinions of quality over time are not idiosyncratic. Think of the peer-review process as the home-plate umpire in a baseball game. Any given call (e.g., strike) may be incorrect, arguable, and misguided. And any given pitcher or batter suffers unfairly as a result of that call. As reviewers (the umpires) make the call on your manuscript (rejection, you strike out), you too may have that occasional bad call. But over time, it is unlikely that all manuscripts an author submits receive a misguided call. Pitchers and batters earn their reputations by seeing how they perform over time, across many umpires, and many games. One looks for patterns to emerge, and this can be seen in the publication record of an active researcher.

6.2 You Receive the Reviews

Alas, the editorial process is completed (typically within three months after manuscript submission) and the reviews are in. You receive an email (or in olden days a printed letter) from the editor noting whether the paper is accepted for publication and if not whether it might be if suitably revised. It is possible that the letter will say the manuscript is accepted as is (no further changes) and praise you for your brilliance. The letter may comment further that the reviewers were awed by how the study was executed and how well the manuscript was written. If this occurs, it is the middle of the night and you are dreaming. Remain in this wonderfully pleasant state as long as you can. When you awake, your spouse, partner, or significant (p < .05) other reads the email and you read one of the three decisions noted previously.

If the manuscript is accepted, usually some changes are needed. These do not raise problems. More often than not, the manuscript is rejected. There are individual differences in how one reacts to this decision. Typically, one feels at least one of these: miffed, misunderstood, frustrated, or angry at the reviewers. Usually one has only the email comments and has limited avenues (e.g., scrutiny of the phrasing and language) for trying to identify who could have possibly rejected the manuscript. If a hard (printed) version of the reviews was sent to you, one can scrutinize the font style, key words, possible DNA remnants of the reviewers’ comments sheets, and molecules on the pages that might reveal pollutants associated with a specific city in the country. (I myself would never stoop to such behaviors but I have a “friend” who, over the years, was able to identify two not-so-friendly reviewers who unwittingly left clues that I – I mean my friend – was able to decipher.) To handle a rejection verdict, some authors select one of the very effective psychotherapies or medications for depression; others use coping strategies (e.g., anger management training, stress inoculation therapy) or complementary and integrative medicines (e.g., acupuncture, mineral baths, vegan enemas). (I myself use all these routinely with their order balanced in a Hyper-Graeco-Latin Square Design.)

The task is to publish one’s work. Consequently, it is useful and important to take from the reviews all one can to revise the manuscript. Maladaptive cognitions can harm the process. For example, when reading a review, the author might say, the reviewer misunderstood what I did or did not read this or that critical part. These claims may be true, but the onus is always on the author to make the study, its rationale and procedures, patently clear. A misunderstanding by a reviewer is likely to serve as a preview of the reactions of many other readers of the article. Indeed, most readers may not read with the care and scrutiny of the reviewers. If the author feels a rejected manuscript can be revised to address the key concerns, by all means write to the editor and explain this in detail and without righteous indignation and affect.

Authors often are frustrated at the reactions of reviewers. In reading the reactions of reviewers, the authors usually recognize and acknowledge the value of providing more details (e.g., further information about the participants or procedures). This is the descriptive facet of manuscript preparation I discussed previously. However, when the requests pertain to explanation and contextualization, authors are more likely to be baffled or defensive. This reaction may be reasonable because much less attention is given to these facets in graduate training and explanation and contextualization are much less straightforward. Also, reviewers’ comments and editorial decision letters may not be explicit about the need for explanation and contextualization. For example, some of the more general reactions of reviewers are often reflected in comments such as: “Nothing in the manuscript is new,” “I fail to see the importance of the study,” or “This study has already been done in a much better way by others.”Footnote ⁵ In fact, the characterizations may be true. Authors (e.g., me) often feel like they are victims of reviewers who wore sleep masks when they read the manuscript, did not grasp key points, and have had little exposure to, let alone mastery of, the pertinent literature. Occasionally two or more of these are true.

As often as not, it is the reviewers who might more appropriately give the victim speech. The comments I noted are great signs that the author has not made the connections among the extant literature and this study and integrated the substantive, methodological, and data-analytic features in a cohesive and thematic way. Reviewers’ comments and less than extravagant praise often reflect the extent to which the author has failed to contextualize the study to mitigate these reactions. The lesson for preparing and evaluating research reports is clear. Describing a study does not establish its contribution to the field, no matter how strongly the author feels that the study is a first.

Let us assume that the manuscript was rejected with an invitation to resubmit. As a rule, I try to incorporate as many of the reviewers’ and editor’s recommendations as possible. My view is that the reviewer may be idiosyncratic, but more likely represents a constituency that might read the article. If I can address several or all issues, clarify procedures that I thought were already perfectly clear, and elaborate a rationale or two, it is advisable to do so. Free advice from reviewers can and ought to be used to one’s advantage.

There are likely to be aspects of the reviews one cannot address. Perhaps reviewers provide conflicting recommendations, or a manuscript page limit precludes addressing or elaborating a specific point. Even more importantly, perhaps as an author one strongly disagrees with the point. Mention these in the letter to the editor that accompanies the revised manuscript. Explain what revisions were or were not made and why. If there are large revisions that alter the text (few sentences), methods or data analyses, help the editor by noting where the change can be found in the manuscript and even submit an extra copy of the manuscript in which the changes are tracked in some editing/word-processing system.

The investigator may receive a rejection letter and decide simply to submit the manuscript as is to another journal. I believe this is generally unwise. If there are fairly detailed reviews, it is to the author’s advantage to incorporate key and often not-so-key points, even if the manuscript is to go to another journal. I have often seen the same manuscript (not mine) rejected from two different journals in which there were no changes after the rejection from the first journal. The authors could have greatly improved the likelihood of publication in the second journal but were a bit stubborn about making any revisions. Even if the manuscript were to be accepted as is in the second journal, it is still likely the author missed an opportunity to make improvements after the first set of reviews was provided. In general, try to take all the recommendations and criticisms from the reviews and convert them to facets that can improve the manuscript. Obstacles to this process may stem from our natural defensive reactions as authors or a negativity bias and the occasional brutish way in which reviewers convey cogent points. (I remember being highly offended the first two or three times reviewers noted such comments, “the author [me] would not recognize a hypothesis if it fell on his lap” and “the design of this study raises very important issues, such as whether it is too late for the author [me] to consider a career change.” I have come to refer to all this as the pier-review process to underscore how often reviewers have made me want to jump off one.)

There is an additional reason to encourage taking advantage of the review process and trying to improve a manuscript we might think is perfect. For those researchers who remain in academia, one’s published studies occasionally are read as part of a promotion process. As an author we might feel relieved that a study or two was published and view that automatically as things are great. In some ways it does, but as this study is read later we still want to be sure the case was made in a compelling fashion and reviewer suggestions might help. My view is to incorporate as many recommendations, changes, and comments as possible. I begin with the view that reviewers are experts and their recommendations, concerns, and misunderstandings are facets of the manuscript it behooves me to address.

It is worthwhile and highly rewarding to publish one’s research. The process takes time and persistence. Also, contact with others through the review process can greatly improve one’s work. In my own case, reading the reviews occasionally has stimulated the next studies that I carried out. In one case, I befriended a person who had been a reviewer of my work earlier in my career. Over time and from following his work, it was very clear that he was behind an influential review although his identity had been masked. Years later over dinner, I mentioned his review in a distant past, the study it generated, and the very interesting results and, of course, expressed my gratitude. His suggestion actually led to a few studies. (His review of my manuscript was not entirely positive, which probably is the main reason I hid in the bathroom of the restaurant until he paid the check for dinner.) The lesson is more than getting one’s manuscript published. Reviews can be very educational and it is useful to let the comments sit for a while until the rage over rejection subsides.

The journal review process is not the only way to obtain input on one’s manuscript. Once in a while, I send a penultimate draft of a manuscript to experts in the field whom I do not know. I convey in a letter what I am trying to accomplish and ask if they would provide feedback. I have done this on several occasions and cannot recall any colleague who has refused to provide comments. The comments are usually detailed and quite constructive and have a different tone from those that emanate from the journal review process. The comments in turn can be used to devise the version that is submitted for publication.