Skip to main content Accessibility help

The Negative Effects Questionnaire: psychometric properties of an instrument for assessing negative effects in psychological treatments

  • Alexander Rozental (a1) (a2), Anders Kottorp (a3), David Forsström (a4), Kristoffer Månsson (a1) (a5) (a6), Johanna Boettcher (a7), Gerhard Andersson (a1) (a8), Tomas Furmark (a6) and Per Carlbring (a5) (a9)...



Psychological treatments provide many benefits for patients with psychiatric disorders, but research also suggests that negative effects might occur from the interventions involved. The Negative Effects Questionnaire (NEQ) has previously been developed as a way of determining the occurrence and characteristics of such incidents, consisting of 32 items and six factors. However, the NEQ has yet to be examined using modern test theory, which could help to improve the understanding of how well the instrument works psychometrically.


The current study investigated the reliability and validity of the NEQ from both a person and item perspective, establishing goodness-of-fit, item bias, and scale precision.


The NEQ was distributed to 564 patients in five clinical trials at post-treatment. Data were analysed using Rasch analysis, i.e. a modern test theory application.


(1) the NEQ exhibits fairness in testing across sociodemographics, (2) shows comparable validity for a final and condensed scale of 20 instead of 32 items, (3) uses a rating scale that advances monotonically in steps of 0 to 4, and (4) is suitable for monitoring negative effects on an item-level.


The NEQ is proposed as a useful instrument for investigating negative effects in psychological treatments, and its newer shorter format could facilitate its use in clinical and research settings. However, further research is needed to explore the relationship between negative effects and treatment outcome, as well as to test it in more diverse patient populations.


Corresponding author

*Corresponding author. Email:


Hide All
Andersson, G. (2016). Internet-delivered psychological treatments. Annual Review of Clinical Psychology, 12, 157179. doi:
Andersson, G., Carlbring, P. and Hadjistavropoulos, H. D. (2017). Internet-based cognitive behavior therapy. In Hofmann, S. G., and Asmundson, G. J. G. (eds), The Science of Cognitive Behavioral Therapy (pp. 531549). San Diego, CA: Academic Press.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561573. doi:
Boettcher, J., Rozental, A., Andersson, G. and Carlbring, P. (2014). Side effects in internet-based interventions for social anxiety disorder. Internet Interventions, 1, 311. doi:
Bond, T. G. and Fox, C. M. (2013). Applying the Rasch Model: Fundamental Measurement in the Human Sciences. New Jersey, NJ: Psychology Press.
Boswell, J. F., Kraus, D. R., Miller, S. D. and Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: benefits, challenges, and solutions. Psychotherapy Research, 25, 619. doi:
Castonguay, L. G., Boswell, J. F., Constantino, M. J., Goldfried, M. R. and Hill, C. E. (2010). Training implications of harmful effects of psychological treatments. American Psychologist, 65, 3449. doi:
Crawford, M. J., Thana, L., Farquharson, L., Palmer, L., Hancock, E., Bassett, P., et al. (2016). Patient experience of negative effects of psychological treatment: results of a national survey. British Journal of Psychiatry, 208, 260265. doi:
Fisher, W. (1992). Reliability, separation, strata statistics. Rasch Measurement Transactions, 6, 238. doi:
Hansen, N. B., Lambert, M. J. and Forman, E. M. (2002). The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329343. doi:
Hofmann, S. G., Asnaani, A., Vonk, I. J. J., Sawyer, A. T. and Fang, A. (2012). The efficacy of cognitive behavioral therapy: a review of meta-analyses. Cognitive Therapy and Research, 36, 427440. doi:
Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537567. doi:
Ladwig, I., Rief, W. and Nestoriuc, Y. (2014). What are the risks and side effects to psychotherapy? – development of an Inventory for the assessment of Negative Effects of Psychotherapy (INEP). Verhaltenstherapie, 24, 252263. doi:
Lerdal, A., Kottorp, A., Gay, C., Aouizerat, B. E., Lee, K. A. and Miaskowski, C. (2016). A Rasch analysis of assessments of morning and evening fatigue in oncology patients using the Lee Fatigue Scale. Journal of Pain and Symptom Management, 51, 10021012. doi:
Linacre, J. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 85106.
Linacre, J. M. (2004). Rasch model estimation: further topics. Journal of Applied Measurement, 5, 95110.
Linacre, J. M. (2005). Rasch dichotomous model vs. one-parameter logistic model. Rasch Measurement Transactions, 19, 1032.
Lindner, P., Frykheden, O., Forsström, D., Andersson, E., Ljótsson, B., Hedman, E., et al. (2016). The Brunnsviken Brief Quality of life scale (BBQ): development and psychometric evaluation. Cognitive Behaviour Therapy, 45, 182195. doi:
Loerinc, A. G., Meuret, A. E., Twohig, M. P., Rosenfield, D., Bluett, E. J. and Craske, M. G. (2015). Response rates for CBT for anxiety disorders: need for standardized criteria. Clinical Psychology Review, 42, 7282. doi:
Lovibond, P. F. and Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33, 335343. doi:
Löwe, B., Kroenke, K., Herzog, W. and Gräfe, K. (2004). Measuring depression outcome with a brief self-report instrument: sensitivity to change of the Patient Health Questionnaire (PHQ-9). Journal of Affective Disorders, 81, 6166. doi:
Mantel, N. (1963). Chi-square tests with one degree of freedom; extensions of the Mantel-Haenszel procedure. Journal of the American Statistical Association, 58, 690700. doi:
McHugh, R. K. and Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. American Psychologist, 65, 7384. doi:
Mechler, J. and Holmqvist, R. (2016). Deteriorated and unchanged patients in psychological treatment in Swedish primary care and psychiatry. Nordic Journal of Psychiatry, 70, 1623. doi:
Moritz, S., Fieker, M., Hottenrott, B., Seeralan, T., Cludius, B., Kolbeck, K., et al. (2015). No pain, no gain? Adverse effects of psychotherapy in obsessive-compulsive disorder and its relationship to treatment gains. Journal of Obsessive-Compulsive and Related Disorders, 5, 6166. doi:
Muris, P. and Merckelbach, H. (1996). A comparison of two spider fear questionnaires. Journal of Behavior Therapy and Experimental Psychiatry, 27, 241244. doi:
Parker, G., Fletcher, K., Berk, M. and Paterson, A. (2013). Development of a measure quantifying adverse psychotherapeutic ingredients: the Experiences of Therapy Questionnaire (ETQ). Psychiatry Research, 206, 293301. doi:
Patomella, A-H., Tham, K. and Kottorp, A. (2006). P-drive: assessment of driving performance after stroke. Journal of Rehabilitation Medicine, 38, 273279. doi:
Raîche, G. (2005). Critical eigenvalue sizes in standardized residual principal components analysis. Rasch Measurement Transactions, 19, 1012.
Rheker, J., Beisel, S., Kräling, S. and Rief, W. (2017). Rate and predictors of negative effects of psychotherapy in psychiatric and psychosomatic inpatients. Psychiatry Research, 254, 143150. doi:
Rozental, A., Andersson, G., Boettcher, J., Ebert, D. D., Cuijpers, P., Knaevelsrud, C., et al. (2014). Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interventions, 1, 1219. doi:
Rozental, A., Boettcher, J., Andersson, G., Schmidt, B. and Carlbring, P. (2015). Negative effects of Internet interventions: a qualitative content analysis of patients’ experiences with treatments delivered online. Cognitive Behaviour Therapy, 44, 223236. doi:
Rozental, A., Kottorp, A., Boettcher, J., Andersson, G. and Carlbring, P. (2016). Negative effects of psychological treatments: an exploratory factor analysis of the Negative Effects Questionnaire for monitoring and reporting adverse and unwanted events. PLoS One, 11, e0157503. doi:
Rozental, A., Magnusson, K., Boettcher, J., Andersson, G. and Carlbring, P. (2017). For better or worse: an individual patient data meta-analysis of deterioration among participants receiving internet-based cognitive behavior therapy. Journal of Consulting and Clinical Psychology, 85, 160177. doi:
Shea, T. L., Tennant, A. and Pallant, J. F. (2009). Rasch model analysis of the Depression, Anxiety and Stress Scales (DASS). BMC Psychiatry, 9, 21. doi:
Smith, A. B., Rush, R., Fallowfield, L. J., Velikova, G. and Sharpe, M. (2008). Rasch fit statistics and sample size considerations for polytomous data. BMC Medical Research Methodology, 8(33). doi:
Spector, P. E. (1992). Summated Rating Scale Construction: An Introduction. Newbury Park, CA: Sage.
Spitzer, R. L., Kroenke, K., Williams, J. B. W. and Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder. Archives of Internal Medicine, 166, 1092. doi:
Strupp, H. H. and Hadley, S. W. (1977). A tripartite model of mental health and therapeutic outcomes. With special reference to negative effects in psychotherapy. The American Psychologist, 32, 187196. doi:
Suh, C. S., Strupp, H. H. and O’Malley, S. S. (1986). The Vanderbilt process measures: the Psychotherapy Process Scale (VPPS) and the Negative Indicators Scale (VNIS). In Greenberg, L. S. (ed), The Psychotherapeutic Process: A Research Handbook (pp. 285323). New York, NY, USA: Guilford Press.
Vlaescu, G., Alasjö, A., Miloff, A., Carlbring, P. and Andersson, G. (2016). Features and functionality of the Iterapi platform for internet-based psychological treatment. Internet Interventions, 6, 107114. doi:
Waugh, R. and Chapman, E. (2004). An analysis of dimensionality using factor analysis (true-score theory) and Rasch measurement: what is the difference? Which method is better? Journal of Applied Measurement, 6, 8099.
Wright, B. (1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement, 14, 97116. doi:
Wright, B. (1996). Comparing Rasch measurement and factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 3, 324. doi:
Wright, B. and Masters, G. N. (1982). Rating Scale Analysis: Rasch Measurement. Chicago, IL: MESA Press.



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed

The Negative Effects Questionnaire: psychometric properties of an instrument for assessing negative effects in psychological treatments

  • Alexander Rozental (a1) (a2), Anders Kottorp (a3), David Forsström (a4), Kristoffer Månsson (a1) (a5) (a6), Johanna Boettcher (a7), Gerhard Andersson (a1) (a8), Tomas Furmark (a6) and Per Carlbring (a5) (a9)...
Submit a response


No Comments have been published for this article.


Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *