Hostname: page-component-8448b6f56d-qsmjn Total loading time: 0 Render date: 2024-04-24T14:47:29.576Z Has data issue: false hasContentIssue false
  • English
  • Français

Difference in Response Effort Across Sample Types: Perception or Reality?

Published online by Cambridge University Press:  28 July 2015

Shan Ran ,
Mengqiao Liu ,
Lisa A. Marchiondo  and
Jason L. Huang
Shan Ran*
Affiliation:
Wayne State University
Mengqiao Liu
Affiliation:
Wayne State University
Lisa A. Marchiondo
Affiliation:
Wayne State University
Jason L. Huang
Affiliation:
Wayne State University
*
Correspondence concerning this article should be addressed to Shan Ran, 5057 Woodward Avenue, Department of Psychology, Wayne State University, Detroit, MI 48202. E-mail: shan.ran@wayne.edu
Get access Rights & Permissions [Opens in a new window]

Extract

Landers and Behrend (2015) question organizational researchers’ stubborn reliance on sample source to infer the validity of research findings, and they challenge the arbitrary distinctions researchers often make between sample sources. Unconditional favoritism toward particular sampling strategies (e.g., organizational samples) can restrict choices in methodology, which in turn may limit opportunities to answer certain research questions. Landers and Behrend (2015) contend that no sampling strategy is inherently superior (or inferior), and therefore, all types of samples warrant careful consideration before any validity-related conclusions can be made. Despite sound arguments, the focal article focuses its consideration on external validity and deemphasizes the potential influence of sample source on internal validity. Agreeing with the position that no samples are the “gold standard” in organizational research and practice, we focus on insufficient effort responding (IER; Huang, Curran, Keeney, Poposki, & DeShon, 2012) as a threat to internal validity across sample sources.

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 8 , Issue 2 , June 2015 , pp. 202 - 208
Copyright
Copyright © Society for Industrial and Organizational Psychology 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bagger, J., & Li, A. (2014). How does supervisory family support influence employees’ attitudes and behaviors? A social exchange perspective. Journal of Management, 40, 1123–1150. doi:10.1177/0149206311413922Google Scholar
Barger, P., Behrend, T. S., Sharek, D. J., & Sinar, E. F. (2011). I-O and the crowd: Frequently asked questions about using Mechanical Turk for research. The Industrial–Organizational Psychologist, 49 (2), 11–17.Google Scholar
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800–813. doi:10.3758/s13428-011-0081-0Google Scholar
Chiaburu, D. S., Huang, J. L., Hutchins, H. M., & Gardner, R. G. (2014). Trainees’ perceived knowledge gain unrelated to the training domain: The joint action of impression management and motives. International Journal of Training and Development, 18, 37–52. doi:10.1111/ijtd.12021Google Scholar
Green, S. B., & Stutzman, T. (1986). An evaluation of methods to select respondents to structured job-analysis questionnaires. Personnel Psychology, 39, 543–564. doi:10.1111/j.1744-6570.1986.tb00952.xGoogle Scholar
Green, S. B., & Veres, J. G. (1990). Evaluation of an index to detect inaccurate respondents to a task analysis inventory. Journal of Business and Psychology, 5, 47–61. doi:10.1007/bf01013945Google Scholar
Hough, L. M., Eaton, N. K., Dunnette, M. D., Kamp, J. D., & McCloy, R. A. (1990). Criterion-related validities of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology, 75, 581–595. doi:10.1037/0021-9010.75.5.581Google Scholar
Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299–311. doi:10.1007/s10869-014-9357-6Google Scholar
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort respond to surveys. Journal of Business and Psychology, 27, 99–114. doi:10.1007/s10869-011-9231-8CrossRefGoogle Scholar
Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845. doi:10.1037/a0038510Google Scholar
Johnson, R. C., & Allen, T. D. (2013). Examining the links between employed mothers’ work characteristics, physical activity, and child health. Journal of Applied Psychology, 98, 148–157. doi:10.1037/a0030460CrossRefGoogle ScholarPubMed
King, E. B., Botsford, W., Hebl, M. R., Kazama, S., Dawson, J. F., & Perkins, A. (2012). Benevolent sexism at work gender differences in the distribution of challenging developmental experiences. Journal of Management, 38, 1835–1866. doi:10.1177/0149206310365902Google Scholar
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice. Advance online publication. Retrieved from http://www.siop.org/journal/8.2/landers.pdfGoogle Scholar
Liu, M., Bowling, N. A., Huang, J. L., & Kent, T. (2013). Insufficient effort responding to surveys as a threat to validity: The perceptions and practices of SIOP members. The Industrial–Organizational Psychologist, 51 (1), 32–38.Google Scholar
McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450–470. doi:10.1037/a0019216Google Scholar
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455. doi:10.1037/a0028085CrossRefGoogle ScholarPubMed
Nieminen, L., Kotrba, L., Denison, D., & Carter, N. (2014, May). Insufficient effort responding: Uncovering an insidious threat to data quality. In J. L. Huang & M. Liu (Cochairs), Insufficient effort responding to surveys: From impact to solutions. Symposium presented at the annual conference of Society for Industrial and Organizational Psychology, Honolulu, HI.Google Scholar
Pieterse, A. N., Van Knippenberg, D., & Van Dierendonck, D. (2012). Cultural diversity and team performance: The role of team member goal orientation. Academy of Management Journal, 56, 782–804. doi:10.5465/AMJ.2010.0992Google Scholar
Stetz, T. A., Button, S. B., & Quist, J. (2012). Rethinking carelessness on job analysis surveys: Not all questions are created equal. Journal of Personnel Psychology, 11, 103–106. doi:10.1027/1866-5888/a000061CrossRefGoogle Scholar