Introduction

Structural equation modeling (SEM) is a popular approach to developing and employing theoretical models in predictive research within diverse fields and disciplines. With it, researchers can both construct and validate comprehensive models representing relationships of relative dependence and independence among latent constructs and observed variables through a combination of factor analyses and regressions.

Structural equation modeling is amenable to diverse data types (experimental, nonexperimental, cross-sectional, longitudinal) and is used to identify and map latent variables, that is, factors or constructs that are unobserved and thus not measured directly, through the interrelations of observed variables. Observed variables undergo reliability and validity testing and the resulting measurement model undergoes confirmatory factor analysis prior to model fit evaluation. This is followed by path modeling, whereby simultaneous multiple regression analyses take place and can include moderation, mediation, and other forms of variable relationships, with consideration of observed indicators revealing structural relationships of latent variables (for a recent succinct overview of procedural approaches and related statistical parameters, see Dash & Paul, Reference Dash and Paul2021).

Covariance-based SEM (CB-SEM) is one of the most prominent SEM methods and is widely adopted to explore latent phenomena. In education, this includes inquiry into students and teachers’ perceptions, attitudes, or intentions and their influence on learning or teaching outcomes.

Partial least squares SEM (PLS-SEM) is an alternative SEM approach that has been shown to offer certain advantages over CB-SEM. These include greater flexibility in acceptable sample sizes and data types, including nonnormal and nonmetric data (Hair Jr. et al., Reference Hair, Ringle and Sarstedt2011; Rigdon et al., Reference Rigdon, Sarstedt and Ringle2017), and advantages in handling complex mediation models (Sarstedt et al., Reference Sarstedt, Hair, Nitzl, Ringle and Howard2020) and measurement model confirmation, such as with the confirmatory composite analysis proposed by Hair Jr. et al. (Reference Hair, Howard and Nitzl2020) as a confirmatory factor analysis alternative. In a review article, Chin et al. (Reference Chin, Peterson and Brown2008) offer guidelines for SEM use in marketing that points to advantages of PLS-SEM. They note that PLS overcomes some SEM constraints, specifically by requiring only path specifications and measurement groupings into constructs, providing easier results interpretation based on regression as opposed to model fit indices, and embracing models with more indicators or latent variables relative to sample size (Chin et al., Reference Chin, Peterson and Brown2008, p. 295).

Partial least squares SEM has already been put to wide use in various disciplines, such as psychology, sociology, nursing, (Memon et al., Reference Memon, Cheah, Ting, Chuah and Cham2021), and business fields like marketing (Chin et al., Reference Chin, Peterson and Brown2008), accounting (Lee et al., Reference Lee, Petter, Fayard and Robinson2011), and information systems (Stieglitz et al., Reference Stieglitz, Dang-Xuan, Bruns and Neuberger2014), to name a few. More recently, educational researchers have utilized PLS-SEM to study issues such as learning beliefs (Ji et al., Reference Ji, Yue and Zheng2021), e-learning (see Lin et al., Reference Lin, Lee, Liang, Chang, Huang and Tsai2020, for review), behavior and achievement in flipped classrooms (Wang, Reference Wang2019), and educational service assessment (Schijns, Reference Schijns2021). Given both its operational and analytical attractiveness and possible pitfalls from misapplication, Ghasemy et al. (Reference Ghasemy, Teeroovengadum, Becker and Ringle2020) produced a review of PLS-SEM results reporting in higher education research, offering guidelines to facilitate effective, rigorous adoption in service of realizing its research promise.

CB-SEM and PLS-SEM Comparability and Debate: When and How to Employ PLS-SEM

There has been significant debate about the relative merits of CB-SEM and PLS-SEM, which has yielded scholarly exploration of and elaboration on the research potential of both approaches and their relative limitations and advantages (Dash & Paul, Reference Dash and Paul2021; Rigdon, Reference Rigdon2016; Rönkkö et al., Reference Rönkkö, McIntosh, Antonakis and Edwards2016; Sarstedt et al., Reference Sarstedt, Hair, Ringle, Thiele and Gudergan2016).

Comparisons of procedures and results of CB-SEM and PLS-SEM have been conducted over several decades in fields such as business and management, marketing, and information systems (see Rigdon et al., Reference Rigdon, Sarstedt and Ringle2017, for a summary of conceptual comparisons). More recently, scholars have shifted the focus of discussion to relative appropriateness and method application based on research design and context, as the methods “assume fundamentally different measurement philosophies” (Rigdon et al., Reference Rigdon, Sarstedt and Ringle2017, p. 7). Even when employing identical theoretical models, the differences in statistical models make them noninterchangeable (Rigdon et al., Reference Rigdon, Sarstedt and Ringle2017).

How should scholars decide to adopt the best approach? Several considerations have been offered to date. For example, to encourage appropriate application in marketing research, Hair et al. (Reference Hair, Matthews, Matthews and Sarstedt2017) review the PLS-SEM algorithm and advantages and limitations of the approach and offer several clear distinctions that should determine adoption of CB-SEM or PLS-SEM based on research objectives, proposing that CB-SEM should be used for theory testing and confirmation and PLS-SEM for prediction and theory development.

Elsewhere, Rigdon (Reference Rigdon2016) responded to misconceptions of PLS by clarifying that it is intended for use with composite-based models, in contrast to factor-based models explored with CB-SEM. The author suggests that mixed models may employ consistent PLS-SEM (PLSc-SEM) (Dijkstra & Henseler, Reference Dijkstra and Henseler2015a, Reference Dijkstra and Henseler2015b), which estimates a composite model while also converting estimates for a portion of parameters to bring it into accord with a factor model and was created in response to earlier debates about PLS efficacy.

For further guidance, Dash and Paul (Reference Dash and Paul2021) offer recommendations based on a comparison of CB-SEM, PLS-SEM, and PLSc-SEM, arguing for advantages of PLS-SEM for prediction and PLSc-SEM for model testing and analysis. Rather than attempting to reconcile PLS-SEM with factor model constraints, researchers seeking to “examine … data and evaluate many different configurations … in a situation that is ‘data-rich and theory-skeletal’” (Rigdon, Reference Rigdon2016, p. 13) can apply PLS-SEM for exploratory work for orienting research.

Ultimately, scholars should adopt a method that matches with the model type they wish to estimate and/or based on practical considerations such as variable understanding, data collection experience and instrument use, or stage in the research process (Rigdon, Reference Rigdon2016). To Chin et al. (Reference Chin, Peterson and Brown2008), the researcher’s responsibility is to report sufficient information for replication of analyses, estimates, and fits and provide a list for reporting that they consider essential in this regard (p. 291).

Given both the increasing appeal of PLS-SEM and the historical debate of its relative merits as well as the various dimensions of research design and practice that determine its value in producing rigorous scholarship, we offer this case study of its application in an area of educational research that has thus far seen limited application, that of research in English Medium Instruction (EMI) and related contexts. We proceed this chapter with a case study demonstrating its application in such a context and conclude with a discussion of several existing studies employing PLS-SEM in EMI research (including Choi, Reference Choi2021; Law & Fong, Reference Law and Fong2020; Peng, Reference Peng2022; Tai & Tang, Reference Tai and Tang2021; Zhang et al., Reference Zhang, Wang and Zhou2022) and implications for future inquiry.

Case Study Illustration: Predicting Student Satisfaction in EMI Courses

Over the past decades, the growth of English as a medium of instruction has become one of the most important developments in higher education in countries where English is not the primary language (Wächter & Maiworm, Reference Wächter and Maiworm2014). The two major forces driving EMI growth in higher education are globalization and internationalization (Tsou & Kao, Reference Tsou, Kao, Fenton‑Smith, Humphreys and Walkinshaw2017). EMI is also a result of the increasing emergent status of English as an academic lingua franca of the world (Galloway & Rose, Reference Galloway and Rose2015). Evidence shows English becoming more widely used as a language of university instruction in European countries (Ferguson, Reference Ferguson2007), even in nations with “big” languages like France and Germany (Ammon & McConnell, Reference Ammon and McConnell2002). Asian countries have also adopted the Englishization trend through EMI programs at universities (Yeh, Reference Yeh2014).

With this growth in EMI, how might researchers explore the influence of this trend? One way to consider efficacy is by operationalizing student responses in terms of attitude to EMI programming. Do they feel it meets their needs? The term “customer satisfaction” was first used in marketing to mean a judgment that a feature of a product or service, or the product or service itself, provides a satisfying level of fulfillment related to consumption (Reynoso, Reference Reynoso2010, p. 13). In educational contexts, students are referred as the primary customers of higher education institutions and student satisfaction is determined by the level of expectation and perceived performance (Fernandes et al., Reference Fernandes, Ross and Meraj2013). Regarding the student as customer perspective, there is a new ethical prerogative that students must become customers and therefore, as payers, can reasonably demand that their views be heard and adapted to (William, Reference William2002). In other words, in determining the success of tertiary institutions in delivering products to their primary customers (students), it is essential to measure customer (student) satisfaction (Alemu & Cordier, Reference Alemu and Cordier2017; Karakas, Reference Karakas2017).

As an illustration of the PLS-SEM method for this chapter, this study explores student perspectives regarding current EMI practices in the Taiwan higher education context. Specifically, this study examines student evaluations of their EMI courses and the relationship between EMI effectiveness factors and satisfaction of EMI courses. In that regard, this study asks the following questions:

• Q1: How do students evaluate their satisfaction of EMI courses?

• Q2: How does gender affect student satisfaction?

Data Analysis with SmartPLS

For this study, we conducted the SEM data analysis using the software Smart partial least squares (SmartPLS). The evaluation of the PLS model was done by evaluating first an outer model and then an inner model. The outer model is a measurement model to predict the correlation between indicators or parameters estimated with their latent variables, while the inner model is a structural model for predicting causality between latent variables. In order to answer Q2, multiple group analysis (PLS-MGA) was also conducted and demonstrated.

Model Specification and Path Model Estimation

The PLS path model derived after PLS algorithm calculation with the independent variables (SATUE, SPOIC, and SPOCC), dependent variable (SAT), relationships among variables, and all indicators of variables are shown in Figure 5.1. The reflective nature of variables is indicated by arrow directions.

Figure 5.1 PLS path model

One path coefficient indicates that English communicative ability (SATUE) strongly affects satisfaction (SAT), and its value is 0.390. Another path coefficient shows perceptions of university internationalization (SPOIC) weakly affecting satisfaction, with a value of 0.278.

Evaluation of Measurement Models (Outer Model)

In this study, we used the measurement model approach for analysis to assess the reliability, composite reliability (CR), and average variance extracted (AVE) of the constructs. The factor loading values show the reliability of individual indicators of constructs. It is suggested that the value for factor loadings should exceed 0.6 for acceptability (Fornell & Larcker, Reference Fornell and Larcker1981). The results (Figure 5.1) showed that values for P3 and P16 are 0.324 and 0.551, respectively. These values were dropped for the sake of getting improved results. The factor loading values for all latent variable constructs after eliminating those two indicators are shown in Table 5.2.

Table 5.2 Measurement model

Construct	Item coding	Factor loading	Outer weight	CA	CR	AVE
Student perception of course content (SPOCC)				0.92	0.94	0.76
	P21	0.926	0.244
	P22	0.899	0.238
	P19	0.887	0.238
	P20	0.840	0.234
	P18	0.800	0.194
Student perception of internationalization of the college (SPOIC)				0.836	0.876	0.505
	P1	0.760	0.194
	P13	0.707	0.139
	P14	0.697	0.249
	P2	0.683	0.196
	P4	0.601	0.178
	P5	0.705	0.189
	P6	0.802	0.258
Student ability to use English to communicate in class (SATUE)				0.829	0.886	0.662
	P10	0.883	0.332
	P11	0.852	0.345
	P8	0.684	0.213
	P9	0.820	0.326
Student satisfaction (SAT)				0.928	0.941	0.667
	S1	0.863	0.189
	S2	0.792	0.147
	S3	0.868	0.176
	S4	0.828	0.161
	S5	0.704	0.132
	S6	0.819	0.135
	S7	0.830	0.139
	S8	0.816	0.142

Note: Average variance extracted (AVE); Cronbach’s alpha (CA); Composite reliability (CR).

To measure reliability, we used Cronbach’s alpha (CA) and CR. The results for CA and CR are presented in Table 5.2 for SPOCC (0.920, 0.940), SPOIC (0.836, 0.876), SATUE (0.829, 0.886), and SAT (0.928, 0.941). According to Hair et al. (Reference Hair, Ringle and Sarstedt2011), CA and CR values should be higher than 0.70, and thus values in this study were within an acceptable range.

We assessed both the Fornell and Larcker criterion and heterotrait–monotrait (HTMT) ratio to test the discriminant validity (Fornell & Larcker, Reference Fornell and Larcker1981). The HTMT ratio has recently gained preference over Fornell and Larcker (Baloch et al., Reference Baloch, Meng, Xu, Cepeda-Carrion and Bari2017; Henseler et al., Reference Henseler, Hubona and Ray2016). Fornell and Larcker’s tests in Table 5.3 exhibit values greater than the correlations among the variables. The HTMT ratio results are lower than the 0.090 threshold (see Table 5.4).

Table 5.3 Discriminant validity (latent variable correlation and square root of AVE)

	SPOCC	SAT	SATUE	SPOIC
SPOCC	0.872
SAT	0.670	0.816
SATUE	0.567	0.754	0.813
SPOIC	0.446	0.650	0.617	0.711

Table 5.4 HTMT

	SPOCC	SAT	SATUE	SPOIC
SPOCC
SAT	0.708
SATUE	0.635	0.839
SPOIC	0.468	0.708	0.719

Additionally, we examined the convergent validity to obtain AVE values, and all the values were greater than the 0.50 threshold (for SPOCC, SPOIC, SATUE and SAT, the AVE values were 0.76, 0.505, 0.662, and 0.667, respectively), as suggested by Henseler et al. (Reference Henseler, Hubona and Ray2016; see Table 5.2).

Furthermore, we examined the variance inflation factor (VIF) to assess the problem of multicollinearity in the data. Aiken et al. (Reference Aiken, West and Reno1991) suggested that the values of VIF must be less than 10, and this study found VIF values within the suggested range, depicting no issue of multicollinearity in the data (see Table 5.5).

Table 5.5 Saturated model results

Construct	R2	R2 adjusted	VIF	f2
SPOCC			1.507	0.223
SAT	0.693	0.688
SATUE			1.950	0.293
SPOIC			1.651	0.121

Note: Variance inflation factor (VIF); effect size (f²); determination of coefficient (R²).

Evaluation of the Structural Model (Inner Model)

After finding the estimated model satisfied the outer model criteria, we then analyzed measurement testing the structural model (Figure 5.1) by looking at the value of R-square (R²) on the dependent variable. The results of R² values on variables based on the measurement results are shown in Table 5.5, where it can be seen that R² for student satisfaction was 0.693, indicating that the degree of influence of SPOCC, SATUE, and SPOIC on student satisfaction was 69.3 percent.

The value of f-square (f²) depicts the respective contributions of each construct to the relationships found in the outer model. It also reflects the significance of the influence of each construct on the others along with the degree of this influence. The results of f² analysis are shown in Table 5.5. The value of f² should be less than 0.02 in order to indicate a significant relationship. The relation of SPOCC and SAT shows an f² of 0.223, while SATUE and SAT have f² values of 0.293. Similarly, the relation of SPOIC to SAT shows an f² value of 0.121 (Figure 5.2).

Figure 5.2 Structural model

Bootstrapping is a method that is used to check and test the significance of a model. The value of t-statistics reflects significance of path coefficients (Ringle et al., Reference Ringle, Da Silva and Bido2015). The PLS-SEM findings show that SPOCC (β = 0.322, t = 5.480, p < 0.05), SATUE (β = 0.419, t = 6.131, p < 0.05), and SPOIC (β = 0.248, t = 3.985, p = <0.05) have positive effects on student satisfaction (β = −0.115, t = 0.062, p = 0.265) (Table 5.6).

Table 5.6 Path coefficients

	β	Mean	SD	t-value	p-value
SPOCC	0.322	0.319	0.059	5.480	***
SATUE	0.419	0.424	0.068	6.131	***
SPOIC	0.248	0.248	0.062	3.985	***

^*** p < 0.001, ^**p < 0.01, ^*p < 0.05 (two-tailed test)

MGA

The MGA tests whether predefined data groups have significant differences in their group-specific parameter estimates. SmartPLS provides the outcomes of three different approaches, based on the bootstrapping results from every group: Confidence Intervals (Bias Corrected), PLS-MGA, and Parametric Test.

The Confidence Intervals method was used to analyze the influence of gender as this method is a nonparametric significance test for the difference of group-specific results that builds on PLS-SEM bootstrapping results. A result is significant at the 5 percent probability of error level, if the p-value is smaller than 0.05 or larger than 0.95 for a certain difference of group-specific path coefficients. The PLS-MGA method (Henseler et al., Reference Henseler, Ringle and Sinkovics2009) as implemented in SmartPLS is an extension of the original nonparametric Henseler’s MGA method.

Table 5.7 presents the path coefficient and p-value for both gender groups: male and female. In this table, if the p-value is less than or equal to 0.05, or greater than or equal to 0.95, the path coefficient is considered to be significant. In both male and female groups, all the p-values are less than 0.05, so all the path coefficients are significant. Considering the female group, “student ability to use English to communicate in class” has the highest path coefficient value (SATUE – > SAT = 0.453), followed by “student perception of course content” (SPOCC – > SAT = 0.342). The factor of “student perception of internationalization of the college” has the lowest impact on student satisfaction (SPOIC – > SAT = 0.174). In the male group, “student ability to use English to communicate in class” has the strongest impact on student satisfaction (β = 0.370), whereas “student perception of course content” has a slight impact on trust (β = 0.336). The impact of “student perception of internationalization of the college” on student satisfaction is highly significant (β = 0.355).

Table 5.7 Gender relationship with student satisfaction

	Female			Male
	Path coefficients	t-value	p-value	Path coefficients	t-value	p-value
SPOCC – > SAT	0.342	4.177	0.000	0.336	4.596	0.000
SATUE – > SAT	0.453	6.036	0.000	0.370	4.125	0.000
SPOIC – > SAT	0.174	2.524	0.012	0.355	3.864	0.000

Table 5.8 highlights that the impacts of SPOCC, SATUE, and SPOIC on SAT were not significantly different between male and female groups. For SPOCC and SAT the male group is higher than the female group (SPOCC – > SAT = −0.006, SATUE – > SAT = −0.083), whereas for SPOIC and SAT the female group is higher than the male group (SPOIC – > SAT = 0.180). Overall, the male group is higher than the female group in satisfaction (Figures 5.3–5.5).

Table 5.8 Path coefficient difference for gender

	Path coefficients – difference (male vs. female)	t-value (male vs. female)	p-value (male vs. female)
SPOCC – > SAT	−0.006	0.052	0.959
SATUE – > SAT	−0.083	0.717	0.474
SPOIC – > SAT	0.180	1.613	0.109

Figure 5.3 Moderating effect of gender: SPOCC on SAT

Figure 5.4 Moderating effect of gender: SATUE on SAT

Figure 5.5 Moderating effect of gender: SPOIC on SAT

Discussion

In this chapter, we demonstrate the PLS-SEM approach to explore its potential application in EMI research. For a field in which topics of inquiry engage contexts of particular complexity, in offering advantages in service of researchers investigating smaller sample sizes, nonnormal or nonmetric data, complex mediation analysis, and confirmatory factor analysis (Choi, Reference Choi2021; Dash & Paul, Reference Dash and Paul2021; Lin et al., Reference Lin, Lee, Liang, Chang, Huang and Tsai2020), PLS-SEM has demonstrable potential advance future EMI research.

While PLS-SEM has been utilized widely in educational studies (Ghasemy et al., Reference Ghasemy, Teeroovengadum, Becker and Ringle2020; Ji et al., Reference Ji, Yue and Zheng2021; Lin et al., Reference Lin, Lee, Liang, Chang, Huang and Tsai2020; Schijns, Reference Schijns2021; Wang, Reference Wang2019), it has only recently been employed to conceptualize, explain, and predict the dynamic developments within teaching and learning through EMI. With EMI research notably interwoven with interdisciplinary work, we invite researchers to nurture space in their thinking for review of and reflection on (1) recent educational studies employing PLS-SEM, (2) recent EMI or language learning studies employing PLS-SEM, and (3) recent EMI studies that, while not using PLS-SEM, inspire questions and directions of inquiry for which PLS-SEM may appropriately be employed to provide explanation and illumination.

In educational research of the past decade, PLS-SEM has been applied to the examination of satisfaction or learning outcomes in higher education (Ghasemy et al., Reference Ghasemy, Teeroovengadum, Becker and Ringle2020) and the exploration of e-learning and online environments (Lin et al., Reference Lin, Lee, Liang, Chang, Huang and Tsai2020). Through the management lens, PLS-SEM is commonly used for examining service quality in an online education setting (Schijns, Reference Schijns2021). According to Lin et al.’s (Reference Lin, Lee, Liang, Chang, Huang and Tsai2020) review of e-learning, PLS-SEM can support researcher explorations of learning contexts as well as development of theory. For instance, a research team (Wang et al., Reference Wang, Wallace and Wang2017) employed PLS-SEM to explore a reward-based online design competition with a study design that included sociocultural factors among learners. Elsewhere, Wang (Reference Wang2019) used PLS-SEM to construct an online behavior model examining behaviors and achievement in flipped classrooms. PLS-SEM is particularly promising as an approach to addressing a growing need for examinations of interplays among diverse factors. For example, Prihandoko (Reference Prihandoko2021) explored online learning experiences by using PLS-SEM to examine learner perceptions, achievement, and motivation. In terms of general studies in education, using PLS-SEM is also considered a helpful research tool for examining learner belief structures (Ji et al., Reference Ji, Yue and Zheng2021) and self-efficacy (Zhao et al., Reference Zhao, Peng and Liu2021).

EMI studies involve dynamic teaching and learning in diverse educational contexts, necessitating analysis techniques that can manage complexity. Choi (Reference Choi2021) particularly found PLS-SEM advantageous in investigating EMI satisfaction, further highlighting the salience of the aspects of lesson preparation and teaching skills. However, PLS-SEM is still an emerging method in EMI research. It has been seen more commonly in previous research on language-oriented classrooms (Ahmadi-Azad et al., Reference Ahmadi-Azad, Asadollahfam and Zoghi2020), and the approach has much to offer in examining the relationships between factors implicated in how language is taught and learned (Tai & Tang, Reference Tai and Tang2021). Later we present a range of topics related to language learning and teaching that may benefit from deeper exploration with this approach when applied to EMI contexts in the future.

Current research interests center on evaluating student perceptions and proposing explanatory and predictive models to better understand the dynamism of student learning. A recent study from a group of applied linguistic researchers (Law & Fong, Reference Law and Fong2020) used PLS-SEM to investigate undergraduate students’ perceptions of their learning transfer in an English for Academic Purpose context. Their study demonstrated the advantage of PLS-SEM when investigating the relationships between perceived variables, such as how students perceived content, understanding, and applicability of their learning. They further established an empirical research model to uncover learning transfer in language education. Similarly, a group of foreign language researchers (Zhang et al., Reference Zhang, Wang and Zhou2022) utilized PLS-SEM to investigate student perceptions of course satisfaction in an English Speaking Instruction context in order to further develop a speaking system. They examined how students perceived their understanding of course cognition, learning behavior, and knowledge mastery. They also included teacher–student communication in the model to explore its mediation and realize better understanding of aspects of instructional environment and overall effectiveness.

In studying college English learning in Korea, Joo (Reference Joo2022) applied PLS-SEM to analyze students’ perceptions of English learning, including how learners perceived their motivation, interest, attitude, and satisfaction after engaging in different learning materials such as TED talk videos. In another study, Joo (Reference Joo2021) also conducted PLS-SEM, this time examining students’ perceived self-efficacy, such as in terms of their learning achievement, in an English writing class. There has also been some additional research examining factors that affect teaching. For instance, a recent study (Peng, Reference Peng2022) proposed using the PLS-SEM approach to respond to the influencing factors that affect oral English teaching. However, this study also revealed that future research might still need to consider limitations, such as being more careful in reviewing nonconvergence of parameter estimations. As such, Peng (Reference Peng2022) suggests reconsidering the quality of data or conflicts between data.

Potential Applications for EMI-Related Contexts

English Medium Instruction is situated in a complex educational system that involves dynamic contextual factors including the intersection of language acquisition and content knowledge (Lo & Lo, Reference Lo and Lo2014), individual differences among learners and instructors (Yuan et al., Reference Yuan, Lu, Jing, Yang and Hao2022), and resources/affordances of learning environments (Min et al., Reference Min, Wang and Liu2019). Table 5.9 presents an overview of interwoven factors dynamically implicated in EMI. Further exploration of their interrelationships and roles would help illustrate a more comprehensive picture of these dynamics for practitioners and reveal what other learning support and professional development might be provided.

Overall, we suggest that the PLS-SEM approach has great potential to address ongoing challenges in research and practice in EMI contexts. We conclude with a suggestion for future researchers to consider three dimensions related to the purpose of their EMI studies for using the PLS-SEM approach:

1. 1. The nature of conducting exploratory research in EMI contexts.

2. 2. The need to examine interrelationships of variables in EMI contexts;

3. 3. the goal of developing theory to conceptualize EMI research.

This chapter provided a demonstration of the application of using PLS-SEM in EMI research by exploring student satisfaction and gender differences with respect to student perceptions of course content, internationalization of college, and the ability to use English. As definitions and operationalizations of satisfaction vary between and in relation to educational contexts, there is still more to explore in this area, as well as many others, in order to deepen our understanding of EMI in various instantiations. We invite researchers to consider their study objectives and design in regard to the advantages and requirements of the PLS-SEM approach.

Introduction

The purpose of this chapter is to demonstrate the application of exploratory factor analysis (EFA) in investigating grammatical complexity in writing assignments from Hong Kong (HK) students for an English Medium Instruction (EMI) course in a university. English has been used as an important medium of professional communication in many different areas, for instance, business, science, and education. Substantial emphasis has been put on science education in HK universities since the government launched an educational policy to promote science, technology, engineering, and math disciplines in 2015 (Pun & Thomas, Reference Pun and Thomas2020). In universities, a scientific English course is often provided through EMI to help the students succeed in professional communication in science areas. This chapter focuses on a scientific English course (i.e., English for Science) that is offered by a government-funded HK university as a required course for students in science-related majors, such as physics, chemistry, data science, and computing mathematics. The course aims to improve students’ communicative competence in varied scientific contexts, and a major course project is a scientific report, which involves analyzing a scientific issue, presenting experimental results, and interpreting the results for specialist audiences. This written genre was selected to be investigated due to the importance of scientific reports in professional communication in science. For example, scientific writing is not only a method to assess students’ knowledge in science but also an important way for students to become engaged in science communities.

An important research construct in this chapter is grammatical complexity, which has received substantial research attention in academic writing studies in the 2000s and the 2010s (Lan et al., Reference Lan and Sun2019). The majority of empirical studies on grammatical complexity has been conducted in multiple EMI contexts, for instance, programs of English for Academic Purposes (e.g., Parkinson & Musgrave, Reference Parkinson and Musgrave2014), programs of English as a Second Language (e.g., Bulté & Housen, Reference Bulté and Housen2014), and first-year composition programs (e.g., Eckstein & Ferris, Reference Eckstein and Ferris2018; Lan & Sun, Reference Lan and Sun2019). Among these studies, grammatical complexity has played three important roles, which are (1) tracking language development in academic writing (e.g., Lu, Reference Lu2011; Staples et al., Reference Staples, Egbert, Biber and Gray2016); (2) predicting writing quality in exam writing tasks (e.g., Crossley & McNamara, Reference McNamara, Graesser, McCarthy and Cai2014; Kyle & Crossley, Reference Kyle and Crossley2018); and (3) distinguishing different written genres and/or registers (e.g., Biber et al., Reference Biber, Gray and Poonpon2011; Yoon & Polio, Reference Yoon and Polio2017).

There are numerous synthesized reviews on the representation of grammatical complexity in second language (L2) writing, for instance, Wolfe-Quintero et al. (Reference Wolfe-Quintero, Inagaki and Kim1998), Ortega (Reference Ortega2003), Norris and Ortega (Reference Norris and Ortega2009), Biber et al. (Reference Biber, Gray and Poonpon2011), and Lan et al. (Reference Lan and Sun2019). While we acknowledge the importance of these reviews, it is equally (if not more) important to have a data-driven investigation on how grammatical complexity is represented in seminal research models. An EFA is used to inductively explore the representation of this important research construct in scientific research reports of HK university students in an EMI English course. The findings shed light on how to produce effective instructions on grammatical complexity in scientific English courses. More importantly, we not only consider this chapter an example using EFA to investigate grammatical complexity but also regard it as an opportunity to leverage similar research in different EMI courses in the future.

Research Design

The research method we used is a corpus-driven approach. A corpus-driven approach refers to analyzing corpus data to develop theories (Cheng, Reference Cheng2012). No study has inductively analyzed the representation of grammatical complexity, particularly in scientific research reports of HK university students. This corpus-driven approach will have several characteristics as mentioned in Biber et al. (Reference Biber, Conrad and Reppen1998), for example, using a large collection of authentic texts, analyzing the use of language patterns in specific contexts, and making extensive use of computers for analysis. From the corpus perspective, an existing corpus is used in this study – a corpus of scientific research reports written by undergraduate students in science majors in an HK university. From the statistical perspective, an EFA is applied to provide an empirical description on the relationship between latent factors of grammatical complexity and their associated grammatical complexity measures. The EFA allows us to model the representation because (1) it is performed when researchers aim to group variables based on patterned correlations and (2) there are no apparent theories that will be relied on (Tabachnick & Fidell, Reference Tabachnick and Fidell2013). Therefore, with the corpus of scientific reports, we model the representation of grammatical complexity with an EFA.

Two research questions are answered:

1. 1. How many latent factors of grammatical complexity are identified by the best factor model?

2. 2. How can the pattern matrix of the final factor solution demonstrate the relationship among different measures and the latent factors?

Quantitative Methods

Results Report

Stage 2: Conducting the EFA

Two specific solutions of the EFA were generated in SPSS: (1) the initial solution of the EFA model and (2) the final solution of the EFA model.

The Initial Solution

First, Principal Axis Factoring was selected as the extraction method, and an oblique rotation method, Promax, was selected for the initial analysis.Footnote ¹ Also, small correlation coefficients with absolute values a less than 0.40 were suppressed so that only the variables that significantly loaded on each factor will be shown in the results. With the current setting, the initial solution was achieved with a four-factor EFA model, based on the following observations: (1) The scree plot shows that beyond four factors, the line’s slope in the scree plot becomes not much less sharp (Figure 6.1); and (2) in terms of the eigenvalues of the extracted factors, only four factors have the eigenvalues greater than 1 (see Appendix B for more information). Both criteria suggest that a four-factor model can be achieved for the EFA as an initial solution. In other words, it is reasonable to conclude that four latent factors have the potential to explain grammatical complexity in our dataset.

Figure 6.1 Scree plot

Second, the pattern matrix was generated to present how the variables load on the four extracted factors. This pattern matrix allows the investigation of relationships between the fourteen measures and the four latent dimensions of grammatical complexity. The pattern matrix reports seven significant loading variables for Factor 1, five for Factor 2, two for Factor 3, and three for Factor 4. The factor loadings range from 1.077 to 0.502. However, three variables (MLS, MLT, C/S) cross-load on two factors. The presence of cross-loading variables suggests that the initial solution does not fit the current dataset well, and further action is needed.

The Final Solution

To make the EFA model fit our data and be interpretable, the initial solution is adjusted to achieve a final solution. First, we further adjusted the four-factor model in the initial solution: The cross-loading variables were deleted one by one to reach clear relationships between the factors and the variables (i.e., the grammatical complexity measures). Two cross-loading variables were finally excluded from the EFA model, which are MLS and T/S.Footnote ² When a variable is excluded, the factor loadings of the remaining variables often change. As a result, with MLS and T/S deleted, we finally achieved a three-factor model with a clear pattern among the remaining twelve measures and their three latent factors of grammatical complexity. This adjusted EFA model explains 90.451 percent of the total variance of grammatical complexity. In other words, this three-factor model with twelve measures can explain 90.451 percent of the variance of this theoretical construct. Second, it is important to explain one tricky variable: MLT. In the adjusted EFA model, eleven of the twelve remaining variables significantly load on individual factors, and the only exception is MLT, which cross-loads significantly on Factor 1 (0.544) and Factor 2 (0.677). We decided to keep MLT and finalized our simple solution as a three-factor model with twelve variables. This decision is based on a reason: MLT has always been the most frequent measure of grammatical complexity in L2 writing since the 1990s (Lan et al., Reference Lan and Sun2019; Lu, Reference Lu2011; Wolfe-Quintero et al., Reference Wolfe-Quintero, Inagaki and Kim1998). The exclusion of MLT will make the EFA model contradict empirical findings in existing writing studies. To handle this tricky variable, MLT was included as a cross-loading variable on Factor 1 and Factor 2 in the final EFA model.

Third, the final solution is presented by the pattern matrix of the three-factor model (see Table 6.2). The finalized model includes three factors and the twelve measures of grammatical complexity features, and the three-factor model suggests that there are three latent constructs of grammatical complexity. Factor 1 includes six measures, which are DC/T, C/T. DC/C, CT/T, C/S, and VP/T. Factor 2 includes four measures, namely CN/C, MLC, CN/T, and MLT. Factor 3 includes two measures, which are CP/T and CP/C.

Table 6.2 Pattern matrix

Variables	Factor 1	Factor 2	Factor 3
DC/T	0.998
C/T	0.989
DC/C	0.910
CT/T	0.800
C/S	0.835
VP/T	0.811
CN/C		0.991
MLC		0.885
CN/T		0.850
MLT	0.544	0.677
CP/T			0.976
CP/C			0.960

Implications of Using EFA for EMI

The evidence-based EFA model has both research and pedagogical implications for EMI. English has been used as a medium for English instruction in HK universities. For research implications, this EFA model is developed based on scientific research reports from an EMI English course, English for Science. The three latent factors of grammatical complexity suggest the three major types of grammatical structures that the HK students should handle in English writing for this specific context. Traditionally, as a theoretical construct, grammatical complexity has been primarily observed by clausal complexity in academic writing in L2 writing (Biber et al., Reference Biber, Gray and Poonpon2011). In EMI courses of science subjects, we also suggest investigating nominal phrasal complexity and coordinate phrasal complexity to make the representation of grammar use complete. In addition, as English is used as a medium of instruction in academic contexts in HK universities, it is also suggested to conduct an EFA on grammatical complexity with different written genres to expand our understanding of grammar use in other discipline-specific writing. Some important EMI contexts in HK universities include but are not limited to English in business, English in law, English in journalism, and English in humanities and social sciences.

According to the pedagogical implications, the three latent factors of grammatical complexity align with different stages of writing development. In terms of language development in academic writing, there are four sequential stages from beginning to advanced: (1) the use of coordinate structures (Norris & Ortega, Reference Norris and Ortega2009); (2) the use of finite subordinate clauses; (3) the use of nonfinite dependent clauses; and (4) the use of dependent phrases primarily in compressed noun phrases (Biber et al., Reference Biber, Gray and Poonpon2011). As a result, different latent factors of grammatical complexity should be focused on for students at different stages of language development. In the science-related EMI context, it is suggested that instructors first ensure that students can master clausal subordination structures and then move on to offering instructions on the nominal phrasal structures, for example, compressed noun phrases with head nouns modified by phrasal modifiers (e.g., prepositional phrases as postmodifiers). The focus of specific latent factors of grammatical complexity in EMI will facilitate the students to effectively develop their grammar use.

Conclusion

In this chapter, we conducted an EFA to explore the latent factors of grammatical complexity in a corpus of scientific research reports from an EMI English course for science students. The EFA model includes three latent factors, which are clausal (subordination) complexity, nominal phrasal complexity, and coordinate phrasal complexity. Each latent factor is associated with corresponding measures of grammatical complexity from Lu (Reference Lu2010). The only tricky measure is MLT that has cross-loaded on both clausal (subordination) complexity and nominal phrase complexity. There is a clear pattern between the remaining eleven measures and their associated latent factors. This model has implications on research and pedagogy for academic writing in EMI contexts, including the application of EFA in other EMI contexts in HK universities (e.g., business English) and the instruction on different latent factors (e.g., nominal phrasal complexity) to effectively facilitate the students’ grammar use in the science-related EMI context.

The successful construction of questionnaires, which is a critical step toward gathering research data, would greatly benefit from the guidance of questionnaire design theories. Researchers have advocated for the implementation of practices such as building item pools and conducting pilot studies (Curle & Derakhshan, Reference Curle, Derakhshan, Pun and Curle2022; Dörnyei & Taguchi, Reference Dörnyei and Taguchi2009). In addition, statistics in support of questionnaire validity, which often include Cronbach’s alpha and item analysis statistics, are often reported together with research results to control the quality of questionnaire items.

Successful interpretation of data collected from questionnaires is indispensable to the presentation of research results. A widely used approach to process questionnaire data is computing the mean and standard deviation of each item. The fundamental assumption is that participants have been using the scales on the questionnaires as interval ones. It is highly possible, however, that the scales are used in an ordinal manner. For example, on a 5-point Likert scale ranging from “1 = strongly disagree” to “5 = strongly agree” participants’ perception of the difference between “1 = strongly disagree” and “2 = disagree” may vary from person to person. An average score of 1.5 does not necessarily indicate an equal distance to “1 = strongly disagree” or “2 = disagree.” In this regard, comparing the average score of 1.5 to 1.75 between two different items, or between two different groups, will not always yield accurate results for interpretation.

To provide possible solutions of transforming ordinal scales into interval scales, this chapter introduces Rasch modeling as a method to explain data gathered from questionnaires. In addition to explicating basic knowledge about Rasch Models and model validation statistics, this chapter also features a pilot study measuring undergraduate students’ attitudes toward English Medium Instruction (EMI) in mainland China. Data collected will be analyzed through Many-facet Rasch Measurement (MFRM), with step-by-step instructions provided for future reference.

Fundamental Facts and Principles about Rasch Models

First introduced by Georg Rasch for analyzing responses to dichotomous items, Rasch Models are grounded on the assumption that different people’s responses to a number of items are stochastically independent. Both person and item parameters, which are unknown to researchers, decide the probability of providing a positive response to a question item. Rasch Models are usually explained as additive logistic models, which give recognition to the latent nature of the person parameter. This latent nature can be understood as “a random effect explaining the covariation among items” (Kreiner, Reference Kreiner, Chirstenson, Kreiner and Mesbah2013, p. 10) and is also called “variable” or “trait” interchangeably at times. With the latent nature being the focus for investigation, Rasch Models are characterized by the presence of both item parameter and person parameter on one logit scale. They take both human factors and item performances into account, thus offering explanations for person ability as well as item difficulty (Bond & Fox, Reference Bond and Fox2001).

Though Rasch Models were first devised to process dichotomous data, Polytomous Rasch Models have been used more frequently, especially for processing Likert scales with values of “strongly agree,” “agree,” “disagree,” and “strongly disagree.” Researchers have mentioned conflicting views on whether to categorize Rasch Models as a one-parameter logistic model within the camp of item response theory (IRT) (Andrich, Reference Andrich, Smith and Smith2004; Shaw, Reference Shaw1991). However, Rasch aligns with IRT in transforming ordinal data through a mathematical framework supported by logistic probability. Besides the prerequisite that all measured constructs are valid and well justified, Rasch Models demand an additional list of requirements to be fulfilled, which include the following:

• Unidimensionality. An underlying principle for Rasch Models is unidimensionality, which is the quality of measuring one trait or attribute of one person at one time.

• Monotonicity. As the latent trait of a person grows, the expected item score the person receives would also increase.

• Homogeneity. All participants would unanimously find the most difficult item the hardest to respond to. In other words, the ranking order of the item parameters would remain the same across different levels of latent trait.

The list also includes two other requirements for researchers’ and practitioners’ consideration. First, items on a measurement scale have to be independent of each other; that is, the response given to one item will not be influenced by responses to other items. Second, items demonstrating differential item functioning (DIF) (i.e., people of different subgroups do not find the items to be equally difficult) fail to meet the requirement of Rasch Models. Researchers and practitioners have to carefully evaluate different model options for data processing when these requirements cannot be fulfilled. The Graphical Loglinear Rasch Model (GLLRM), for instance, is often recommended when locally dependent items or items of DIF need to be included for analysis (Nielson et al., Reference Nielson, Garcia and Alastor2021; Upegui-Arango et al., Reference Upegui-Arango, Forkmann, Nielson, Hallensleben, Glaesmer, Spangenberg, Telsmann, Jukef and Boecker2019).

After considering and weighing the prerequisites for Rasch Models, researchers are also faced with model selection decisions. How are Rasch Models further categorized in terms of their functions? Which models have been used in extant literature for analyzing questionnaire data? Table 7.1 provides a classification scheme of commonly used Rasch Models and features brief explanations for their functioning mechanisms. It also enlists the model’s leniency toward restrictions such as unidimensionality, monotonicity, homogeneity, and local independence.

To examine the functioning of the selected Rasch Models, researchers would often report a series of statistics that demonstrate the models’ appropriateness. Related statistics are explained as follows:

1. (a) Estimates for Model Fit: Researchers might be challenged by the fact that empirical observations do not perfectly fit a selected Rasch Model, which occurs more frequently in statistical calculation. The estimates of model fitness are thus carefully evaluated and are often calculated as log-likelihood chi-square in Rasch analysis output. Model unfitness is demonstrated through a small p-value. Another approach is to observe the difference between expected responses and observed responses, which is usually measured by standard deviation according to Eckes (Reference Eckes2011). Model fitness is guaranteed only if the standard deviation of observed data is within the range between −2 and +2.

2. (b) Estimates for Person Fit and Item Fit: In contrary to “fit,” “misfit” happens when observed data do not match with expected results generated by the models. A “misfitting” person might be a questionnaire respondent who selected “disagree” across all the items on a Likert scale or chose his or her answer in an unpredictable manner in comparison with other participants. A “misfitting” item, however, could be answered correctly with less difficulty for respondents with lower ability, which contradicts the assumption of monotonicity.

   Also included in the report of Rasch model statistics are infits and outfits, which are chi-square statistics that check the association between the model and data and are also considered as important indices in the explanation of “fit.” As Boone et al. (Reference Boone, Staver and Yale2014, p. 164) explained, outfit is “a fit statistic sensitive to outliers” such as guessing or thoughtless errors, while infit “focuses less on outliers but more on responses near a given item difficulty (or person ability).” Among all the fit statistics provided by Rasch Model statistics report, both Boone et al. (Reference Boone, Staver and Yale2014) and Linacre (Reference Linacre2012) recommended a closer investigation of Item Outfit MNSQ (Mean-Square) and Person Outfit MNSQ. Another important estimate is standard residuals, which also reveals the difference between observed data and expected data.

   In addition to the commonly used statistics that reflect model fitness, Boone and Staver (Reference Boone and Staver2020) synthesized a collection of more advanced indices, or statistical techniques, to demonstrate the robustness and appropriateness of Rasch Models. These statistics also help provide evidence in support of the fundamental assumptions of the Rasch Models, such as unidimensionality and homogeneity. For example:

3. (c) Principal Component Analysis of Residuals: The purpose of conducting principal component analysis of residuals (PCAR) is to check the dimensionality of a questionnaire instrument. When applying a Rasch Model for measurement purposes, an ideal situation would be that the latent measure will explain all the data information. This is often checked through the indices of Person Fit and Item Fit. The unexplained part of the data, or the residuals, should conform to a normal distribution after standardization.PCAR results indicate that there are more unexplored reasons that could explain the residuals, and researchers need to consider whether additional factors are playing a role in the dataset.

4. (d) Item Characteristics Curve: Appearing in the form of a nonlinear curve that plots expected item score (y axis) against item measure (x axis), the item characteristics curve (ICC) is adopted as a practice to control the quality of questionnaire items. The expected item score (y axis) presents the raw score a respondent might obtain, while the item measure (x axis) corresponds to the transformed logit scale through Rasch modeling. The ICC thus functions as a bridge between interval numbers and ordinal scale, where researchers can conduct more analysis on the functioning of questionnaire instruments for measurement purposes.

Depending on various research purposes, Rasch Models have been used in connection with other statistical methods such as factor analysis and effect size calculation. The next section of this chapter includes three studies that focus on the development of scales and examination of their use, where Rasch modeling has assisted researchers in retrieving more relevant information.

The Application of Rasch Modeling in Processing Questionnaire Data – Three Case Reports

In the research field of language testing and assessment, Rasch Models have usually been recognized as a method of examining rater reliability and test item validity (Khabbazbashi, Reference Khabbazbashi2017; Yan, Reference Yan2014). This statistical method has also been frequently applied in processing questionnaire data with a psychometric approach. Three studies are discussed in this section, which focus on the development and validation of scales by using Rasch Models.

The Replication of Japanese English Medium of Instruction Attitude Scale in the Higher Education Context of Mainland China – A Case Study

To further explain the application of Rasch modeling in EMI-related research, this chapter features a replicated use of Japanese English Medium of Instruction Attitude Scale (JEMIAS) (Curle, Reference Curle2018) in the higher education context of mainland China. Similar to the situation in Japan, Chinese universities are also experiencing growth in the development of EMI courses and programs. The challenges encountered by instructors, students, and other stakeholders have been described and analyzed in a wide range of studies, such as the regional disparities rooted in the general English education landscape (Kong & Wei, Reference Kong and Wei2019) and divergent language support provided by different types of EMI institutions in the country (McKinley et al., Reference McKinley, Rose and Zhou2021).

The purpose of conducting this case study was twofold: First, the researcher used Rasch modeling to interpret survey data, thus offering more diverse statistical options for examining the validity and reliability of questionnaires. Also, the JEMIAS instrument implemented among Chinese students was slightly adapted such that the data collected will provide useful information for investigating the efficiency of JEMIAS in measuring students’ attitudes across different contexts. The following are the research questions for this case study :

• Research Question 1: How can Rasch Models be applied in analysing questionnaire data with the guidance of step-by-step instructions

• Research Question 2: Will students’ attitudes toward EMI be influenced by their academic majors?

• Research Question 3: With which JEMIAS items would the student respondents agree or disagree the most regarding their attitudes toward EMI?

Questionnaire Results Analysis and Discussion

• Research Question 1: How can Rasch Models be applied in analysing questionnaire data with the guidance of step-by-step instructions

Step 4: Subscales and Dimensions

Factor analysis, which has been implemented during the construction phase of JEMIAS, is pivotal in confirming the structure of a questionnaire instrument. An important function of factor analysis is helping researchers identify subscales, or different groups of items. The significance of extracting subscales from one instrument has also been emphasized in Krägeloh et al. (Reference Krägeloh, Medvedev, Hill, Webster, Booth and Henning2018) and Zhao et al. (Reference Zhao, Huen and Chan2017). The “positive” wording and “negative” wording of questionnaire items, however, are also of vital importance to the successful interpretation of questionnaire data. For example, JEMIAS contains items such as “I think that increasing English-taught programs taught in Japan is not practical,” where the statement sentence is phrased in the form of negative wording. Participants’ responses to these items would deviate from those for items such as “Because I learn through English, I can remember and retain my English.” The possible influence of item wording on the item score is also in need of evaluation when researchers attempt to analyze participants’ responses.

In this case study, the researcher first conducted MFRM on the five items grouped as Factor 1: “Negative impact of EMI,” where “negatively impact” and “less” are the keywords for phrasing the item texts. In contrast with “strongly agree,” responses of “strongly disagree” would lead to a positive evaluation of EMI.

Figure 7.1 demonstrates a Wright map generated from MFRM, which presents a preliminary view of the synergic effects of the three facets on the item score. The first column of asterisks (“*”) represents the number of participants who responded to the questionnaire, and the number “0” is set to represent the origin of the logit scale. In the second column, however, it could be noticed that participants’ academic major of “Liberal arts,” “Medical science,” “Science,” and “Social science” are located at approximately the same level. “Engineering,” however, appears beneath the level of “0.” As the script in Table 7.4 specifies, “For facet 1 (respondent), higher measures mean higher scores. For the other two facets (academic major and questionnaire item), higher measures mean low scores.” In the data analysis procedure, a higher logit scale generated from “academic major” or “item” usually leads to a lower item score, and the participants are leaning toward the end of disagreement rather than agreement. In other words, engineering majors tend to endorse a higher score for the items on the questionnaire. They incline to be on the “agree” side in acknowledging the negative impact of EMI and are more likely to perceive the negative influence of EMI.

Figure 7.1 MFRM Wright map for the items of Factor 1

The third column includes a ranking of the five items selected for MFRM analysis. Items located in the upper section indicate a higher logit scale, and participants are more likely to select a lower score for the item. In this scenario, respondents tend to disagree more with Item 46: “I think that an increase in courses taught through English in Chinese universities has had a negative impact on teaching and learning.” In comparison, Item 3, “Because I learn courses in English, I have less time to do things I am interested in” has the lowest logit scale, and the respondents are inclined to agree with the statement.

The last column offers researchers with information about participants’ actual use of the scale. For the five items grouped in Factor 1, respondents are relatively consistent in endorsing “2” and “8” across the items. The distance between “1” and “2” and that between “8” and “9”, however, are much wider, which implies that “1” and “9” are much less frequently used. If the short lines under “2” or over “8” fluctuate violently across items, researchers may need to investigate more participants’ use of scales. In this case, participants have their own interpretation of the scale and cannot agree to a consensus. It is also possible that the internal validity of each item calls for a more detailed examination.

Concluding Remarks

As one of the categories of Rasch Models, MFRM has provided researchers with a practical option to process questionnaire data. Through transforming ordinal scales into interval scales, Rasch modeling preserves the property of “ranking orders” for questionnaire items. However, the final logit scale still enables researchers to conduct numerical and even parametric comparison across different groups of participants or items. In addition, MFRM captures the concept of “facet” and presents Wright maps that integrate all related facets (e.g., individual participants, participants’ background information, items) into a panoramic picture. Researchers are offered more opportunities to narrate the “story” behind scales, thus expanding abundant space for data interpretation.

The pilot study included in this chapter acts as an instructional case for unpacking the whole process of conducting MFRM analysis. The interpretation of logit scales, however, also elicits more questions regarding the EMI context in the Chinese mainland. First, the adapted JEMIAS does not reveal prominent differences across Chinese undergraduate students’ disciplinary background. The scale is measuring student participants’ most fundamental perceptions and attitudes toward EMI. Second, individual items related to specific factors, such as the relationship between EMI instruction and China’s international identity as well as EMI-related teaching practices, call for further investigation. A more granular analysis of Chinese students’ concerns, opinions, and attitudes toward EMI would hopefully lead to stronger instructional support and more efficient plans for designing relevant programs in the future.

Introduction

The emergence of English Medium Instruction (EMI) education attracts a group of language education scholars who adopt a wide range of research methodologies to investigate various EMI issues, including the effectiveness of EMI programs (e.g., Lo & Lo, Reference Lo and Lo2014) and perceptions, experience, and characteristics of different stakeholders (e.g., Farrell, Reference Farrell2020; Pun, Reference Pun2022). One popular research methodology, which is widely adopted in the current EMI literature, is questionnaire-based research (e.g., Macaro et al., Reference Macaro, Akincioglu and Han2019; Xie & Curle, Reference Xie and Curle2022). While questionnaire findings depict certain features of a large population (e.g., EMI student motivation), questionnaire-based EMI research may be doubted regarding its validity and reliability. Such critiques are due to researchers’ inadvertent negligence of important steps during questionnaire development, validation, and analysis procedures (Curle & Derakhshan, Reference Curle, Derakhshan, Pun and Curle2021). For example, many EMI researchers are not aware of the necessity of a pilot study, or the validity of the questionnaire is not well documented (Macaro et al., Reference Macaro and Akincioglu2018).

To offer some suggestions on how to conduct questionnaire-based EMI research, this chapter reports an empirical study on Chinese EMI students’ attitudes and motivation toward learning the content subject matter in English. With the EMI study as an example, the researchers intend to demonstrate specific steps in questionnaire development (Dörnyei, Reference Dörnyei2007) and methods of questionnaire validation and analysis in the pilot stage.

The Research Project

This research project is to examine and compare Chinese EMI students’ motivation and attitudes toward EMI education. The role of different demographic variables (i.e., discipline, gender, study level) in EMI learner motivation is also investigated. Methodologically, it demonstrates how quantitative approaches and methods (mainly the questionnaire study) can be adopted in EMI research and how the quantitative data can be analyzed with inferential statistical analyses. The motivations behind the research project are multifold. First, along with the emergence of EMI as a vibrant research field (Macaro, Reference Macaro2022), a mushrooming literature has explored EMI policymaking and curriculum design as well as EMI teachers’ classroom practice and continuing development (e.g., De Costa et al., Reference De Costa, Green-Eneix and Li2020; Pun & Thomas, Reference Pun and Thomas2020; Yuan et al., Reference Yuan, Li, Peng and Qiu2022). While such a line of inquiry sheds light on the effective implementation and reforms of EMI in higher education, it is equally crucial to consider students’ perspectives and experiences in diverse EMI programs. In particular, in view of the pivotal role of motivation in shaping students’ learning choices and subsequent engagement and efforts (Gardner, Reference Gardner2004), there is a need to investigate students’ motivation and attitudes as well as the influencing factors in EMI environments.

Second, a cluster of recent studies (e.g., Kim et al., Reference Kim, Kweon and Kim2017; Zhang & Pladevall-Ballester, Reference Zhang and Pladevall-Ballester2022) has revealed students’ positive attitudes toward EMI, their lack of self-confidence about English use, and their tendency to rely on first language (L1) in EMI classrooms. One notable limitation of these studies is their focus on students’ general attitudes and motivation (e.g., about course experience and language use) without probing the complexities involved in their perceptions shaped by various demographic factors, personal dispositions, and contextual forces. For instance, Jiang et al. (Reference Jiang, Zhang and May2019) investigated students’ motivation toward EMI in the Chinese context, and their research design and focus were primarily concerned with students’ language learning needs as the major determinant of students’ EMI motivation. In Macaro and Akincioglu’s (Reference Macaro and Akincioglu2018) survey study, students’ EMI motivation was examined in relation to different variables including gender, year group, and university type. Although the findings demonstrated the strong mediating effects of such factors on students’ choices to enter EMI programs, the study cannot fully capture students’ motivation influenced by intricate, dynamic sociocultural environments. Such a gap gives impetus to the present research project, which, informed by relevant motivational theories, looks into the complexities of students’ EMI motivation in Chinese higher education settings. Particularly, our study not only focuses on the linguistic aspect of EMI (e.g., students’ attitudes toward English and their language learning needs) but also treats EMI as a powerful form of educational experience, in which students’ multiple types of motivation (e.g., integrative and instrumental motivation) may emerge and evolve in their situated institutional and sociocultural settings.

Third, China presents an interesting and meaningful context for EMI research given its fervent pursuit of internationalization of higher education in the past decades. As reported by Jablonkai and Hou (Reference Jablonkai and Hou2023) in their recent review article on EMI in China, around 98 percent of Chinese universities have made active attempts to launch bilingual or EMI programs, while some top universities even offer over 200 EMI courses in a variety of disciplines to attract international students and enhance their level of internationalization. On the other hand, EMI is not without controversy in China. For instance, given the status of English as a Foreign Language (EFL) in China, some university teachers and students may show resistance toward EMI because it may hinder students’ comprehension and application of content knowledge. Some even hold the perception that English as a colonizing and imperialist language can pose threats to the local knowledge and culture in Chinese society (Yuan, Reference Yuan2020). Such beliefs speak to the complexities embedded in the provision of EMI, which not only is a matter of instruction language but also intertwines with diverse intuitional, social, and cultural issues. By exploring students’ motivation and attitudes toward EMI against such a complex backdrop, the project can bring relevant and practical insights to EMI teachers and university managers in both China and similar EFL settings (e.g., Japan and Korea).

Quantitative Research Design

To shed light on university students’ EMI motivations and attitudes in Chinese universities, the research team adopted a mixed methods approach, which involved a large-scale questionnaire survey and follow-up case study (e.g., interviews and observations). Given the scope of the book focusing on quantitative research methods in EMI research, this chapter presents and discusses the design and implementation of a questionnaire survey with practical implications and directions for future research.

We chose to conduct the questionnaire survey based on the following reasons. First, the questionnaire survey study is cost-effective (Dörnyei & Taguchi, Reference Dörnyei and Taguchi2009), and it enables the team to grasp a general understanding of motivation data from EMI students in a wide range of contexts and disciplines. Second, the researchers can conduct inferential statistical analyses to examine EMI motivation and attitudes and explore the role of demographic variables within the constructs. The researchers’ bias can be avoided. Third, the questionnaire survey also serves as the first stage of a larger EMI research project. Based on the findings, the researchers interviewed and observed some of the questionnaire participants to capture in-depth information about their motivational constructs (cf. the explanatory sequential design in Creswell, Reference Creswell2015).

Method in Detail

Data Reporting

Implications of Conducting EMI Questionnaire-Based Research

Questionnaire-based research is an appropriate research methodology for investigating various EMI research topics, including but not limited to EMI student motivation (Rose et al., Reference Rose, Curle, Aizawa and Thompson2020) and self-efficacy beliefs (Thompson et al., Reference Thompson, Aizawa, Curle and Rose2022). These questionnaire-based studies not only provide valid and reliable measurements for EMI scholars and language teaching practitioners but also depict a descriptive and generalizable picture of a large population. Correlation and causal relationships of different learner/teacher variables can also be examined (e.g., Soruç et al., Reference Soruç, Pawlak, Yuksel and Horzum2022).

While the significance of questionnaire-based EMI research is acknowledged (Curle & Derakhshan, Reference Curle, Derakhshan, Pun and Curle2021), its effectiveness could be optimized by strictly following established guidelines in questionnaire development and employment (e.g., Dörnyei, Reference Dörnyei2007). This process should ideally include the creation of an item pool, selection of appropriate question items, piloting of these items, and conducting item analysis before questionnaire validation. Administering the questionnaire to different student populations and performing inferential statistical analyses on the data (e.g., EFA, the reliability test) are also necessary. It should be reminded that EFA is typically suitable for validating questionnaires in their pilot stage, as demonstrated in the current study, while confirmatory factor analysis (CFA) can be further conducted when testifying the validity of a scale in the follow-up stage. In addition, various analytical methods can be adopted to examine questionnaire data and/or investigate the relations of different variables and components. These methods include but are not limited to parametric and non-parametric tests (e.g., t-test, ANOVA [analysis of variance], path analysis, structural equation modeling, and facet Rasch measurement), with researchers selecting the method that is most appropriate for their specific research questions.

The questionnaire survey design can be an integral part of the mixed methods research design in educational studies (Creswell & Clark, Reference Creswell and Clark2017). Three models of mixed methods are proposed. The first model (the explanatory design) is that questionnaire data are collected first, which inform what kinds of qualitative data are needed to supplement and explain the quantitative results. In Qiu and Fang’s (Reference Qiu and Fang2022) mixed methods research on EMI students’ perceptions of their native and nonnative English-speaking content subject teachers, the researchers first administered a questionnaire survey to 101 EMI students and 12 out of the 101 participants were interviewed on their perceptions of EMI teachers based on their input in the questionnaire survey. This is an example of the explanatory design where the interview findings reflected the reasons underlying the participants’ questionnaire input.

In the second model (the exploratory design), for example, interview data with a small group of EMI students can be collected, which inform the development of larger-scale questionnaires on a related topic. Although this method is relatively rare in EMI research, Yuan and Qiu’s (Reference Yuan, Qiu, Sah and Fang2023) design of their questionnaire on EMI teachers’ language attitudes followed this pattern. Interviews with three EMI teachers were conducted, which served as an exploratory pilot study. The three teachers’ attitudes toward L1 use in EMI classrooms also informed the development of the questionnaire survey, and the researchers integrated some of their views into the survey to explore whether the views were generally accepted by other EMI teachers in China.

The questionnaire data and the qualitative data (e.g., interview) may also be collected simultaneously (Creswell, Reference Creswell2015). In Baker and Hüttner’s (Reference Baker and Hüttner2019) comparative study of EMI programs in Europe and Asia, the researchers surveyed 121 international students from different EMI programs in different countries (Thailand, Austria, United Kingdom) and interviewed four EMI lecturers and eight EMI students. The survey results and interview findings supplemented each other.

To conclude, this chapter discusses how questionnaire-based research can be designed in the context of EMI and how questionnaire data can be quantitatively analyzed to explore EMI students’ attitudes and motivation and to reveal the different EMI attitudes and motivation degrees of students with different demographic information. It suggests that questionnaire-based studies are appropriate for EMI research, but their potential would be maximized if they could be supplemented by qualitative data.

Introduction

It is widely assumed by English Medium Instruction (EMI) researchers that EMI is beneficial to learners’ language development (e.g., Kong & Wei, Reference Kong and Wei2019; Macaro et al., Reference Macaro, Curle, Pun, An and Dearden2018; Pun & Curle, Reference Pun, Curle, Pun and Curle2022; Yang, Reference Yang2014). The effectiveness of EMI on language learning has been a classic and extensively discussed topic, with various methods used to address it (see Lo & Lo, Reference Lo and Lo2014, for a research synthesis). Most studies to date have used questionnaire or language tests to measure learners’ learning outcomes and ascertain the effectiveness of EMI. Recently, a new research method has emerged in Content and Language Integrated Learning (CLIL) research to investigate the effects of CLIL on students’ language learning outcomes, namely corpus-based analysis. Compared to traditional methods, corpus-based analysis can unveil learners’ longitudinal linguistic development as instruction progresses, providing more direct and convincing evidence on the benefits of CLIL (Hafner & Wang, Reference Hafner and Wang2018; Lahuerta Martínez, Reference Lahuerta Martínez2017). For example, Bulté and Housen (Reference Bulté and Housen2019) investigated pupils’ linguistic development in a CLIL program over a period of nineteen months. Their study shows that learners’ written productions exhibit noticeable growth in syntactic complexity, such as longer production units and the use of more subordinated structures after the CLIL education. This research paradigm may also have great potential in EMI research, offering researchers an alternative perspective to explore the benefits of EMI. As such, this chapter aims to introduce the application of this quantitative research method (i.e., corpus-based analysis) in EMI research with three major tasks. First, it will summarize relevant literature exploring the effects of EMI on English learning in the second section. The third section will focus on how to use corpus-based analysis to conduct relevant studies, including corpus construction, linguistic analysis instruments, and statistical analyses. Lastly, the fourth section will present one example study that demonstrates the use of corpus-based analysis in EMI research. The example study aims to use corpus-based analysis to investigate learners’ longitudinal development of phraseological competence in an EMI course in the Chinese context.

Previous Studies on the Effects of EMI on Learners’ Linguistic Development

As pointed out by Pun and Curle (Reference Pun, Curle, Pun and Curle2022), the effect of EMI education on language learning is a pivotal research area within the field of EMI. Many researchers contend that due to the constant exposure to authentic language materials, learners’ language proficiency could be enhanced through EMI education (e.g., Aizawa, et al., Reference Aizawa, Rose, Thompson and Curle2020; Bolton & Botha, Reference Bolton and Botha2015; Jiang et al., Reference Jiang, Zhang and May2016; Marsh et al., Reference Marsh, Hau and Kong2000). Lo and Lo (Reference Lo and Lo2014) used several theories to explain the positive impact of EMI on language learning. To be more specific, according to the comprehensive input hypothesis, the provision of high-quality input is conducive to language learning. Meanwhile, many psychological theories assert that learners’ affective factors play a critical role in language learning, such as a higher level of language motivation. They further summarized that EMI can expose learners to substantial language input and enhance their language learning motivation, leading to successful learning outcomes. Nonetheless, the relevant discussion on the effectiveness of EMI education is overall “long on claims and short on empirical research” (Kong & Wei, Reference Kong and Wei2019, p. 44). Most existing empirical studies mainly use experimental designs to investigate the differences in the language outcomes between learners in EMI programs and those in non-EMI programs. In other words, the benefits of EMI programs are mainly established by comparing the achievements of learners in EMI and non-EMI programs (Lei & Hu, Reference Lei and Hu2014). For example, Lin and Morrison (Reference Lin and Morrison2010) compared the academic vocabulary knowledge of Hong Kong EMI and non-EMI learners. Participants’ lexical knowledge is measured by two aspects: receptive word knowledge and productive word knowledge. Their results indicated that learners from EMI programs outperformed those from Chinese Medium Instruction programs in both the vocabulary tests. Thus, they concluded that EMI is more beneficial to language learning. However, studies of this type are limited in the following two aspects. In the first place, the longitudinal development of learners’ language competence has been neglected. As a result, the relevant studies fail to inform us of the extent to which and how learners’ language competence has developed longitudinally in the EMI program. In the second place, language proficiency is normally measured by the scores of language tests in previous studies (e.g., Lei & Hu, Reference Lei and Hu2014; Lo & Murphy, Reference Lo and Murphy2010; Yuksel et al., Reference Yuksel, Soruç, Altay and Curle2023). Compared with measures of linguistic complexity in learners’ productions, test scores are more large-grained measures of language proficiency, which may not demonstrate the nuanced picture of learners’ language competence (Bulté & Housen, Reference Bulté and Housen2019). Therefore, more empirical data of learners’ longitudinal linguistic development are needed to better demonstrate the benefits of EMI education.

Recently, there emerged a new research trend in CLIL research, that is, using corpus-based techniques to analyze the longitudinal changes in linguistic complexity measures in learners’ written and spoken productions so as to demonstrate CLIL benefits. Linguistic complexity refers to the absolute sophistication of linguistic features in learners’ productions. Relevant measures can quantify the number of linguistic constructions in their productions, thus gauging their linguistic competence (Bulté & Housen, Reference Bulté and Housen2014; Lu, Reference Lu2011). Additionally, comparing complexity scores of students’ productions over different periods can measure the development of their linguistic competences with the progression of CLIL. Along similar veins, Jablonkai (Reference Jablonkai, Pun and Curle2021, p. 102) remarks that the analysis of lexical and syntactic complexity measures may unveil learners’ “gains in terms of discipline-specific language competence.” Table 9.1 presents the details of the relevant studies, including the context and participants of the study, linguistic complexity measures, statistical analyses, and major findings. The main objective of research of this line is to track CLIL learners’ longitudinal linguistic development, ultimately providing empirical evidence on CLIL benefits. Since EMI and CLIL share a similar pedagogical goal and an overall design (Wannagat, Reference Wannagat2008), it is assumed that the corpus research paradigm holds potential in research on EMI benefits. Therefore, as pointed out earlier, the subsequent section aims to elaborate on the application of corpus-based analysis in EMI research.

Furthermore, although several corpus-based studies have been conducted to examine the impact of CLIL on language learning, these studies are subject to three limitations. Firstly, the existing studies are mainly concerned with the longitudinal development of learners’ syntactic and lexical competence. However, researchers contend that linguistic complexity is a multidimensional construct that encompasses at least four components: syntactic, lexical, morphological, and phraseological complexity (Bulté & Housen, Reference Bulté and Housen2019; Lu, Reference Lu2012; Paquot, Reference Paquot2018). In other words, previous studies have neglected other linguistic complexity components, such as phraseological complexity, in addition to syntactic and lexical complexity. A more comprehensive understanding of learners’ longitudinal development of linguistic competence in the immersion programs need further exploration. Secondly, most of these studies have small sample sizes, such as five participants in Bulté and Housen (Reference Bulté and Housen2019) and fifteen in Lázaro and García Mayo (Reference Lázaro and García Mayo2012). Lastly, as for the results, virtually all the studies demonstrate that CLIL has a positive impact on learners’ linguistic development. That is, the complexity measures in learners’ productions show increasing tendencies as the instruction proceeds (see Table 9.1). However, the corresponding reasons for the growth of learners’ linguistic knowledge in CLIL programs remain underexplored. The existing studies are mostly descriptive, and few explanatory studies have been conducted to investigate the relevant factors accounting for CLIL benefits.

To summarize, more attention should be paid to the longitudinal development of leaners’ other linguistic competence, such as phraseological competence in language immersion programs. Data of larger sample sizes may also render the results more representative and generalizable. These limitations of CLIL research should also be avoided in the research of EMI benefits. Accordingly, we have carried out a study to investigate eighty-nine students’ longitudinal development of phraseological complexity measures during a semester-long EMI course in the Chinese context. Linguistic features of learners’ language input are also examined to show the possible effects of textbook input on learners’ linguistic gains in EMI education. This study will be reported in detail in the section titled “The Study.” The introduction of this study will hopefully demonstrate the potential of corpus-based analysis in understanding the extent and manner in which EMI benefits learners’ language development. Before the example study is presented, the following section will provide an overview of the methodology employed in relevant studies. This will offer target readers an understanding of how to conduct corpus-based analyses in this area.

Methodology of Corpus-Based Studies on EMI Effectiveness

This section is dedicated to the methodology of studies utilizing corpus-based analysis to investigate the effectiveness of EMI. It commences with a discussion of data collection, followed by an overview of the measures and instruments. Finally, it concludes with the statistical analyses used to reveal differences in linguistic complexity measures and demonstrate the longitudinal gains in linguistic competence. The methodology details outlined in this section may assist researchers and students in designing their own studies.

The Study

This section aims to introduce the example study for two reasons. Firstly, presenting this study will provide readers with a more profound comprehension of the potential of corpus-based analysis in EMI research. Secondly, since this study has addressed the limitations of previous research, it may offer theoretical implications for relevant studies. The example study intends to investigate eighty-nine students’ longitudinal development of phraseological competence in a four-month-long EMI course in an Eastern Chinese university. The corresponding reason for EMI benefits is also examined in this study. More specifically, this study chooses to elaborate on one important variable, that is, learners’ input to show the extent to which textbook input would influence learners’ linguistic competence in an EMI context. The following are the detailed research questions of this study:

1. 1. How does learners’ phraseological competence change with the progression of EMI?

2. 2. To what extent would textbook input influence learners’ linguistic performance?

The data of this study consist of students’ written responses to several short answer questions in an EMI course, whose name is English Novel Reading, along three data collection times. Ninety-five students participated in this study but eighty-nine of them completed all three assignments. Apart from taking the compulsory English as a Foreign Language classes, participants chose to enroll in this EMI course in order to boost their English proficiency and to gain more knowledge of British and American literary works. The eighty-nine students come from different science and humanities majors. Seventy-four percent of them are second-year university undergraduate students while the remaining 36 percent are first-year or third-year undergraduates. Based on the test scores of the College English Test (Band-4), they are of intermediate language proficiency. As can be seen from Table 9.3, our learner data comprise the written productions of these eighty-nine students, which amount to 32,417 words. To address the second research question, we also built a textbook corpus, which consists of the excerpts of novels for learners to read at each data collection point. These excerpts of original novels are used by instructors to teach, and students’ written productions are related to the excerpts.

This study centers on learners’ phraseological competence for the following three reasons. In the first place, as argued earlier, phraseological complexity is understudied in EMI literature. In the second place, the results of previous studies indicate that compared with syntactic and lexical complexity measures, phraseological complexity measures are more correlated with learners’ language competence for intermediate-level and advanced-level learners (Paquot, Reference Paquot2018). In the third place, some studies suggest that the appropriate and idiomatic use of phraseological units such as collocations and n-grams pose challenges for second language (L2) learners (Jiang et al., Reference Jiang, Bi, Xie and Liu2021). In conclusion, investigations of the development of phraseological competence may show a more comprehensive picture of EMI’s effects on learners’ linguistic development. Since there are various types of phraseological units (collocations, bigrams, formulaic sequences, etc.), it is not feasible to include all of them in a single study. Therefore, this study focuses on the widely researched phraseological unit, namely n-grams (i.e., bigrams and trigrams). Bigrams refer to the contiguous sequences of two words and trigrams refer to contiguous units containing three words. Learners’ phraseological competence is conceptualized by the sophistication of bigrams and trigrams. To quantify this sophistication, this study adopts two approaches suggested by Paquot (Reference Paquot2018): association strength-based measures (such as MI-score and T-score) and the proportion of bigrams/trigrams in a reference corpus. Association strength-based measures aim to quantify the likelihood for the two individual words of a bigram to appear together in a reference corpus (Jiang et al., Reference Jiang, Bi, Xie and Liu2021; Paquot, Reference Paquot2018). The higher the scores, the more complex the n-grams. Therefore, regardless of the operationalization of phraseological complexity, a reference corpus is necessary to calculate the scores of phraseological complexity measures. The Corpus of Contemporary American English (COCA) has been frequently used as a reference corpus due to its large size and representativeness of contemporary English. Thus, COCA is used as a reference corpus in this study as well to compute the scores of phraseological complexity measures. Eight specific phraseological complexity measures are adopted in this study: bi_MI, bi_T, tri_MI, tri_T, bi_prop_30k, bi_prop_60k, tri_prop_30k, and tri_prop_60k (see Table 9.4). The instrument used to analyze these phraseological complexity measures is TAALES (Kyle et al., Reference Kyle, Crossley and Berger2018). One thing to note is that TAALES calculates the scores of phraseological complexity measures in reference to the different subcorpora of COCA, such as the COCA magazine corpus, the COCA news corpus, and the COCA academic corpus. Since our study is conducted in the context of an academic EMI course, we selected only the measures calculated in reference to the COCA academic corpus. Scores of the eight phraseological complexity measures in learners’ written productions along the three collection times will be compared by the use of repeated measures ANOVA to demonstrate the development of learners’ phraseological competence.

Table 9.5 presents the descriptive statistics of the eight phraseological complexity measures in learners’ written productions along the three data collection times. The development trends of the eight measures through the three time points are presented in Figure 9.1. As is clear from Table 9.5 and Figure 9.1, four measures show clear ascending tendencies with the progression of EMI, that is, bi_MI, bi_T, tri_MI, and tri_prop_60k. There are no obvious development patterns observed for the following two measures: tri_T and bi_prop_30k. Surprisingly, two measures, that is, bi_prop_60k and tri_prop_30k, show a striking descending tendency as time proceeds. Several repeated measures ANOVA tests were also conducted to ascertain whether time or EMI education exerts significant effects on the eight phraseological complexity measures. The results indicate that there exist significant differences in six measures along the three data collection times: bi_MI (F = 13.571, p = 0.000^*), bi_T (F = 10.823, p = 0.000^*), tri_MI (F = 11.916, p = 0.000^*), bi_prop_60k (F = 8.985, p = 0.000^*), tri_prop_30k (F = 4.703, p = 0.01^*), and tri_prop_60k (F = 5.457, p = 0.005^*). No significant differences among the three times were observed for the other two measures: tri_T (F = 0.679, p = 0.412^*) and bi_prop_30k (F = 0.142, p = 0.868^*).

Figure 9.1 Development trends of the eight phraseological complexity measures along three data collection times

In summary, the results presented here indicate that learners’ phraseological knowledge exhibits growth in EMI education, especially with regard to the association strength of bigrams and trigrams. Subsequently, we examined the distribution of high-frequency bigrams and trigrams in the textbooks and learners’ written productions. The similarities of phraseological units between the input and output enabled us to understand the effects of the input on EMI learners’ language use. Many high-frequency bigrams consist of two functional words such as of the, which may not be informative with regard to the input effects. Therefore, those bigrams that consist of two functional words were removed from analysis. Tables 9.6–9.8 list the most frequent twenty bigrams and trigrams in the textbook and learners’ written productions. It could be summarized from Tables 9.6–9.8 that leaners tend to use some bigrams and trigrams that appear in the textbook, showing that input would influence students’ language use. Also, as time proceeds, the effect is getting larger in that at data collection time 2 and time 3, more bigrams and trigrams in learners’ productions are the same as those of textbook input (see Tables 9.7 and 9.8). It is hypothesized that the influence of textbooks may allow learners to produce more complex phraseological units, leading to positive outcomes in their phraseological competence in English as a Foreign Language learning.

However, the analysis on input effects presented here is exploratory, and there are some limitations to this analysis. Firstly, the comparison includes only the most frequent twenty n-grams between the input and output, which may introduce bias and not fully capture the influences of the input on learners’ written productions. Secondly, the correspondence of phraseological units between the input and output is just one explicit way to demonstrate the effects of textbook input. Future studies may employ alternative methods to operationalize the textbook effects to triangulate the findings.

Concluding Remarks

This chapter outlines the application of corpus-based analysis in exploring the impact of EMI on learners’ linguistic development. It details the methodology of such analyses. To summarize, researchers need to consider the design of the corpus, the sustainability of measures, and the robustness of the statistical analyses. Additionally, an example study is presented to offer readers a clear understanding of the value of corpus-based analysis in EMI research. Future studies may compare the longitudinal linguistic gains of EMI learners with those of non-EMI learners to provide further empirical evidence of EMI benefits.

Introduction

Research in the field of language education has a long-standing history of utilizing structural equation modeling (SEM) to examine intricate relationships among variables related to language learners. It is suggested that models developed in SEM methodology “are usually, and should best be, based on existing or proposed theories that describe and explain phenomena under investigation” (Raykov & Marcoulides, Reference Raykov and Marcoulides2006, p. 6). As one of the SEM methods, path analysis intends to explain causal relations formulated within the model based on existing knowledge on what a researcher wants to study without considering latent variables (Collier, Reference Collier2020). Under the assumption that variables in a hypothesized model are linearly connected, path analysis can simultaneously analyze complex relationships between dependent variables and independent variables and reveal direct and indirect effects of an independent variable on a dependent variable (Jeon, Reference Jeon2015). Therefore, researchers widely use this technique for examining and developing theories of interest in their research domain.

Despite the enduring popularity of path analysis, there has been limited research in the context of English Medium Instruction (EMI) to illustrate established theories. Moreover, researchers have yet to incorporate statistical data to refine the theoretical models and better elucidate the causal relationships between various factors that potentially influence EMI students’ academic achievement. To fill this gap, this study aims to develop and analyze a well-fitted model that could account for contingent links between variables that directly and indirectly affect EMI students’ academic achievement in science. Drawing on survey data from eight EMI secondary schools in Hong Kong, the current study identified interplayed roles of students’ English proficiency, language use in the science classroom, self-perceived English difficulty in the science classroom, self-concept of science learning, and science achievement.

Theoretical Background on EMI

Method

The data for this study were collected through a large-scale survey administered to senior secondary students attending EMI secondary schools in Hong Kong. The researchers carefully selected eight EMI schools that matched in terms of geographical location, average academic performance, curriculum design, and regional socioeconomic status. Prior to the recruitment process, the researchers sent invitation emails to the schools, requesting their participation in the study. All eight EMI secondary schools responded to the invitation and confirmed their willingness to cooperate.

In Hong Kong, secondary schools are categorized into three bands based on academic performance, with Band 1 representing above-average schools where students maintain high academic performance and are eligible to learn content subjects in English. In this study, all participating schools were from the Band 1 category, located in close proximity to each other, and shared similar demographic features.

Data Analysis

This study employed a three-phase data analysis approach. Firstly, a correlation analysis was conducted to examine the relationships between the variables LISC, EP, SA, PEDSC, and SCSL. This analysis was performed using Statistical Package for the Social Sciences (SPSS) 21.0 software, providing a foundation for model development and aligning with existing theories.

Based on the results of the correlation analysis and existing theories, three hypothesized models were constructed to capture the potential relationships between LISC, EP, PEDSC, and SCSL, and their impact on SA (refer to Figure 10.1). Model fit indices were then employed to compare and select the best-fitting model among the three alternatives.

Figure 10.1 Hypothesized models

Note: *English proficiency→EP; MOI (medium of instruction) interaction→LISC; English difficulty→PEDSC; self-concept→SCSL; science achievement→SA.

In the third phase, a path analysis using SEM was conducted for the selected model. Analysis of Moment Structures (AMOS) 21.0 software was utilized for this analysis. The path analysis allowed for an examination of the total, direct, and indirect effects of the independent variables on learners’ science achievement. The bootstrapping technique was employed to identify these effects accurately.

By employing these three phases of data analysis, this study aimed to explore the relationships among the variables and determine the factors that significantly influence students’ science achievement. The combination of correlation analysis, model development, and path analysis using SEM provides a robust approach to investigating these relationships and uncovering the underlying mechanisms.

Results

Effect Strengths in the Path Diagram

Based on existing theories on EMI and the results of the correlation analysis, three hypothesized models were developed. Figure 10.1 illustrates these models:

• Model 1: In this model, students’ SA is directly and indirectly influenced by the other variables. Both LISC and SCSL mediate the effects of EP and PEDSC on students’ science achievement.

• Model 2: This model considers both affective variables (SCSL and PEDSC) as mediating variables that directly impact students’ SA. In this model, EP and LISC have direct and indirect effects on students’ SA, but LISC does not have a direct effect on PEDSC.

• Model 3: Building upon Model 2, Model 3 includes an additional direct effect of LISC on PEDSC. This addition is based on the possibility that students’ LISC, which is significantly correlated with EP, may have an influence on PEDSC as it reflects the amount of English exposure in the science classroom.

These three models were developed to explore and understand the potential relationships between the focal variables and their effects on students’ science achievement. The models consider different pathways and mediating variables to examine the complex interactions among EP, LISC, PEDSC, SCSL, and SA in the context of EMI. Further analysis and evaluation of these models will help determine which model best explains the structural relationships between the variables and how students’ science achievement can be explained based on the chosen model.

Based on the model fit indices presented in Table 10.3, it was found that among the three hypothesized models, Model 3 provided the most appropriate fit in capturing the causal connections between the variables. The fit indices for Model 3 indicated a good fit: χ²(1) = 1.132, p > 0.05, Root Mean Square Error of Approximation (RMSEA) = 0.019, Tucker-Lewis Index (TLI) = 0.994, Adjusted Goodness of Fit Index (AGFI) = 0.981, and Comparative Fit Index (CFI) = 0.999.

Table 10.3 Fit indices for the research model

	χ2(p)	df	RMSEA	TLI	AGFI	CFI
Model 1	1.749 (.186)	1	0.045	0.965	0.972	0.996
Model 2	4.147 (.126)	2	0.054	0.949	0.967	0.990
Model 3	1.132 (.287)	1	0.019	0.994	0.981	0.999

χ² measures the goodness of fit based on the statistical significance of the data. However, it is important to note that χ² is sensitive to sample size. Therefore, it is recommended to consider other fit indices when analyzing SEM results. In this study, fit indices such as RMSEA, TLI, AGFI, and CFI were utilized, as they are relatively stable across different sample sizes. A desirable fit is typically indicated by an RMSEA value of 0.05 or less. Values greater than 0.95 for TLI, AGFI, and CFI suggest a reasonably good fit (see Blunch, Reference Blunch2013; Kline, Reference Kline2005). In the case of Model 3, the fit indices met these criteria, indicating that the model provides a satisfactory fit to the data. These findings support the use of Model 3 to further analyze the relationships between the variables and understand students’ science achievement in the context of EMI.

The results of the path analysis revealed several significant findings (Table 10.4). Firstly, participants’ EP and their self-perceptions, including SCSL and PEDSC, had statistically significant positive effects on their SA. The standardized coefficients were as follows: βEP → SA = 0.57 (p < 0.001), βSCSL→SA = 0.25 (p < 0.05), and βPEDSC→SA = 0.09 (p < 0.001). These findings suggest that students with higher English proficiency and more positive self-concept, and more challenges in using English in the science classroom, tend to have better science achievement.

Table 10.4 Maximum likelihood parameter estimates for the research model

Model path		Unstandardized coefficient (SE)	Standardized coefficient
LISC→PEDSC		−0.078 (.057)	−0.070
LISC→SCSL		−0.030 (.049)	0.033
LISC→SA		−0.042 (.063)	−0.028
SCSL→SA		0.415 (.068)***	0.253
PEDSC→SA		0.119 (.058)*	0.087
EP → PEDSC		−0.180 (.037)***	−0.254
EP → SCSL		0.078 (.031)*	0.132
EP → SA		0.553 (.042)***	0.569

Note: ^*p < 0.05, ^***p < 0.001.

Furthermore, when examining the effect of participants’ EP, it was found that it had a statistically significant positive path coefficient for SCSL (βEP → SCSL = 0.13) and a significantly negative influence on PEDSC (βEP → PEDSC = −0.25). This means that higher English proficiency was associated with more positive self-concept in the science classroom and lower perceived English difficulty. Overall, these results highlight the importance of English proficiency and self-perceptions in shaping students’ science achievement in the EMI context (see Figure 10.2).

Figure 10.2 Results of path analysis (standardized coefficients)

Note: *English proficiency→EP; MOI interaction→LISC; English difficulty→PEDSC; self-concept→SCSL; science achievement→SA.

Table 10.5 shows the decomposition of the standardized effects of research Model 3. The standardized total influence of each variable on participants’ science achievement was measured by evaluating its indirect and direct effects. For instance, the total effect of participants’ English proficiency on science achievement was calculated as follows: direct effect (0.569) + indirect effect mediated by students’ self-perceptions (0.011) = 0.58. The results showed that participants’ English competence was the factor with the strongest effect on science learning outcomes in EMI secondary schools. Students’ self-concept and their perceived level of English difficulty in the science classroom also positively affected their achievement in science subjects.

Table 10.5 Decomposition of standardized effects for the model

		LISC	SCSL	PEDSC	EP
SA	Direct effect	−0.028	0.253	0.087	0.569
	Indirect effect	−0.014	−	−	0.011
	Total effect	−0.042	0.253	0.087	0.580

Note: Numbers in bold indicate statistical significance.

Additionally, participants’ SCSL had a positive direct effect on SA with a value of 0.253. This suggests that positive self-concepts in science learning contribute to better science achievement. Moreover, participants’ PEDSC also showed a positive direct effect on SA, although its magnitude was relatively smaller (0.087). This indicates that students’ perceived language difficulty in understanding English in the science classroom contributes to better science achievement.

Discussion

The findings from path analysis, based on statistical data from eight EMI secondary schools in Hong Kong, provide insights into the relationships between English proficiency, language for interaction in the science classroom, self-perceived English difficulties, science learning self-concept, and academic achievement in science for secondary students.

At the individual level, English proficiency emerged as the strongest predictor of students’ science achievement. This finding aligns with previous research in the EMI context, highlighting the positive association between English competence and academic success (e.g., Lo & Lo, Reference Lo and Lo2014; Macaro, Reference Macaro2018; Rose et al., Reference Rose, Curle, Aizawa and Thompson2019). As studying content subjects in English requires an understanding and application of knowledge, a lack of English proficiency can hinder students’ ability to absorb and utilize knowledge of the subject matter (e.g., Beckett & Li, Reference Beckett and Li2012; Doiz et al., Reference Doiz, Lasagabaster and Sierra2013). The results underscore the importance of developing students’ academic English proficiency to enhance their learning of scientific knowledge in the EMI setting.

In contrast, the path analysis itself did not find a significant impact of communication mode in the science classroom on students’ scientific knowledge attainment. While previous studies have emphasized the effectiveness of mixed language use in promoting comprehension and learning (Wu & Lin, Reference Wu and Lin2019; Xie & Curle, Reference Xie and Curle2020), the findings suggest that language output types may not necessarily reflect students’ internal learning processes.

Regarding affective variables, students with a stronger science learning self-concept demonstrated higher science achievement. Additionally, contrary to expectations (Marsh et al., Reference Marsh, Trautwein, Lüdtke, Köller and Baumert2005; Shen & Tam, Reference Shen and Tam2008), students’ perceived level of English difficulty in science learning was positively associated with scientific knowledge acquisition. It is possible that the challenge of learning science subjects in English serves as a motivating factor that drives students to invest more effort. Students may recognize the importance of studying content subjects in English and be inspired to devote time and energy to science disciplines (Lasagabaster, Reference Lasagabaster2016). Although the current data cannot provide precise explanation for this phenomenon, it becomes evident that negative perceptions of EMI should not always be considered obstacles to effective learning of subject content.

Furthermore, the study revealed that both science learning self-concept and perceived English difficulty partially mediate the effect of English proficiency on science achievement. This underscores the importance of not only fostering students’ English skills but also paying attention to their psychological well-being and providing an effective learning environment.

Conclusion

The current study contributes to the EMI literature by examining the complex relationships between variables that can impact students’ science learning outcomes. The empirical evidence obtained in this research supports previous findings while also providing new insights into the roles of each independent variable in the EMI secondary education context in Hong Kong. Based on the findings from path analysis, the following conclusions are drawn:

• Consistent with previous findings, English proficiency remains a potent factor for effective learning of science subjects in the EMI context. To ensure good learning of science subjects, EMI secondary schools should make efforts to promote students’ English skills.

• The type of language used in the science classroom may influence students’ understanding of subject content, but it does not guarantee the quality of their actual acquisition of scientific knowledge. Despite the widespread support for the translanguaging approach in EMI context, empirical evidence about its effectiveness in learning is lacking (Cummins, Reference Cummins, Juvonen and Källkvist2021). Good learning of science subject knowledge may extend beyond language use in the EMI classroom.

• While previous EMI research has often highlighted the relationship between students’ positive attitudes toward EMI and academic achievement, the results in this study reveal that students’ perceived difficulty with English does not necessarily lead to negative science learning outcomes among EMI students. Although the data from the current study do not provide further explanations about the phenomenon, it is worth delving into the motivational mechanisms under the surface, which are still underexplored in the EMI research area.

While the study provides a holistic understanding of the relationships between the variables of interest, there are several limitations that should be addressed in future research. Firstly, self-report surveys can be subject to trustworthiness issues, and additional research using different methods is needed to enhance the credibility of the findings. Secondly, the study includes a small number of EMI secondary schools in Hong Kong, limiting the generalizability of the results. Future research should involve a larger sample of local EMI schools to validate the current findings. Lastly, the study focused on a limited set of variables in the path analysis. To gain a deeper understanding of the mechanisms promoting EMI students’ science achievement, future studies should explore more comprehensive models with a broader range of variables.

Introduction: What Is a Questionnaire Survey?

As one of the most popular methods in social science, survey research seeks to understand a population as a whole through investigating a sample of individuals within this group. It tends to follow the postpositivist paradigm, grounded in the theoretical assumption that researchers can integrate certain pieces of knowledge conveyed by participants with other pieces of knowledge to generate new understandings of a phenomenon. In the field of English Medium Instruction (EMI), survey has been used extensively in a wide variety of studies (e.g., Drljača Margić & Vodopija-Krstanović, Reference Drljača Margić and Vodopija-Krstanović2018; Graham et al., Reference Graham, Eslami and Hillman2021; Jiang et al., Reference Jiang, Zhang and May2019; Kim & Thompson, Reference Kim and Thompson2022; Macaro et al., Reference Macaro, Tian and Chu2020; Qiu & Fang, Reference Qiu and Fang2022; Xie & Curle, Reference Xie and Curle2022).

Survey data can be gathered through structured interviews or self-administered questionnaires; it is the latter that is at the forefront of the discussion in this chapter. A questionnaire survey refers to “any written instrument that presents participants with a series of questions or statements to which they should react either by selecting from existing possibilities or writing out their answers” (Brown, Reference Brown2001, p. 6). According to the definition, participants are required to provide answers to prompts that take shape in either statements or questions. The popularity of questionnaire surveys in EMI research can be partly attributed to the fact that it is reasonably straightforward to construct and is capable of collecting vast quantities of information readily available for analysis. This is especially true when the questionnaire surveys are administered to a diverse group of participants, for example, a mix of EMI lecturers and students, or EMI students from different sociocultural backgrounds. In this case, data collection can be completed within hours. Considerable investment of time and effort can be saved when using online questionnaire surveys, for researchers just need to send emails and regular reminders to participants. Researchers are free from entering data one by one; rather, such numerical data can be downloaded and transformed to various formats. Compared to the classic pen-and-pencil questionnaire surveys, which require printing, mailing, and possibly dispatching by researchers themselves, a good return can be expected on the cost when online survey software packages and services are used.

Apart from enhancing efficiency with regard to researcher time, effort, and cost, one contributing factor to the widespread use of questionnaire survey in EMI studies is its versatility that facilitates researchers’ measurement of abstract constructs. These abstracts are assumed to exist and can be difficult to be measured otherwise. A brief glance at research outlets (e.g., academic journals, conference proceedings, theses, dissertations) reporting empirical evidence of EMI reveals a diverse application of questionnaire survey. Researchers use questionnaire survey to collect data on learner beliefs, learner motivation, learner characteristics, parents’ perspectives toward EMI implementation, learner expectations, learner attitudes, learners’ EMI learning experience, instructor needs, learners’ self-efficacy, learner anxiety, feedback practices, learner and instructor satisfaction, instructors’ challenges and coping strategies, just to mention a few. These constructs cannot be obtained through direct observation, and the understanding of them is a preliminary step toward designing a targeted intervention to improve the quality of EMI programs.

How to Use Questionnaire Survey in EMI Research?

Case Study

Student Engagement and Online Learning

The role of student engagement in HE has received increased attention. In HE contexts, studies of student engagement in learning often focus on what students are doing and its effect on students’ academic performance (Carini et al., Reference Carini, Kuh and Klein2006). Student engagement can be represented in “the development of critical thinking skills, higher grades and a general embracing of learning by taking responsibility and actions to achieve intrinsically motivated goals” (O′ Shea et al., Reference O′ Shea, Stone and Delahunty2015, p. 43). However, central to the discussion is what engagement means and what it is that university students are engaging with. This is of particular relevance since learning is often assumed to take place as the level of engagement demonstrated by students rises (Beldarrain, Reference Beldarrain2006; O′ Shea et al., Reference O′ Shea, Stone and Delahunty2015). Prior studies have noted that instructors are also required to engage in teaching so that reciprocal benefits can be substantial in both teaching and learning (Dyment et al., Reference Dyment, Downing and Budd2013; Pittaway, Reference Pittaway2012). When it comes to online contexts, however, engagement is manifested in different forms because of insufficient face-to-face interaction and the ways in which teaching and learning activities are supported by technology.

Drawing on an online questionnaire, Swan et al. (Reference Swan, Shea, Fredericksen, Pickett, Pelz and Maher2000) identify three factors that exert influence upon the success of asynchronous online learning, namely, design of the course interface, quality interaction with course instructors, and active discussion with classmates. Likewise, Anderson (Reference Anderson, Moore and Anderson2003) sees interaction as key to student engagement and it should be nurtured in online teaching and learning. Interactions that promote student engagement include learner–learner, learner–instructor, and learner–content interactions (Moore, Reference Moore, Harry, John and Keegan1993). They are useful in kindling students’ interest in distance education and in driving them toward the best possible educational outcomes (Chen et al., Reference Chen, Gonyea and Kuh2008). Previous studies such as those outlined earlier have established the importance of online student engagement for student success (Anderson, Reference Anderson, Moore and Anderson2003; Dennen et al., Reference Dennen, Aubteen Darabi and Smith2007; Robinson & Hullinger, Reference Robinson and Hullinger2008; Swan et al., Reference Swan, Shea, Fredericksen, Pickett, Pelz and Maher2000). Creating an online learning environment that is cohesive and interactive (Gaytan & McEwen, Reference Gaytan and McEwen2007) can help address students’ feelings of isolation (Lewis & Abdul-Hamid, Reference Lewis and Abdul-Hamid2006; Madeline et al., Reference Madeline, Ricky, Tracy, Roberts and Emily2005; Yu & Huang, Reference Yu, Huang, Pun, Curle and Yuksel2022), as it enables students to connect with teachers, students, and course content (Young, Reference Young2006).

The scholarly attention given to student engagement gives rise to strong theoretical foundations and useful models. According to Kuh (Reference Kuh2003, p. 25), student engagement involves “the time and energy students devote to educationally sound activities inside and outside of the classroom, and the policies and practices that institutions use to induce students to take part in these activities.” This conception serves as a basis for the development of the National Survey of Student Engagement (NSSE). Created to incorporate the language of effective pedagogical practices into efforts aimed at improving collegiate quality, the NSSE measures five dimensions of engagement: level of academic challenge, active and collaborative learning, student interaction with faculty members, enriching educational experience, and a supportive campus environment (Robinson & Hullinger, Reference Robinson and Hullinger2008). The NSSE looks into students’ collegiate experience as a whole, both inside and outside the classroom. Other instruments that attempt to measure student engagement pay more attention to the classroom context. For example, Handelsman et al. (Reference Handelsman, Briggs, Sullivan and Towler2005) developed a measure of in-class student engagement. Their study revealed four distinct dimensions of student engagement: skills engagement (general learning strategies used by students for intrinsic and extrinsic rewards); emotional engagement (emotional involvement with course material); participation/interaction engagement (class participation and communication with instructors and other students); and performance engagement (levels of classroom performance) (Handelsman et al., Reference Handelsman, Briggs, Sullivan and Towler2005).

Drawing on these previous incarnations of student engagement in traditional classroom settings, Dixson (Reference Dixson2015) sees online student engagement as involving students (1) devoting time and energy to learning teaching materials and acquiring skills, (2) interacting with instructors and peers in a meaningful way (enough so that other people become “real”), and (3) becoming emotionally involved with their studies (i.e., getting excited about an idea, enjoying the learning and/or interaction). In this way, online engagement consists of students’ attitudes, viewpoints, and learning behaviors as well as communication with others (see Figure 11.1). Student engagement in online courses is therefore concerned with students devoting time, energy, ideas, efforts, and, to some degree, emotions to their studies. Following this perspective, Dixson (Reference Dixson2015) develops and validates the Online Student Engagement Scale (OSE), which measures how students behave in learning as well as how they perceive their learning process and the connections they are making with the course content, the teacher, and fellow students in the light of skills, participation, performance, and emotion.

Figure 11.1 Components of online student engagement

Methods

Due to growing trends toward academic internationalization and the overwhelming dominance of English as a lingua franca in academia, Chinese universities are caught up in a rush to adopt EMI (Yu & Pun, Reference Curle, Derakhshan, Pun and Curle2021). In 2001, the Ministry of Education of China issued several guidelines aimed at promoting bilingual teaching, formulating the goal that within three years, 5–10 percent of content area courses should be delivered in a foreign language, mainly English (Ministry of Education of the People’s Republic of China, 2001). During the past decade, EMI has evolved in HE from being a bilingual teaching experience in economically developed areas to being widely used on a national scale (Jiang et al., Reference Jiang, Zhang and May2019). With EMI becoming an essential part of the ongoing HE internationalization, Chinese universities are offering specialized undergraduate courses in English to increase students’ global competitiveness and to attract overseas students. While benefiting from the native-like English-speaking environment, university students enrolled in EMI programs are often troubled by language and academic problems (Skyrme, Reference Skyrme2007), which may influence their in-class engagement. The online learning environment further adds to the difficulties that university students encounter in EMI settings, and one of the major challenges faced by instructors is how to activate students’ academic engagement during ERT.

Participants were recruited from an undergraduate-level EMI course entitled “scientific writing” offered in the spring semester in 2021 by a public research-intensive university in mainland China. The course is a compulsory one covering academic writing skills required in medical research. Forty-six second-year medical students (eighteen females, twenty-eight males) taking the course participated in this pre–post intervention study. The instructor taught academic writing for various academic contexts (research papers, conference presentations, technical reports) through video conference-delivered lectures. During this one-semester course, students were randomly assigned to twelve groups to collaboratively complete a review paper via an online platform called Tencent Docs. Tencent Docs is an online document platform designed for multiperson collaboration, offering a word processor and spreadsheet editor like Microsoft Word and Excel with functions of editing, viewing track changes, sharing, translating, and downloading. At the beginning of the course, the instructor provided five recommended topics and paper structures for students’ reference. Student groups gathered information, searched medical literature using different databases, located relevant literature, and collaboratively worked with their group members online to complete the group assignment. Students were free to negotiate writing timelines and progress within the group. At the end of the course, each group was required to submit a full-fledged paper and to make an oral presentation online.

The two course instructors are specialized in academic writing education and have at least five years of EMI teaching experience. They have been working closely with practicing doctors in developing and refining the course content and in-class activities.

With the help of the course instructors, the nineteen-item OSE (Dixson, Reference Dixson2015) was distributed to participants to measure their perceived engagement in this online course (see Appendix). This self-rating questionnaire used a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree), where students reported how well each learning behavior, thought, or emotion was characteristic of them or their behavior before and at the end of the course (Dixson, Reference Dixson2015). In the questionnaire (Cronbach’s alpha = 0.92), statements included descriptions of thoughts and feelings such as “finding ways to make the course material relevant to my life” and “finding ways to make the course interesting to me”; those of skills such as “looking over class notes between getting online to make sure I understand the material” and “taking good notes over readings, PowerPoints, or video lectures”; those of participation such as “helping fellow students” and “posting in the discussion forum regularly”; and those of performance items such as “doing well on the tests/quizzes” and “getting a good grade.”

After data collection, statistical analyses were performed using Statistical Package for the Social Sciences (SPSS) 22 and self-contrast methodology. The means and standard deviations were used to describe the total scores and single scores for each item listed on the questionnaire. The paired t-test was used to compare pre- and post-activity scores. A significant difference was considered to exist if the p-value was less than 0.05 (two-tailed).

Findings

From March to June 2021, ninety-two questionnaires were made available online and were distributed. All participants in the EMI course finished the collaborative writing assignment and participated in this survey research. They completed the questionnaire prior to and at the end of the course.

At the beginning of the course, many students considered the collaborative writing assignment as a way of releasing pressure during ERT. The true aim was to help students cope with ERT, where the challenges of insufficient interaction and lack of emotional support have to be overcome (Yu & Huang, Reference Yu, Huang, Pun, Curle and Yuksel2022), and reducing stress was just a natural part of this process. Students were encouraged to communicate with other course participants and make full use of computer-supported collaborative learning and then apply these in other online courses.

The questionnaire was completed in full by all forty-six out of forty-six students in this EMI course. The results reveal that the majority of the participating students had significant improvements (p < 0.05) in online engagement in terms of skills, emotion, and participation (Table 11.1). Ratings for each item before and after the collaborative writing assignment were analyzed using the paired t-test, and the results were significantly different. No significant difference was observed between the implementation of online collaborative writing and students’ performance in tests and quizzes (p > 0.05).

Discussion

In the HE sector, the outbreak of COVID-19 had given rise to a considerable disruption in the teaching modality. To control the spread of this rapidly growing and highly infectious disease, HE institutions had moved education online. This shift to ERT meant that a great number of university students could keep learning, albeit situated in a context of social injustice, inequity, and the digital divide (Bozkurt et al., Reference Bozkurt, Jung, Xiao, Vladimirschi, Schuwer, Egorov, Lambert, Al-Freih, Pete, Olcott, Rodes, Aranciaga, Bali, Alvarez, Roberts, Pazurek, Raffaghelli, Panagiotou, de Coëtlogon, Shahadu, Brown, Asino, Tumwesige, Ramirez Reyes, Barrios Ipenza, Ossiannilsson, Bond, Belhamel, Irvine, Sharma, Adam, Janssen, Sklyarova, Olcott, Ambrosino, Lazou, Mocquet, Mano and Paskevicius2020). Instructors were required to adapt schedules to ERT environments, usually with minimal if any prior experience. Despite instructors’ and students’ varying degrees of preparedness, this crisis “has been, and will continue to be, both a massive challenge and a learning experience for the global education community” (Schleicher, Reference Schleicher2020, p. 7). This study introduced in detail the teaching methods for collaborative writing and systematically evaluated these techniques for improving student engagement in online courses.

The results yielded a positive association between students’ practice of collaborative writing and self-reported emotions in ERT. This resonates with the notion that the measures to control the spread of the coronavirus have an influence on students’ psychological well-being, such as social isolation and loneliness arising from the disruption in their everyday routines and interpersonal communication (Labrague et al., Reference Labrague, De Los Santos and Falguera2021). The prolonged social distancing measures made it hard for students to satisfy one of their basic needs – the need to associate with others (Bavel et al., Reference Bavel, Baicker, Boggio, Capraro, Cichocka, Cikara, Crockett, Crum, Douglas, Druckman, Drury, Dube, Ellemers, Finkel, Fowler, Gelfand, Han, Haslam, Jetten, Kitayama, Mobbs, Napper, Packer, Pennycook, Peters, Petty, Rand, Reicher, Schnall, Shariff, Skitka, Smith, Sunstein, Tabri, Tucker, Linden, Lange, Weeden, Wohl, Zaki, Zion and Willer2020; Yu & Huang, Reference Yu, Huang, Pun, Curle and Yuksel2022). In view of the months-long closure of many HE institutions, students have fewer opportunities for peer-to-peer communication on campus and may feel less connected with their friends (Elmer et al., Reference Elmer, Mepham and Stadtfeld2020), which may consequently affect the way they engage in online courses. The findings generated from this study highlight the significance of relational factors, such as peer support during collaborative writing, for students’ mental health and their engagement in distance education during the time of the crisis. Collaborative writing helps enhance university students’ perception of social support from and interaction with peers free of violations against the ongoing anti-epidemic measures, for instance, by encouraging students to work with peers on an academic project via social media. Collaborative writing afforded students the opportunity to exchange ideas and feedback with each other.

A comparison of the scores shows that collaborative writing activities are effective in promoting students’ skills engagement. Such skills include (1) making sure to study on a regular basis; (2) staying up on the readings; (3) looking over class notes between getting online to make sure I understand the material; (4) being organized; (5) taking good notes over readings, PowerPoints, or video lectures; and (6) listening/reading carefully. As is shown in previous research, the success of online learning is positively conditioned by active learners who are in charge of their own learning process (Hiltz & Shea, Reference Hiltz, Shea, Hiltz and Goldman2005). In taking charge of their studies, students are implementing self-regulated learning, which refers to “self-generated thoughts, feelings, and actions that are planned and cyclically adapted to the attainment of personal goals” (Zimmerman, Reference Zimmerman, Boekaerts, Pintrich and Zeidner2000, p. 14). Self-regulated learning is gaining increasing significance in online education, as distance learning sets a great demand on students’ agency and ownership of their own learning (Çebi & Güyer, Reference Çebi and Güyer2020). A systematic review of online student engagement during ERT reveals that the levels of self-regulated learning in the online contexts were higher than or similar to those of traditional face-to-face courses (Salas‐Pilco et al., Reference Salas‐Pilco, Yang and Zhang2022). The results of this study indicate that collaborative writing activities are important in helping students enhance their self-regulation in the EMI course. In a rather similar line, Su et al. (Reference Su, Zheng, Liang and Tsai2018) point out the key role of online collaboration in fostering students’ self-regulation. It might be argued that collaborative writing, as a technology-mediated type of assignment, creates a convenient and user-friendly environment for students to apply self-regulated strategies, such as goal orientation, self-monitoring, and self-assessment in order to heighten self-regulation. According to Boykin et al. (Reference Boykin, Evmenova, Regan and Mastropieri2019), students with varying knowledge levels and needs are able to collaborate effectively and efficiently on the writing tasks and promote their self-regulation during the learning process. In addition, online collaborative writing is student-centered in nature and allows the students to take charge of and make plans for the academic activities, which could further help them actively engage in their studies.

The findings of this study demonstrate that students’ participation/interaction engagement is positively associated with online collaborative writing activities. The six aspects in focus are (1) having fun in online chats, in discussions, or via email with the instructor or other students; (2) participating actively in small-group discussion forums; (3) helping fellow students; (4) engaging in conversations online (chat, discussions, email); (5) posting in the discussion forum regularly; and (6) getting to know other students in the class. When producing a jointly written text, student participants are following active and collaborative learning, which refers to students’ efforts of contributing to class discussions, working with other students, and engaging in other class events (Kuh, Reference Kuh2001). The results support previous research into this area, which views collaboration as critical to facilitating learning in promoting learning in the online classroom (Conrad & Donaldson, Reference Conrad and Donaldson2011; Robinson & Hullinger, Reference Robinson and Hullinger2008). The online classroom has often been perceived as a learning community, where learners are expected to devote collaborative efforts to their studies (Barker, Reference Barker and Cothran2002; Benbunan-Fich et al., Reference Benbunan-Fich, Hiltz, Harasim, Hiltz and Goldman2005; Dede, Reference Dede and Hanna2000). The results of the study confirm that collaborative writing promotes online learners’ active engagement in online learning through thinking, talking, and interacting with the other students. This corroborates the ideas of Palloff and Pratt (Reference Palloff and Pratt2001) that collaborative assignments empower and engage students to the extent that this affects students’ subsequent learning behaviors. In completing a collaborative project, group exchanges can solicit different perspectives and encourage idea sharing, which are generally considered as effective learning techniques (Benbunan-Fich et al., Reference Benbunan-Fich, Hiltz, Harasim, Hiltz and Goldman2005; Conrad & Donaldson, Reference Conrad and Donaldson2011; Robinson & Hullinger, Reference Robinson and Hullinger2008). As a core part of collaborative learning, learning with peers enables students to freely pose their own inquiry in respective study groups (Kemery, Reference Kemery and Aggarwal2000).

The results indicate that students’ performance in online courses appears not to have been affected by collaborative writing. Students’ attitudes toward their grades, tests, and quizzes seem to remain the same, as the quantitative results were not statistically significant. A few reasons might be plausible to explain these results. Firstly, although students were required to collaboratively work on the writing tasks following a tight schedule, to students this was a course designed for scientific writing and the assignment merely constituted a small part of the total grade, so they tended to focus more on their own essays for better marks, not the quality of collaborative writing. Apart from this, technical vocabulary and word spelling were not explicitly taught in class, but their importance was emphasized by the instructor in class. Hence, participating students tended to use online editing services with which they could check for spelling and grammatical errors and to consult online dictionaries for definitions, sample sentences, and synonyms. To a certain extent, the use of online proofreading tools limits peer-tutoring during the collaborative activities. Secondly, group work may not lead to productive outcomes associated with some language skills. Shehadeh (Reference Shehadeh2011) shows that compared with independent learning, collaboration may be less helpful in the acquisition of some straightforward skills such as spelling. This may also apply to students’ basic language knowledge, vocabulary being an example (Li & Mak, Reference Li and Mak2022). As is shown here, however, collaboratively working on a writing project may promote university students’ understanding of pragmatics, for instance, word usage in writing. Moreover, students’ priorities may differ in certain language aspects in their collaboration with peers. This is in line with Kessler et al. (Reference Kessler, Bikowski and Boggs2012), who point out that students appear to neglect mistakes that they regard as less important in jointly producing a text, despite the fact that they could make the corrections. However, these conclusions are only suggestive, given the small-scale nature of the study and the limited number of examinations the students attended. The findings do suggest that the effects of collaboration on the students’ in-class performance online need to be investigated further with a larger sample size and more forms of assessment.

This chapter reports the experiences using online collaborative writing for engaging students in the EMI field. Several limitations of this case study should be noted. Firstly, it included a limited number of learners and the results were based on a single EMI course. Therefore, no systematic conclusions can be made. Future research is needed to include more subjects and participants and to examine a broader range of disciplines and interventions. In addition, detailed insights into the student collaborative setting are required for a better understanding of the learning process. Future research may seek to analyze video data of real-time online interaction (e.g., instructor–student interaction, student–student interaction). Even though the data collected with questionnaires shed light on students’ perception of their engagement in online classes, the generalizability of these findings is limited. Moreover, the small number of participants may cause a certain bias. It is suggested that different settings of EMI be investigated in future studies to provide more food for thought on the creation of an engaging online learning environment.

As Hughes (Reference Hughes2007) points out, while the technology-mediated education is “more welcoming of diversity” (p. 710) and has the potential for students’ flexible engagement, there remains the possibility of disengaging. With the current educational disruption resulting from the COVID-19 pandemic fueling a great deal of investments in and the use of instructional technologies designed for ERT (Asare et al., Reference Asare, Yap, Truong and Sarpong2021), online learning is likely to become more feasible for university students, in consideration of safety, cost, time, and distance. Further studies carried out at a multi-institutional or national level for an understanding of the nature of student engagement in the online context are therefore recommended. Qualitative analysis that foregrounds students’ and instructors’ perspectives may provide more insights into the myriad factors that affect student engagement in online courses and may also serve as a foundation for interventions aimed at nurturing and maximizing student engagement.

Concluding Remarks

Given their potential in gathering large amounts of data in a short time with minimum researcher involvement, questionnaire surveys have gained immense popularity in EMI research. Bearing the goal of constructing or testing a theory in mind, data collected from questionnaire surveys can be used to explain and interpret trends or determine how two or more variables may be related, for example the relationship between EMI teaching practices and students’ learning motivation. Also, they can be used to make a comparison between different groups of research participants, such as support needs of EMI instructors across different countries, or to investigate an individual participant or a particular group of participants over a period of time, such as students’ self-report of attitudes toward EMI during years of study. Moreover, questionnaire-based research is effective in measuring the influence of several independent variables on the dependent variable, for example, how EMI-related language ideologies, academic skills support provision, and language use anxiety may impact student performance. Despite their extensive application in EMI research, questionnaire surveys have become somewhat “problematizing,” with a range of disparate strands to account for. Questionnaire development and enactment can be complicated, can be difficult to adapt in agreement with research aims, and is likely to require applications of questionnaire theory, since data accuracy is positively conditioned by adequate and well-documented validity and reliability properties (Curle & Derakhshan, Reference Curle, Derakhshan, Pun and Curle2021). Such challenges are complicated by the dynamic and evolving nature of EMI in the current era characterized by the globalization and marketization of HE, yet they offer exciting new perspectives and opportunities for EMI researchers and practitioners.

Project Overview and Research Context

Research has increasingly focused on the teaching of academic subjects such as science, medicine, and engineering in English Medium Instruction (EMI) universities (Macaro et al., Reference Macaro, Curle, Pun, An and Dearden2017). EMI broadly refers to the use of “the English language to teach academic subjects (other than English itself) in countries where the first language of the majority of the population is not English” (Macaro & Akincioglu, Reference Macaro and Akincioglu2018, p. 256). Governments and educational institutions have implemented EMI policies aimed at promoting the integration of content and language learning (Dafouz et al., Reference Dafouz, Camacho and Urquia2013) in response to the “Englishisation” of higher education globally (Galloway et al., Reference Galloway, Numajiri and Rees2020, p.395). EMI is increasingly applied in tertiary education, and thus understanding and addressing the requirements of students in terms of their learning experiences and the linguistic challenges they face are important to fully address the varying levels of English proficiency in EMI classrooms (Galloway et al., Reference Galloway, Numajiri and Rees2020; Kamaşak et al., Reference Kamaşak, Sahan and Rose2021).

This project focuses on the dimension of gender in language experiences in EMI university settings. Recent EMI studies have explored the relationship between the motivation to learn English and gender (Lasagabaster, Reference Lasagabaster2016) and gender differences in the self-reported challenges of using academic English (Pun & Jin, Reference Pun and Jin2021). Female students have been found to be more motivated in English as a Foreign Language (EFL) settings (Meece et al., Reference Meece, Glienke and Burg2006). However, Lasagabaster (Reference Lasagabaster2016) argued that the EMI setting can dilute the gender-related differences in language learning motivation, as “the use of the language to learn content is equally prone to motivate male and female students,” and that the EMI approach can support “both male and female students to form vivid images of themselves as authentic users and speakers of the language” (Lasagabaster, Reference Lasagabaster2016, p. 328). Lasagabaster’s (Reference Lasagabaster2016) study indicates that in language learning in EMI university contexts, gender-related differences in levels of motivation tend to disappear. Pun and Jin’s (Reference Pun and Jin2021) study involved seventy-three randomly recruited science students from two EMI universities in Hong Kong, in which both females and males reported English language difficulties in academic speaking and writing. However, male students demonstrated better peer communication in English than females in a science classroom setting, at a statistically significant level. Pun and Jin (Reference Pun and Jin2021) suggested that females were likely to feel more stressed or shy when actively presenting their opinions in front of people, rather than any actual gender difference in English-speaking ability. Thus, in this project we focus on male and female students’ perceptions rather than gender differences in language motivation or proficiency.

Although the literature has recently addressed the connection between gender and students’ perceptions of English language use in EMI universities, further research is required. Students’ perceptions are socially and culturally situated and based on gender-related social experiences. In countries such as Japan, more females join EMI programs than males, as internationalism and English are considered to provide them with a degree of freedom in a male-centered society (Bradford et al., Reference Bradford, Shimauchi and Brown2017). However, females in other countries, such as Finland, were found to be less confident in their English skills than males (Bukve, Reference Bukve2020). Chou (Reference Chou2018, p. 615) analyzed studies published in recent decades and found that students often feel discouraged from actively participating in class, due to the “anxiety resulting from a lack of confidence in speaking and a fear of losing face in public.”

Students’ experiences are likely to significantly influence their feelings regarding EMI. Many students express that they feel anger, due to “not understanding the content knowledge, failing in EMI courses and losing concentration” (Turhan & Kırkgöz, Reference Turhan and Kırkgöz2018, pp. 272–273). However, successful EMI courses and teaching strategies can also lead to feelings of hope and enjoyment (Turhan & Kırkgöz, Reference Turhan and Kırkgöz2018). The personal characteristics of learners, including gender, can potentially affect their experiences and consequently how they perceive the challenges of using academic languages. Although some studies have assessed the types of challenges that students in EMI universities face, the relationship between these challenges and students’ characteristics remains unclear. The contribution of this project is to explore the effects of gender on students’ perceptions of the language difficulties they encounter.

In terms of the methodology applied, questionnaire surveys are useful tools for measuring students’ characteristics, including gender, and the challenges they perceive when studying in EMI universities (Jiang et al., Reference Jiang, Zhang and May2016; Tatzl & Messnarz, Reference Tatzl and Messnarz2013). For example, a recent study investigated the linguistic challenges faced by 498 undergraduate students in an EMI university in Turkey using a questionnaire survey (Kamaşak et al., Reference Kamaşak, Sahan and Rose2021). Second- and fourth-year undergraduate students were found to perceive reading English as more challenging than first-year students, but interestingly male students found understanding EMI instruction more challenging than female students. Pun and Jin (Reference Pun and Jin2021) used a similar method in their study of seventy-three undergraduate students in two EMI universities in Hong Kong, and through a questionnaire survey they identified gender differences in the EMI experience, particularly in terms of communication. Chu et al. (Reference Chou2018, p. 9) also used a questionnaire in their study of 278 undergraduate students in Taiwan and proposed that the “majority of local students’ command of English may not be compatible” with that of their international peers, in terms of “verbally expressing themselves fluently.” They found that local students whose first language was not English were reluctant to use English in class participation and social interaction, to “prevent themselves from making mistakes or embarrassing themselves” (Chu et al., Reference Chou2018, p. 9). Tsui and Ngo (Reference Tsui and Ngo2017, p. 69) used a questionnaire survey to explore perceptions of EMI instructions among 606 undergraduate students in Hong Kong and found that many were worried that the instructions could diminish their “academic results, motivation to learn, learning atmosphere and in-class discussion.” Thus, questionnaire surveys can provide valuable insights into the various aspects of students’ language learning experiences in EMI settings. However, additional statistical measures are required to further explore the effect of gender on students’ perceptions.

Questionnaire surveys can be combined with statistical techniques such as analysis of variance (ANOVA) or multivariate ANOVA (MANOVA) to compare group differences, in terms of gender and years of study, in various language-related challenges (see examples from Kamaşak et al., Reference Kamaşak, Sahan and Rose2021). For example, Turhan and Kırkgöz (Reference Turhan and Kırkgöz2018) combined a questionnaire with MANOVA to reveal differences in the motivations of students toward EMI instructions. The MANOVA test was used to investigate whether the year of study was a determinant of the motivation of students, and although they found that the year did not influence the motivation levels of the participants, final-year university students were the most highly motivated toward EMI (Turhan & Kırkgöz, Reference Turhan and Kırkgöz2018). Similarly, Chen and Kraklow (Reference Chen and Kraklow2014) applied MANOVA and found significant group differences in terms of both intrinsic motivation and English learning engagement between EMI and non-EMI students. In addition, Kamaşak et al. (Reference Kamaşak, Sahan and Rose2021) used ANOVA and MANOVA to explore group differences in language-related challenges, including students’ gender, field of study, year of study, first language background, prior EMI experience, and language proficiency test score. Their study revealed significant differences, with male students struggling more with listening, social sciences students facing greater challenges with reading and writing, second- and fourth-year students having greater reading-related challenges, Turkish students encountering more linguistic challenges than international students, and students with no prior EMI experience finding more challenges with writing, reading, speaking, and listening. Informed by these studies, we apply a one-way MANOVA as an additional statistical method to investigate group differences in our research. This enables us to examine the effects of gender on the perceived writing and speaking challenges of students in a Hong Kong EMI university. Achievements in different elements of English language skills are shown to be correlated (Stotsky, Reference Stotsky1983). In this chapter, we explore the mean differences between groups for a combination of perceived language challenges.

What Is MANOVA?

Multivariate ANOVA is a technique for measuring the linear effects of categorial variables on multiple dependent variables (D’Amico et al., Reference D’Amico, Neilands and Zambarano2001; Keselman et al. Reference Keselman, Huberty, Lix, Olejnik, Cribbie, Donahue and Levin1988). It differs from ANOVA (introduced in Chapter 2 of the book), which involves only one dependent variable. MANOVA is essentially an expanded form of ANOVA, and the main distinction between an ANOVA test and a MANOVA test is that an ANOVA test considers only one dependent variable (the measure that an experimenter is evaluating to see if it changes when other variables change), whereas a MANOVA test includes two or more dependent measures (Emerson, Reference Emerson2018). Moreover, in MANOVA, the independent variable is categorical while the dependent variable should be an interval or scale. Figure 12.1 illustrates the concepts that guide researchers to select different family members of ANOVA. In our case, gender is the independent variable, while writing and speaking challenges are the two dependent variables. Gender (male/female) is a categorical variable, while writing and speaking challenges are measured on an interval/scale. As no control variables are included in this study and we have one independent variable, a one-way MANOVA is most appropriate.

Figure 12.1 Considerations in using MANOVA/MANCOVA

Use of MANOVA in EMI Research in the University Context

Of late, MANOVA has increasingly been applied in research into EMI in universities. We searched for three phrases in the Google Scholar search engine: “MANOVA,” “university students,” and “English Medium Instruction.” Table 12.1 shows some of the recent studies that have used MANOVA in the context of EMI universities. Several were conducted in Taiwan (Chen & Kraklow, Reference Chen and Kraklow2014; Chou, Reference Chou2018), Turkey (Kamaşak et al., Reference Kamaşak, Sahan and Rose2021; Soruç et al., Reference Soruç, Altay, Curle and Yuksel2021; Turhan & Kırkgöz, Reference Turhan and Kırkgöz2018), and Japan (Aizawa et al., Reference Aizawa, Rose, Thompson and Curle2020). Statistical Package for Social Sciences (SPSS) and R were the two most common software packages used in these studies. Most used several rounds of one-way MANOVAs to study the effects of independent variables on more than one dependent variable. Other future studies may extend to two-way or three-way MANOVA, two-way multivariate analysis of covariance (MANCOVA) or three-way MANCOVA. Unlike MANOVA, MANCOVA includes control variables. Table 12.1 reports research into the effects of different types of EMI instruction on the posttest academic achievements and motivation of university students, when controlling for pretest achievement. MANCOVA has two main advantages over MANOVA: (1) it can explain within-group variance and (2) confounding factors can be controlled for (George & Mallery, Reference George and Mallery2019).

A wider range of independent variables than dependent variables was included in these studies. The independent variables can be classified into three types: learners’ characteristics (Chu et al., Reference Chu, Lee and Obrien2017; Kamaşak et al., Reference Kamaşak, Sahan and Rose2021; Turhan & Kırkgöz, Reference Turhan and Kırkgöz2018); the context of study (Chen & Kraklow, Reference Chen and Kraklow2014; Chou, Reference Chou2018); and English language level (Aizawa et al., Reference Aizawa, Rose, Thompson and Curle2020; Soruç et al., Reference Soruç, Altay, Curle and Yuksel2021). Perceived ease, anxiety, and challenges of language use were the dependent variables most often studied (Aizawa et al., Reference Aizawa, Rose, Thompson and Curle2020; Chou, Reference Chou2018; Kamaşak et al., Reference Kamaşak, Sahan and Rose2021; Soruç et al., Reference Soruç, Altay, Curle and Yuksel2021). Affective aspects, such as motivation and engagement, were also investigated as dependent variables (Chen & Kraklow, Reference Chen and Kraklow2014; Turhan & Kırkgöz, Reference Turhan and Kırkgöz2018). However, MANOVA is seldom used to examine the effects of different types of EMI instruction on students’ learning outcomes and language-related challenges.

The findings of studies that applied MANOVA as the statistical technique reveal various patterns. For example, similarities can be identified in two studies focusing on Japan and Turkey (Aizawa et al., Reference Aizawa, Rose, Thompson and Curle2020; Soruç et al., Reference Soruç, Altay, Curle and Yuksel2021), as language-related challenges were found to differ across year groups in the EMI university settings of both. Researchers can compare the results across EMI studies of how groups differ in terms of the joint effects of the dependent variables. They can also compare results from different national contexts, as in the example of Japan and Turkey. We jointly examined the studies of Chen and Kraklow (Reference Chen and Kraklow2014) and Turhan and Kırkgöz (Reference Turhan and Kırkgöz2018) and found that both indicate that the year of study does not predict the motivation toward EMI but that EMI programs motivate and engage students in EMI instruction. Thus, the application of MANOVA enables the findings across studies to be combined.

Research Question and Hypothesis Formulation

The research question we address in this study is: How does gender predict language-related challenges in terms of writing and speaking?

Research Design and Methodology

A questionnaire survey was designed to measure demographic information and students’ perceived speaking and writing challenges in an EMI university. We recruited seventy-five university students from two Hong Kong universities who were studying science subjects such as engineering or natural science. Of these, twenty-four were female and fifty-one were male. A majority of students (n = 53) were in their first undergraduate year and the remainder (n = 22) were in their second year or above. A majority of the students (n = 69) reported that their first language was Cantonese, and the remainder reported Mandarin as their first language. Our preliminary results were reported in Pun and Jin (Reference Pun and Jin2021), while in this book chapter we aim to illustrate how we can incorporate one-way MANOVA into our study of the effects of gender on perceived writing and speaking challenges.

The scales of the questionnaire survey were adapted from the literature. The survey comprised 149 items, of which seven sought the respondents’ background information and the remaining 142 were self-assessment questions on the respondents’ experiences in EMI universities (Pun & Jin, Reference Pun and Jin2021). The questionnaire was reviewed by several education researchers and science lecturers (Pun & Jin, Reference Pun and Jin2021) and was piloted with forty-eight undergraduates (Pun & Jin, Reference Pun and Jin2021).

When conducting one-way MANOVAs, researchers commonly identify one categorial independent variable and two scale dependent variables. The aim of a one-way MANOVA is to identify the effects of the independent variable on both dependent variables, and the two dependent variables are conceptually related to each other (Pallant, Reference Pallant2020). In our case, the independent variable is gender, while the two dependent variables are perceived writing difficulty and perceived speaking difficulty. As mentioned in the previous section, these dependent variables are conceptually related (Cleland & Pickering, Reference Cleland and Pickering2006). Cronbach’s alpha was calculated to validate the reliability of the scale of perceived writing and speaking challenges, and the values are given in Table 12.2.

Taber (Reference Taber2018) suggested that a scale with a Cronbach’s alpha of 0.7 is considered reliable. Thus, the scales in Table 12.2 can reliably reflect perceived writing and speaking difficulties.

Assumptions of MANOVA

University students’ responses to the items were coded on a scale of 1 (very easy) to 5 (very difficult). An average score was computed for each dependent variable, with a higher score representing a higher level of perceived difficulty. A one-way MANOVA is appropriate because we are interested in how gender affects students’ perceived writing and speaking challenges. Table 12.3 provides a summary of the means and standard deviations of perceived speaking and writing difficulties for university students.

Pallant (Reference Pallant2020) suggested that researchers must ensure various assumptions are met before conducting a one-way MANOVA:

1. 1. The sample size should be large and higher than the required number of cases per cell.

2. 2. Univariate and multivariate normality should be ensured. Multivariate normality can be indicated by Mahalanobis distances.

3. 3. Univariate and multivariate outliers should be checked.

4. 4. Linear relationships between dependent variables should be ensured.

5. 5. Each pair of dependent variables should have moderate correlations.

6. 6. Homogeneity should exist in variance and covariance matrices.

In the following subsections, we describe the tests we conducted to ensure that these assumptions were met.

Sample Size

A large sample is important because it reduces the risk of violating other assumptions, such as univariate and multivariate normality. The smallest number of cases in each cell is two in our study, as there are two dependent variables. Therefore, we have a total of four cells (two levels of independent variable and two dependent variables). As the descriptive statistics reveal, the sample size is much greater than the required number of cases.

Univariate and Multivariate Outliers and Normality

The normality of the groups can be assessed in terms of their perceived writing and speaking challenges using Shapiro–Wilk tests (Pallant, Reference Pallant2020). The significance levels are 0.179 for females and 0.319 for males regarding perceived speaking challenges and 0.140 for females and 0.576 for males regarding perceived writing challenges. All of the significance values are greater than 0.05, indicating that the dependent variables are normally distributed across males and females. For univariate outliers, we checked the boxplot for outliers in terms of the perceived writing and speaking challenges, as represented by a circle or an asterisk at the end.

In Figure 12.2a, “case 46” is circled for the perceived writing challenges in the male group; in Figure 12.2b, “case 8” (female) and “case 40” (male) are both circled. These indicate three outliers. We must then consider whether these cases should be excluded or whether the variables should be transformed. Alternatively, we can decide whether to exclude these three cases after examining the multivariate outliers, as indicated in the following discussion.

Figure 12.2 Boxplots of (a) perceived writing challenges and (b) perceived speaking challenges

Multivariate outliers can be identified by calculating the Mahalanobis distance, which is a measure of the distance between a case and the centroid of the other cases (Tabachnick and Fidell, Reference Tabachnick and Fidell2013). Tabachnick and Fidell (Reference Tabachnick and Fidell2013) noted that the critical values at α = 0.001 vary according to the number of dependent variables. These values are typically listed in a chi-square table and can be read off according to the number of dependent variables. As we have two dependent variables in this study, the critical value is 13.82. The maximum value of the Mahalanobis distance obtained from IBM SPSS version 28 is 11.5, which is lower than the critical value. This may suggest an absence of multivariate outliers, but if there is only one case with a score higher than the critical value, then we should consider whether to include or exclude that case in the statistical analysis by assessing whether the difference between the critical value and the individual score is large (Pallant, Reference Pallant2020). The presence of multiple cases of multivariate outliers may require their removal from the dataset.

Linearity

A linearity test was conducted to assess whether a linear relationship exists between the two dependent variables of perceived speaking challenges and writing challenges. The scatter plot in Figure 12.3 shows that the relationship is linear, with an R² value of 0.211. As indicated in Table 12.4, the significance level in the column of deviation from linearity is greater than 0.05, confirming that the relationship between these two variables is linear.

Figure 12.3 Plot showing the relationship between perceived speaking and writing challenges

Multicollinearity and Singularity

The dependent variables in MANOVA should be moderately correlated. French et al. (Reference French, Macedo, Poulsen, Waterson and Yu2008) argued that if the dependent variables are highly correlated, insufficient variance will remain. Too low a correlation, however, indicates that running MANOVA may be unnecessary, because a univariate analysis would suffice to identify the effects of the independent variable on each dependent variable. We ran a Pearson’s correlation between our two dependent variables of perceived writing and perceived speaking challenges. The Pearson’s correlation coefficient of r_p = 0.460 (p < 0.05) indicates that the two dependent variables are moderately correlated, and thus running MANOVA may yield meaningful results. However, a correlation in the range of 0.80 to 0.90 would be a concern (Pallant, Reference Pallant2020).

Levene’s Test of Equality of Error Variances

An important assumption of MANOVA is that the error variances should be equal. Levene’s test of equality of error variances in Table 12.5 indicates that no significant value was lower than 0.05. Therefore, equal error variances were assumed.

Homogeneity of Variance–Covariance Matrices

Box’s test of equality of covariance matrices reflects the homogeneity of variance–covariance matrices (Pallant, Reference Pallant2020). If the value of significance is larger than 0.001, the assumption of homogeneity of these matrices has not been violated (Pallant, Reference Pallant2020). The significance value in our study is 0.185, and thus the assumption of homogeneity of variance–covariance matrices is valid.

Multivariate Tests

The one-way MANOVA results are presented in Table 12.6. The four statistics that can be selected are Pillai’s trace, Wilk’s lambda, Hotelling’s trace, and Roy’s largest root. Tabachnick and Fidell (Reference Tabachnick and Fidell2013) suggested that if the data violate any of the MANOVA assumptions such as a large sample size, researchers can consider using Pillai’s Trace. Otherwise, Wilks’ Lambda should be generally used (Tabachnick and Fidell, Reference Tabachnick and Fidell2013). In the second “Gender” column the Wilks’ lambda row is in bold. Wilks’ lambda is consulted when conducting a one-way MANOVA (Pallant, Reference Pallant2020), and a significance level lower than 0.05 indicates a significant difference between the gender groups, while a level higher than 0.05 indicates no significant difference. Table 12.6 shows that the significance (“Sig.”) level is 0.037, and thus lower than 0.05. We can conclude from this that the students’ overall perceptions of language challenges were affected by their gender.

Tests of Between-Subjects Effects

Tests of between-subjects effects can provide researchers with a more detailed view of each dependent variable, as shown in Table 12.7. The perceived speaking and writing challenges are the dependent variables in this study. This test includes separate analyses of each dependent variable. To reduce the risk of a Type I error (claiming significant relationships when in fact there are none), we applied a stricter Bonferroni adjustment (Pallant, Reference Pallant2020). We divided our original alpha level of 0.05 by 2, thus obtaining a new alpha level of 0.025. Our results can be considered as significant only if the significance level falls below 0.025. As indicated by the bold text in Table 12.7, perceived speaking challenges have a significance level of 0.012, while perceived writing challenges have a significance level of 0.503. This indicates that gender has a statistically significant effect on perceived speaking challenges (F (2, 75) = 6.658; p < 0.025; partial η² = 0.084) but not on perceived writing challenges (F (2, 75) = 0.454; p = 0.503; partial η² = 0.006). The partial eta squared of the effects of gender on perceived speaking challenges is 0.084, which indicates a moderate effect size.

Multiple Comparisons

Table 12.8 shows the pairwise comparisons between females and males for the perceived speaking and writing challenges. The mean scores are significantly different for males and females for perceived speaking challenges (p < 0.05) but not for perceived writing challenges (p = 0.503). The mean score for perceived speaking challenges is 0.333 higher for females than for males. Although the mean of perceived writing challenges for females is higher than the mean of perceived writing challenges for males by 0.111, the difference is not significant.

Writing up the Results

The following paragraph indicates how we wrote up the results according to the format suggested by Pallant (Reference Pallant2020):

The following components should be included when writing up the results: (1) the F-value; (2) the p-value; (3) the partial eta squared; and (4) Wilks’ lambda. These give a good indication of whether the independent variable has a significant effect on the interaction between the two dependent variables.

Discussion of the Use of MANOVA in EMI Research Studies

One major challenge of MANOVA is that many complex assumptions need to be met. EMI researchers may therefore ask why we cannot simply perform a series of ANOVAs to test the effect of an independent variable on several dependent variables. MANOVA has the advantage of reducing the likelihood of committing a Type I error, which involves claiming that relationships are significant when in fact there are no significant relationships. MANOVA is also more powerful than ANOVA, in the sense that researchers can examine the effects of an independent variable on a linear combination of outcome variables (Keselman et al., Reference Keselman, Huberty, Lix, Olejnik, Cribbie, Donahue and Levin1988). The two outcome variables should be moderately correlated. In this study, we are interested in how gender affects the linear combination of perceived writing and speaking challenges. The two challenges are conceptually related (Cleland & Pickering, Reference Cleland and Pickering2006) as they are both types of perceived linguistic challenges.

In addition to examining the outputs of multivariate tests, researchers should also examine the outputs of the test of between-subjects effects and pairwise comparisons. Although we have a combined effect on the dependent variables in our study, there is no statistically significant effect on perceived writing challenges. Thus, this approach enables a finer examination of the effects of gender on each dependent variable. Further insights can be drawn from post hoc pairwise comparisons.

In summary, a one-way MANOVA can be extremely useful when examining the effects of a categorical variable on various dependent variables, including the perceived challenges, motivation, or language competence of EMI university students. The discussions in this chapter suggest that future research into the use of MANOVA for analyzing results from questionnaire surveys is required. In addition, MANCOVA should be considered if variables are included as covariates.

Introduction

In recent years, the learning experience of students transitioning from first language (L1)-medium secondary schools to English Medium Instruction (EMI) tertiary education has attracted growing scholarly attention (e.g., Evans & Morrison, Reference Evans and Morrison2011, Reference Evans and Morrison2016; Lin & Morrison, Reference Lin and Morrison2010; Zhou & Rose, Reference Zhou, McKinley, Rose and Xu2021; Zhou & Thompson, Reference Zhou, Thompson and Zhou2023). Students might experience tremendous difficulties in their first experiences listening to EMI academic lectures, and their perceptions of difficulties may change during the first semester – a critical transition period reported to shape students’ perceptions, beliefs, and behaviors (Ding & Stapleton, Reference Ding and Stapleton2016; Zhou et al., Reference Zhou, Thompson and Zhou2023). Although the importance of transitional issues has been well recognized in EMI research (Macaro et al., Reference Macaro, Curle, Pun, An and Dearden2018), very few studies have used a longitudinal quantitative design to provide an overview of changes in students’ learning patterns during transition.

In this chapter, a case study will be presented that draws on a longitudinal quantitative design to investigate students’ perceptions of lecture listening difficulties during their first semester transitioning into an EMI university in China. Ortega and Iberri-Shea (Reference Ortega and Iberri-Shea2005) elucidated four criteria that qualify a study as longitudinal: first, the duration of the study; second, the utilization of multiwave data collection; third, the conceptual emphasis on capturing change through design; and fourth, the emphasis on creating antecedent–consequent relationships through continuous monitoring of the phenomenon within its environment, instead of relying on experimental controls or comparisons. Echoing this definition of a longitudinal design, the study collected data from the same group of students on their perceptions of lecture listening difficulties across multiple time points (i.e., the beginning, halfway, and the end) during the transition semester and analyzed patterns of change through statistical tests. Such a design can broaden the methodological scope of extant EMI research that investigates students’ learning experience and may be especially applicable to situations where changes in perceptions or behaviors are likely to occur such as the transition period.

The rationale underpinning the longitudinal quantitative design of the study stems from two underresearched issues in extant EMI literature. First, although academic challenges in EMI settings have been well documented in the past two decades (e.g., Aizawa & Rose, Reference Aizawa and Rose2020; Evans & Morrison, Reference Evans and Morrison2016; Hellekjær, Reference Hellekjær2010; Kim & Yoon, Reference Kim and Yoon2018; Kırkgöz, Reference Kırkgöz2005; Zhou et al., Reference Zhou and Rose2021), studies adopting a longitudinal design are scarce. Data have usually been collected through cross-sectional designs to illustrate learning difficulties at a particular time point for a selected cohort of students, for example, first-year undergraduate students (e.g., Aizawa & Rose, Reference Aizawa and Rose2020; Evans & Morrison, Reference Evans and Morrison2016), undergraduate students from mixed year levels (e.g., Kim & Yoon, Reference Kim and Yoon2018; Kırkgöz, Reference Kırkgöz2005), or a combination of undergraduate and postgraduate students (e.g., Hellekjær, Reference Hellekjær2010; Zhou et al., Reference Zhou and Rose2021). As such, these studies can only offer a one-shot description of learning issues at a fixed time point, whereas little is known about whether and how students’ perceived difficulties evolve as they adapt to studying via EMI. Especially for students who just arrive at an EMI university, the usually adopted one-off surveys are unable to capture the dynamics and complexities of their transitional learning experience (see Evans & Morrison, Reference Evans and Morrison2011).

Another issue relatively underexplored in EMI research is how students entering with different English proficiency levels may vary in their transitional experience. It has been found that students from EMI and L1-medium secondary schools reported different challenges when listening to EMI lectures at a tertiary level of education (e.g., Aizawa & Rose, Reference Aizawa and Rose2020; Evans & Morrison, Reference Evans and Morrison2016). Since students coming from EMI secondary schools might be more proficient in English than their peers from L1-medium schools, their English proficiency upon entry might be a covariate (or confounding variable) that explains the differences in the perceptions of challenges identified in these studies. Hence, to disentangle the respective trajectories of students’ lecture listening associated with their level of English proficiency during transition, the study aims to answer the following questions:

1. 1. What are the difficulties experienced by students in listening to lectures upon entering an EMI university?

2. 2. Are there changes in students’ perceptions of listening difficulties during their first semester transitioning into the university?

3. 3. To what extent is students’ English listening proficiency associated with their longitudinal patterns of change in listening difficulties during transition?

In what follows, the chapter will introduce a longitudinal quantitative study that aims to answer these research questions. The overall design of the study will first be outlined, followed by a detailed description of procedures for implementing the study. Important issues such as sampling, the design of measures, and the selection of statistical tests will be discussed, detailing the considerations and rationale behind each methodological decision. Key dilemmas such as handling attrition and missing data will be illustrated, while sharing suggestions for addressing these problems. The chapter will conclude with the methodological implications of the study for future EMI research and offer recommendations informed by the methodological limitations of the study. This case study focuses on the methodological design of the study, and findings are reported separately in Zhou and Rose (Reference Zhou and Rose2024).

A Case Study on EMI Longitudinal Quantitative Research

Conclusion: Methodological Implications and Suggestions

The chapter introduces a longitudinal quantitative study design that investigates change in students’ perceptions of lecture listening difficulties during the critical transition into EMI higher education. It outlines the key procedures for designing and implementing the study, including decisions about research context and sampling, the development of measures, and the selection of appropriate statistical tests for data analysis. Approaches for reducing attrition and handling missing data in longitudinal quantitative research were described, and the strategies for mitigating these problems were also shared. The longitudinal quantitative design of the study could be applied to future EMI research that aims to (1) explore the longitudinal change in students’ learning analytics such as perceptional, behavioral, and motivational changes and (2) examine how the shape of change over time is associated with individual difference variables.

Some methodological limitations of the study should be noted, which could inform the design of future longitudinal quantitative research in EMI contexts. First, due to the outbreak of COVID-19 at the end of 2019, the study only collected data from the first semester via three time points. Future research can extend the time frame and collect data from more time points to capture the change over a longer duration such as an academic year. Second, the study only sampled from disciplines of social sciences and humanities (commonly regarded as “soft” subjects), which limits the capacity of the generalization of the findings. It would be interesting for future research to also sample from “hard” subjects (e.g., physics, engineering, statistics) that depend much more on mathematical formulaic expressions or graphical illustrations, because students’ perceptions of lecture listening difficulties could vary substantially from those reported in this study. Finally, in longitudinal designs where the same measures are repetitively administered, the “practice effect” should be cautioned against (Barkaoui, Reference Barkaoui2014). In other words, the observed changes might be initiated by increased familiarity with the measurement instead of authentic changes in the constructs being measured. Future research with a longer duration can consider lengthening the time interval between different time points of data collection to reduce participants’ perceived familiarity with the research instrument to lower chances of the “practice effect.”

Introduction

Globally, the use of English as the medium of instruction in various educational contexts has grown exponentially in the last decade. The majority of studies in the field concentrate on issues relating to the implementation of English Medium Instruction (EMI) programs, teachers’ and students’ beliefs, challenges, and first language (L1) use (Curle et al., Reference Curle, Jablonkai, Mittelmeier, Sahan, Veitch and Galloway2020; Ismailov et al., Reference Ismailov, Chiu, Dearden, Yamamoto and Djalilova2021; Jablonkai & Hou, Reference Jablonkai and Hou2021; Macaro et al., Reference Macaro, Curle, Pun, An and Dearden2018). However, there has been very little research into the actual language used in EMI classrooms to date. The few existing studies in this area have highlighted a number of relevant language-related issues, for example, identifying language and discourse functions in EMI classroom discourse, determining learning outcomes in terms of discipline-specific communicative competence, and investigating context and discipline-specific language use (Dalton-Puffer & Smit, Reference Dalton-Puffer and Smit2013; Macaro, Reference Macaro2019). The analysis of variation in EMI language use can reveal contextual and disciplinary differences in information and discourse organization that might be the result of a pedagogical approach or academic culture. Such refined understanding of linguistic features and variation can then be translated into pedagogical practices or can inform the support systems of EMI contexts. In her most recent work, Jablonkai (Reference Jablonkai, Pun and Curle2021) argues that a corpus-based approach is well placed to focus on such aspects of language use in EMI. This chapter demonstrates how a specific corpus-based analytical framework, Biber’s (Reference Biber1988) multidimensional (MD, henceforth) analysis, can be applied to explore variation in language use in an EMI context.

Methodology

The steps of an additive MD analysis and cluster analysis are summarized here. In what follows, each step of the analysis is discussed in detail.

1. 1. Build a corpus.

   1. a. Record class sessions.

   2. b. Transcribe recorded sessions based on preestablished transcribing conventions and save them as text files.

2. 2. Provide a situational analysis of the context.

3. 3. Prepare files for linguistic analysis.

   1. a. Run a grammatical tagger on all texts (in this case, the Biber tagger).

   2. b. Correct tags for a few grammatical features using the Fixtagger interactive program.

4. 4. Calculate dimension scores.

   1. a. Calculate a normed count and a Z-score for each linguistic feature on each of the existing dimensions of variation previously identified (in this case, Csomay, Reference Csomay2021).

   2. b. Calculate dimension scores for each class session.

   3. c. Describe texts grouped around extratextual features (e.g., discipline, level of instruction).

5. 5. Identify clusters.

   1. a. Run K-means clustering program on a software called Statistical Package for the Social Sciences (SPSS).

   2. b. Name the clusters (text types; in this case, lecture types).

6. 6. Report on patterns.

   1. a. Describe the texts in your corpus based on the clusters and look for patterns based on aspects of the situation (e.g., discipline, level of instruction).