2.1. Multidimensional Word Knowledge

There are multiple aspects of word knowledge to consider when assessing how well a reader knows a word (Pearson, Hiebert, & Kamil, Reference Pearson, Hiebert and Kamil2007; Perfetti, Reference Perfetti2007). For example, knowledge of a word’s definition, use, connotations, levels of abstractness, multiple meanings, and links to other related words are a few aspects to consider. Our modeling takes into consideration the multidimensional nature of word knowledge by considering appropriate measures of these multiple aspects, specifically multiple-choice synonym knowledge, self-report of meaning knowledge, and consideration of words within a morphological families through production of related words.

2.2. Doubly Multilevel Data

Within a single classroom, there is great variability of student skills and needs. Students are often grouped into smaller groups where instruction of specific skills or words occurs. Classroom teachers or specialized individuals such as reading specialists or second language certified teachers teach these different groups. As such, a multilevel structure on the reader side arises from the fact that teaching words commonly takes place in the context of small groups that are nested within a classroom teacher that is nested within a school. A multilevel structure on the item side arises from the fact that word knowledge is multidimensional (see above). Therefore, multiple aspects of word knowledge need to be assessed for the same word: items assessing each aspect are nested within words. Because research indicates word learning is incremental (Nagy et al. , Reference Nagy, Anderson and Herman1987), different readers, words, and instructional characteristics could lead to the development of different aspects of word knowledge, making it so that the multilevel structure on the item side is particularly important to include in studies of word learning.

2.3. Longitudinal and Multigroup Data from a Pretest–Posttest Study Design

Word learning often occurs in the context of different types of instruction that is differentially effective (Graves, Reference Graves2007). For example, recent standards movements like the Common Core (CCSS, National Governors Association Center for Best Practices & Council of Chief State School, 2010) suggest different instructional strategies to support word learning and help adolescents better comprehend texts. Experimental studies are needed to determine whether a word learning intervention shows superiority over instruction-as-usual controls and/or other research-based comparison instruction. Therefore, when considering instructional characteristics that contribute to word learning, both longitudinal and multigroup structures must be accounted for within the data. Longitudinal data are needed because differences between pretest and posttest word knowledge must be explored. Multigroup structures are also needed because word learning in the two different instructional conditions must be compared to determine which instruction is most effective at supporting word learning.

2.4. Word-Specific Word Knowledge

When studying word learning, it is important to consider what an individual knows about a specific word rather than about words in general. For example, a reader may be familiar with the definition of effortlessly but not know how to use the word or not know a specific meaning that is being conveyed within the text. For example, the reader may be familiar with effortlessly completing homework, but may not realize that a dictator could effortlessly torture individuals, and that meaning may come up when reading history texts. Word knowledge is word-specific in that when a reader is applying word knowledge to oral or reading comprehension, the reader needs to be able to access information about that particular word, otherwise comprehension challenges ensue. Perfetti’s (Reference Perfetti2007) lexical quality hypothesis emphasizes the importance of considering word-specific knowledge when investigating reading. This is because while general literacy skills like word reading are important, reading comprehension is dependent on quick and easy accessing of word-specific information, like accessing the meaning of effortlessly as related to the context it is being used within the text.

2.5. Empirical Research Questions

The current study explores word learning between pretest and posttest for two instructional groups where readers are nested within teachers and items are nested within words. We are guided by the following research questions:

1. (a) What is the dimensionality structure of word knowledge and if there are different dimensions, does significant word learning occur for each aspect of word knowledge across the two hours of instruction? To explore this, we will look specifically at

   1. (i) the relationship between different measures of aspects of word knowledge (multiple-choice, self-report, and production of morphological relatives) for a given word, and

   2. (ii) the learning for each aspect of word knowledge across the two hours of instruction examining mean and variability in word learning for each aspect at posttest.

2. (b) How is word learning affected by word characteristics and instructional contexts? Here we will look specifically at

   1. (i) how word characteristics like derived-word frequency, root word frequency, opaqueness, number of morphemes, number of morphological relatives, and affix frequency explain variance in word easiness across words, and

   2. (ii) whether significant group differences exist between readers in the morphological instruction in combination with comprehension strategy instruction condition (i.e., intervention group) versus readers in the comprehension strategy instruction alone condition (i.e., comparison group), controlling for readers’ background. Our approach unites earlier research that has looked at how single aspects of word knowledge are learned and which has explored either instructional effects or word characteristics that affect word learning. As such, it provides a more comprehensive understanding of word learning.

3.1. Model Specification

In this section, we describe the generalized mixed effect models with data representation, an equation, and a diagram. To frame this data structure within the multilevel literature (e.g., Bryk & Raudenbush, Reference Bryk and Raudenbush1992, Ch. 8), item responses at Level 1 are cross-classified with readers and items at Level 2 and are cross-classified with teachers and words at Level 3. Readers at Level 2 are nested within teachers at Level 3 on the reader side, whereas items at Level 2 are nested within words at Level 3 on the item side. In the model specification, it is assumed that the same three measures were administered at pretest and posttest and an item is loaded on one measure (as suggested by the empirical illustration).

Figure 1.

Data representation for the binary longitudinal item responses ( $y$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {y}}$$\end{document} ) with doubly multilevel structure in the exemplar case when $K=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=2$$\end{document} , $J=6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=6$$\end{document} , $G=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G=2$$\end{document} , $I=6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I=6$$\end{document} , $T=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T=2$$\end{document} , and $D=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D=3$$\end{document} . k is an index for a teacher ( $k=1,2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k=1, 2$$\end{document} ); j is an index for a reader ( $j=1,2,3,4,5,6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j=1, 2, 3, 4, 5, 6$$\end{document} ); g is an index for a word ( $g=1,2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=1, 2$$\end{document} ); i is an index for an item ( $i=1,2,3,4,5,6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1, 2, 3, 4, 5, 6$$\end{document} ); t is an index for a time point ( $t=1,2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t=1, 2$$\end{document} ); d is an index for a measure ( $d=1,2,3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=1, 2, 3$$\end{document} ); k[j] indicates that a reader j is nested within a teacher k; and g[i] indicates that an item i is nested with a word g.

To show data complexity, data representation of binary longitudinal item responses ( $y$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {y}}$$\end{document} ) with doubly multilevel structure is presented in Figure 1.Footnote 3 Denote a binary item response by $y_{k\left[j\right]g\left[i\right]td}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{k[j]g[i]td}$$\end{document} for a teacher k ( $k=1,\dots ,K$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k=1, \ldots , K$$\end{document} ), a reader j ( $j=1,\dots ,J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j=1, \ldots , J$$\end{document} ), a word g ( $g=1,\dots ,G$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=1, \ldots , G$$\end{document} ), an item i ( $i=1,\dots ,I$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1, \ldots , I$$\end{document} ), a time point t ( $t=1,\dots ,T$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t=1, \ldots , T$$\end{document} ), and a measure (of an aspect of word knowledge) d ( $d=1,\dots ,D$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=1, \ldots , D$$\end{document} ). The subscript k[j] indicates that a reader j is nested within a teacher k and the subscript g[i] indicates that an item i is nested with a word g. For simplicity, the representation in Figure 1 shows an exemplar case of $K=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=2$$\end{document} , $J=6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=6$$\end{document} , $G=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G=2$$\end{document} , $I=6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I=6$$\end{document} , $T=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T=2$$\end{document} , and $D=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D=3$$\end{document} . Furthermore, it is assumed that the same item set was given across time points. In this example, there are 72 (= 6 readers $\times$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document} 6 items $\times$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document} 2 time points) item responses, $y$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {y}}$$\end{document} . Possible dependency in the item responses are due to the doubly multilevel structure (i.e., teacher clustering and word clustering), measure clustering, and time points (i.e., repeated measures).

In the specification of the generalized mixed effect model for binary longitudinal data with doubly multilevel structure, a binary item response ( $y_{k\left[j\right]g\left[i\right]td}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{k[j]g[i]td}$$\end{document} ) has an independent Bernoulli distribution with a mean $P(y_{k\left[j\right]g\left[i\right]td}=1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P(y_{k[j]g[i]td}=1)$$\end{document} conditional on the random effects we specify below. The linear predictor is

(1) $\begin{array}{c}{\eta }_{k\left[j\right]g\left[i\right]td}=X_{td}^{\prime }{\beta }_{td}+S_{td}^{\prime \left(2\right)}{s}_{k\left[j\right]td}^{\left(2\right)}+S_{td}^{\prime \left(3\right)}{s}_{ktd}^{\left(3\right)}+W_{td}^{\prime \left(2\right)}{w}_{g\left[i\right]td}^{\left(2\right)}+W_{td}^{\prime \left(3\right)}{w}_{gt}^{\left(3\right)},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \eta _{k[j]g[i]td} = X_{td}^{\prime }\beta _{td} + S_{td}^{\prime (2)}s_{k[j]td}^{(2)} + S_{td}^{\prime (3)}s_{ktd}^{(3)} + W_{td}^{\prime (2)}w_{g[i]td}^{(2)} + W_{td}^{\prime (3)}w_{gt}^{(3)}, \end{aligned}$$\end{document}

where

• • the superscript (2) refers to Level 2,

• • the superscript (3) refers to Level 3,

• • $X_{td}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{td}$$\end{document} is the design matrix for fixed effects,

• • $S_{td}^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{td}^{(2)}$$\end{document} is the design matrix for random reader effects,

• • $S_{td}^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{td}^{(3)}$$\end{document} is the design matrix for random teacher effects,

• • $W_{td}^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{td}^{(2)}$$\end{document} is the design matrix for random item effects,

• • $W_{td}^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W_{td}^{(3)}$$\end{document} is the design matrix for random word effects,

• • ${\beta }_{td}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{td}$$\end{document} is a fixed effect; ${\beta }_{\left(\right[TD]\times 1)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\beta }}_{([TD] \times 1)}$$\end{document} is the vector of fixed effects,

• • $s_{k\left[j\right]td}^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]td}^{(2)}$$\end{document} is a random reader effect for a time point t and a measure d at the reader level; $s_{k\left[j\right]\left(\right[TD]\times 1)}^{\left(2\right)}={[s_{k\left[j\right]11}^{\left(2\right)},\dots ,s_{k\left[j\right]td}^{\left(2\right)},\dots ,s_{k\left[j\right]TD}^{\left(2\right)}]}^{\prime }\sim MN(0_{\left(\right[TD]\times 1)},{\Sigma }_{2\left(\right[TD]\times [TD\left]\right)})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{s}_{k[j]([TD] \times 1)}^{(2)}=[s_{k[j]11}^{(2)}, \ldots , s_{k[j]td}^{(2)}, \ldots , s_{k[j]TD}^{(2)}]' \sim MN(\mathbf{0}_{([TD] \times 1)}, \Sigma _{2([TD] \times [TD])})$$\end{document} ,

• • $s_{ktd}^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{ktd}^{(3)}$$\end{document} is a random teacher effect for a time point t and a measure d; $s_{k\left(\right[TD]\times 1)}^{\left(3\right)}={[s_{k11}^{\left(3\right)},\dots ,s_{ktd}^{\left(3\right)},\dots ,s_{kTD}^{\left(3\right)}]}^{\prime }\sim MN(0_{\left(\right[TD]\times 1)},{\Sigma }_{3\left(\right[TD]\times [TD\left]\right)})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{s}_{k([TD] \times 1)}^{(3)}=[s_{k11}^{(3)}, \ldots , s_{ktd}^{(3)}, \ldots , s_{kTD}^{(3)}]' \sim MN(\mathbf{0}_{([TD] \times 1)}, \Sigma _{3([TD] \times [TD])})$$\end{document} ,

• • $w_{g\left[i\right]td}^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]td}^{(2)}$$\end{document} is a random item effect for a time point t and a measure d; $w_{g\left[i\right]td}^{\left(2\right)}\sim N(0,{\sigma }_{3td}^2)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]td}^{(2)} \sim N(0,\sigma _{3td}^{2})$$\end{document} , and

• • $w_{gt}^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{gt}^{(3)}$$\end{document} is a random word effect for a time point t; and $w_{tg}^{\left(3\right)}\sim N(0,{\sigma }_{4t}^2)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{tg}^{(3)} \sim N(0, \sigma _{4t}^{2})$$\end{document} .

Variance–covariance matrices of the random reader effects and the random teacher effects (i.e., ${\Sigma }_{2\left(\right[TD]\times [TD\left]\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{2([TD] \times [TD])}$$\end{document} and ${\Sigma }_{3\left(\right[TD]\times [TD\left]\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{3([TD] \times [TD])}$$\end{document} , respectively) are unstructured and should be positive definite to ensure the inverse of the matrices (i.e., the computation formulas in lme4) exists (Bates, Reference Bates2010).

The logit link function is selected in this study. With the specification above, the generalized mixed models for doubly multilevel structure can be described as follows:

(2) $\begin{array}{cc}\mathrm{logit}[P(y_{k\left[j\right]g\left[i\right]td}=& 1|s_{k\left[j\right]td}^{\left(2\right)},s_{ktd}^{\left(3\right)},w_{g\left[i\right]td}^{\left(2\right)},w_{gt}^{\left(3\right)})]\\ =& X_{td}^{\prime }{\beta }_{td}+S_{td}^{\prime \left(2\right)}{s}_{k\left[j\right]td}^{\left(2\right)}+S_{td}^{\prime \left(3\right)}{s}_{ktd}^{\left(3\right)}+W_{td}^{\prime \left(2\right)}{w}_{g\left[i\right]td}^{\left(2\right)}+W_{td}^{\prime \left(3\right)}{w}_{gt}^{\left(3\right)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathrm{logit}\Bigg [P\Big (y_{k[j]g[i]td}= & {} 1|s_{k[j]td}^{(2)}, s_{ktd}^{(3)},w_{g[i]td}^{(2)},w_{gt}^{(3)}\Big )\Bigg ]\nonumber \\= & {} X_{td}^{\prime }\beta _{td} + S_{td}^{\prime (2)}s_{k[j]td}^{(2)} + S_{td}^{\prime (3)}s_{ktd}^{(3)} + W_{td}^{\prime (2)}w_{g[i]td}^{(2)} + W_{td}^{\prime (3)}w_{gt}^{(3)}. \end{aligned}$$\end{document}

Initial status at pretest and learning at posttest were modeled in our specification (Embretson, Reference Embretson1991). To measure the initial status and learning, a design matrix, as presented below, is specified for each measure. In addition, the same design matrix is imposed on $s_{k\left[j\right]}^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{s}_{k[j]}^{(2)}$$\end{document} and $s_k^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{s}_{k}^{(3)}$$\end{document} , respectively, which indicates that the initial status and learning are measured at each level on the person side. An element of the matrix specified below is 1 if a random effect is modeled for a measure in a time point and 0 otherwise. For three measures ( $D=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D=3$$\end{document} , multiple-choice response [MC], self-report [SR], and production of morphological relatives [MR]) and two time points ( $T=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T=2$$\end{document} , pretest and posttest), $S^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S^{(2)}$$\end{document} and $S^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S^{(3)}$$\end{document} can be specified as follows:

$\begin{array}{c}\left(\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\\ 1& 0& 0& 1& 0& 0\\ 0& 1& 0& 0& 1& 0\\ 0& 0& 1& 0& 0& 1\end{array}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \begin{array}{cccccc} 1&{}\quad 0&{}\quad 0&{}\quad 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 0\\ 1&{}\quad 0&{}\quad 0&{}\quad 1&{}\quad 0&{}\quad 0\\ 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 1&{}\quad 0\\ 0&{}\quad 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 1\\ \end{array} \right) , \end{aligned}$$\end{document}

where the first column is for initial status on MC, the second column is for initial status on SR, the third column is for initial status on MR, the fourth column is for learning on MC, the fifth column is for learning on SR, and the sixth column is for learning on MR.

For unconditional models (without any covariates), X is a full copy of either $S^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S^{(2)}$$\end{document} or $S^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S^{(3)}$$\end{document} . In addition, ${\beta }_{\left(\right[TD]\times 1)}={[{\beta }_{11},\dots ,{\beta }_{td},\dots ,{\beta }_{TD}]}^{\prime }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\beta }}_{([TD] \times 1)}=[\beta _{11}, \ldots , \beta _{td}, \ldots , \beta _{TD}]'$$\end{document} is the vector of fixed intercepts, a logit for the probability of a correct response of an ‘average’ reader on an ‘average’ item with a logit link function. For conditional models (with readers and word covariates), additional columns and rows are added to X.

To measure learning, the pretest and posttest must have the same scale, which is called the measurement invariance assumption (e.g., Meade, Lautenschlager, & Hecht, Reference Meade, Lautenschlager and Hecht2005). The measurement invariance assumption can be met by having the same item effects (e.g., item easiness or item difficulty) across time points. Specifically, the item parameters are redefined with the measurement invariance assumption as follows:

(3) $\begin{array}{c}{w}_{g\left[i\right]d}^{\left(2\right)}\sim N(0,{\sigma }_{3d}^2)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w_{g[i]d}^{(2)} \sim N \big (0,\sigma _{3d}^{2}\big ) \end{aligned}$$\end{document}

for $d=1,\dots ,D$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=1, \ldots , D$$\end{document} and

(4) $\begin{array}{c}{w}_g^{\left(3\right)}\sim N(0,{\sigma }_4^2).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w_{g}^{(3)} \sim N\Big (0,\sigma _{4}^{2}\Big ). \end{aligned}$$\end{document}

The design matrix for the random item effects, $W^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^{(2)}$$\end{document} , can be specified as follows:

$\begin{array}{c}\left(\begin{array}{cccccc}1& 0& 0& 1& 0& 0\\ 0& 1& 0& 0& 1& 0\\ 0& 0& 1& 0& 0& 1\\ 1& 0& 0& 1& 0& 0\\ 0& 1& 0& 0& 1& 0\\ 0& 0& 1& 0& 0& 1\end{array}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \begin{array}{cccccc} 1&{}\quad 0&{}\quad 0&{}\quad 1&{}\quad 0&{}\quad 0\\ 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 1&{}\quad 0\\ 0&{}\quad 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 1\\ 1&{}\quad 0&{}\quad 0&{}\quad 1&{}\quad 0&{}\quad 0\\ 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 1&{}\quad 0\\ 0&{}\quad 0&{}\quad 1&{}\quad 0&{}\quad 0&{}\quad 1\\ \end{array} \right) , \end{aligned}$$\end{document}

where the first column is for MC at pretest, the second column is for SR at pretest, the third column is for MR at pretest, the fourth column is for MC at posttest, the fifth column is for SR at posttest, and the sixth column is for MR at posttest. In addition, the design matrix for the random word effect, $W^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^{(3)}$$\end{document} , is ( $\left[TD\right]\times \left[TD\right]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[TD] \times [TD]$$\end{document} ) unit matrix with the measurement invariance assumption. In this study, the measurement invariance assumption is first tested and then assumed in parameter estimation in the illustration. In the lmer function, the time-invariant random item and word effects can be specified by having the same item and word indicator variables across time points involved in estimating ${\sigma }_{3d}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{3d}^{2}$$\end{document} and ${\sigma }_4^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{4}^{2}$$\end{document} .

Figure 2.

A diagram of a generalized linear mixed model for doubly multilevel binary longitudinal data. TRT treatment variable, word word characteristic variables, MC item responses for multiple-choice, SR item responses for self-report, MR item responses for production of morphological relatives, $s_{k\left[j\right]11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]11}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random reader effect for MC at pretest, $s_{k\left[j\right]12}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]12}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random reader effect for SR at pretest, $s_{k\left[j\right]13}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]13}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random reader effect for MR at pretest, $s_{k\left[j\right]21}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]21}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random reader effect for MC at posttest, $s_{k\left[j\right]22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]22}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random reader effect for SR at posttest, $s_{k\left[j\right]23}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]23}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random reader effect for MR at posttest, $s_{k11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k11}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random teacher effect for MC at pretest, $s_{k12}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k12}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random teacher effect for SR at pretest, $s_{k13}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k13}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random teacher effect for MR at pretest, $s_{k21}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k21}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random teacher effect for MC at posttest, $s_{k22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k22}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random teacher effect for SR at posttest, $s_{k23}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k23}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random teacher effect for MR at posttest, $w_{g\left[i\right]1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]1}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random item effect for MC, $w_{g\left[i\right]2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]2}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random item effect for SR, $w_{g\left[i\right]3}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]3}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random item effect for MR, and $w_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g}$$\end{document} $\text{a}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {a}$$\end{document} random item-group effect. All latent variables at the reader level and at the teacher level are correlated at each level.

The doubly multilevel item response data structure is presented in Figure 2 related to the example of measuring and explaining word learning. Rectangles in the figure represent (binary) item responses for the three measures (i.e., MC, SR, and MR) and circles represent random effects. Dependency in item responses from J ( $j=1,\dots ,J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j=1, \ldots , J$$\end{document} ) readers nested within K ( $k=1,\dots ,K$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k=1, \ldots , K$$\end{document} ) teachers, and I items ( $i=1,\dots ,I$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1, \ldots , I$$\end{document} ) nested within G words ( $g=1,\dots ,G$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=1, \ldots , G$$\end{document} ) for D measures ( $d=1,2,3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=1, 2, 3$$\end{document} ) at T ( $t=1,2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t=1, 2$$\end{document} ) time points (i.e., $J\times I\times T=JIT$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J \times I \times T = JIT$$\end{document} item responses) is explained by 16 random effects: 6 reader-level random effects ( $s_{k\left[j\right]11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]11}$$\end{document} , $s_{k\left[j\right]12}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]12}$$\end{document} , $s_{k\left[j\right]13}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]13}$$\end{document} , $s_{k\left[j\right]21}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]21}$$\end{document} , $s_{k\left[j\right]22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]22}$$\end{document} , $s_{k\left[j\right]23}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]23}$$\end{document} ), 6 teacher-level random effects ( $s_{k11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k11}$$\end{document} , $s_{k12}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k12}$$\end{document} , $s_{k13}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k13}$$\end{document} , $s_{k21}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k21}$$\end{document} , $s_{k22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k22}$$\end{document} , $s_{k23}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k23}$$\end{document} ), 3 item-level random effects ( $w_{g\left[i\right]1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]1}$$\end{document} , $w_{g\left[i\right]2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]2}$$\end{document} , $w_{g\left[i\right]3}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]3}$$\end{document} ), and 1 item group-level random effect ( $w_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g}$$\end{document} ). On the reader side, the total random effect for each measure is decomposed into a reader-level random effect and a teacher-level random effect at each time point (at pretest, $s_{k\left[j\right]11}+s_{k11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]11}+s_{k11}$$\end{document} for MC, $s_{k\left[j\right]12}+s_{k12}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]12}+s_{k12}$$\end{document} for SR, and $s_{k\left[j\right]13}+s_{k13}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]13}+s_{k13}$$\end{document} for MR; at posttest, $s_{k\left[j\right]21}+s_{k21}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]21}+s_{k21}$$\end{document} for MC, $s_{k\left[j\right]22}+s_{k22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]22}+s_{k22}$$\end{document} for SR, and $s_{k\left[j\right]23}+s_{k23}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]23}+s_{k23}$$\end{document} for MR). At both the reader and teacher level, the six random effects are correlated (shown in the same rectangle) because the same readers responded to the three measures repeatedly (i.e., at pretest and posttest). On the item side, there are three item random effects (denoted by $w_{g\left[i\right]1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]1}$$\end{document} , $w_{g\left[i\right]2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]2}$$\end{document} , and $w_{g\left[i\right]3}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]3}$$\end{document} ) nested within a word random effect (denoted by $w_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g}$$\end{document} ) modeled to explain dependency in item responses from items nested within words at both time points with measurement invariance assumed over time points. Random effects of readers and items are crossed random effects at Level 2 and random effects of teachers and words are crossed at Level 3. A reader characteristic such as an instructional treatment variable (denoted by TRT in the figure) can be used to explain a reader-level random effect for each measure at each time point (possibly controlling for other reader characteristics). Word characteristics (denoted by word in the figure) can be used to explain variability in item easiness across words.

3.2. Intraclass Correlations

Intraclass correlations (ICC) (denoted by $\rho$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho $$\end{document} ) can be calculated to show dependency in latent item responses among readers and items due to teachers and words, respectively. Let there be a latent item response $y_{k\left[j\right]g\left[i\right]td}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{k[j]g[i]td}^{*}$$\end{document} such that the observed item response is $y_{k\left[j\right]g\left[i\right]td}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{k[j]g[i]td}=1$$\end{document} if $y_{k\left[j\right]g\left[i\right]td}^{\ast }\ge 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{k[j]g[i]td}^{*} \ge 0$$\end{document} , and $y_{k\left[j\right]g\left[i\right]td}=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{k[j]g[i]td}=0$$\end{document} otherwise (e.g., Milanzi, Molenberghs, Alonso, Verbeke, & De Boeck, Reference Milanzi, Molenberghs, Alonso, Verbeke and De Boeck2015). With the measurement invariance assumption for item parameters (i.e., $w_{g\left[i\right]d}^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g[i]d}^{(2)}$$\end{document} and $w_g^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{g}^{(3)}$$\end{document} ), it is assumed that

(5) $\begin{array}{c}{y}_{k\left[j\right]g\left[i\right]td}^{\ast }=X_{td}^{\prime }{\beta }_{td}+S_{td}^{\prime \left(2\right)}{s}_{k\left[j\right]td}^{\left(2\right)}+S_{td}^{\prime \left(3\right)}{s}_{ktd}^{\left(3\right)}+W_{td}^{\prime \left(2\right)}{w}_{g\left[i\right]d}^{\left(2\right)}+W_{td}^{\prime \left(3\right)}{w}_g^{\left(3\right)}+{ϵ}_{k\left[j\right]g\left[i\right]td}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} y_{k[j]g[i]td}^{*}=X_{td}^{\prime }{\varvec{\beta }}_{td} + S_{td}^{\prime (2)}s_{k[j]td}^{(2)} + S_{td}^{\prime (3)}s_{ktd}^{(3)} + W_{td}^{\prime (2)}w_{g[i]d}^{(2)} + W_{td}^{\prime (3)}w_{g}^{(3)} + \epsilon _{k[j]g[i]td}. \end{aligned}$$\end{document}

In addition, the error, ${ϵ}_{k\left[j\right]g\left[i\right]td}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon _{k[j]g[i]td}$$\end{document} , is assumed to follow a logistic distribution (mean $=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=0$$\end{document} , variance $=\frac{{\pi }^2}{3}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=\frac{\pi ^{2}}{3}$$\end{document} ) with a logit link.

The ICC due to teachers for each measure d at a time point t ( $\rho {\left(R\right)}_{td}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{td}$$\end{document} ) can be defined as the correlation among latent item responses for the same teacher k, but for different readers j and $j^{\prime }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j'$$\end{document} . It is conditional on random effects at the item and word levels, and random effects for the other measures and other time points than a time point t and a measure d (denoted by $s_{k\left[j\right]t^{\prime }{d}^{\prime }}^{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{k[j]t'd'}^{(2)}$$\end{document} and $s_{k{t}^{\prime }{d}^{\prime }}^{\left(3\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{kt'd'}^{(3)}$$\end{document} ):

(6) $\begin{array}{cc}\rho {\left(R\right)}_{td}=& Corr(y_{k\left[j\right]g\left[i\right]td}^{\ast },y_{k\left[j^{\prime }\right]g\left[i\right]td}^{\ast })\\ =& \frac{Cov(y_{k\left[j\right]g\left[i\right]td}^{\ast },y_{k\left[j^{\prime }\right]g\left[i\right]td}^{\ast }|s_{k\left[j\right]t^{\prime }{d}^{\prime }}^{\left(2\right)},s_{k{t}^{\prime }{d}^{\prime }}^{\left(3\right)},w_{g\left[i\right]}^{\left(2\right)},w_g^{\left(3\right)})}{\sqrt{Var(y_{k\left[j\right]g\left[i\right]td}^{\ast }|s_{k\left[j\right]t^{\prime }{d}^{\prime }}^{\left(2\right)},s_{k{t}^{\prime }{d}^{\prime }}^{\left(3\right)},w_{g\left[i\right]}^{\left(2\right)},w_g^{\left(3\right)})}·\sqrt{Var(y_{k\left[j^{\prime }\right]g\left[i\right]td}^{\ast }|s_{k\left[j\right]t^{\prime }{d}^{\prime }}^{\left(2\right)},s_{k{t}^{\prime }{d}^{\prime }}^{\left(3\right)},w_{g\left[i\right]}^{\left(2\right)},w_g^{\left(3\right)})}}\\ =& \frac{{\sigma }_{3td}^2}{{\sigma }_{2td}^2+{\sigma }_{3td}^2+\frac{{\pi }^2}{3}},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho (R)_{td}= & {} Corr\big (y_{k[j]g[i]td}^{*},y_{k[j']g[i]td}^{*}\big ) \nonumber \\= & {} \frac{Cov\Big (y_{k[j]g[i]td}^{*},y_{k[j']g[i]td}^{*}|s_{k[j]t'd'}^{(2)},s_{kt'd'}^{(3)},\mathbf{w}_{g[i]}^{(2)}, w_{g}^{(3)}\Big )}{\sqrt{Var\Big (y_{k[j]g[i]td}^{*}|s_{k[j]t'd'}^{(2)},s_{kt'd'}^{(3)},\mathbf{w}_{g[i]}^{(2)},w_{g}^{(3)}\Big )} \cdot \sqrt{Var\Big (y_{k[j']g[i]td}^{*}|s_{k[j]t'd'}^{(2)},s_{kt'd'}^{(3)},\mathbf{w}_{g[i]}^{(2)},w_{g}^{(3)}\Big )}} \nonumber \\= & {} \frac{\sigma _{3td}^{2}}{\sigma _{2td}^{2}+\sigma _{3td}^{2}+\frac{\pi ^{2}}{3}}, \end{aligned}$$\end{document}

where ${\sigma }_{2td}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{2td}^{2}$$\end{document} and ${\sigma }_{3td}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{3td}^{2}$$\end{document} are the variances of ${\Sigma }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{2}$$\end{document} and ${\Sigma }_3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{3}$$\end{document} , respectively. The $\rho {\left(R\right)}_{td}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{td}$$\end{document} can also be interpreted as the proportion of the variance of the latent item response $y^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y^{*}$$\end{document} for a measure d at a time point t that is accounted for by the teacher clustering.

As in our empirical data, each item is assumed to be related to only one measure d in the calculation the ICC due to words. ICC due to words ( $\rho \left(I\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (I)$$\end{document} ) can be defined as the correlation among latent responses for the same word g, but for different items i and $i^{\prime }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i'$$\end{document} , conditional on random effects at the reader and teacher levels:

(7) $\begin{array}{cc}\rho \left(I\right)=& Corr(y_{k\left[j\right]g\left[i\right]td}^{\ast },y_{k\left[j\right]g\left[i^{\prime }\right]td}^{\ast })\\ =& \frac{Cov(y_{k\left[j\right]g\left[i\right]td}^{\ast },y_{k\left[j\right]g\left[i^{\prime }\right]td}^{\ast }|s_{k\left[j\right]}^{\left(2\right)},s_k^{\left(3\right)})}{\sqrt{Var(y_{k\left[j\right]g\left[i\right]td}^{\ast }|s_{k\left[j\right]}^{\left(2\right)},s_k^{\left(3\right)})}·\sqrt{Var(y_{k\left[j\right]g\left[i^{\prime }\right]td}^{\ast }|s_{k\left[j\right]}^{\left(2\right)},s_k^{\left(3\right)})}}\\ =& \frac{{\sigma }_4^2}{{\sum }_{d=1}^D{\sigma }_{3d}^2+{\sigma }_4^2+\frac{{\pi }^2}{3}}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho (I)= & {} Corr\big (y_{k[j]g[i]td}^{*},y_{k[j]g[i']td}^{*}\big ) \nonumber \\= & {} \frac{Cov\Big (y_{k[j]g[i]td}^{*},y_{k[j]g[i']td}^{*}|\mathbf{s}_{k[j]}^{(2)},\mathbf{s}_{k}^{(3)}\Big )}{\sqrt{Var \Big (y_{k[j]g[i]td}^{*}|\mathbf{s}_{k[j]}^{(2)},\mathbf{s}_{k}^{(3)}\Big )} \cdot \sqrt{Var \Big (y_{k[j]g[i']td}^{*}|\mathbf{s}_{k[j]}^{(2)},\mathbf{s}_{k}^{(3)}\Big )}} \nonumber \\= & {} \frac{\sigma _{4}^{2}}{\sum _{d=1}^{D}\sigma _{3d}^{2} +\sigma _{4}^{2}+\frac{\pi ^{2}}{3}}. \end{aligned}$$\end{document}

Also, the $\rho \left(I\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (I)$$\end{document} can be seen as the explained variance of the latent item response $y^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y^{*}$$\end{document} that is explained by the word clustering.

4.1. Data Description

Table 1 shows descriptive information for all variables described in detail below. This study is an extension of Goodwin (in press) and shares some overlapping data.

Table 1.

Descriptive statistics.

4.2. Word Covariates

Word characteristics served as time-invariant word covariates. Relevant word characteristics were identified from the literature (Carlisle & Katz, Reference Carlisle and Katz2006; Goodwin, Gilbert, Cho, & Kearns, Reference Goodwin, Gilbert, Cho and Kearns2014; Nagy et al. , Reference Nagy, Anderson and Herman1987) and data were secured for the 16 target words described in Table 5.

Morphologically Complex Word Frequency (MWF) The Standard Frequency Index (SFI) from the Educators Word Frequency Guide (Zeno, Ivens, Millard, & Duvvuri, Reference Zeno, Ivens, Millard and Duvvuri1995) was used to represent how often a word is used within written academic texts (in this case, within a corpus of 60,527 academic texts). SFI values can be interpreted where 50 % of the words in sixth-grade texts have SFI values of 32 or greater. These values are based on D, which is a measure of dispersion of a word across subject areas. SFI values are logarithmic transformations of U, which is the frequency of the type per million tokens weighted by D.

Root Word Frequency (RWF) Root words for each target word were identified from Becker, Dixon, and Anderson-Inman (Reference Becker, Dixon and Anderson-Inman1980). Then SFI values were obtained for each root word as described above.

Morphological Family Size (NUMREL) Morphological family members for each target word were identified using Becker et al. ’s (Reference Becker, Dixon and Anderson-Inman1980) database. The database was digitalized and then searched for words that had the same root word as the target word including inflections, derivations, and compounds. An example is for effortlessly, the words effortless, effortful, and effortfully all had the same root word effort.

Number of Morphemes (NUM) The number of morphemes within each target word was identified from Becker et al. ’s (Reference Becker, Dixon and Anderson-Inman1980) database. The morphographic breakdown was used with the example effortlessly shown to contain four morphemes as in ef+(fort)+less+ly.

Average Affix Frequency (FAFFIX) The Becker et al. ’s (Reference Becker, Dixon and Anderson-Inman1980) database was also used to determine the average frequency of the affixes contained within the target word. The frequency of each affix was identified within the database and then the average frequency was calculated. An example is effortlessly, which has the suffixes less (frequency of 219) and ly (frequency of 1101), making the average affix frequency for effortlessly 660.

Most Frequent Affix (HIGHAFFIX) To determine the most frequent affix, the frequencies of each affix were determined using Becker et al. (Reference Becker, Dixon and Anderson-Inman1980). The frequency of the affix with the highest frequency value was recorded, with for example, the frequency of 1101 for ly recorded for effortlessly.

Opaqueness (OPAQUE) Transparency of each target word was determined as opaque (value  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  1) if there were spelling and phonological changes between the target word and its root word like circulate (whose root word is circle) or as transparent (value  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0) if there were no spelling or phonological changes between the target word and its root word like effortlessly (whose root word is effort).

4.3. Analysis

To answer research question (a)i, the dimensionality structure of word knowledge was first investigated at each time point. Three hypothesized structures stemming from the vocabulary literature were explored (i.e., a 1-dimensional [1D] model, a 2-dimensional [2D] model [multiple-choice , self-report  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  breadth, related word production  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  depth], and a 3-dimensional [3D] model). The unidimensional perspective represents the view taken by many researchers that word knowledge can be assessed in many ways, yet each measure is assessing a general construct of word knowledge. The many studies that assess word knowledge via a single assessment represent this view (see Pearson et al. , Reference Pearson, Hiebert and Kamil2007 for further details). The hypothesized two dimensional model represents findings that there are two specific dimensions of word knowledge that must be considered. These are breadth, which conveys how many words students have basic knowledge of, and depth, which refers to how much knowledge a person has about words (Ouellette, Reference Ouellette2006; Tannenbaum, Torgesen, & Wagner, Reference Tannenbaum, Torgesen and Wagner2006). Here, a person may recognize command but not be able to define or interpret or use the word across multiple contexts. Depth of word knowledge appears to be more important than breadth in supporting reading comprehension (Ouellette, Reference Ouellette2006). The three dimensional model takes into account findings like those from Kieffer and Lesaux (Reference Kieffer and Lesaux2012) that indicate word knowledge is also composed of additional dimensions including a dimension for morphological awareness (i.e., knowledge of units of meaning like root words [e.g., command] and affixes [e.g., er in commander] and links to morphological relatives [e.g., command, commander, commanding]).

The three comparison models have the intercept, random effects at the reader level, and random effects at the item level, respectively. To determine which multilevel structure is needed to be modeled (i.e., cross-classified tutors effects, nested teacher effects, and nested word effects), the models with and without nested random effects and crossed random effects were compared at each time point based on which hypothesized structure was shown to fit the data best (i.e., dimensionality). Akaike information criterion (AIC; Akaike, Reference Akaike1974) and Bayesian information criterion (BIC; Schwarz, Reference Schwarz1978) were used to identify the best- fitting model for the dataset. In the calculation of BIC, the number of readers ( $J=202$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=202$$\end{document} ) was used to adjust the sample size for model complexity.

For the finalized unconditional model based on results of research question (a)(i), the measurement invariance was investigated by comparing word easiness at the word level between the pretest and the posttest. When the correlation coefficient between pretest and posttest word easiness is high, we conclude that there is evidence of measurement invariance because the relative ordering of the word easiness did not change between the two time points.

Based on model selection and measurement invariance results, the generalized mixed effect model for doubly multilevel structure that was specified earlier was fit to answer research question (a)ii, which explored whether significant word learning (i.e., intercept coefficients at posttest) occurred for each measure of word knowledge. Also, variability in initial knowledge and learning for each measure of word knowledge was interpreted. Using estimates of the model, word easiness values (technically, the conditional modes of the random effects) were calculated using the extractor function ranef of arm package version 1.3-06 (Gelman et al. , Reference Gelman, Su, Yajima, Hill, Pittau, Kerman and Zheng2010) in R to show variability in item easiness across words. To evaluate the model fit, we calculated item fit (the mean of the standard residuals over persons) and person fit (the mean of the standard residuals over items) statistics to confirm our measurement model represented each test item and each person well. Standardized residuals smaller than $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 2 or larger than 2 would indicate possible misfit at 5 % level. Harrell’s c calculated using somers2 function in Hmisc package (Harrell, Reference Harrell2015) was also calculated as a measure of the ordinal predictive power of the model. In addition, ICCs were calculated to show the degree of dependency due to teachers or words. The finalized measurement model (i.e., the unconditional model) was labeled as Model 1.

To answer research question (b) (i.e., the effects of word and instructional characteristics on word easiness and word learning, respectively), we added reader covariates like instructional treatment and word covariates to the unconditional model from research question (a)ii (i.e., Model 1). We focused on instructional contexts and word characteristics because preliminary analyses showed that the effects of all reader covariates except language background were not significant covariates of readers’ differences in learning. Thus, only the language background covariate will be used for further analyses among covariates of reader characteristics. Two dummy variables were created for language background with native English speakers as the reference group: ELL variable (English speaker $=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=0$$\end{document} , ELL $=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=1$$\end{document} ) and LMY variable (English speaker $=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ = 0$$\end{document} , LMY $=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=1$$\end{document} ). A dummy variable was also created for intervention status with the comparison group being the reference group (comparison group $=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=0$$\end{document} , intervention group $=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=1$$\end{document} ). All continuous word covariates were standardized. The model for research question (b) was labeled as Model 2.

4.4. Results

4.4.1. Results of Research Question (a)

Table 2 presents the number of parameters (Num.), the log-likelihood (LL), AIC, and BIC of the three candidate models regarding the number of dimensions and the four candidate models regarding the different random effects at each time point. At both time points, the 3D model having random effects over readers, teachers, items, and words was the best-fitting model. The correlation coefficient between word easiness values at pretest and posttest was 0.90, which provides evidence that there was the measurement invariance necessary to measure learning. Related to research question (a)i, results indicate word knowledge is best conceptualized as multidimensional (i.e., the 3D model) where multiple choice, self-report, and related words production are related but assess different aspects of word knowledge.

Table 2.

Results of model selection for pretest (top) and posttest (bottom) (Results of research question (a)i).

Num. number of parameters to be estimated, LL log-likelihood, 1D 1-dimensional model, 2D 2-dimensional model with multiple-choice , self-report representing meaning and related word production representing interrelatedness, 3D 3-dimensional model, Word random word (item group) effects, Tutor (cross-classified) random tutor effects, and Tch random teacher (cluster) effects.

Results for the generalized mixed effect model for doubly multilevel structure are displayed in Table 3 (fixed effects) and in Table 4 (population parameters of random effects).Footnote 4 Based on results reported in Model 1 of Table 4, ICCs were calculated:

$\rho {\left(R\right)}_{11}=0.21/(0.61+0.21+3.29)=0.051$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{11}=0.21/(0.61+0.21+3.29)=0.051$$\end{document} , $\rho {\left(R\right)}_{12}=0.29/(2.12+0.29+3.29)=0.051$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{12}=0.29/(2.12+0.29+3.29)=0.051$$\end{document} , $\rho {\left(R\right)}_{13}=3.00/(5.74+3.00+3.29)=0.249$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{13}=3.00/(5.74+3.00+3.29)=0.249$$\end{document} , $\rho {\left(R\right)}_{21}=0.02/(0.15+0.02+3.29)=0.006$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{21}=0.02/(0.15+0.02+3.29)=0.006$$\end{document} , $\rho {\left(R\right)}_{22}=0.11/(1.47+0.11+3.29)=0.023$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{22}=0.11/(1.47+0.11+3.29)=0.023$$\end{document} , $\rho {\left(R\right)}_{23}=0.55/(9.17+0.55+3.29)=0.042$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)_{23}=0.55/(9.17+0.55+3.29)=0.042$$\end{document} , and $\rho \left(I\right)=(1.51/[0.15+0.06+0.59]+1.51+3.29)=0.270$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (I)=(1.51/[0.15+0.06+0.59]+1.51+3.29)=0.270$$\end{document} . The ICC values of $\rho \left(R\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (R)$$\end{document} indicate that 5.1 and 6.0 % of the latent item response for the multiple choice (MC) measure were explained by teacher clustering at pretest and posttest, respectively; 5.1 and 2.3 % of the latent item response for the self-report (SR) measure were explained by teacher clustering at pretest and posttest, respectively; and 24.9 and 4.2 % of the latent item response for the related word production (MR) measure were explained by teacher clustering at pretest and posttest, respectively. Also, the ICC value of $\rho \left(I\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (I)$$\end{document} suggest that 27 % of the latent item response was accounted for word clustering. A common rule of thumb in educational research is that ICCs over 0.05 indicate that there is nonignorable dependency due to clusters (e.g., Jak, Oort, & Dolan, Reference Jak, Oort and Dolan2013). According to this rule of thumb, dependency due to teachers for initial status and due to words is of concern. It is interesting that lower ICCs were found in learning due to teachers. Our interpretation of the lower ICCs are that the main word learning that was occurring was related to the intervention or comparison instruction, which was delivered by tutors.

Table 3.

Results for fixed effects of unconditional and conditional models.

Significance in bold at the 5  % level. INT intercepts, TRT treatment variable (comparison group  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0, intervention group  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  1), ELL  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  ELL variable (English speaker  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0, ELL  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  1), LMY LMY variable (English speaker  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0,  LMY  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  1), MWF morphologically complex word frequency variable, RWF root word frequency variable, NUMREL morphological family size variable, NUM number of morphemes variable, FAFFIX average affix frequency variable, HIGHAFFIX most frequent affix variable, OPAQUE opaqueness variable, MC multiple-choice, SR self-report, and MR production of morphological relatives

In Table 3, significance for the fixed effects is presented in bold. Significance at the 5 % level two-tailed test was determined by whether the absolute value of the t-statistic exceeded 2. Word learning results are shown in Model 1 of Table 3 for fixed effects. Related to research question (a)ii, results suggest there was significant learning of words as shown by the significant learning intercept coefficient for each aspect of word knowledge at posttest (EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.704, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.078 for multiple choice; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  1.308, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.141 for self-report; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  2.124, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.310 for related words production). These effects on the probability scale (transformed from the logit scale) are 0.169 ( $=[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=[1/(1+$$\end{document} exp $(-0.704)]-[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(-0.704)]-[1/(1+$$\end{document} exp(0))]), 0.287 ( $=[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=[1/(1+$$\end{document} exp $(-1.308)]-[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(-1.308)]-[1/(1+$$\end{document} exp(0))]), and 0.393 ( $=[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=[1/(1+$$\end{document} exp $(-2.124)]-[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(-2.124)]-[1/(1+$$\end{document} exp(0))]) for multiple choice, self-report, and related words production, respectively.

Table 4.

Results for random effects of unconditional and conditional models.

MC multiple-choice, SR self-report, and MR production of morphological relatives.

As shown in Table 4, we found that students differed in their knowledge and learning of different aspects of word knowledge. For example, we found small variability in initial knowledge and learning for multiple choice responses compared to large variability for generation of morphologically related words. This likely reflects differences between breadth and depth measures as one would expect less variability in a receptive recognition measure compared to a production measure (Anderson & Freebody, Reference Anderson and Freebody1981). It also likely reflects the incremental nature of word learning (Nagy et al. , Reference Nagy, Anderson and Herman1987), as words were taught with different intensities and with different aspects of knowledge emphasized. A single exposure to a word may build initial meaning knowledge (Carey & Bartlett, Reference Carey and Bartlett1978; Markson & Bloom, Reference Markson and Bloom1997), whereas multiple experiences may be required to build other aspects of word knowledge, like further depth of knowledge (Elleman, Lindo, Morphy, & Compton, Reference Elleman, Lindo, Morphy and Compton2009). These differences in initial knowledge and learning highlight how assessment, instruction, and experiences with words would need to be adapted depending on the type of word knowledge being examined and built.

We also explored word easiness as it relates to word learning. Table 5 displays the predicted word easiness as well as values for each word characteristic. In the table, all word characteristics except OPAQUE were standardized. Words varied in terms of easiness as shown by the value of 1.51 of the variance estimate of the random word easiness and by the easiness values shown in Table 5 where words ranged from easy (uncomfortable, 1.827) to hard (provision, $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 2.572). This variation is what will be explored using word covariates.

Table 5.

Word easiness variability over words from the unconditional generalized mixed effect model.

All word characteristics except OPAQUE were standardized. MWF morphologically complex word frequency, RWF root word frequency, NUMREL morphological family size, NUM number of morphemes, FAFFIX average affix frequency, HIGHAFFIX most frequent affix, and OPAQUE opaqueness.

For the unconditional model used to answer research question a(ii) (i.e., Model 1), standardized residuals for persons ranged from $-0.135$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.135$$\end{document} to 0.117 and standardized residuals for items ranged from $-0.036$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.036$$\end{document} to 0.072, suggesting good fit to the dataset. Harrell’s c was 0.91, which indicates the model fit is relatively satisfactory.

4.4.2. Results of Research Question (b)

Instructional effects on word learning controlling for language background as well as effects of word characteristics on word easiness were investigated in research question (b). Estimates of fixed effects are presented in the Model 2 column of Table 3 and estimates of population parameters of random effects are shown in the Model 2 column of Table 4. When controlling for instruction as well as the other covariates, there were significant differences in learning between the native speakers and ELLs for self-report and related word production (EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}   $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 0.837, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.329 for the self-report; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}   $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 1.583, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.688 for the related word production), however, there was no significant difference in learning between the native speakers and ELLs for multiple choice performance (EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}   $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 0.104, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.182). In addition, there were no significant differences in learning between the native speakers and LMY for all three word knowledge measures (EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.205, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.149 for multiple choice; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.005, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.251 for self-report; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}   $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 0.526, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.559 for related word production). These results indicate that morphological instruction was able to support ELL students in building multiple choice knowledge and to support LMY students in building for all three word knowledge measures.

Controlling for the other covariates, there was no significant instructional group difference in initial performance for all three word knowledge measures (EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.008, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.141 for multiple choice; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.245, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.227 for self-report; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.561, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.368 for related word production). However, controlling for the other covariates, there were significant instructional group differences in learning for all three word knowledge measures (EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.472, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.134 for multiple choice; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  1.120, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.210 for self-report; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  1.303, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.480 for related word production). On the probability scale, these effects are 0.116 ( $=[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=[1/(1+$$\end{document} exp $(-0.472)]-[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(-0.472)]-[1/(1+$$\end{document} exp(0))]), 0.254 ( $=[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=[1/(1+$$\end{document} exp $(-1.120)]-[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(-1.120)]-[1/(1+$$\end{document} exp(0))]), and 0.286 ( $=[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=[1/(1+$$\end{document} exp $(-1.303)]-[1/(1+$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(-1.303)]-[1/(1+$$\end{document} exp(0))]) for multiple choice, self-report, and related word production, respectively. This indicates intervention students were much more likely to show more word knowledge at posttest versus comparison students. At the reader level, the explained variances in learning for each measure compared to Model 1 was 33.3, 27.4, and 7.3 % for multiple choice, self-report, and related word production, respectively.

Figure 3.

Visualization of the significant partial effects of NUMREL (top) and OPAQUE (bottom), adjusted to the 0 values of the other continuous covariates, the reference level of the other categorical covariates, and the 0 values of all random effects.

Controlling for the other covariates in the model, two word characteristics had significant effects on word easiness: morphological family size (NUMREL; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.798, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.310) and opaqueness (OPAQUE; EST  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}   $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 2.666, SE  $=$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$=$$\end{document}  0.800). The effect of 0.798 means that word easiness is 0.798 easier for every 1SD change in morphological family size. The effect of $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 2.666 indicates that words that are opaque are harder by 2.666 compared to words that are not opaque. Figure 3 presents the significant partial effects of NUMREL (top) and OPAQUE (bottom) on the probability scale with 95 % confidence interval (CI) (shown with dotted lines for NUMREL and with a vertical bar for OPAQUE), adjusted to the 0 values of the other continuous covariates, the reference level of the other categorical covariates, and the 0 for all random effects. Including word characteristics was particularly important in explaining word easiness. The explained variance for word easiness using word characteristics (compared to Model 1) is 0.621, which indicates that 62.1 % of variance in word easiness was explained by word characteristics. When word easiness values were predicted on the basis of the word characteristic fixed effects, the correlation with the estimated word easiness in Model 1 was 0.78, which means that there is moderately high explanatory power of the word characteristics.