Switching schools can affect students cognitively, socially and emotionally. In Maryland’s public school system, more than a third of students make such a change between grades 9 and 12. Yet researchers don’t have an accurate way of accounting for these shifts in large studies assessing academic outcomes like achievement scores or college enrollment. The College of Education’s Quantitative Methodology: Measurement and Statistics (QMMS) program is working to change that.
QMMS Ph.D. student Tessa Johnson, Department of Human Development and Quantitative Methodology Chair and Professor Laura Stapleton Ph.D. ’01, and Associate Professor Tracy M. Sweet co-authored a study published in the journal Applied Developmental Science that used a statistical model to adjust for school transitions in data from the Maryland Longitudinal Data System (MLDS). Established in 2012, MLDS links education and workforce data from more than 4.8 million Marylanders; it can be used to improve the state’s education system.
The study represents one of the first applications of this analytical approach to a state longitudinal data system, the authors say. QMMS worked with the MLDS Center — an independent state agency that partners with university researchers — to conduct the investigation.
The team used a method called “multiple membership modeling” to account for dynamic school environments in the statewide data. Researchers need sophisticated strategies for separating cause and effect as precisely as possible when working with comprehensive datasets like MLDS that track multiple elements in a hierarchical, or “nested,” structure.
“When folks analyze data that are nested — of kids within schools, kids within classrooms — and they don’t address the fact that you have this complex nesting, what are your options?” says Stapleton.
As Johnson explains, “students don’t behave the way you want them to” — at least not if you’re an education researcher looking to tease out the impact of specific variables like teaching environments on long-term endpoints like graduation rates. For example, if a teen switches schools due to a parental divorce, their grades might suffer regardless of the quality of instruction.
“It’s hard to isolate a teacher effect when the life of a student is so complex,” Johnson says. But doing so “can be really important for students who are minoritized or who are in situations that are not very supportive in school,” Johnson adds.
Optimizing the value of data is what QMMS does best. “What we do is look for problems in how data are analyzed and suggest better ways of doing it,” explains Stapleton.
She and Johnson use the metaphor of a microwave’s pre-set popcorn button: However well-intentioned its design, the popcorn button sometimes scorches your kernels. “Someone clearly put this popcorn button there for a reason,” says Johnson. “My job as the quantitative methodologist is to put popcorn in a microwave multiple times and figure out which settings to use to get my popcorn out exactly the way I want it to be.”
In the study, the figurative popcorn buttons are standard research models that simplify analyses by lumping mobile students in with everyone else or excluding them altogether. The authors applied a more nuanced numerical model and compared it to these standard approaches.
They found only small differences between standard and multiple membership models. In other words, the popcorn buttons worked just fine in this case. They believe that’s because Maryland’s mobile students tend to move between similar schools. However, the team hopes their work raises awareness about the importance of considering dynamic school contexts in educational research — and they say multiple membership models could be useful in other research scenarios, as well.
Using this method in dynamic contexts “would be applicable to any study where individuals are grouped within some higher-level organization,” such as hospitalized or incarcerated populations, says Angela K. Henneberger ’06, the study’s lead author and research director and principal investigator of the MLDS Center Research Branch and research associate professor at the University of Maryland School of Social Work.
Additional co-authors included Yi Feng M.A. ’18, Ph.D. ’23, associate professor at the University of California, Los Angeles, and Bess A. Rose, statistician, and Michael E. Woolley, professor, both at the UMD School of Social Work.
This research was supported by the Institute of Education Sciences.
Photo by iStock
