By Jenna Somers
Educators and researchers around the world, especially in countries with limited resources, need cost-effective, scalable tools for assessing early child development. Direct assessments, a commonly used approach, require children to complete a range of activities that measure academic, physical and social-emotional skills. These assessments paint a nuanced picture of early child development and are useful to evaluate programs and policies. However, administering direct assessments often takes 30 or more minutes, making them impractical for measuring large populations. Practitioners and researchers need more practical methods.
In a paper recently published in Early Childhood Research Quarterly, Vanderbilt professor Jonathan Seiden offers a versatile framework for creating quicker versions of tests (often called "short forms") measuring early child development with direct assessment. Short forms use a subset of questions from the full assessment. However, selecting which questions to retain is a challenge. Rather than simply focus on questions that are the most statistically reliable and easiest to administer, Seiden's framework encourages test developers to evaluate questions along three dimensions. The goal is to create short forms that are
- conceptually valid and cover the full range of skills being assessed,
- statistically reliable and generate consistent results and
- practical to administer at lower cost and complexity.
An outdated but still used approach to short forms focuses on selecting statistically reliable questions, but that can come at the expense of a test's validity by failing to measure the full breadth of skills and failing to maximize the feasibility of use.
A more modern approach, Automated Test Assembly, relies on an algorithmic method for selecting a subset of questions. While this approach is ideal for advanced test developers, it requires advanced coding skills and creating hard constraints about how the short form should function.
"Rather than giving various constraints to an algorithm to find the ideal set of questions for a short form, I wanted to provide test developers with as much information in as simplified a manner as possible to make decisions themselves and weigh trade-offs holistically," said Seiden, professor of early childhood policy in the Department of Leadership, Policy and Organizations at Vanderbilt Peabody College of education and human development.
Seiden's human-centered framework can be used to create what he characterizes as a "balanced" short form, a curated set of questions that retains the full scope of skills from the full assessment while reducing administrative burdens as much as possible.
Application to the International Development and Early Learning Assessment
In the paper, Seiden applied his framework to the International Development and Early Learning Assessment (IDELA)-a popular direct assessment of early child development which has been used in over 100 countries-to create a balanced short form IDELA. The short form includes 8 tasks that
- cover all key skill domains (motor, social-emotional, literacy and numeracy),
- are judged as highly relevant by child development experts
- can be administered quickly (about 8 minutes).
A traditional short form might select only questions on academic skills, which offer more statistically reliable measurements, but that comes at the cost of other important measures like motor and social-emotional development, resulting in a less valid assessment. The traditional short form also doesn't reduce the length of administration as much as the balanced short form.
The balanced short form may be less reliable by including harder-to-measure skills but more valid because it covers the breadth of skill domains. It sacrifices a small degree of reliability but still results in scores that are sufficiently precise for use in large-scale measurement exercises.
Designed for Practitioners
Importantly, Seiden's framework offers a way forward for practitioners to make contextual decisions about tradeoffs that make sense for their specific situations and budgets. He got the idea for the framework when he was working with IDELA as an education research specialist with Save the Children.
"I advocated for trying to get a better understanding of how difficult it is to administer each item. Typically, in psychometrics, we treat questions interchangeably, but with direct assessments of early childhood development, there is huge variability in how long it takes to administer different questions. I wondered how we could make IDELA simpler and easier to use, and how might we select the best subset of questions to do that," Seiden said.
As he began designing the framework for Save the Children's use of IDELA, other psychometricians expressed interest in its application for other early childhood development assessments.
Real-World Impact
The framework has already moved from theory to practice. A recent study applied Seiden's methodology to an assessment tool in Nepal to create a balanced short form of an early learning and development assessment used nationally. Clearly, the approach is resonating with researchers facing similar constraints.
Extending beyond early childhood assessment, the framework could potentially be adapted to create short forms across other fields. Given the resource constraints faced by researchers, policymakers, and practitioners in many settings, Seiden's framework offers a practical path forward for more valid short forms.
