Rethinking edtech evaluation

Two-thirds of teachers use AI, but only one in five edtech products has evidence of improving outcomes. In this episode: Nearly two-thirds of teachers use AI, but only 20% of AI edtech products have evidence of improving outcomes, underscoring the urgent need for better AI education research. Traditional randomized controlled trials (RCTs) are often too slow and rigid for evaluating rapidly evolving AI tools, necessitating new education research methods. Stacey Alicea and Meghan McCormick propose 'implementation research and development' as a robust framework for assessing AI tool effectiveness through iterative testing and refinement. Three guiding principles for AI edtech evaluation are: building evidence in stages, asking 'how it works' before 'whether it works,' and letting specific research questions dictate the methodology. The Research Partnership for Professional Learning's Shared Measures Toolkit demonstrates effective, iterative evaluation, building measurement infrastructure crucial for responsible AI in classrooms. Chapters: 00:00 — Cold open & welcome 00:45 — The problem: AI use outpaces AI edtech evaluation 01:30 — Why traditional RCTs fail for AI education research 02:45 — Introducing implementation research and development (R&D) for AI tool effectiveness 03:45 — National efforts embracing iterative AI edtech evaluation 04:30 — Principle 1: Build AI evidence in stages (feasibility first) 05:30 — Principle 2: Ask 'how it works' before 'whether it works' for AI in classrooms 06:45 — Principle 3: Let research questions drive the education research methods 08:00 — Implications for school leaders and the need for faster evidence 09:00 — Example: Research Partnership for Professional Learning's Shared Measures Toolkit How can we evaluate new AI tools in education more effectively? To evaluate new AI tools effectively, educators should shift from relying solely on slow randomized controlled trials to iterative 'implementation research and development' that rapidly tests and refines tools in real-world settings. Why are traditional education research methods not working for AI? Traditional education research methods like randomized controlled trials are often too slow and designed for static interventions, making them unsuitable for the rapid and continuous evolution of AI tools in education. What is implementation research and development for AI in education? Implementation research and development (R&D) is an approach that prioritizes rapid testing, feedback, and refinement of early-stage AI products to understand their design, delivery, and real-world usage, providing initial evidence on their effects before large-scale trials. Featuring: Dan Fitzpatrick, Stacey Alicea, Meghan McCormick, Institute of Education Sciences, Leanlab Education, Boston University's EVAL initiative, Teaching Lab, Research Partnership for Professional Learning, Shared Measures Toolkit. Follow AI in Education with Dan Fitzpatrick for more on AI in education.