The new wave of AI-powered EdTech products has a problem: Too many companies are asking, “How can I get my product in front of teachers?” They should be asking instead, “Given that teachers are excited about AI, why don’t they want to use my product?”
The usability of AI products is just as important as their effectiveness in improving student learning. If a tool improves student outcomes but teachers are uninterested in learning to use it, it will fall victim to what Laurence Holt calls the 5% problem. By the same token, if a tool is usable but does not generate an impact, you might acquire a lot of users, but kids will fail to learn how to read, do math on grade level, think like scientists, and so forth.
Last year we launched the Teaching Lab Studio with the goal of building tools that do both. The Studio seeks to invent the learning environments of the future one step at time, first by creating AI-enabled tools that actually have an impact and that teachers and students love to use.
See a representation of this below:
IMAGE: Usability and Effectiveness Framework
Here are three lessons we’ve learned over the past year:
- Co-Create and Engage in User-Centered Design with Educators
AI tools that teachers love to use must be built with and not for teachers. Our team built teachinglab.ai as an R&D space for teachers, coaches, and curriculum experts to create education-focused GPTs and test them in context.
Over the last year, we convened educators, created at least 50 different prototypes and iterations of tools, collected feedback, and invented more usable products based on what we learned. For example, we support teachers to use the open educational resource Illustrative Math, which is a beautifully-designed and evidence-based math curriculum. Yet teachers still encounter pain points when using it: students need additional rigorous practice problems, which are hard to find or develop; teachers spend extra time creating new tasks to develop student prerequisite skills or knowledge; teachers continuously struggle to meet the needs of multilingual learners, etc. By listening and learning from educators, we have now developed more sophisticated web extension products that allow adaptations of open educational math curricula to the unique needs of their students, embedded within interfaces like Google Docs that teachers frequently use.
Because we should never stop evaluating tools in context, we’re currently testing and iterating on these products in our math coaching partnerships across the country, as well as continuing to lead workshops with teachers to develop new tools and improve existing ones. See here for our framework on how to measure the impact of AI-based tools from invention to use in the classroom.
- Build on and for Open Educational Resources (OER)
Generative AI has enormous potential for fulfilling the idealistic goals of expanding broad access to open educational resources.
One tool we’ve built to help realize that potential is The Writing Pathway. TWP leverages AI to build writing skills through sequencing, scaffolding, and content generation for all content areas in 3rd-10th grade. TWP was co-designed and refined by researchers and educators, with significant user testing from Teaching Lab’s 300-member Educator Partner Network. A recent study of 21 middle school classrooms during the 2023-2024 school year, conducted by Dr. Steve Graham at ASU, found that students with teachers using TWP improved their writing nearly 9 times more than those whose teachers did not, with an effect size of .64.
Not only is TWP an open resource itself, but it also strengthens the implementation of open educational curricula like EL Education or OpenSciEd. OER originally promised that teachers could adapt curricula to meet the needs of all students; at times, however, OER curriculum has been customized to reduce rigor. The original promise can be realized if learning science-backed guardrails are put in place. When an evidence-based frame is overlaid onto a customization tool like The Writing Pathway, teachers gain agency and students gain personalization.
- Embrace Sector-wide R&D Collaboration
We are not only living through a generational shift in technological possibility, but a never-before-seen acceleration of the rate of change–-building tools atop underlying technological infrastructure that changes from month to month. We’ve had two realizations about the accelerated EdTech development cycle: 1) we need shared, sector-wide benchmarks and model evaluation mechanisms to ensure AI tools consistently produce high quality outputs; and 2) we at the Teaching Lab Studio cannot produce these standards and benchmarks alone.
For example, Teaching Lab is engaged in two small benchmarking projects with AI and machine-learning researchers: one on assessing math student work and another on assessing the quality of lesson and unit plans. But the field needs to engage in perhaps thousands of these benchmarking projects, followed by open-sourcing the findings to democratize assessment of whether AI tools are creating high-quality resources—ones every caregiver and educator would feel good about putting in front of kids.
Additionally, we need better systems for continuous model evaluation. For instance, one of our AI in Education Fellows, Agasthya Shenoy, partnered with OpenAI to establish a series of evaluations for different LLMs (including GPT-4o mini) aligned with numerous education-focused use cases. The idea is that if these evaluations are run continuously, the LLM used in a product can be optimized—fine-tuning for a specific use case, or building it into an agent network for a specific task.
Co-creating with educators and students, committing to open educational tools and content, and a sector-wide focus on R&D can be time-intensive and expensive. But the alternative is that AI hype will churn out tools that are unviable and undesirable for use in the field. AI has transformational potential for students and teachers, but it won’t be realized unless it is informed by the ongoing expertise of educators and a prolonged investment in collective and open R&D.