There is mounting evidence that Speak Agent impacts both content and language learning. If you are looking for ESSA evidence criteria, please see here.
Study #1. Large-Scale Math and ELD Impact Study Findings
Summary
A groundbreaking third-party impact study, "Effect of Speak Agent on Math and English Language Proficiency Scores at Middle Schools in Prince George’s County Public Schools," found that Speak Agent's math literacy program, Math+Language, leads to significant improvements in Math scores for middle grade students. The research also found a significant, positive effect on multilingual learners' English Language Proficiency scores.
Find an executive summary here →
Research Design
The goal of the research was to evaluate the effect of Speak Agent's math literacy program on student outcomes in both Mathematics and English Language Proficiency across grades 6 to 8 within Prince George’s County Public Schools (PGCPS).
The quasi-experimental data analysis was conducted by Leanlab Education, a nonprofit research organization. The sample used in the analysis was provided by PGCPS from 86 middle schools, K-8 academies, and 6th-grade elementary schools. There are 1,597 class sections represented in the sample accounting for 24,180 unique students enrolled in grades 6 to 8 during the 2021-2022 school year. All classes and students had full-year access to Speak Agent. Students who took none of the three Math Benchmark tests nor the WIDA ACCESS test in SY 2021-2022 were excluded from the sample. Some students had no usage, whereas others had varying amounts of usage. The amount of usage was based on teacher and student choice in real-world classroom environments.
Key demographic data for grades 6 to 8 at PGCPS:-
Black: 55.4%
-
Latino: 37.3%
-
Multilingual: 16.0%
-
Title I: 38.3%
-
Free and Reduced Meals (FARMS): 67.6%
Measures
The data analysis measured the impact of Speak Agent usage on Math and English Language Proficiency outcomes using two assessments that are produced by independent third parties:
-
Math outcomes were measured by PGCPS' quarterly Math Benchmark assessments (N=46,929), developed by Pearson. Students took the Benchmark up to 3 times in SY 2021-2022. This explains why there were 24,180 students, but 46,929 observations.
-
English Language Proficiency was measured by ACCESS for ELLs (N=5,361), produced by WIDA. Multilingual students took this test once in SY 2021-2022.
How to Interpret the Results
- Cross-sectional analysis predicts the score of a randomly selected student at one point in time, controlling for all variables except for usage. It does not account for a student's baseline score.
- Longitudinal analysis predicts how much the score of a randomly selected student will change from one academic quarter to the next. It does account for a student's baseline score and tells you how it increases or decreases. In this case, it tells you how a score would change if a student changed their usage of Speak Agent.
Math Impact: Summary
Cross-sectional analysis of 46,929 Math Benchmarks test scores found that, on average, each additional Speak Agent activity module that a student completed corresponded with an increase of 1.0% on the Math Benchmark score. Completion of 10 activities corresponded with an average math score increase of 10.3%. This finding is significant at a 99.9% confidence level or p<0.001 (Huebert, 2022).
Time-series analysis of the same 46,929 Math Benchmarks test scores found that, on average, an increase in the number of Speak Agent activities completed by a student will increase the test score by 0.6% per activity. Completion of 10 additional activities are predicted to lead to a math score increase of 6% on average. The reverse is also true: A student who completed fewer Speak Agent activities than in the previous quarter will, on average, see a decline in scores at the same rate. This finding is also significant at a 99.9% confidence level (Huebert, 2022).
Is there a point of diminishing returns? Yes, there will be diminishing returns at some level of Speak Agent usage. The current data set, however, shows no evidence of a statistically significant, nonlinear relationship. This is likely due to the fact that very few students exceeded the recommended dosage and "maxed out" the program's impact.
English Language Proficiency Impact: Summary
Cross-sectional analysis of 5,361 WIDA ACCESS for ELLs test scores found that, on average, each additional Speak Agent activity that a multilingual student completed corresponded with an increase of 1.7 points on the Overall Scale Score. Completion of 10 activities corresponded with a score increase of 17.1. This finding is significant at a 99.9% confidence level (Huebert, 2022).
The same statistically significant positive effect of Speak Agent applies to all WIDA ACCESS domain scale scores as well, at a 99.9% confidence level (Huebert, 2022).
| WIDA Domain | Increase Per Activity Completed |
| Speaking | 2.2 |
| Listening | 2.0 |
| Writing | 1.8 |
| Reading | 1.3 |
| Comprehension | 1.5 |
| Literacy | 1.6 |
| Oral | 2.1 |
Study #2. Science Impact Study Findings
A quasi-experimental design (QED), two-phase impact study (Li, 2018), sponsored by the National Science Foundation (award #1632488), found significant gains in elementary Science concepts and vocabulary using Speak Agent Science+LanguageSM. Five second-grade classes totaling 74 students, 59 (80%) of whom were ELLs, at two schools at a central Texas public school district were compared. The district administered randomized versions of pre/post-tests designed by independent evaluator Rockman et al in September and December 2017. A second set of pre/post-tests were administered in January and April 2018. Each test consisted of 16 to 20 varied question types covering 40+ academic concepts from second-grade Science units such as matter, energy, force and motion, the natural world, the water cycle, weather, and living organisms.
Second-grade students in the Fall 2017 experimental group used Speak Agent for an average of less than six hours over a 12-week period (< 30 minutes per week). The control group received regular Science instruction over the same 12 weeks. The experimental group acquired 3.1 times more Science concepts than the control group did. In Spring 2018, both the experimental and control groups were allowed to use Speak Agent. On average, these students acquired 4.6 times more Science concepts over 12 weeks in Spring 2018, as compared to the Fall 2017 control group. Again, students had access to Speak Agent for an average of less than 30 minutes of usage per week over a 12-week period. Both effects had significance at p < .01 (Li, 2018). The graph shows the average gains in Science concept acquisition for each second-grade group. Speak Agent content addressed 102 second-grade concepts in seven units of study.
Testing with third graders had similar findings but were not statistically significant, likely due to constraints with population size, fidelity, and dosage. Read the full study report by Rockman et al >
Teacher Observations
Independent evaluator Rockman et al conducted eight teacher interviews as part of the NSF study. Teachers reported that Speak Agent helped students learn complex and challenging Science concepts. Engagement with Speak Agent increased connections to the content in significant and impactful ways, as Li (ibid.) reports: "[Teachers] observed students engaging in more Science content, asking questions about challenging topics, and wanting to look up additional information on their own time. Students would create their own experiments (e.g., model volcanos) based on content introduced in Speak Agent, or they would go to the library to find additional information about a concept discussed through the activities. Students also seemed to remember information from Speak Agent and apply it to class lessons and discussions." Teachers "liked that the Speak Agent tool focused on individual students’ needs and helped teachers assess where students struggled and thrived." They lacked time, however, for Science teaching and also lacked equipment. The classrooms had student-to-device ratios of 1:2 to 1:5. Some rooms lacked working projectors. Students had access to Speak Agent for less than 30 minutes per week and teachers were allowed only one hour of training and professional development. Significant results were achieved in spite of these barriers.
The above teacher comments are verbatim, as recorded by interviews in the NSF study.
Follow-On Science Research
Results from Excel Beyond the Bell, the afterschool program for Montgomery County, Maryland, showed similar findings despite using a wholly different second-grade Science curriculum. The change score was also 15% growth with the same student population, dosage, and 12-week timeframe.
Reading Research
The Montgomery County, Maryland afterschool program, Excel Beyond the Bell, also found a highly promising 31% growth in reading vocabulary over the same period.
This unexpected finding led us to pursue a small-scale study in middle-grade reading intervention in partnership with KIPP Kansas City and LEANLAB Education. The KIPP study found a gain of 25% in reading assessment scores among 7th graders over just a nine-week period. Find the small-scale reading study summary here >
Please see our References page for a full A to Z listing of all research cited on speakagent.com.
