Research Design
A National Science Foundation-funded study found significant gains in science concept knowledge using Speak Agent's game-based learning program after just a few hours of classroom use. Four second-grade classes and two third-grade classes at two schools in Manor Independent School District participated. More than 90% of the students were English language learners (ELLs).
Manor ISD teachers administered a paper & pencil pre-test and post-test designed by independent evaluator Rockman et al in September and December 2017, respectively. The tests consisted of 20 varied questions covering 34 science concepts. All definitions and images were carefully selected to avoid any overlap with those used in the Speak Agent product. Rockman et al graded all of the tests.
Students in the experimental group used Speak Agent for an average of approximately 6 hours over a 12-week period using tablets, laptops, and desktops (varying by classroom and availability). Students in the control group spent the same amount of time receiving standard science instruction without Speak Agent.
Preliminary Findings - 2nd Grade Science
There were 59 second-graders, with 31 students in the experimental group and 28 in the control group. The Speak Agent content covered three second-grade science units about Investigating Matter, Investigating Force and Motion, and Earth Materials and Natural Resources. These units aligned to the scope, sequence, and pacing of Manor ISD's science curriculum. All units were aligned to Texas Essential Knowledge and Skills (TEKS), the state standards. The second-graders were able to access Speak Agent in their home language (Spanish).
Second-grade students in the experimental group improved their test performance by an average of 10.3% vs. 2.8% improvement for the control. The chart below shows the average pre- and post-test performance of second-graders on the assessment:
Preliminary Findings - 3rd Grade Science
There were 26 third-graders, with 12 students in the experimental group and 14 in the control. (There were originally more, but some students dropped due to lack of access to devices for them to use.) The Speak Agent content covered four third-grade science units: Investigating Properties of Matter, Investigating Energy, Investigating Force & Motion, and Investigating the Natural World. As with second-grade, these units aligned to Manor ISD's scope, sequence, and pacing, as well as the TEKS standards. Unlike the second-graders, the third-graders only had access to Speak Agent content in English.
Third-grade students in the experimental group improved their test performance by an average of 9.9% vs. 1.9% improvement for the control. The chart below shows the average pre- and post-test performance of third-graders on the assessment:
Next Steps
A second phase of the NSF study will continue at Manor ISD through end of April 2018. In addition, Speak Agent is in the middle of two other studies with partners in California and Maryland. These studies will cover reading and math content. The studies will evaluate whether the preliminary findings at Manor ISD may be reproduced at other sites and in other content areas.
Extra Goodies for Statistics Wonks
Second Grade: A two-way ANOVA was used to examine the effects of condition (experimental & control) and assessment period (Pre & Post) on second-grade test performance. All effects were reported as significant at p < .05. There was a significant main effect of condition, F(1, 57) = 5.32, with students in the experimental condition (M = 68.0%, SE = 2.2%, 95% CI: 63.6% - 72.3%) scoring higher on the vocabulary assessments than students in the control condition (M = 60.7%, SE = 2.3%, 95% CI: 56.1% - 65.3%). There was a significant main effect for assessment period, F(1, 57) = 9.18, with students scoring higher on the Post assessment, (M = 67.0%, SE = 2.1%, 95% CI: 62.9% - 71.1%) compared to the Pre assessment (M = 61.6%, SE = 1.5%, 95% CI: 58.5% - 64.7%). Finally, there was also a significant interaction of Condition X Assessment Period, F(1, 57) = 7.27. This indicates that the performance between the Pre & Post assessments differed between conditions. There was a greater change in performance from Pre to Post assessment for the experimental group (10.3% change) than the control group (2.8% change).
A follow-up independent t-test compared differences in change scores (difference between Pre and Post assessment scores) between the two conditions, and a significant effect was found, t(57) = 2.70, p < .01. Students in the experimental condition, on average, demonstrated greater positive change scores (M = 10.2%, SE = 1.9%) than those from the control condition (M = 0.6%, SE = 3.1%).
Third Grade: The results of the two-way ANOVA for effects of condition on test performance between assessment periods revealed a significant main effect for assessment period, F(1, 25) = 8.57, p < .05, with students (combined across conditions) having higher scores on the Post assessment (M = 70.2%, SE = 2.2%, 95% CI: 65.7% - 74.6%) compared to the Pre assessment (M = 64.4%, SE = 2.0%, 95% CI: 60.3% - 68.6%) There was also a significant interaction for Condition X Assessment period, F(1,25) = 4.43, p < .05, suggesting that the difference in scores between Pre and Post assessment was significantly different between conditions. In fact, students in the experimental condition had a higher average gain (9.9%) on the Post assessment compared to students from the control condition (1.7%). There was no significant main effect for condition due to the insufficient sample size.
An independent t-test was used to compare differences in change scores between the two conditions, and a significant effect was observed, t(25) = 2.10, p < .05. The average change in responses for students in the experimental condition (9.9%) was greater than the average change in responses for students in the control condition (1.6%).
The study was funded by National Science Foundation award IIP-1632488.
Written by Ben Grimley
Ben is CEO and Co-Founder of Speak Agent, Inc.
