Privately funded, Randomized Control Trials for policy

Results from the first four RCT's funded by Arnold Ventures.

Here are the results for charter schools:

The study found that students who won a KIPP middle school admissions lottery were 6 percentage points more likely to enroll in a four-year college than students who lost the lottery (47% of lottery winners enrolled vs. 41% of lottery losers). We view this finding as highly suggestive but not yet strong evidence of an effect because it did not quite reach statistical significance (p=0.085). The study also found a 4 percentage point increase in the rate of persistence through the first two years of a four-year college (30% vs. 26%), but this finding was not statistically significant and so is preliminary and not reliable (p=0.23). These effects of winning a KIPP lottery (i.e., the “intention-to-treat” effects) are the primary study findings based on the researchers’ pre-registered analysis plan.

However, only 68% of students who won a KIPP lottery actually enrolled in a KIPP school. In an exploratory analysis, the study found that the effect on these 68% (i.e., the “treatment-on-treated” effect) was a 9 percentage point increase in enrollment in a four-year college and a 6 percentage point increase in persistence. The enrollment effect approached statistical significance (p=0.085); the persistence effect did not. [2]

I am left wondering whether this effect is biased due to the presence of competition, e.g., there is some evidence that public schools get better when a competing charter schools opens up.  If so, control group students who went to a public school that also gets better, would bias the estimated effect towards zero.   

HT:  David S.