Today, most large internet companies run 10,000 or more experiments a year to improve things like user engagement, ad conversions, and the like.
In comparison, how many experiments are done each year in the US studying the effectiveness of the elementary math curriculum? Is this satisfactory, considering the US has some 90,000 primary schools, and 33 million elementary students?
Arguably, the present status quo in testing and innovation is simply immoral. After all, it's not as if the US faced no issues whatsoever in education (see chart)
Admittedly, the problems in elementary education extend beyond "What works". Still, so long as innovation remains a large part of the answer, the question arises: How do we bring internet-scale innovation to the offline world? One answer I've been focusing on is distributed learning or "edge experimentation".
This is best illustrated by example. Consider a Walmart store manager with no background in math or statistics. Using software like the one we developed at Cambridge Social Science Decision Lab he could design, implement, and analyze a simple randomized controlled trial without further supervision. Now, if each Walmart store manager does just 2 simple experiments a year, that is over 10,000 experiments per year across Walmart. This is what we mean by "edge experimentation".
One advantage of edge experimentation is scale. Achieving internet-scale experimentation in the offline world will require delegating down, to store managers, or school boards. I doubt a single centralized lab, nay a few dozen education researchers, could ever achieve such internet-scale experimentation on their own. A second advantage of edge experimentation is innovation. For example, automated multi-armed contextual bandits, of the sorts deployed at Google etc., can select the right treatment for the right person at scale. However, they cannot dream up new treatments. At least not yet.
This is important because in highly successful companies like McDonalds, such innovation often comes from those closest to the customer. In their case the franchisees. It was them who invented the Big Mac, Filet-o-Fish, the Breakfast Burrito, or Ronald MacDonald, among other.
Our edge experimentation framework empowers those innovators closest to customers to experiment rigorously, at scale. Admittedly, there are may problems yet to be solved, and skepticism to overcome in developing edge experimentation. However, I believe 10,000 wasted opportunities a year are 10,000 too many. Arguably, the relevant question is not "will this fly?" but rather "how do we make it fly?".
After all, where there is a will, there is a way.