Is game-based learning assessment a Big Data problem? Lessons from piles of horse manure.

loading horse manure

Several years ago I was invited to participate in a panel on Big Data and Machine Learning in on-line learning games at the annual Neural Information Processing Systems meeting. NIPS has become THE annual meeting place for the leaders of the Machine Learning field as well as the leaders of Google, Facebook and others looking to hire the leaders in the field. Introduced at the workshop as someone who has been involved in on-line learning for many years, a participant in the front row remarked that it was interesting that one of the founders of the NIPS meeting, way back in 1986, had my same name. I responded that that was more interesting than he knew. J

It turns out that about the same time I was helping to found NIPS, our idea that computerized games could be used to assess science learning started the research and development effort that resulted, 15 years later, in the founding of Now an additional 17 years later, it turns out we have been thinking about and working on game-based learning and assessment for a very long time.

Perhaps for this reason, this year at the annual South by Southwest EDU festival in Austin Texas, I was invited, this time by the U.S. Department of Education, to participate in a workshop on the potential use of on-line games for assessing student learning. Several in the room introduced themselves as being interested in applying NIPS-like Machine Learning approaches to the enormous amounts of data potentially generated by learning games. However, when it came my turn to introduce myself, I decided to self identify instead with my latest commercial enterprise: selling high quality horse manure compost in Southern Oregon.

Judging from the reaction, several participants appeared to regard this as some kind of confession: and then seemed surprised when I went on to suggest that the process of making high quality organic horse manure compost was potentially relevant to the question at hand. How is that?


With the growth in backyard gardening and reduce, reuse and recycle (in Texas, blast it, burn it, and bury it J), more and more are familiar with the seemingly miraculous biological process by which table scraps turn into organic soil.

My own experience with this biological process is even more historical than my involvement in game-based learning, going back 60+ years to my early life on the family dairy and horse farm in upstate New York. One can argue about how much I know about game-based learning, but I do know how to compost horse manure.

So how does that process work? First, it is important to start with high quality horse manure from horses fed good food, free of as many pesticides and other drugs as possible. Many pesticides in particular go straight through your horse, into the compost, to then suppress plant growth where it is eventually applied. Likely, the level of exercise and fitness of the horses matters as well, although I dont know anyone who has studied that. In other words, the quality of the horse manure you start with is critical.

For any biological process, the quality of the eventual product produced depends strongly on establishing the best possible environment for further development. In the case of horse manure, ideal conditions for the microbes that actually do the work is a pile approximately 5 feet high and 5 feet across, and as long as you want.

In any biological process, then, the trickiest part is process control, i.e. knowing how to monitor the process and when and how to intervene to keep it going to its full and best completion. In the horse manure business, progress is monitored using temperature. At start, if you have the right manure and the right conditions, the pile reaches 100 degrees (F). When the temperature drops down to about 80 degrees, it is time to turn the pile to push optimal composting. At that point the temperature can go up to as high as 160 degrees, where decomposition is at maximum and weed seeds, fly larvae and other organisms potentially carrying diseases are also killed. As the temperature starts to drop again you turn the pile one more time and wait for it to return to ambient temperature. The whole process takes about 3 months.


Learning as a biological composting process

So what possible relevance could any of this have to game-based learning and its assessment? First, abundant research suggests that the physical and emotional state of children is critical to success in school. Recent evidence suggests that the right conditions for learning actually start in utero; a big problem in a country, like the US, where there are huge and growing disparities in the quality of life of young children.

Second, optimal learning depends on the quality and sophistication of the overall learning environment. was built on the assumption that optimal on-line game-based learning requires a sophisticated overall environment, not just a series of one-off games. As in an effective classroom, many different forms are available for learning, not just one. The sophistication with which those different forms interact to promote learning is key to success.

Third, success takes time. Current classroom structure puts time to do anything sustained at a premium, and as a result many learning games have been designed accordingly. However, children, like the microbes in horse compost, need sustained involvement for success. Whyville is being used increasingly as an outside classroom supplement for sustained involvement in game-based learning.

Fourth and perhaps most importantly, the sophistication of the methods used to monitor progress is critical. In the world of formal learning, the process of monitoring progress is referred to as formative assessment, and, as in horse manure composting, it is essential for success. Applying current trends in Big Data to horse composting, one could see yet another computational opportunity: instrument the trillions of microbes, collect all that data, push it into the cloud and use machine learning to figure out the best metric for assessing progress. Similarly, many of the discussions I have heard about big data in on-line learning imagine collecting every possible bit of data, pushing it to the cloud, and using some analysis method (Machine Learning for example) to figure out what is there. In the worst examples, all that big data is simply pushed at teachers on the assumption they will be able to figure it out.

Fortunately, when humans began composting (likely a very long time ago), we didnt have sophisticated measuring devices, computers, the cloud, etc. Instead, temperature became the global measure for composting, telling you when conditions were optimal to engage the microbes to do their thing and when conditions need to be changed for re-engagement.

It is my assertion that the best and most successful teachers effectively use engagement measures (i.e. temperature), to measure progress in their classrooms. However, for this global measure to work, it is necessary for the learner to be in a sophisticated well-designed learning environment. If all you want is heat, you can construct a compost pile that goes over 160 degrees; however, at those temperatures the beneficial organisms for decomposition are immobilized. Similarly, if all you want is maximum addictive human engagement in games, you can certainly design the games accordingly, but it is not likely that your users will learn much that is useful. On the other hand, as in manure composting, a wonderful thing about biological organisms is if they are properly prepared and the environment is properly constructed, monitored and adjusted, and they are given time, they will naturally do what they evolved to do. For humans, that is learning. Skilled teachers who use engagement measures to monitor their classrooms know that if you engage children, they will learn. If you engage them in a rich and well-designed learning environment, temperature is an ideal global measure of their progress. For this reason, we have spent considerable effort in Whyville developing measures of engagement.

Finally, there is one more assessment lesson from composting that I would like to suggest could and should be applied more vigorously to learning games. In addition to Formative assessment, we also care about Summative assessment, both to know if the manure composted and to know what our children have learned. In the case of horse manure composting, there is one age old measure of the quality of the final product you pick it up in your hands and smell it if it still smells like horse shit, the composting didnt work. It might be useful to apply a similar smell test to learning games.