Scaling the MOOC

I read with interest Audrey Watters’ commentary on Scaling College Composition. Some of the work I did in this area (I call it Connectivist Metrics) and the recent discussions I had with Stephen Downes in New Delhi during the EDGEX conference around intelligent environments for assessment, as well as all the great work that is happening in Learning Analytics by George Siemens and others, leads me to a few key thoughts and ideas.

It seems like the right time to take a critical look at the notion of assessment. The context of the traditional education system, and of most new age systems that leverage the online medium, suggests a dominant way of thinking about assessments.

Assessments are performed by somebody (the instructor, board, the learner or system) on someone (the learner). The purpose of the assessment depends upon the intended use of the assessment (the why) while the subject of the assessment (the what) defines the boundaries of what may be assessed. The where, when and the how questions demand answers for the modality of the assessment and the which question demands answers on aspects such as the level or complexity of the assessment.

The order that permeates the thinking on assessment precludes emergence and chaos. What would emergent assessments look like? They would be assessments that are not pre-designed, but may result in the some of the same competencies being demonstrated as in the traditional “designed” assessments or in outcomes that provide alternate manifestations of competencies. They would be governed more by the same principles that underly complex systems design.

My favorite example from school is of a fellow student who had enough time in his exams to provide three different ways of solving the same math problem, one of which was really the “expected” method. For those of us who have had fun in marking automagically some of the open ended assessments types (like essays and multi-step tasks based items), this chaos is challenging – and this is at a micro scale – at the scale of the individual learner.

The corresponding thought around content runs deeper into curricula and how they are planned. In my estimates, school students spend less than an hour each year on a single topic of instruction on average (or something close). There is simply no way in which there can be any learning chaos at a systemic level within the traditional system.

So systems that want to assess at scale range from the adaptive testing systems at the single learner level, to systems that utilize the power of the network (peer reviews, ratings), automated graders and of learning analytics (dashboards, mining).

But I am not sure the scaling of assessments reduces to development of systems for authoring items & exams, compiling and evaluating scores. Somehow, we must put the focus on systems, particularly in the MOOC, that recognize evidence of competency. To do this, we must allow an educator to define what is meant by that competency in a manner that is open and expressive.

Can we look at defining a language of assessments like that which goes beyond the traditional elements of measurement (the multiple choice, the essay) and allows educators to pick on a constellation of recognizable evidences sequenced and stitched together in a particular way? Systems could then be based on more objectively mark-able and error-free mechanisms.

Such a language would have interesting implications. Just like we would build software to do tasks, we could engage with a community of developers to solve smaller problems – like figuring out if the student interacted with the community or if she used a specific technique to solve a problem. Each smaller problem would then be associated with competencies and evaluation would be a mix of possibilities (yes/no, subrange, enumeration types).

Over time, and with an engaged community, there could be thousands of competencies that could be assessed in this manner and thousands of patterns of assessments that could be created and shared. These patterns could include an ever-expanding list of criteria/behaviors. Perhaps these assessment patterns could themselves be aggregated meaningfully to derive more complex patterns and intelligence.

This would also solve a critical need for the assessment types and tools to evolve. In effect, this could pave the way for unifying learning and assessment. It would allow us to scale downwards to the individual learner and upwards to a MOOC environment. It would focus attention on what constitutes competence or proficiency by analysis of patterns that educators use for assessments (and in that sense, open up hitherto esoteric assessment mechanisms). Perhaps it could also work well with learners who want to express competence in a manner that others understand.

It would then be the task of systems to understand and react to such assessment patterns. That itself, would be the basis for understanding how MOOCs could be responsive to learning needs.

When such systems, based on open thinking, languages and architecture, permeate education, will there be transformation. Perhaps until then, we would mutter under our breath, like George Siemens did:

The concepts that I use to orient myself and validate my actions were non-existent on summit panels: research, learner-focus, teacher skills, social pedagogy, learner-autonomy, creativity, integration of social and technical system, and complexity and network theory. Summit attendees are building something that will impact education. I’m worried that this something may be damaging to learners and society while rewarding for investors and entrepreneurs.