Planning for evaluation

At our fourth bootcamp workshop we had a masterclass from the OU team on how they designed in evaluation as part of the module planning. Student feedback, a student reference group, analytic dashboards (and custom reports), and tutor reports come together to build a picture of how the module performs and informs decisions about the following presentation.

During our Jamboard exercise we thought through some of the ways we could plan evaluation into our SML module. Here we aren’t dealing with students at a distance, instead there will be lots of opportunities to actually see and hear how they are getting on:

While we don’t have a reference group we do have lots of opportunities to gather feedback:

  • reviews of content pre-run
  • regular informal in-class check-ins
  • end of module survey / focus group
  • student reflections on the process and what they are learning
  • views from student reps via Student Voice / Student Staff Committees

Thinking about this further, its obvious that one size doesn’t fit all. Yes, there is a need for standard questions in centrally run surveys, but you need to know what the module is about to design how to evaluate it effectively. And, in many respects where our focus is on skills development -the “mentor” role of the academic leads will give an immediacy to feedback and permit in-flight corrections.

Parameters for a NCL Bootcamp

We have also begun the process of thinking about how we gather up our learning to present an in-house variant of the Bootcamp. Some things are clear:

  • it needs to reflect our focus as a predominantly campus based university (blended is normal, online is rare)
  • tools and techniques need to complement and extend our existing module approval processes
  • any learning design frameworks or approaches need to be easy to pick up and easy to pass on (i.e. Made to Stick)
  • centred on a mentored or action learning approach – a supported journey
  • it needs a pacemaker – a structure and metronome to enable teams to complete the design (too fast and we will drop people, too slow and people will disengage)
  • it needs to be offered as a pilot, and developed with feedback.
  • it need to be flexible enough to support our gloriously diverse range of disciplinary cultures

We have lots of options. In some ways this is freeing – we can take the best of our current practice, add in elements that are helpful, and document other approaches to come at the design task from a different angle.

However, if we are to develop something that could have traction we need input from stakeholders, and that is what is next on the agenda.

Storyboards, episodes and patterns

One of the really interesting ideas from last week’s “Developing your Design” bootcamp was that of considering an “episode level design” between the module level design and the the detail of the activities. The episode could represent a week (or maybe a fortnight) and it could have one or more design patterns giving a rhythm and predictability to teaching.

Thinking about some of our examples, a pattern for an episode (a week) could be something like:

  • Big question
  • Unpacking theory and practice
  • Group activity
  • Q&A
  • Discussion and reflection

Once learning outcomes are authored for each episode, then then, tools like ABC activity cards, CoDesign cards, or OU’s Activity cards can help to structure each each element of the pattern into a set of tasks and content (e-tivities) geared towards meeting the learning outcomes.

But, before learning outcomes can be in the drivers seat, we were reminded of the importance of carefully crafting them – with active verbs, defining a level – so we can evaluate whether the proposed activities will work i.e. enable students to meet these learning goals.

I’ve really enjoyed working with ABC as storyboarding tool, but am aware that people can get confused on the level they are working. Some are happy to abstract them “this is the pattern for weeks 2-6” but in other situations I’ve found participants bogged down in the detail of what week 3 will contain and, in the time constrained workshop, be unable to see the sweep of the module. As a facilitator, you do encourage participants to work at the overview level, but placing an emphasis on a pattern for the week as a prelude, or additional step, may well help.

Lots to think about…