FMS Feedback – From Excel sheets to detailed online feedback

The MBBS programme collects a lot of assessment information that needs to be displayed in a way that is useful for the students, so that they can improve their grades through focusing on the right areas. They called upon the web skills of the FMS TEL team to design and implement a web application that could take assessment results, process them, and show them to the students in a visual and interactive way.

Using templates

The first step towards this was agreeing a template that the results could be stored in. The types of assessments the MBBS programme uses are varied and quite detailed in their scope. They required a means to take all of this variation and detail and create something useable. The assessment team started by using complex Excel documents to collect and store all the assessments results. Each assessment type (WriSkE, MOSLER, SBA, OSCE…) needed a unique Excel template to store all the student scores and a way to map the assessment structure to the curriculum outcomes.

spreadsheet template — Example WriSkE template

Processing the Spreadsheets

Once we had agreed on the template structure, we could then focus on how we would process them for use in a web application. We planned for the heavy concurrent use the web application would undergo when all the students logged in and tried to access their grades at the same time. The best solution for this was to minimise the use of complex database structures and instead store the results in pre-processed files, one for each student per assessment. The format we chose for this was JSON and this allowed us to rely on the speed of the server to provide the data.

Custom processing scripts were written for each assessment type to create these files. This would mean an administrator from the assessment team could log in to the site, go to the admin tools, choose an assessment type, set up a few settings including a release date, attach the assessment to an uploaded taxonomy (this attached the exam structure to learning outcomes), attach Excel file and process. The site would then go through the spreadsheet and create each assessment file ready for the students.

Display the Results

The final step was to decide how to display this detailed assessment information to the student. We chose to use an online chart library called highcharts. This allowed us to utilise a whole suite of charts and graphs to display the raw results in an interactive way.

One of the core charts we used was quartile (boxplot) graphs, which allowed us to plot the students scores against the cohorts. This means that students can see how they are performing in the context of their cohorts, which many of them appreciate. We also heavily use bar charts you can drill down into, and spiderweb charts that could show the same information in a visually different way. Letting the students modify and change the display to their preferences was also key.

Finally, we added tabs to show the exam structure and an area to which support documents could be added, such as assessor comments.

Each time a student clicked on an assessment, the site would load their file and convert into a visually rich interface of charts, graphs and links to drill down into the data.

System Success and Expansion

The FMS Feedback system has been successfully used by the MBBS programme since 2014. It is held in high regard by the administrative team who appreciate its flexibility and ease of use. For students, the detailed data displayed allows them to have a very granular understanding about their levels of mastery of individual skills and topic areas, and to spot patterns in their performance. This means they can focus their studies on areas identified for improvement, with a view to increasing their overall competency.

It has since been expanded to cover a few assessments from dental programmes that had a similar structure and process to the original MBBS assessments, such as SBA and OSPE’s.

If you would like to learn more about the FMS Feedback system, please contact the FMS TEL Team.

Instant Feedback – Lindsey Ferrie

This post is about the use of Menti – a pretty polling tool that can show responses in real time on screen. Lindsey Ferrie used this to gather student feedback during an in-person event. Link to guides included.

In Biomedical, Nutritional and Sport Sciences this year there is a focus on improving assessment and feedback. Including student voice in this development work is key. As well as the normal routes for student voice, such as staff-student committees and module evaluation forms, there were certain questions that staff had in mind which could reveal a lot about how students were feeling about this aspect of their course.

The formal module evaluation forms don’t always capture a large sample from the student body, and as with any survey, generating a high response rate is very difficult, and can sometimes end up only revealing the most polarised responses. Informal live response tools were a potential way to gather large amounts of more representative data, as well as demonstrating that feedback is wanted, heard, and acted upon within a shorter cycle. It is also much more convenient to capture a response immediately when students are in the room.

Having seen Menti in use during the FMS TEL Conference, the decision was taken to try it out in a much bigger setting – the inductions for all programmes in the School. Overall, this includes around 1500 students.

A useful aspect of Menti in this situation was the capacity to provide a wide range of question types, as well as the ability to see answers as they are coming in. Naturally, this has some disadvantages too – students’ responses may be coloured by what they see on screen if this is displayed. Possibly due to its anonymous nature, students provided some genuinely challenging feedback which proved very useful.

When in use, Menti allowed for the sessions to be more interactive as sharing live responses is quite immersive. It also allowed for the presenters to directly react and respond to things as they came in and broke the ice between students and staff. This can be useful for quick questions about content as well as student voice activities, for example polling the level of understanding of a concept at the start and end of a teaching session.

In terms of providing evidence to share with other colleagues, or at committees, Menti retains the responses for you to review, and can generate lists of entered comments. It isn’t designed as a statistical analysis tool, so comments don’t come in a convenient format for analysis. Graphs can be screenshotted for simple sharing if needed. It is of course possible to read through the comments and type up an overview of common themes yourself if needed.

The instant feedback received highlighted that some students weren’t entirely satisfied with the feedback they were receiving on their lab reports. Based on the responses, the School was able to put on a dedicated feedback session to further explore this, and to find out what could be improved, and have already been able to implement some changes.

Tips

Consider adding a delay before sharing answers on screen
For free text answers, it may be prudent to summarise rather than share the screen
Consider numbers when selecting your question types – it’s not easy to read 30+ typed responses quickly.
Don’t overuse it – keep use to key points so the novelty doesn’t wear off
Just try it – and be prepared for the honesty in the responses!
Menti works best in the moment in a session, or when you want to gather responses instantly

Can I try it?

You can create a free account on Mentimeter.com. This free account has limits on how many individual questions you can make, but you can always make multiple presentations and switch to them when sharing your screen. The university does not have any subscription to this service. As such, do not ask students to input any personal or sensitive information as this won’t be covered by our data policies.

When presenting, simply ask the audience to visit the site and enter the provided code that you show on the screen. When you’re ready, you can display the results to your audience. See the FMS TEL Community for a full walkthrough.

Other Ideas for Use

Gauging the experience already in the room before starting a session (free text or poll)
A quick litmus test on an opinion – yes/no/not sure (could be done at the beginning and end to see how things change)
Challenging students to define a term
Choices and branching scenarios
Feeding back results on a solo exercise
Anonymous feedback / rating for an aspect of the session

Contacts

Lindsey Ferrie, Senior Lecturer, School of Biomedical, Nutritional and Sport Sciences

Resources

Mentimeter presenter site
Example of a text answer with summary. Newcastle staff only – via Recap, start at 7:40.
Example of a multiple choice Menti with shared result screen. Newcastle staff only – via Recap, start at 28:20.
Menti guide on the FMS TEL Community on Canvas or the MLE.

Conversations for Learning – Kay McAlinden

This case study concerns the current module Psychosocial Issues in Cancer and Palliative care (ONC8006) as it is taught in 2020-21 academic year. This module is part of the Cancer and Palliative care programme and as such deals with emotionally challenging topics, including breaking bad news, loss, responding with empathy, screening for distress, anxiety and depression, families and carers, body image issues, survivorship, and interventions such as Cognitive Behavioural Therapy and Mindfulness. It is currently a small module with less than 20 students.

Next year it will be merging with another module, Handling Loss Grief and Bereavement (ONC 8010) and will become Psychosocial issues in Advanced Disease (ONC8030). Several elements of professional development, including a session run by Manchester University, and the FMS TEL Humanising the Online Experience webinar, helped to inspire and implement a few new ways of working which will be carried forward.

Choosing Priorities

Clear themes emerged from the sessions attended, showing what students want from their online courses. Prompt feedback from tutors is clearly valued, as well as ease of access and clear information about assignments. Students want to feel the tutor’s presence in their course and want to be able to give feedback afterwards.

Given that the online environment can be stark and unemotional, and people do not always want to discuss these topics – even in classrooms, teaching this content online was always going to be a challenge. It was important to try and model the communication skills, behaviours and support that students would be learning about, allowing this to stand as an example of good practice.

Zoom Tutorials

The stress and emotional load on students this year has been particularly high both personally and professionally given the impact of COVID on service delivery and home life. Students often mentioned the difficulties of delivering bad news by phone and trying to support patients and families when there was minimal access to psychosocial services. This prompted the decision to offer Zoom appointments or phone calls to anyone who wanted them. These were offered rather than being compulsory, and not all students took up the offer. These were offered via Canvas announcements.

This year most chats only lasted 10-15 minutes but allowed students to consider how they would approach their essay. Spending this one-to-one time with students, even on the phone or via Zoom, has meant that getting to know them has been a lot easier, and students feel more supported. In some cases, a chat over email was enough, but sometimes it was much more time-effective just to call the student and have a quick chat, rather than spending time drafting and re-drafting a long or complex email.

Many of these chats took the form of one-to-one tutorials on assignments, explaining what was expected and how to do a good job of the assignment, as well as offering reassurance. Careful monitoring of the discussion boards has also allowed issues to be picked up and addressed.

Use of Discussion Boards

This module has always made good use of discussion boards. Participation in the discussion boards has been good, and regular and swift responses to students’ posts have helped further this. Questions such as ‘If you were diagnosed with cancer, who would be around to support you? What if they weren’t there?’ push students to think very deeply about their own personal situations, and to connect compassionately with what they are learning.

Some tasks had options which are less emotive, but most students chose to answer the most emotive task – naturally tutor feedback for this was very rich. This meant it was time-consuming, but also extremely valuable. Adding a simple ‘like’ or a quick remark would have been far too dispassionate for a module like this! Other students were also able to respond in a way that supported each other, thanking one another for their stories.

Further discussion board use included the choice for students to respond in their own private journal area rather than more publicly. This setup can help to engage students when topics are personal or reflective in nature.

Advice and Support from FMS TEL

The advice and support about what is possible, and how to implement things technically has been invaluable. This has allowed for modernisation, development and innovations in the methods used, as well as some experimentation within the module. When trying new things, make sure you set yourself up to succeed by starting small, and starting with what you’re comfortable with, adding more new elements as your confidence grows.

Feedback and Next Steps

This year the use of regular Canvas announcements has been a good way of keeping things on track. The participation in discussion boards and tutorials has been great, and students have said they feel supported. As the module will look very different next year, more adaptations will need to be made. For example, in future it might be useful to offer assignment guidance as a small group tutorial, especially if more students are taking the module. This way it can be scaled up and students can still feel supported.

Now it feels easier to interact with students online than it did at the beginning. Getting to know them as people beyond their studies has been very rewarding for everyone. Small things like being confident enough to use humour, or to share a funny cartoon about the subject, have become easier to judge as time has gone on. This can be tricky to judge if you can’t see faces or are worried about cultural faux pas, but once again, getting to know your class by building community really helps with this.

Resources

Newcastle University Digital Learning – Technology Guides

Enrol in the FMS TEL Canvas Community for access to materials and Canvas notifications of new resources and blog posts.

Humanising the Online Experience Webinar Materials (watch, listen or read)

Discussion Boards Webinar Materials (watch, listen or read) – including examples from this module

Discussion Boards Guides – how to implement different types of boards for different purposes in Canvas and Padlet (for MLE users)