Dan – FMS TEL

FMS Feedback – From Excel sheets to detailed online feedback

The MBBS programme collects a lot of assessment information that needs to be displayed in a way that is useful for the students, so that they can improve their grades through focusing on the right areas. They called upon the web skills of the FMS TEL team to design and implement a web application that could take assessment results, process them, and show them to the students in a visual and interactive way.

Using templates

The first step towards this was agreeing a template that the results could be stored in. The types of assessments the MBBS programme uses are varied and quite detailed in their scope. They required a means to take all of this variation and detail and create something useable. The assessment team started by using complex Excel documents to collect and store all the assessments results. Each assessment type (WriSkE, MOSLER, SBA, OSCE…) needed a unique Excel template to store all the student scores and a way to map the assessment structure to the curriculum outcomes.

spreadsheet template — Example WriSkE template

Processing the Spreadsheets

Once we had agreed on the template structure, we could then focus on how we would process them for use in a web application. We planned for the heavy concurrent use the web application would undergo when all the students logged in and tried to access their grades at the same time. The best solution for this was to minimise the use of complex database structures and instead store the results in pre-processed files, one for each student per assessment. The format we chose for this was JSON and this allowed us to rely on the speed of the server to provide the data.

Custom processing scripts were written for each assessment type to create these files. This would mean an administrator from the assessment team could log in to the site, go to the admin tools, choose an assessment type, set up a few settings including a release date, attach the assessment to an uploaded taxonomy (this attached the exam structure to learning outcomes), attach Excel file and process. The site would then go through the spreadsheet and create each assessment file ready for the students.

Display the Results

The final step was to decide how to display this detailed assessment information to the student. We chose to use an online chart library called highcharts. This allowed us to utilise a whole suite of charts and graphs to display the raw results in an interactive way.

One of the core charts we used was quartile (boxplot) graphs, which allowed us to plot the students scores against the cohorts. This means that students can see how they are performing in the context of their cohorts, which many of them appreciate. We also heavily use bar charts you can drill down into, and spiderweb charts that could show the same information in a visually different way. Letting the students modify and change the display to their preferences was also key.

Finally, we added tabs to show the exam structure and an area to which support documents could be added, such as assessor comments.

Each time a student clicked on an assessment, the site would load their file and convert into a visually rich interface of charts, graphs and links to drill down into the data.

System Success and Expansion

The FMS Feedback system has been successfully used by the MBBS programme since 2014. It is held in high regard by the administrative team who appreciate its flexibility and ease of use. For students, the detailed data displayed allows them to have a very granular understanding about their levels of mastery of individual skills and topic areas, and to spot patterns in their performance. This means they can focus their studies on areas identified for improvement, with a view to increasing their overall competency.

It has since been expanded to cover a few assessments from dental programmes that had a similar structure and process to the original MBBS assessments, such as SBA and OSPE’s.

If you would like to learn more about the FMS Feedback system, please contact the FMS TEL Team.

Personal Tutoring, an example of rapid application development?

What is Personal Tutoring?

Personal Tutoring is the process of assigning the availability of university staff for student tutoring. It is not actually the process of assigning individual students to individual staff. Staff in the faculty are assigned to programme groupings/pots (not individual programmes), these groupings have a lead administrator that can then work out by looking at the expected student intake whether they need more staff resource, or can free it up for others.

The faculty admin team have undertaken this process for many years using excel spreadsheets and email communications to pass the information back and forward. This kinds of activity is both time consuming and prone to errors caused by duplicate copies of data and missed communications.

This is the perfect example of a process that can be done better using a web application, the kind of work the Technologies Developers in FMS TEL undertake all the time.

No time to plan properly

Sometimes a project comes to the unit that needs to be completed in a short time frame. Ideally the amount of time spent on a new website would be evenly spread between specifications, design, implementation, testing and support/improving., it may even go through many loops of these processes.

When a project does not have the luxury of time, then all these steps need to be compressed and decisions made on which steps need to be prioritised . In the case of personal tutoring the design phase and specifications where collaborated from the old Excel spreadsheets and turned into a simple tabular wireframe display. These spreadsheets where also identified as the origin of the data the site would be based off and import scripts planned accordingly. No complex interface features where offered, just a clean display of the data, with filters and stats to help the tutoring assignments. As for the implementation of the site, we decided to host the new site on top of another, saving time on hosting framework and infrastructure. We chose a site that had similar tools (FMS Projects, which has a statistics section) and tools we could utilise. We also based the core of the new site off knowledge and experience the team was used to (API’s and data tables, spreadsheet importing / exporting). Finally the testing and support side was compressed, keeping the interface simple, reducing the need for support and documentation. The limited number of users the site may have, also helps support as we can offer short term direct guidance.

With all these measures, we managed to reduce a development process that could take 6 months down to 3.

The Results

The Personal Tutoring site is due to go live in July 2022. We managed to write and get a functional version of the site done in roughly 2 months. This left 1 month planning the release, testing and show casing the site to the customers and making improvements from their feedback. Overall we are pleased with the structure and quality of the site. The code design is based on solid principles and should offer a degree of flexibility when Personal Tutoring gets used and the inevitable suggested improvements come through.

If you are interested in this topic and wish to learn more, please contact:

Dan Plummer, Learning Technologies Developer, dan.plummer@newcastle.ac.uk

Dealing with extra sensitive data in the Medical Learning Environment (MLE)

Most FMS sites run by the unit contain and maintain personal data that needs to be kept private. Techniques such as securely certified websites and authentication/authorisation portals are usually sufficient in keeping this data safe.

The Challenge

With the introduction of the new year 4 in the MBBS curriculum and the move to more blended learning, a higher degree of sensitive data was required to be stored on the Medical Learning Environment (VLE for MBBS). Year 4 students are now asked to keep electronic records of patients and interactions as part of the Advanced Clinical Experience module. This data contained personal contact details such as address, telephone and email of patients the students would follow on the clinical journey, and let them reflect upon this experience throughout year 4.

So before the start of Year 4, in the summer of 2020, we investigated and implemented an enhanced way of storing this patient information in the MLE.

The Solutions

First we investigated how the data was stored in the backend database. Most information is stored in databases as unencrypted data due to the lack of sensitive nature of the data.

This new data required something else. It was decided that parts of the data that could contain personal patient information should be encrypted, both in transit and at rest.

For parts of the ACE model (the data structure we use for the ACE section of MLE) we replaced the open text fields with this new encrypted field. This now meant that when data was entered and saved, before it was added to the database, the system would replace the open text with a encrypted data set using a secure key. To read the data again it would need the use of the decrypt method, that only the MLE could do by using the secure key.

The second part we investigated was to detach any personal patient information from the student’s reflections. Once the student had completed the recording of the patient’s details, the direct link in the website was removed and generic patient information used from that point onwards to identify the individual records. This kept the sensitive information separate from the day to day recording of patient interactions.

The students also uploaded consent forms signed by patients who agreed to take part in the ACE module. Final versions of consent forms highlighted that these would also contained sensitive information.

After further investigation the development team included these static files in the encryption methods used to support ACE. In order to allow students to verify the uploaded consent forms, the MLE allows a short window before encryption and archiving of consent forms takes place. Once this process completes the consent forms are no longer accessible via the website (MLE) and recovery if required is performed by a limited number of staff in FMS TEL.

These methods used may be a little extreme for the day to day data stored on most FMS sites, but the investigations and lessons learned from the ACE data has provided us with options for other sites in the future.

If you are interested in this topic and wish to learn more, please contact:

Dan Plummer, Learning Technologies Developer, dan.plummer@newcastle.ac.uk

John Moss, Technology Enhanced Learning Manager, john.moss@newcastle.ac.uk