Posted on by Christian Lawson-Perfect

Numbas guidance for support staff

Numbas is an e-assessment system developed by the e-learning unit in the school of mathematics, statistics and physics.

It’s been in use for around a decade, within Newcastle and around the world.

This page contains information for professional services staff supporting modules which use Numbas for assessment.

Contents

How it works

Questions and exams are written in the Numbas editor, at https://numbas.mathcentre.ac.uk/

Note: “Exam” is the name for any package of Numbas questions, whether it’s practice material or a high-stakes assessment.

The Numbas LTI tool manages the running of assessments. It’s available as an external tool in Canvas, or in the Numbas app.

A staff member creates the Numbas LTI link, uploads the exam package, and sets options like number of attempts and availability dates.

Students click the LTI link, answer the questions, and their scores are saved to the LTI tool.

The LTI tool can report scores directly back to Canvas. Staff members can click the LTI link to see all attempts, statistics, download scores, review individual attempts, change settings, or remark.

Settings

There are a variety of options in the Numbas editor and the LTI tool to control what feedback students get, and when.

In the editor

When editing an exam, you can set:

  • A password to begin the exam. Documentation
  • Whether the student can do “Try another question like this one”. Documentation
  • The duration of time allowed to complete the exam. Documentation
  • How much immediate feedback students get on their answers. Documentation
  • How much feedback to show in review mode, once the exam is over. Documentation

In the LTI tool

Once you have uploaded an exam to the LTI tool, you can set the following options:

  • The number of attempts students may make at the exam. Documentation
  • For exams which allow multiple attempts, how to calculate the student’s total score. Documentation
  • When students can see their total exam score on the attempts summary. Documentation
  • When students can re-enter the exam in review mode. Documentation
  • When students can attempt the exam. Documentation

Documentation

Numbas editor: https://docs.numbas.org.uk/en/latest/

Numbas LTI tool: https://numbas-lti-provider.readthedocs.io/en/latest/instructor/resources.html

The Numbas app

Numbas exams run entirely in the student’s browser. For normal browsers, that means students can use the browser’s built-in developer tools to cheat by seeing correct answers or modifying their score.

The Numbas app is a locked-down browser, like the Respondus locked-down browser that is used elsewhere in the university. It doesn’t have any developer tools, so students can’t cheat that way. It’s the only way of accessing our locked-down Moodle server (Moodle is a virtual learning environment like Canvas), so we can be sure that students don’t have access to developer tools for credit-bearing assessments.

There are instructions on installing the Numbas app at https://www.mas.ncl.ac.uk/numbas/help/.

There are versions of the app for Windows, Mac, Linux, iOS and Android. It’s installed on the Windows Virtual Desktop and cluster PCs.

It’s policy in MSP that all in-course Numbas assessment that counts towards module credit must run through the Numbas app.

Both students and staff log in to the Numbas app with the same username and password as for other Newcastle IT services. For students, it’s usually of the form b0000000 or c0000000, and for staff it’s usually of the form nxxx.

For a module to use the Numbas app, the e-learning unit must do some setup work. Student enrolments are automatically synchronised with the university Active Directory.

Lecturers and PS staff must be given access to modules individually by the e-learning unit.

Support

Numbas is supported by the e-learning unit in maths, stats and physics.

Any queries about Numbas should go to the numbas@ncl.ac.uk, so they reach the whole team.

LTDS does not at the moment have anyone assigned to supporting Numbas – they forward all queries to the MSP e-learning unit. We’re happy to support Numbas use anywhere in the university.

The school of engineering has a Numbas champion, Simon Lambert (simon.lambert@ncl.ac.uk), who can help with day-to-day problems, or forward more complicated queries to the MSP e-learning unit.

We’d like other schools who use Numbas heavily to appoint Numbas champions, to help spread the support workload.

What you might need to do

Set up an assessment

Documentation

When: the lecturer has created an exam in the Numbas editor, and asked you to make it available to students.

First, check that the settings for the exam in the Numbas editor are correct.

Availability dates and limits on the number of attempts are set once you’ve uploaded the exam to Canvas or the Numbas app.

At the moment, if some students are allowed extra time for a time-limited assessment, you must set up a second copy of the exam with the expanded time limit.

On Canvas

  1. Open the exam in the Numbas editor, and click Download SCORM package.
  2. Navigate to the appropriate module in Canvas, and create either an Assignment or an item in a Module.
  3. For assignments: Set “submission type” to “External tool”, then click Find and select “NUMBAS”. Tick Load this tool in a new tab. For modules: Add “External tool” and select “Numbas”. Tick “Open in a new tab”.
  4. Open the link you created.
  5. In Upload exam package, upload the SCORM .zip file you downloaded from the editor, and press Submit.
  6. Set the availability and review dates.

On the Numbas app

  1. Open the exam in the Numbas editor, and click Download SCORM package.
  2. Open the Numbas app, and navigate to the appropriate module. (The e-learning unit might need to give you access)
  3. Click Turn editing on.
  4. Click Add activity or resource.
  5. Click Numbas.
  6. Write a name in Activity name, then click Save and display.
  7. In Upload exam package, upload the SCORM .zip file you downloaded from the editor, and press Submit.
  8. Set the availability and review dates.

Create a paper version of an exam

The exam author should use the “Printable worksheet theme” to create a PDF version of the exam. If they like, they can create a different version of the exam for each student, or just one version for everyone.

Here’s a video showing how to do it:

Reopen a student’s attempt

Documentation

When: a student accidentally ends their attempt at an exam, you can reopen it for them.

You shouldn’t do this for high-stakes assessments where the student could have seen feedback or worked solutions after ending their attempt.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Attempts, and search for the student’s name.
  3. On the line corresponding to the completed attempt, click Reopen.

Allow a student another attempt at an assessment

Documentation

When: the module leader wishes to allow a student another attempt at an assessment which is normally limited to one attempt.

You don’t need to delete their first attempt.

  1. Open the exam in Canvas or the Numbas app.
  2. Click View individual student progress and grant access tokens.
  3. On the line with the student’s name, click the + icon to grant an access token.
  4. When the student next opens the assessment, they will have the option to start a new attempt.

Check a student’s answers have been recorded

When: because of computer or internet connection troubles, or just cautiousness, a student is not sure if all of their answers have been saved.

If the student gives you a receipt code

Documentation

When a student finishes an attempt, Numbas gives them a receipt code, a string of letters and digits, which contains encrypted information about their score. If a student sends you one of these codes, you can check it against the score saved in the database.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Validate a receipt code.
  3. Paste the code into the text box.
  4. Click Validate this code.
  5. You are shown the information from the receipt code and the information saved in the database. If they don’t match, then the student’s device may still have some unsaved data on it. They should access the attempt again and Numbas will try again to save data to the database. If they can’t do that, then you should ask for evidence of their work such as screenshots or handwritten working.

Review the student’s attempt

Documentation

  1. Open the exam in Canvas or the Numbas app.
  2. Click Attempts, and search for the student’s name.
  3. On the row corresponding to the student’s attempt, click Review. This shows the student’s attempt in review mode, with all their answers and any feedback visible. Click through the exam’s questions to see what answers are saved.

Update an exam package

When: there was a mistake setting up the marking or wording of a question, and the lecturer has fixed this in the Numbas editor.

You must decide if the fixed exam can safely be used as a replacement for the old one. A replacement is only unsafe if questions are added or removed, or the parts structure of a question changes due to parts being added, moved or removed. If you make an unsafe replacement, existing attempts will continue to use the old version, while new attempts will use the new one. If the change is just to marking settings or the text prompts, the replacement is safe, and all attempts will use the new package.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Settings.
  3. Click Replace exam package.
  4. Tick This is a safe replacement for the previous exam package if appropriate.
  5. Click Update from editor.

Automatically remark an assessment

When: there was a mistake setting up the marking of a question, you’ve updated the exam package with a fixed version, and existing attempts need to be remarked.

The remarking interface shows you all attempts at the assessment, along with their current scores, and allows you to remark them using the latest version of the exam. You must update the exam package if you haven’t already done so – see the section above.

Once you choose to save a changed score for an attempt, the original data is still kept, so this can be undone if necessary. At the moment, undoing a remarked score needs someone from the e-learning unit, but eventually it’ll be available in the web interface.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Remark.
  3. Click Remark all attempts.
  4. Remarking can take a long time. You will see a progress bar and an estimated finish time. No new scores are saved at this point.
  5. Once the remarking has finished, you can compare the new marks against those saved in the database. You can choose to show all attempts, just changed attempts, or just those with increased/decreased scores.
  6. Either click Save all changed attempts, or click the Save buttons corresponding to individual attempts.

Manually remark an assessment

When: the lecturer wants to apply their discretion to award a student a different mark for an answer.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Attempts.
  3. Search for the student’s name.
  4. In the row corresponding to the student’s attempt, click Remark.
  5. The table lists the student’s scores for each part of each question of the exam. Find the row corresponding to the part you want to remark, and click the pencil icon.
  6. Enter the new score in the score field.
  7. Click outside the score field to save it. (This interface needs to be improved)

Mark a student’s unsubmitted answers

When: a student entered answers for one or more question parts, but didn’t press Submit.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Remark.
  3. Tick Use unsubmitted answers.
  4. Remark either an individual student’s attempt, or all attempts.
  5. If the total score changes, you must click Save to keep this new score.

Discount part of a question

When: student have already started attempts at an assessment, but the lecturer decides that part or whole of a question should not count towards the total score.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Discount question parts
  3. In the row corresponding to the question part you want to discount, click Discount this part.
  4. If this assessment is in Canvas, you will need to report the changed scores:
    1. Click Dashboard.
    2. Click Report scores back to VLE.

Produce summary statistics for an assessment

When: an assessment’s deadline has passed, and you want to know summary statistics such as the average score and score quartiles.

  1. Open the exam in Canvas or the Numbas app.
  2. Click Statistics.
  3. You’re shown:
    • A table of the numbers of completed and incomplete attempts.

Upload scores to NESS

When: an assessment’s deadline has passed and the scores should be uploaded to NESS for release to students.

  1. Open the Numbas LTI link for the assessment. Click Download scores as CSV.
  2. Go to NESS. Click Coursework -> Marking -> Upload marks.
  3. Select the appropriate module and assessment.
  4. File format: comma separated; Student no index: 4; Mark index: 5 for percentage, 6 for raw score.
  5. Make sure the marks available in Numbas and NESS match.
  6. Select the CSV file.
  7. Click Upload.
  8. The scores are now in NESS, but not released to students. Release the scores when appropriate.

Frequently asked questions

The following are answers to the most common questions from students.

How are incomplete attempts handled?

Incomplete attempts are included in the CSV score exports, so students won’t lose out if they forget to click “End Exam”.

Can students leave an assessment and carry on later?

Yes, no matter what the settings for an assessment are, students can always reopen it before the deadline and carry on where they left off.

Their answers are saved automatically. Students should click Submit after entering each answer, but Numbas makes its best effort to save even unsubmitted answers.

What happens if the student’s computer breaks during an attempt?

They can reopen the attempt later on, on the same device or a different one.

Data is saved to the student’s computer before being sent to the Numbas server. If there was any data that hadn’t been received by the server when the student’s computer broke, it will be sent again when they reopen the attempt on the same device.

What happens if the student’s internet connection breaks during an attempt?

All the marking happens on the student’s device, so they can continue working through an assessment even when there’s no internet connection.

If no connection is available for a long period (about a minute), a red warning box pops up telling the student that their data has not been saved to the database. It’s still saved to their device.

When the internet connection is restored, all data is sent to the server. Once all the data has been saved to the server, the red box disappears.

If the student closes the exam while there’s no internet connection, their data is not completely lost: the next time they open that attempt on the same device, the data will be restored from local storage and Numbas will try again to save it to the database.

What happens if the deadline passes while a student is attempting an exam?

The attempt ends immediately. Any unsubmitted answers are not marked. If necessary, you can use the automatic remarking tool to mark unsubmitted answers.

What happens if the availability dates change while students are attempting an exam?

Each device with the exam open is sent the new dates, so all attempts will end at the right time.

What happens if the student opens the same attempt on two different devices?

The first device to open the attempt is kicked out, assuming it’s still connected to the internet.

Can students change their answers?

Numbas does not limit how many times a student can submit answers. Students can change their answers to any question part at any time until the attempt ends.

Leave a Reply