Numbas v7 and our own lockdown app

Collage of screenshots showing new Numbas features

It’s been a while since our last development update. The reason for that is that we’ve been working on some big changes to every bit of the Numbas software. We’ve released v7 of the Numbas app and editor, and a new lockdown app to integrate with the LTI provider which ensures that students can only access assessments in a restricted environment. There’s also a Safe Exam Browser integration, for in-person invigilated exams.

Laura Midgley has joined the team, replacing George Stagg, who left for an exciting job with RStudio.

Four people stood in a line, facing the camera
The e-learning unit now: from left to right, Chris Graham, Christian Lawson-Perfect, Laura Midgley and Aamir Khan.

Numbas runtime

Unit tests for the exam runtime

This won’t affect most Numbas users directly, but I’m mentioning it here to reassure you that we’re taking efforts to make sure new changes don’t break things!

There are now some automatic unit tests checking the functionality of exams. Previously there were only unit tests covering the JME language.

I’ve set up continuous integration, so that the tests are run automatically each time we commit a change to the runtime’s GitHub repository, and we’re alerted if any of the tests fail. It’s already helped me catch a few breaking changes!

A testing regime is only as good as the tests it carries out, so this will grow to be more useful over time, as we add tests to cover more aspects of the code.

Conflicts between question variables and marking parameters

When a question variable has the same name as a parameter to a part marking algorithm, an error is now thrown.

This has broken some existing questions. The reason for the change is that when an error was not thrown, affected questions could end up marking answers incorrectly because the question variable overrode the marking parameter, or vice versa.

There isn’t a solution to this that doesn’t break something: either we ignore the values of question variables, or we rename marking algorithm variables, with either way potentially breaking custom algorithms that already use them.

In retrospect, I should have made question variables only available through a dictionary, so there can’t be a conflict. Too late now!

If you encounter this error, the solution is to rename the conflicting question variables.

Other changes


  • Thanks to Chris Theron for translating the interface text into Afrikaans!
  • You can set an adaptive marking variable replacement for a gap-fill, and it applies to all of the gaps. (issue from 2016!)
  • There is an option in the matrix entry part to pre-fill some cells. (issue from 2015!)
A 2×2 matrix input with the first row filled in with 5, 4, and non-editable.

A matrix entry part with the top row pre-filled.
  • Functions can be marked as ‘deterministic’ – they always produce the same output given the same input. Using this, question variables which can be deterministically reproduced from other variables are not saved to the SCORM suspend data, saving a lot of space. (issue)
  • Number values store the precision that they represent, either as decimal places or significant figures. When they’re displayed to the student, this precision is used, including trailing zeros if necessary. (issue)
  • There’s a “show penalty hint” option for next part options in explore mode. (issue, documentation)
  • You can put 0 in the “initial number of rows/columns” settings for matrix entry parts to use the same dimensions as the expected answer. (issue)
  • You can turn off the hint saying “lose N marks” on “next part” buttons. (issue)
  • In marking algorithms, you can give the apply function several names of notes to apply. (issue)
  • There’s a new JME function random_integer_partition, to produce a random partition of an integer into a given number of parts. (issue, documentation)
  • There is now a display flag matrixCommas to control whether commas are used between elements in the same row of a matrix or vector. (issue, documentation)
  • There are two new simplification rules, turned off by default, called rationalDenominators and reduceSurds. (issue, documentation)
  • There’s a new JME function, largest_square_factor, which finds the largest perfect square factor of a given integer. (documentation)
  • A JME variable name can have a Greek letter in a subscript. (issue)
  • There are two new convenience functions in marking algorithms: add_marking_if and multiply_credit_if. (documentation)
  • You can change the names of questions in an exam in “sequential” navigation mode, like you can in “menu” mode. We’ve used this to have an unnumbered “formula sheet” question at the start of exams. (issue)
  • The number entry part type now has a “display answer” setting, for cases where a custom marking algorithm means that the expected answer should be displayed differently to the standard format, or when you don’t want to use the mean of the minimum and maximum expected value. (issue, documentation)
  • A wider variety of characters representing parentheses is accepted, making input more convenient for users with keyboards in non-English locales. (code)
  • There is a new display flag, timesSpace, which causes implicit multiplication to be rendered as a small empty space instead of the × symbol. (issue, documentation)
  • When lists are displayed in LaTeX, including function arguments, the symbol used to separate elements of the list is localised: locales which use a comma for the decimal separator use a semi-colon as the list separator. (issue)
  • There’s a new pipe operator, |>, which can be used to compose transformations of a single value. ( (documentation)

Other changes

  • The code uses more recent JavaScript features, dropping support for old browsers like Internet Explorer 11, as described above. We’ve made the decision to aim for compatibility with 95% of devices currently in use, as measured by To help us make decisions about whether to start using new browser features, I made a tool called “Can I also use”, which will be useful for any web development project. The list of supported browsers in the documentation has been updated to reflect this.
  • In exams with a time limit, the remaining time is always visible. (issue)
  • “Number entry” parts deal with precision restrictions better when the answer is given in scientific notation: only the precision of the significand is considered. (issue)
  • Functions added by an extension are only available in questions using that extension. (issue)
  • You can use implicit multiplication after a postfix operator. (issue)
  • When a question variable has more than one definition, an error is thrown. (issue)
  • The decimal function rewrites all number literals in its definition to be decimal values, so precision isn’t unexpectedly lost by an intermediate calculation in floating-point number values. (issue)
  • Fractions with a minus sign on the denominator are accepted. (issue)
  • There’s a more helpful error message when a reference is made to another part that doesn’t exist. (issue)
  • When resuming an attempt, the check that each question’s name is the same is no longer run: this is not a necessary or sufficient criterion for an updated exam package being compatible with the saved attempt data. (issue)
  • In the question preview theme, the page footer is horizontally centred. (code)
  • The “display options” button is shown in the footer of the question preview theme. (issue)
  • When a part has a single step, that step is not named. (issue)
  • The adaptive marking variable replacement routine throws an error if there is a circular reference. (issue)
  • The default theme uses the browser’s built-in sans-serif font instead of Source Sans, eliminating an external dependency. (issue)
  • There are new error messages for: a custom JavaScript function definition specifies an invalid language; the value passed into the vector constructor is not an array of numbers; the value passed into the matrix constructor is not a matrix object. (issue)
  • In the default theme, just an error message is displayed when the stylesheet doesn’t load. (code)
  • In the default theme, the part feedback area no longer has a maximum width or height. (issue)
  • The per-item feedback for multiple response parts is displayed in the order in which the corresponding tickboxes are displayed. (code)
  • The documentation for each of the built-in part types now describes the data type of the studentAnswer variable in marking algorithms. (issue)
  • There is an error message for the case when a question requires an extension but it hasn’t been included in the exam package. (code)
  • The way that values are substituted into the expression in the \simplify command has changed, using substitutions into the compiled expression tree instead of into the JME string. This means in particular that decimal values are displayed nicely, instead of being wrapped in `dec(…)`. (code)
  • The decimal function when applied to a number literal does not first parse the number as a JS number object, so no precision is lost. (code)

Bug fixes

  • When substituting variables into an expression value, the “plain” number notation is always used, avoiding errors when your locale’s default number notation is not the English one. (issue)
  • Improved the accessibility of modal dialogs: the header is read out, and focus is on the start of the text. (issue)
  • Variables are substituted into the labels on “next part” option buttons. (issue)
  • In menu mode, LaTeX in question names is rendered. (issue)
  • When pattern-matching a function name, the scope’s case-sensitivity setting is respected. (issue)
  • Simplification rules collecting numbers together avoid incorrectly producing ‘infinity’ when exceeding the maximum value representable by a floating-point number. (issue)
  • Error messages in parsing the exam’s XML are escaped, so they’re not accidentally interpreted as HTML. (code)
  • Fixed a typo in the “relative difference” checking function which meant it didn’t work when the input numbers have type decimal. (code)
  • When construction a JME sub-expression with the expression function, variable substitution into strings uses the JME substitution method instead of the display-text method. (issue)
  • Fixed a bug where evaluating a note in a marking script would could lead to variables being saved to the parent scope. (code)
  • The JME function root(-x,n) only returns a real value if n is odd. (code)
  • Fixed a typo in multiple response parts which led to the wrong marking method being used. (issue)
  • The JME function mark_part uses the part’s JME scope, instead of the one in which mark_part was called. This avoids variables from the containing scope leaking into the part’s marking. (code)
  • Fixed a case where the message for an error produced by a part would sometimes show the part’s name twice. (code)
  • I rewrote the algorithm for inferring the types of free variables in an expression. It’s now much faster, and won’t fail on large expressions. (issue)
  • Fixed a typo in the definition of log(x,base) when x is a decimal value. (code)
  • Fixed a bug when rendering expression values as LaTeX. (code)
  • When rendering an expression as JME code, numbers with a * symbol in their rendering are surrounded by brackets. This fixes an operation precedence issue when numbers are rendered in scientific notation. (code)
  • When rendering decimal values as LaTeX, they are no longer converted to JS numbers first, losing precision. (issue)
  • If a gap in a gap-fill part has an adaptive marking variable replacement referencing a part which must be answered but has not, the error message is shown in the part feedback, just like for non-gap parts. (issue)
  • When a custom JME function is defined, it always sets the custom findvars behaviour, fixing an issue in the editor where it looked like a parameter name was considered to be an external variable. (issue)
  • The cmi.time_spent key in attempt SCORM data was incorrectly adding 1 hour to the time spent. The code has been rewritten. (code)

Numbas editor

New layout

Screenshot of the Numbas question editor

In the exam, question and custom part type editors, the tab navigation runs horizontally at the top of the page, leaving more width for the main editor.

When adding a part, the available part types are shown in a vertical list along with descriptions, instead of as small buttons in a single line.

The tree of parts has been rearranged, with the aim of reducing clutter and making it easier to read.

The part’s marking algorithm and the area to test answers and run unit tests are now in separate tabs.

In the question editor, only one function’s definition is shown at a time, like the variable editor.

When you’re editing a variable, its full value is displayed under the definition. When the variable represents an interactive element, you can interact with the element. This will help when using long lists or dictionaries, and while creating randomised diagrams.

Screenshot of the variable editing interface. At the top is the JME code editor, and underneath in a box labelled "Generated value" is a rendered diagram, showing a line, a circle and a point on a set of axes.
You can now see randomised diagrams while editing them.

The “Variable testing” tab in the question editor is now accessed from the bottom of the list of parts, instead of being a separate tab on its own.

The editor for custom functions now only shows one function at a time, similar to the variables editor.

The code editor for theme and extension files now occupies the entire viewport, so there aren’t two competing vertical scrollbars.

Screenshot of the extension editor interface. At the top is a header "Edit extension "working-out"". Beneath is an interface with a sidebar on the left listing files, and a code editor on the right.
The rejigged code extension editor.

Web-based first setup

Laura’s first job was to convert the command-line script used to set up an instance of the Numbas editor to a web-based form.

Screenshot of the Numbas editor setup form

It looks good!

As a little bonus, instead of using a progress bar while you wait for the installation to finish, I made it show you the factorisation of the elapsed time:

Hopefully this will make it easier for other people to join in with Numbas development.

Easier question testing

Screenshot of the editor interface: a button labelled "Run all unit tests" above a table with columns "Part", "Test" and "Passed?". There are 5 rows. The first four all end with a green tick and "Passed", while the last ends with a red cross and "Failed"
Run all the unit tests in a question at once

There is now a top-level Testing tab in the question editor, showing all unit tests associated with the question’s parts, and a button to run them all at once.

Additionally, each part now automatically gets a unit test checking that the expected answer is marked as correct. (issue)

This should make it much easier to check that questions do what they should.

Other changes


  • When a gap is defined but not used in the part’s prompt, a warning message is shown. (issue)
  • The custom part type editor loads all extensions used by the part type, so their functions show up in the code autocompleter. (code)
  • The marking matrix field for “match choices with answers” parts now shows the text of the choices and answers. (issue)
  • The “Saving/Saved” popup notification in the question and exam editors has been replaced by a permanent badge at the top-right of the screen, so you can always see whether your changes have been saved to the database. (issue)
  • In the theme and extension editors, you can drag and drop files onto the buttons to upload or replace a file. (code)

Other changes

  • There’s a little pencil icon next to the name in the question and exam editors, since it’s not obvious that you have to go to the “settings” tab to change it. (code)
  • The header on the question/exam preview page is now a neutral grey, instead of scary red. (code)
  • In the question editor, the “Statement” and “Advice” tabs have different icons. (issue)
  • The custom part type editor now shows when there’s an error loading a required extension. (issue)
  • On the list of your extensions, extensions are ordered alphabetically. (issue)
  • The way that variables are substituted into \simplify has changed, to improve the rendering of decimal values. (code)

Bug fixes

  • The marking matrix editor for “match choices with answers” parts now scrolls if it gets too wide. (issue)
  • Fixed a bug where custom constants weren’t found by the routine to identify undefined variables. (code)
  • Fixed a bug with applying CSS rules in content previews when the question’s CSS preamble contains an at-rule. (issue)
  • The routine for detecting undefined variables is now aware of variables added by an extension. (issue)
  • Fixed a bug where the values of locked variables weren’t discarded before regenerating the variable preview. (code)
  • Fixed a typo in the embed view which set the viewport scale incorrectly. (code)



  • Updated Pyodide to v0.21.0.
  • In Python, added a matplotlib_preamble pre-submit task to clear any output of matplotlib from previous runs. (issue)
  • There’s an option to turn off variable substitution in the “expected answer” field of the Code part type, so you can easily use curly braces in code.
  • The webR runner queues all blocks of code at once, so that .Last.value works as you’d expect. (issue)
  • Errors thrown while executing the postamble are shown in the part feedback. (issue)
  • The code editor widget has a minimum height. (code)
  • The webR runner removes ANSI escapes from the output, and carriage return characters from code to run. (code)

Random person

  • Jesse Hoobergs wrote a new compression scheme, reducing the size of the extension. (pull request)

Numbas LTI provider

Numbas lockdown app and Safe Exam Browser

One of Numbas’s main strengths, the entirely client-side exam runtime, is also a weakness when you’re carrying out summative assessments: if the student runs an exam in a normal web browser, they can use the browser’s developer tools to inspect the code and see expected answers, or manipulate their score.

The solution to this is to require students to use a specific locked-down browser app which doesn’t allow access to the underlying code.

For several years, we have used a locked-down browser app to deliver summative Numbas assessments to our students. This app was hard-coded only to work with a Moodle server we set up for the purpose, so we couldn’t share it with other institutions.

Over the last year, we’ve been working on a new Numbas lockdown app which integrates with the LTI provider, so it can be used anywhere.

The Numbas lockdown app will launch automatically when a student launches a Numbas LTI resource which requires it. The app presents a minimal browser interface, and launches straight into the Numbas exam after the student enters a password to decrypt the launch settings.

Students see this screen when they launch a resource which requires the Numbas lockdown app.

The app doesn’t need any configuration on the server apart from a default password, which can be overridden. We recommend that it is used for any summative assessment contributing to course credit.

Setting an exam to require the Numbas lockdown app, and then the process of launching the exam in the app as a student.

At Newcastle University, we have a policy of using another locked-down browser app, Safe Exam Browser, for in-person invigilated exams. Safe Exam Browser doesn’t run on as many platforms as the Numbas app, but does completely control the computer interface, so students can’t use other programs while completing an exam.

We’ve also added support for Safe Exam Browser to the Numbas LTI provider. A server admin has to create a settings file using the SEB config tool and upload it to the Numbas LTI provider, and then any instructor can require SEB when students launch a particular resource.

Other changes