Numbas

Christian Lawson-Perfect

Development update: August 2020

We’re continually working on Numbas, fixing bugs and adding new features. I thought that it would be a good idea to post more regularly about updates to the Numbas software, rather than waiting until each year’s major version release to list everything that’s changed.

I’m going to go into a bit more detail on changes than I normally would in major-release posts, to give an idea of what day-to-day development on Numbas looks like.

I’d like these posts to be monthly, but I’m about to embark on a month of paternity leave, so don’t expect another one until the end of October.

So, here’s what’s changed since Numbas v5.0 was released in June.

Numbas runtime

General

  • Enhancement: The logic around the question “how long is this variable name?” has been improved greatly. This matters when you’re using things like Greek letters, which take several characters to write in JME, but are displayed as a single symbol. (code)
  • Enhancement: The ‘expand juxtapositions’ logic (used to implement the implicit multiplication and composition rules) makes use of this enhanced logic to better split up long names into single-letter variables. (code)
  • Bugfix: The “scientific notation” number notation style no longer puts spaces between groups of digits. (code)

Interface

  • Enhancement: Updated the translations, adding Indonesian.
  • Enhancement: When you end the exam, you’re now always shown the “results” screen – the “you may close this window” screen is no more. Whether you see the question score breakdown depends on the “show the results page” exam option. This means that students are now always shown the button to print a transcript of their exam, even if they don’t get any feedback. (code)
  • Enhancement: The default theme now has a favicon. (code)
  • Bugfix: Content areas containing only an image or an iframe no longer count as ’empty’. (code)

Marking

  • Enhancement: For part types that do some parsing (mathematical expression, number entry, matrix entry), the interpreted_answer note now fails if the student’s answer is not valid. This matters in explore mode, or in complicated gap-fill marking algorithms. (code)
  • Enhancement: The “extension” part type no longer assumes you’re going to write a custom marking script in JavaScript – you can write a JME marking algorithm instead. The GeoGebra extension now uses this, providing some JME functions to query a worksheet. (code)

JME

New functions

  • reorder and shuffle_together make it easier to take several lists and apply the same permutation to them. (issue)
  • replace_regex replaces all substrings that match the given regular expression with a given value.
  • atan2(y,x) implements the widely-used function which is equivalent to $\arctan(y/x)$, but ensures the result is in the right quadrant.

Other changes

  • Enhancement: The random function returns an integer when given a range of integers. (issue)
  • Enhancement: The functions ceil, round and floor return integer values, not number. (code, issue)
  • Enhancement: When substituting values into content areas, data types that can automatically convert to HTML do so. This makes it more convenient to use the new geogebra data type provided by the GeoGebra extension. (code)
  • Enhancement: Within a part (content areas and the marking algorithm), the JME variable part_path is now defined, giving the unique ID of the part. Interactive elements such as GeoGebra worksheets can use this value to link to the part they’re embedded in. (code)
  • Bugfix: The output of dpformat and sigformat functions is displayed as plain text when substituted using curly braces, or TeX code when substituted inside a LaTeX environment. (code)
  • Bugfix: Upper-case Greek letters in JME expressions are displayed in upper-case. Previously they were incorrectly converted to lower-case. (code)
  • Bugfix: Rational values are displayed as fractions, rather than decimals. (code)
  • Bugfix: The exec function now evaluates lazily, so it doesn’t try to evaluate function tokens. It’s still quite awkward to use, so I prefer substitute. (code)
  • Enhancement: You can define several variables at once: write a list of names separated by commas, and ensure that the definition produces a list of at least that many items. This makes it much easier to work with “scenario” questions, where you’ve got a finite list of scenarios to choose from, each with several associated variable values. (code)
Variable editor interface. The name field contains "a,b,c" and the value field "shuffle(1..10)". The description says "Three distinct numbers between 1 and 10". On the right, the variable preview shows three rows a,b and c with generated values 5, 10 and 1.
  • Bugfix: The scientificnumberlatex function marks its output as “safe”, so the curly braces around the exponent don’t get interpreted as variable substitution. (code)
  • Bugfix: Expressions with a complex number on the left-hand side, such as i*x, don’t omit the multiplication symbol. (code)

JavaScript:

  • Bugfix: The question signal HTMLAttached only fires after all the question’s HTML has been attached, including parts. (code)
  • Bugfix: When loading a mathematical expression part, the ‘answer simplification string’ setting has a default value. (code)
  • Bugfix: Part.markAlternatives copes with the marking function returning an invalid result. (This affected old questions with custom JavaScript marking scripts) (code)
  • Enhancement: When a JME marking algorithm produces an error, it’s exposed in the result object. This allows the editor to show the error in detail rather than just “the answer is invalid”. (code)
  • Bugfix: jme.calculus.differentiate now checks for compatible types, rather than being fixed to the ‘number’ data type. This makes it work on the number-like types such as integer, rational and decimal. (code)
  • Bugfix: … and I fixed the partial differentiation rules, which never properly worked. (code)
  • Enhancement: JME function signature definitions can label arguments, to make it easier to establish which optional arguments have been provided. (code)
  • Enhancement: Within a JME function signature, you can specify a list with optional entries. (code)
  • Bugfix: When casting the arguments to a function to match a signature, list elements are also cast. (code)

Extensions

  • I almost entirely rewrote the GeoGebra extension. You can now create a GeoGebra worksheet entirely in JME or from a .ggb file uploaded as a question resource, rather than loading a worksheet from geogebra.org. The link between worksheets and parts has been improved – you can fill in the input for a number entry or matrix entry part with the value of a GeoGebra variable, and you can query a GeoGebra worksheet from within a part’s marking algorithm. (Documentation, demo questions)
  • I spent some time improving the JSXGraph extension, but didn’t get quite as far as I did with GeoGebra. You can now create a JSXGraph diagram entirely in JME, or using JSXGraph’s JessieScript language. (Documentation, demo questions)
  • I fixed a couple of bugs in the Eukleides extension.
  • I added some new functions to the permutations extension, and a custom part type which assesses a permutation written in cycle notation. I don’t have a public demo of this extension, sorry!
  • Prompted by a request on the numbas-users list, I wrote a small extension which provides a function to write a number out in words. (Documentation, demo)

Outstanding issues

There are 93 open issues on GitHub, 18 of which were added after v5.0 was released.

Numbas editor

Enhancements

  • Superusers no longer see every extension in the question editor. (This is an enhancement for me, because I was seeing dozens!) (code)
  • When testing a part’s marking algorithm, errors in the mark note are caught and displayed, rather than just saying “your answer is invalid.” This makes it easier to debug marking algorithms. (code)
  • In the non-editing view of a gap-fill part’s prompt, gaps are displayed nicely as boxes with the gap’s name, rather than a number inside square brackets. (issue)
  • When you upload an image as a question resource, you’re asked to give a text alternative for anyone who can’t see the image. (issue)
Screenshot of editor interface. On the left is an image, and on the right are two fields labelled "URL" and "Text alternative"
  • There’s now a “long plain text string” definition template for variables. This makes it easier to store strings that span multiple lines. I’m always keen to add new templates for the variable editor. (issue)
  • The editor now uses Django 3.1 and other Python dependencies are up to date. (code)

Bug fixes

  • When editing a new file in an extension, it’s included in the list of files. (issue)
  • The list of files in an extension is shown when viewing the Options and Access tabs. (issue)
  • The item preview and embed views now have a favicon like other pages. (code)
  • When you click to write a comment, the rich text editor loads immediately, rather than requiring another click. (issue)
  • Fixed the help link for the mathematical expression part setting “pattern student’s answer must match”. (issue)
  • The “Parts mode” select box when creating a question is now styled the same as other select boxes. (code)

Outstanding issues

There are 42 open issues on GitHub, 6 of which were opened after v5.0 was released.

Numbas LTI provider

We released version 2.7 of the LTI provider. It adds a global dashboard view, an option to only show completed attempts when viewing the statistics for a resource, client-side completion receipt codes for attempts, and some bug fixes around time zone handling, sending email receipts on attempt completion, cross-site security and text formatting.

There are 32 open issues on GitHub, one of which was opened after v5.0 was released.

Documentation


That’s it for now. See you in October!

Numbas LTI provider v2.7

We’ve just released a minor new version of the Numbas LTI tool provider.

Changes in this version include:

  • A global dashboard view, showing resources which are active today.
  • An option to only show completed attempts when viewing the statistics for a resource.
  • When the student ends their attempt, they’re shown an encoded receipt which they can use as evidence of their score, even if no receipt email is sent.
  • Bug fixes around time zone handling, sending email receipts on attempt completion, cross-site security and text formatting.

We recommend updating any servers running the LTI provider as soon as possible, by following the upgrade instructions.

The installation instructions for Ubuntu, Red Hat Enterprise Linux and Heroku have all been updated.

Recordings of Numbas training sessions

Here are the recordings of our recent training sessions held over Zoom.

First, there was a beginners’ session on the 15th of July, covering the Numbas editing interface, how to deliver exams to students, and writing a first question:

And an advanced session on the 22nd, covering explore mode, custom marking algorithms, and custom part types:

Numbas training sessions

We’re planning on running another Numbas training session for beginners, following the series we ran in March and April.

The first session is scheduled for 10:00 BST on Wednesday 15th July. It’ll run over Zoom and last about two hours. No previous knowledge of Numbas will be assumed.

Following demand, we’ve scheduled a more advanced training session, for experienced Numbas users.

This will take place at 10:00 BST on Wednesday 22nd July. Again, it’ll run over Zoom and last about two hours. Christian will demonstrate a few things, then there’ll be time to work through question-authoring problems from attendees.

Register for a training session:

If you’re looking to start using Numbas in a hurry because of the CoViD-19 crisis, have a look at our blog post about how Numbas can help.

Numbas v5.0

Montage of screenshots of Numbas v5

It’s time for another major version of Numbas. This release includes a radically new way of presenting questions, as well as a few other helpful new features.

We’ve made a short video to demonstrate the new features:

We’ve also made a demo exam so you can try it out yourself.

Recording of online Numbas training session

This morning we ran the second of our online Numbas training sessions organised in response to the coronavirus crisis.

Here’s the recording of the session:

There’s at least one more session to go, at 10:00 BST next Wednesday, April 8th. You can still register to attend. If there’s demand for it, we might schedule more sessions.

How Numbas can help during the COVID-19 crisis

Now that face-to-face teaching in many institutions has been cancelled in response to the coronavirus crisis, many lecturers are looking for ways to move their teaching and assessment online. 

We’ve put this page together as a reference to describe how Numbas could be used to help your students and to replace paper-based assessments. 

What Numbas is 

Numbas is a free, open-source e-assessment system for mathematical subjects. 

Key features include: 

There’s an editor at numbas.mathcentre.ac.uk where you can create your own questions or reuse existing questions. It’s quick to sign up and we have lots of documentation and tutorials to get you started. 

How it works 

  • Write questions using the Numbas editor. 
  • Collect questions into an exam. 
  • Share the exam with students. 
  • Students attempt the exam. They can pause or leave the exam at any point and return to it later.  
  • Scores are automatically sent back to your VLE’s grade book. 

Numbas exams run entirely on the student’s device, so the student’s experience is not affected by a spotty internet connection. Once the exam starts, the only communication with the server is to report scores back (even this is optional – if you don’t want to track students’ scores, Numbas can run entirely standalone). Other systems need constant communication back to the server to render mathematical notation or to handle marking and navigation. 

And if you’re using the Numbas LTI tool, instructors can: 

  • See statistics about the class’s performance. 
  • Review individual students’ attempts exactly as the students saw them. 
  • Download a full breakdown of students’ attempts in CSV format. 

What can I use it for?

Numbas was originally designed for use in our mathematics and statistics programmes.

However, Numbas has also been used for many other subjects, including engineering; chemistry; physics; business studies; psychology; sports science; accounting; and biomedical science.

Even non-mathematical subjects can benefit from Numbas by making use of its several multiple-choice question types.

Why should I use it? 

Automatic assessment gives students immediate feedback, allowing them to work at their own pace. 

Randomised questions are good both for practice and for exams where you don’t want students to be able to copy off each other. 

There’s a lot of free material available to use under a permissive Creative Commons licence. Use the Explore page or the search facility to find content. 

How do I get started? 

Follow the tutorials at docs.numbas.org.uk. They will show you how to browse the question database, write your question, and deliver an exam to your students. 

Once you’ve done that, there’s full documentation on every feature of the editor, as well as ‘how-to’ instructions for common tasks. A good next step is our document on effectively planning a Numbas question

Where can I get help? 

Post your questions to the numbas-users mailing list

We’re happy to answer questions emailed to numbas@ncl.ac.uk, but please understand we’re quite busy at the moment! 

Delivering Numbas tests to students 

There’s more detailed info in the Numbas documentation

For purely formative material, you can just share a link to a Numbas exam. Students can complete the test, but their scores won’t be saved anywhere. This is a good solution if you want to quickly provide a bank of revision material. 

To save scores, you need a virtual learning environment that supports either SCORM (Moodle and Blackboard do, among others) or LTI (pretty much everything). 

The table below shows which standards are supported by common VLEs. 

VLE SCORM LTI 
Blackboard Yes, but don’t rely on it for high-stakes Yes – admin must make connection 
Moodle Yes Yes – teacher can make connection 
Canvas No Yes – teacher can make connection 
Desire2learn Brightspace Yes Yes  – admin must make connection 
ILIAS Yes Yes, with a plugin 
Sakai Yes, with a plugin Yes – teacher can make connection 
Schoology Yes Yes – teacher can make connection 

For SCORM, no setup is required: you download the package from the Numbas editor and upload it to your VLE. 

For LTI, you might need your VLE administrator to set up a connection. The Numbas LTI software needs to run on your own server. There’s full documentation about what you need and how to set it up

Considerations for high-stakes assessment

If you’re thinking of using Numbas for a high-stakes assessment, consider the following:

Robustness

Do everything you can to avoid losing student data. If using your VLE’s built-in SCORM player, it is responsible for saving attempt data.

Blackboard’s SCORM player can not be relied upon. Consistent reports describe Blackboard failing to save data for around 10% of students’ attempts.

We have fixed several bugs in Moodle’s SCORM player, and are now quite confident that it is reliable.

The Numbas LTI tool has been heavily tested and has several features to ensure no data is lost:

  • Data is saved to the student’s local storage as well as the remote server. If the student’s device crashes, they can resume where they left off after restarting.
  • After finishing, the student is shown a large warning box until all data has been confirmed to be saved.
  • Students can be automatically emailed a receipt containing a code which the instructor can use to validate their score.

Additionally, you can instruct your students to click the “Print this results summary” button after they have finished the exam, and print to a PDF file. They could email this to you if there’s a discrepancy with the score saved in your VLE.

Security

Because Numbas exams run in the browser, it’s possible that students could use their browser’s developer tools to modify their scores. For high-stakes assessments, we have always recommended using a locked-down browser.

Safe Exam Browser is an open-source tool which provides a locked-down browser.

Validity

Think carefully about what questions you can ask. You might need to think creatively about how to assess questions which would normally be answered with long passages of text. These questions could be reconfigured as multiple-choice, or you might just need to ask different questions.

Training

We’re going to run online training sessions, open to all. These will be informal, with a flexible format.

At the moment, we’ve scheduled the following sessions:

  • 10:00 GMT, Wednesday March 25th.
  • 10:00 GMT, Wednesday April 1st.
  • 10:00 GMT, Wednesday April 8th.

We’ll begin by running through the Numbas

The sessions will run over Zoom. You’ll need to register in advance, using this link.

Accessibility statements for Numbas

A few people have asked us for an accessibility statement before they start using Numbas, so we’ve written one.

In fact, we’ve written two: one for the exam interface seen by students, and one for the editor.

The statements describe the accessibility requirements that we’ve designed Numbas around, some tips on how to use Numbas effectively with different assistive technologies, and details on the outstanding accessibility issues that we know about.

If you’ve got any feedback about the statements or accessibility in Numbas, please let us know.

Documentation

Numbas features full documentation which is always in line with the most recent version.

Numbas Documentation

Numbas Blog

We regularly update our blog with articles about new and future features, as well as other useful information.

Numbas Blog