It’s been a while since our last development update. The reason for that is that we’ve been working on some big changes to every bit of the Numbas software. We’ve released v7 of the Numbas app and editor, and a new lockdown app to integrate with the LTI provider which ensures that students can only access assessments in a restricted environment. There’s also a Safe Exam Browser integration, for in-person invigilated exams.
Laura Midgley has joined the team, replacing George Stagg, who left for an exciting job with RStudio.
Unit tests for the exam runtime
This won’t affect most Numbas users directly, but I’m mentioning it here to reassure you that we’re taking efforts to make sure new changes don’t break things!
There are now some automatic unit tests checking the functionality of exams. Previously there were only unit tests covering the JME language.
I’ve set up continuous integration, so that the tests are run automatically each time we commit a change to the runtime’s GitHub repository, and we’re alerted if any of the tests fail. It’s already helped me catch a few breaking changes!
A testing regime is only as good as the tests it carries out, so this will grow to be more useful over time, as we add tests to cover more aspects of the code.
Conflicts between question variables and marking parameters
When a question variable has the same name as a parameter to a part marking algorithm, an error is now thrown.
This has broken some existing questions. The reason for the change is that when an error was not thrown, affected questions could end up marking answers incorrectly because the question variable overrode the marking parameter, or vice versa.
There isn’t a solution to this that doesn’t break something: either we ignore the values of question variables, or we rename marking algorithm variables, with either way potentially breaking custom algorithms that already use them.
In retrospect, I should have made question variables only available through a dictionary, so there can’t be a conflict. Too late now!
If you encounter this error, the solution is to rename the conflicting question variables.
- Thanks to Chris Theron for translating the interface text into Afrikaans!
- You can set an adaptive marking variable replacement for a gap-fill, and it applies to all of the gaps. (issue from 2016!)
- There is an option in the matrix entry part to pre-fill some cells. (issue from 2015!)
- Functions can be marked as ‘deterministic’ – they always produce the same output given the same input. Using this, question variables which can be deterministically reproduced from other variables are not saved to the SCORM suspend data, saving a lot of space. (issue)
- Number values store the precision that they represent, either as decimal places or significant figures. When they’re displayed to the student, this precision is used, including trailing zeros if necessary. (issue)
- There’s a “show penalty hint” option for next part options in explore mode. (issue, documentation)
- You can put 0 in the “initial number of rows/columns” settings for matrix entry parts to use the same dimensions as the expected answer. (issue)
- You can turn off the hint saying “lose N marks” on “next part” buttons. (issue)
- In marking algorithms, you can give the
applyfunction several names of notes to apply. (issue)
- There’s a new JME function
random_integer_partition, to produce a random partition of an integer into a given number of parts. (issue, documentation)
- There is now a display flag
matrixCommasto control whether commas are used between elements in the same row of a matrix or vector. (issue, documentation)
- There are two new simplification rules, turned off by default, called
reduceSurds. (issue, documentation)
- There’s a new JME function,
largest_square_factor, which finds the largest perfect square factor of a given integer. (documentation)
- A JME variable name can have a Greek letter in a subscript. (issue)
- There are two new convenience functions in marking algorithms:
- You can change the names of questions in an exam in “sequential” navigation mode, like you can in “menu” mode. We’ve used this to have an unnumbered “formula sheet” question at the start of exams. (issue)
- The number entry part type now has a “display answer” setting, for cases where a custom marking algorithm means that the expected answer should be displayed differently to the standard format, or when you don’t want to use the mean of the minimum and maximum expected value. (issue, documentation)
- A wider variety of characters representing parentheses is accepted, making input more convenient for users with keyboards in non-English locales. (code)
- There is a new display flag,
timesSpace, which causes implicit multiplication to be rendered as a small empty space instead of the × symbol. (issue, documentation)
- When lists are displayed in LaTeX, including function arguments, the symbol used to separate elements of the list is localised: locales which use a comma for the decimal separator use a semi-colon as the list separator. (issue)
- There’s a new pipe operator,
|>, which can be used to compose transformations of a single value. ( (documentation)
- In exams with a time limit, the remaining time is always visible. (issue)
- “Number entry” parts deal with precision restrictions better when the answer is given in scientific notation: only the precision of the significand is considered. (issue)
- Functions added by an extension are only available in questions using that extension. (issue)
- You can use implicit multiplication after a postfix operator. (issue)
- When a question variable has more than one definition, an error is thrown. (issue)
decimalfunction rewrites all number literals in its definition to be
decimalvalues, so precision isn’t unexpectedly lost by an intermediate calculation in floating-point number values. (issue)
- Fractions with a minus sign on the denominator are accepted. (issue)
- There’s a more helpful error message when a reference is made to another part that doesn’t exist. (issue)
- When resuming an attempt, the check that each question’s name is the same is no longer run: this is not a necessary or sufficient criterion for an updated exam package being compatible with the saved attempt data. (issue)
- In the question preview theme, the page footer is horizontally centred. (code)
- The “display options” button is shown in the footer of the question preview theme. (issue)
- When a part has a single step, that step is not named. (issue)
- The adaptive marking variable replacement routine throws an error if there is a circular reference. (issue)
- The default theme uses the browser’s built-in sans-serif font instead of Source Sans, eliminating an external dependency. (issue)
- In the default theme, just an error message is displayed when the stylesheet doesn’t load. (code)
- In the default theme, the part feedback area no longer has a maximum width or height. (issue)
- The per-item feedback for multiple response parts is displayed in the order in which the corresponding tickboxes are displayed. (code)
- The documentation for each of the built-in part types now describes the data type of the
studentAnswervariable in marking algorithms. (issue)
- There is an error message for the case when a question requires an extension but it hasn’t been included in the exam package. (code)
- The way that values are substituted into the expression in the
\simplifycommand has changed, using substitutions into the compiled expression tree instead of into the JME string. This means in particular that
decimalvalues are displayed nicely, instead of being wrapped in `dec(…)`. (code)
decimalfunction when applied to a number literal does not first parse the number as a JS number object, so no precision is lost. (code)
- When substituting variables into an
expressionvalue, the “plain” number notation is always used, avoiding errors when your locale’s default number notation is not the English one. (issue)
- Improved the accessibility of modal dialogs: the header is read out, and focus is on the start of the text. (issue)
- Variables are substituted into the labels on “next part” option buttons. (issue)
- In menu mode, LaTeX in question names is rendered. (issue)
- When pattern-matching a function name, the scope’s case-sensitivity setting is respected. (issue)
- Simplification rules collecting numbers together avoid incorrectly producing ‘infinity’ when exceeding the maximum value representable by a floating-point number. (issue)
- Error messages in parsing the exam’s XML are escaped, so they’re not accidentally interpreted as HTML. (code)
- Fixed a typo in the “relative difference” checking function which meant it didn’t work when the input numbers have type
- When construction a JME sub-expression with the
expressionfunction, variable substitution into strings uses the JME substitution method instead of the display-text method. (issue)
- Fixed a bug where evaluating a note in a marking script would could lead to variables being saved to the parent scope. (code)
- The JME function
root(-x,n)only returns a real value if
nis odd. (code)
- Fixed a typo in multiple response parts which led to the wrong marking method being used. (issue)
- The JME function
mark_partuses the part’s JME scope, instead of the one in which
mark_partwas called. This avoids variables from the containing scope leaking into the part’s marking. (code)
- Fixed a case where the message for an error produced by a part would sometimes show the part’s name twice. (code)
- I rewrote the algorithm for inferring the types of free variables in an expression. It’s now much faster, and won’t fail on large expressions. (issue)
- Fixed a typo in the definition of
- Fixed a bug when rendering
expressionvalues as LaTeX. (code)
- When rendering an expression as JME code, numbers with a
*symbol in their rendering are surrounded by brackets. This fixes an operation precedence issue when numbers are rendered in scientific notation. (code)
- When rendering
decimalvalues as LaTeX, they are no longer converted to JS numbers first, losing precision. (issue)
- If a gap in a gap-fill part has an adaptive marking variable replacement referencing a part which must be answered but has not, the error message is shown in the part feedback, just like for non-gap parts. (issue)
- When a custom JME function is defined, it always sets the custom
findvarsbehaviour, fixing an issue in the editor where it looked like a parameter name was considered to be an external variable. (issue)
cmi.time_spentkey in attempt SCORM data was incorrectly adding 1 hour to the time spent. The code has been rewritten. (code)
In the exam, question and custom part type editors, the tab navigation runs horizontally at the top of the page, leaving more width for the main editor.
When adding a part, the available part types are shown in a vertical list along with descriptions, instead of as small buttons in a single line.
The tree of parts has been rearranged, with the aim of reducing clutter and making it easier to read.
The part’s marking algorithm and the area to test answers and run unit tests are now in separate tabs.
In the question editor, only one function’s definition is shown at a time, like the variable editor.
When you’re editing a variable, its full value is displayed under the definition. When the variable represents an interactive element, you can interact with the element. This will help when using long lists or dictionaries, and while creating randomised diagrams.
The “Variable testing” tab in the question editor is now accessed from the bottom of the list of parts, instead of being a separate tab on its own.
The editor for custom functions now only shows one function at a time, similar to the variables editor.
The code editor for theme and extension files now occupies the entire viewport, so there aren’t two competing vertical scrollbars.
Web-based first setup
Laura’s first job was to convert the command-line
first_setup.py script used to set up an instance of the Numbas editor to a web-based form.
It looks good!
As a little bonus, instead of using a progress bar while you wait for the installation to finish, I made it show you the factorisation of the elapsed time:
Hopefully this will make it easier for other people to join in with Numbas development.
Easier question testing
There is now a top-level Testing tab in the question editor, showing all unit tests associated with the question’s parts, and a button to run them all at once.
Additionally, each part now automatically gets a unit test checking that the expected answer is marked as correct. (issue)
This should make it much easier to check that questions do what they should.
- When a gap is defined but not used in the part’s prompt, a warning message is shown. (issue)
- The custom part type editor loads all extensions used by the part type, so their functions show up in the code autocompleter. (code)
- The marking matrix field for “match choices with answers” parts now shows the text of the choices and answers. (issue)
- The “Saving/Saved” popup notification in the question and exam editors has been replaced by a permanent badge at the top-right of the screen, so you can always see whether your changes have been saved to the database. (issue)
- In the theme and extension editors, you can drag and drop files onto the buttons to upload or replace a file. (code)
- There’s a little pencil icon next to the name in the question and exam editors, since it’s not obvious that you have to go to the “settings” tab to change it. (code)
- The header on the question/exam preview page is now a neutral grey, instead of scary red. (code)
- In the question editor, the “Statement” and “Advice” tabs have different icons. (issue)
- The custom part type editor now shows when there’s an error loading a required extension. (issue)
- On the list of your extensions, extensions are ordered alphabetically. (issue)
- The way that variables are substituted into \simplify has changed, to improve the rendering of
- The marking matrix editor for “match choices with answers” parts now scrolls if it gets too wide. (issue)
- Fixed a bug where custom constants weren’t found by the routine to identify undefined variables. (code)
- Fixed a bug with applying CSS rules in content previews when the question’s CSS preamble contains an at-rule. (issue)
- The routine for detecting undefined variables is now aware of variables added by an extension. (issue)
- Fixed a bug where the values of locked variables weren’t discarded before regenerating the variable preview. (code)
- Fixed a typo in the embed view which set the viewport scale incorrectly. (code)
- Updated Pyodide to v0.21.0.
- In Python, added a
matplotlib_preamblepre-submit task to clear any output of matplotlib from previous runs. (issue)
- There’s an option to turn off variable substitution in the “expected answer” field of the Code part type, so you can easily use curly braces in code.
- The webR runner queues all blocks of code at once, so that
.Last.valueworks as you’d expect. (issue)
- Errors thrown while executing the postamble are shown in the part feedback. (issue)
- The code editor widget has a minimum height. (code)
- The webR runner removes ANSI escapes from the output, and carriage return characters from code to run. (code)
- Jesse Hoobergs wrote a new compression scheme, reducing the size of the extension. (pull request)
Numbas LTI provider
Numbas lockdown app and Safe Exam Browser
One of Numbas’s main strengths, the entirely client-side exam runtime, is also a weakness when you’re carrying out summative assessments: if the student runs an exam in a normal web browser, they can use the browser’s developer tools to inspect the code and see expected answers, or manipulate their score.
The solution to this is to require students to use a specific locked-down browser app which doesn’t allow access to the underlying code.
For several years, we have used a locked-down browser app to deliver summative Numbas assessments to our students. This app was hard-coded only to work with a Moodle server we set up for the purpose, so we couldn’t share it with other institutions.
Over the last year, we’ve been working on a new Numbas lockdown app which integrates with the LTI provider, so it can be used anywhere.
The Numbas lockdown app will launch automatically when a student launches a Numbas LTI resource which requires it. The app presents a minimal browser interface, and launches straight into the Numbas exam after the student enters a password to decrypt the launch settings.
The app doesn’t need any configuration on the server apart from a default password, which can be overridden. We recommend that it is used for any summative assessment contributing to course credit.
At Newcastle University, we have a policy of using another locked-down browser app, Safe Exam Browser, for in-person invigilated exams. Safe Exam Browser doesn’t run on as many platforms as the Numbas app, but does completely control the computer interface, so students can’t use other programs while completing an exam.
We’ve also added support for Safe Exam Browser to the Numbas LTI provider. A server admin has to create a settings file using the SEB config tool and upload it to the Numbas LTI provider, and then any instructor can require SEB when students launch a particular resource.
- The automatic re-marking tool correctly loads standalone scripts from extensions. (code)
- In the attempt timeline view, each row contains a link to review the attempt as it appeared at that point in time. (code)
- The JSON data export for an attempt uncompresses all the suspend data values. (code)
- I’ve started work on a page in the documentation describing how to move an Ubuntu installation of the LTI provider to Docker.
- Fixed a bug where two records could be created for the same LTI context, when a link from a new context was launched twice in quick succession. (code)
- Added some instructions on connecting to D2L Brightspace.
- Added a note to the installation instructions that you should set up a daily cron job to clear the session cache.
- The routines to compress attempt suspend data now use the diff-match-patch library, which is considerably faster. (code)