By Dr. Nicholas Parker.
a. Introduction
Since 2008 the School of Mathematics and Statistics has incorporated computer-based assessments (CBAs) into its summative, continuous assessment of undergraduate courses, alongside conventional written assignments. These CBAs present mathematical questions, which usually feature equations with randomized coefficients, and then receive and assess a user-input answer, which may be in the form of a numerical or algebraic expression. Feedback in the form of a model solution is then provided to the student.
From 2006 until the last academic year (2011/2012), the School employed the commercial i-assess CBA software. However, this year (2012/2013) the School rolled out a CBA package developed in-house, Numbas, to its stage 1 undergraduate cohort. This software offers greater control and flexibility than its predecessor to optimize student learning and assessment. As such, this was an opportune time to gather the first formal student feedback on CBAs within the School. This feedback, gathered from the stage 1 cohort over two consecutive years, would provide insight into the student experience and perception of CBAs, assess the introduction of the new Numbas package, and stimulate ideas for further improving this tool.
After an overview of CBAs in Section b and their role in mathematics pedagogy in Section c, their use in the School of Mathematics and Statistics is summarized in Section d. In Section e the gathering of feedback via questionnaire is outlined and the results presented. In Section f we proceed to analyze the results in terms of learning, student experience, and areas for further improvement. Finally, in Section g, some general conclusions are presented.
b. A Background to CBAs
Box 1: Capabilities of the current generation of mathematical CBA software.
- Questions can be posed with randomized parameters such that each realization of the question is numerically different.
- Model solutions can be presented for each specific set of parameters.
- Algebraic answers can be input by the user (often done via Latex commands), and often supported by a previewer for visual checking
- Judged mathematical entry (JME) is employed to assess the correctness of algebraic answers.
- Questions can be broken into several parts, with a different answer for each part.
- On top of algebraic/numerical answers, more rudimentary multiple-choice, true/false and matching questions are available.
- Automated entry of CBA mark into module mark database.
Computer-based assessment (CBA) is the use of a computer to present an exercise, receive responses from the student, collate outcomes/marks and present feedback [10]. Their use has grown rapidly in recent years, often as part of computer-based learning [3]. Possible question styles include multiple choice and true/false questions, multimedia-based questions, and algebraic and numerical “gap fill” questions. Merits of CBAs are that, once set up, they provides economical and efficient assessment, instant feedback to students, flexibility over location and timing, and impartial marking. But CBAs have many restrictions. Perhaps their over-riding limitation is their lack of intelligence capable of assessing methodology (rather CBAs simply assess a right or wrong answer). Other issues relating to CBAs are the high cost to set-up, difficulty in awarding of method marks, and a requirement for computer literacy [4].
In the early 1990s, CBAs were pioneered in university mathematics education through the CALM [6] and Mathwise computer-based learning projects [7]. At a similar time, commercial CBA software became available, e.g. the Question Mark Designer software [8]. These early platforms featured rudimentary question types such as multiple choice, true/false and input of real number answers. Motivated by the need to assess wider mathematical information, the facility to input and assess algebraic answers emerged by the mid 1990s via computer-algebra packages. First was Maple’s AIM system [5, 14], followed by, e.g. CalMath [8], STACK [12], Maple T. A. [13], WebWork [14], and i-assess [15]. This current generation of mathematics CBA suites share the same technical capabilities, summarized in Box 1.
Read the rest
We’re giving a day-long workshop titled “Building online maths assessments using Numbas” at the University of York on the 4th of July. It’ll be an introduction to using Numbas, from logging on to the mathcentre editor, through selecting questions to make a test, to eventually writing your own questions.
The workshop is provided by the Sigma North East network for excellence in mathematics and statistics support, and attendance is free.
There’s more information, and a booking form, on the Sigma NE event page.
I’ve just released v1.5 of Numbas on Github. While there have been loads of changes since the last time I remembered to bump the version number up, the biggest change recently is that I have rewritten the default theme to use the knockout.js framework. It makes the underlying code a lot simpler, and allows us to do a few new things that would have been very complicated. In particular, there is now a Review mode which is made available when you have finished an exam – you can click on any question on the feedback page to go back to it and compare your answers with the expected answers, as well as seeing any marking feedback and the model solution in the Advice section.
The version numbers in the Numbas source code repository don’t mean too much since we push updates to the stable branch as soon as they’ve been tested instead of lumping them together, but it’s good to mark progress every now and then.
The mathcentre editor tracks the stable branch, so you can try the new features there now.
The Review mode was something we had in the previous system we used at Newcastle, and it was the one thing that many of the students asked for in a survey conducted at the end of the first semester by Dr. Nick Parker. I’m glad it’s finally in Numbas!
When you finish an exam, you can click on any question to review it.
Marking feedback and the correct answers are shown for each part.
And the model solution given in the Advice section is also revealed.
I’ve just deployed an update to the Numbas editor which makes searching a bit easier, and gives everyone a user profile.
I’ve replaced the old question/exam index pages with a new system which is a lot quicker when the database contains a lot of items (for example, the mathcentre database, which currently contains over 600 questions).
It’s now possible to filter questions by tag, which should make organising and finding questions you’re working on a bit easier. The tags on the question editing page are now clickable, so you can easily find related questions.
Every user has a profile page where they can provide a bio, along with links to their exams, questions, and the tags they’ve used. You can edit your bio by clicking on the “edit your profile” link on your profile page.
Thanks to Joshua Beals for giving me a prod to sort out the searching interface – it was beginning to creak a bit!
You can try out the new features at the mathcentre Numbas editor.
Many people have asked for the ability to hide questions from public view, and to share editing privileges with their colleagues. I spent last week implementing sharing and access controls on both exams and questions in the Numbas editor.
To define who can see and edit your questions, click on the Access tab on their edit pages. For an explanation of the various settings, see the Numbas documentation.
We’ve finally put a news item on the mathcentre front page that the Numbas editor is available for anyone to use. We’ve uploaded over 300 of our own questions to the database for you to pick and choose from, or you can create your own questions using the browser-based editor.
If you just want to run some questions for revision, we’ve put some tests on a few first-year maths topics to Numbas in the main mathcentre database; you can find them by searching for “Numbas” at mathcentre.ac.uk.
Over the Summer we had a project, funded through HESTEM, to put Numbas material and the new editor on Loughborough University’s excellent mathcentre site.
We’ve run quite a bit behind schedule for various reasons but we now have some material on the site for students to use, and a fully functional question editor for teachers, stocked up with over 300 questions from our database developed over the past few years here at Newcastle.
The tests for students are available under the “Test Yourself” header for a few topics. For example, this test on completing the square allows students to try randomly generated questions on the same subject over and over until they feel happy with it. This is exactly the kind of thing e-assessment is good at: repetitive practice at basic skills at the student’s pace.
The Numbas editor is a much bigger deal. It’s online at numbas.mathcentre.ac.uk. It allows a few levels of interaction:
- Anyone can browse the database of questions and tests on the site, and download packages for uploading to VLEs or wherever.
- If you create an account on the site, you can select questions to make into exams suited for your particular needs, or…
- You can write your own questions, using the same tools we use to make our highly successful in-course assessments.
We’re still working on the documentation, and the interface is still being tweaked, but we think that the material and tools we’re making available will be of real use to both students and teachers. If you’ve got any questions, please write a comment here or send an email to numbas@ncl.ac.uk.
Bill has written this report about how Numbas is being used this year for a University committee. I thought it was worth sharing.
Numbas will be used for all first year modules and service teaching in Maths & Stats in 2012/2013. This is in formative mode with an in-course assessment component. Will be extended to all second year modules in 2013/2014. Note that presently this is the most extensive use of formative e-assessment in UK HE and is based on our original award winning use of formative e-assessment.
Transfer to other Universities
Two universities are transferring the technology from Newcastle as part of an HE STEM project: Chemistry at Bradford and Maths at Kingston. This will be finished by August 2012.
Birmingham University are talking to us in early April about using Numbas in their foundation courses and in their maths support system.
There is now a new project starting April 2012 to embed Numbas in mathcentre.
OER and Numbas
We are running an HEA/JISC workshop on preparing OER materials using Numbas on April 10.
This is free and open to all.
We are developing new materials for Maths-Aid using Numbas as part of e-learning and support packages. Also preparing materials and resources (eg DVDs) for revision.
Date: 10 Apr 2012
Time: 09:30 – 16:00
Location: Herschel Building Newcastle University Newcastle upon Tyne Tyne and Wear , UK, NE1 7RU
Booking: The Higher Education Academy website
Contact email: Bill Foster
This event is being hosted as part of the Higher Education Academy’s OER Workshop and Seminar Series Read the rest
With version 9.1 SP6, Blackboard finally fixed their SCORM player. That means that you can upload Numbas exams to Blackboard and it will save students’ grades into the Grade Center, just like any other assignment. Earlier versions claimed to support SCORM, but the implementation was quite broken.
In this post I’ll walk you through the process of putting a Numbas exam on Blackboard. Read the rest
