{"id":864,"date":"2015-07-23T14:21:07","date_gmt":"2015-07-23T14:21:07","guid":{"rendered":"http:\/\/www.numbas.org.uk\/?p=864"},"modified":"2015-07-23T14:21:07","modified_gmt":"2015-07-23T14:21:07","slug":"adaptive-marking-based-on-answers-to-other-question-parts","status":"publish","type":"post","link":"https:\/\/www.numbas.org.uk\/theme-testing\/blog\/2015\/07\/adaptive-marking-based-on-answers-to-other-question-parts\/","title":{"rendered":"Adaptive marking based on answers to other question parts"},"content":{"rendered":"<p>A long-standing limitation of Numbas has been the inability to offer &#8220;error-carried forward&#8221; marking: the principle that we should\u00a0only penalise them once for making an error in an early part. When calculations build on each other, an error in the first part can make all the following answers incorrect, even if the student follows the correct methods from that point onwards.<\/p>\n<p>Numbas now has a system of adaptive marking enabled by\u00a0the replacement of question variables with the student&#8217;s answers to\u00a0question parts.<\/p>\n<p><!--more-->It&#8217;s hard to pin down exactly what &#8220;error-carried forward&#8221; marking means &#8211; what kinds of errors will you\u00a0carry forward, and can you detect them? Once you&#8217;ve decided to take an error into consideration, how do you change the way you mark following\u00a0question parts? Because I could never come up with a good way of thinking about these problems, I didn&#8217;t make any progress on\u00a0designing a system to allow adaptive marking in Numbas.<\/p>\n<p>A few\u00a0months ago, I\u00a0came up with a simple abstraction of a circumstance in which you would want to\u00a0adapt marking based on an error in an earlier part:<\/p>\n<ul>\n<li>The student is presented with some data and asked to perform a calculation whose result they enter in part <b>a<\/b>.<\/li>\n<li>The student is then asked to use that\u00a0value in a further calculation, whose result they enter in part\u00a0<strong>b<\/strong>.<\/li>\n<li>If the student&#8217;s value for\u00a0<strong>a<\/strong> is wrong, then their value for\u00a0<strong>b<\/strong> will probably also be wrong. But the method used to obtain\u00a0<strong>b<\/strong> from\u00a0<strong>a<\/strong>\u00a0should be independent of\u00a0<strong>a<\/strong> (or, for any\u00a0value of\u00a0<strong>a<\/strong>, there is a clear method of obtaining a value of\u00a0<strong>b<\/strong>). So, if the method applies the method correctly and proves that by giving the appropriate value, they should be given\u00a0full credit for part\u00a0<strong>b<\/strong>.<\/li>\n<li>Part\u00a0<strong>b<\/strong> is then assessing only the application of the method, and the student only loses credit in part\u00a0<strong>a<\/strong>. It&#8217;s clear from the marks awarded where the student went wrong.<\/li>\n<\/ul>\n<p>That abstraction doesn&#8217;t consider\u00a0<em>what<\/em> error the student made: that&#8217;s a very hard problem! You could come up with a marking scheme where some errors are penalised more heavily than others (getting a sign wrong in a formula isn&#8217;t as bad as using the wrong formula entirely, for example), but I&#8217;ll solve the easier problem first: ensure that it&#8217;s clear what each question part is assessing, and to make them as independent as possible.<\/p>\n<p>If part\u00a0<strong>b<\/strong> is assessing a method, then we should be able to write down the\u00a0result of that method as a function of the value assessed in part\u00a0<strong>a<\/strong>, and any other information given to the student. (If we can&#8217;t write down this function, then this method can&#8217;t be assessed by a computer and we should\u00a0go home!) At this point, we start looking at the question variables. If there&#8217;s a variable used in the definition of the answer to <strong>b<\/strong>\u00a0which corresponds to the expected answer to part\u00a0<strong>a<\/strong>, then we can carry forward the student&#8217;s error by replacing the value of that variable with the student&#8217;s answer, and recomputing the expected\u00a0answer to part\u00a0<strong>b<\/strong>.<\/p>\n<p>I <a href=\"https:\/\/github.com\/numbas\/Numbas\/issues\/368\">wrote this down in our GitHub issue tracker<\/a>. After a few months working on\u00a0other stuff, I finally got round to implementing it a few weeks ago.<\/p>\n<p>It works really well!<\/p>\n<p>Each question part now has an &#8220;Adaptive marking&#8221; tab, where you can specify a list of variables to replace with the answers to other question parts. When the student submits an answer to the part, the values of the specified variables are replaced with the student&#8217;s answers to the corresponding parts, and then any other question variables depending on those parts are recalculated automatically, before the new &#8220;correct answer&#8221; to the part is calculated. You can decide whether to also accept the original correct answer, or to always make the variable replacements to ensure that\u00a0student&#8217;s answers are\u00a0consistent with each other.<\/p>\n<p>The\u00a0variable replacement is very powerful &#8211; changing one variable can affect many others. This makes it very easy to add adaptive marking to a question &#8211; since Numbas does all the recalculation for you, if your question\u00a0is structured well it&#8217;s a simple matter of\u00a0matching up variables with parts, and then all of the other intelligent stuff Numbas does to mark the student&#8217;s answer carries on as normal.<\/p>\n<p>Here&#8217;s a video showing how I added adaptive marking to a\u00a0fairly complicated question on hypothesis testing:<\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/player.vimeo.com\/video\/134209217\" width=\"500\" height=\"281\" frameborder=\"0\" title=\"Numbas adaptive marking\" webkitallowfullscreen mozallowfullscreen allowfullscreen><\/iframe><\/p>\n<p>This feature is now available in the public\u00a0Numbas editor. I&#8217;ve created <a href=\"https:\/\/numbas.mathcentre.ac.uk\/exam\/2134\/adaptive-marking\/\">a small demo exam<\/a> to show off a few ways you can use adaptive marking &#8211;\u00a0among other things, it shows that variable replacements don&#8217;t just have to be numeric values. I&#8217;ve also written\u00a0some <a href=\"http:\/\/numbas-editor.readthedocs.org\/en\/latest\/question-parts.html#adaptive-marking\">documentation<\/a> to explain how to use this feature in your own questions.<\/p>\n<p>This system has its limitations &#8211; I&#8217;ve mentioned that it can&#8217;t determine\u00a0<em>how<\/em> a student made an error, which would be very useful for feedback; and it assumes that the student will be consistent in what they write, which\u00a0you might not want to do &#8211; however, I believe it will prove to be a very useful addition to Numbas.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A long-standing limitation of Numbas has been the inability to offer &#8220;error-carried forward&#8221; marking: the principle that we should\u00a0only penalise them once for making an error in an early part. When calculations build on each other, an error in the first part can make all the following answers incorrect, even if the student follows the [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[10],"tags":[21],"_links":{"self":[{"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/posts\/864"}],"collection":[{"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/comments?post=864"}],"version-history":[{"count":3,"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/posts\/864\/revisions"}],"predecessor-version":[{"id":867,"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/posts\/864\/revisions\/867"}],"wp:attachment":[{"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/media?parent=864"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/categories?post=864"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.numbas.org.uk\/theme-testing\/wp-json\/wp\/v2\/tags?post=864"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}