We’ve seen a few cases of genAI-generated .exam files on the Numbas editor. At present, generative AI tools do not have the innate ‘knowledge’ to make an exam file and will often lead to invalid or non-functional files.
If you are choosing to use generative AI to generate numbas exams, ensure that you provide links to the schema and documentation, and encourage the model to add appropriate descriptions to every variable and the question itself, including a note that it was generated by a genAI model.
Skill-enabled LLMs can use the json skill and the schema file to validate the schema, but can miss details on substituting in variables and marking algorithms, so you should always ensure to thoroughly test both correct and incorrect answers and look for edge cases, as well as ensuring question content itself makes sense.
We don’t plan to incorporate LLMs into Numbas.