Not so long ago, the examination season was one of the most dreaded periods in the academic calendar for Alias Gitau, the Examination Registrar at the Nyeri National Polytechnic and his team.
Delivering credible assessments seems like a straight forward and easy task but not for Examiners who had to spend numerous hours personally ensuring that each of the more than 8400 assessment tools that the institution prepares per academic year meet the highest standards. For years, the process of generating Continuous Assessment Tests (formative assessment) and End of Term Exanimations (summative assessment) at the polytechnic was manual. The only digital process was type-setting and printing out the tools which doesn’t require a lot of time.
For an assessment to be conducted, a panel comprising internal trainers and industry experts would have to be constituted. The team of industry experts would be the first to sit for days, set the assessment tools and then the panel from the exam registrar’s office would follow suit and validate the assessment tools to ensure that the exams were aligned with the set standards.
“The process would start at the beginning of the academic year when Heads of Departments would submit their course planner which details the number of units per course and the total number of lessons to be taught under that course. The Examinations Office would then prepare the exam schedule, constitute the panels, and organise an event where the tools would be set and validated,” he says.
However, that changed four months ago when the institution decided to embrace technology to revolutionize its exam cycle. Gitau says that they partnered with a local developer to come up with an AI powered Assessment Generating System which they aptly named AGEX.
AGEX works using prompts and configurations. Gitau defines the system as a game-changer as it is tailored to eliminate the time-consuming process of having to physically sit and manually set the tools.
The process is largely the same but the only difference is, unlike many AI systems which scour internet for information and generate responses, the AGEX system fed with the variables it needs -such as the units to be taught per course during the term, the occupational standards which are set by the Kenya National Qualifications Authority, the curriculum and the learning guide which is developed by the trainers- as its knowledge base and point of reference.
“The objective is to ensure that the system does not generate assessments outside the scope of the knowledge base,” says Gitau.
Before generating an assessment tool, Gitau says that the system is given guidelines in the form of configurations to help it establish whether it is expected to generate a CAT or a main examination. The guidelines are configured to help the system allocate even finer details such as the duration of a CAT or the main exam, the minimum and maximum number of questions to set per section, as well as the marks to be allocated per question.
“For instance, if you want to generate a CAT, you just need one prompt for the AGEX system to retrieve all the information it needs from the knowledge base and it will give you a tool for every course being offered in the college. With that one command, the system is able to limit itself to the topics that have already been covered. In the case of a summative assessment, the system analyses the entire learning guide per course and uses it to generate a conclusive tool,” explains the registrar.
“The system is configured in way that it automatically selects the most suitable framing language for each level of study so that when for instance we want to generate say a Level V (the Diploma Level) assessment tool, it should avoid verbs like ‘list’ and use verbs like ‘elaborate’, ‘sketch’ or ‘explain’ based on the level of study. These guidelines are fed to the system and it has been trained to ensure that whenever it receives a command to generate a tool, the outcome should align with those requirements,” he continues.
Not only does the AGEX system generate the exams but also their marking schemes. Once the process is complete, the soft copy versions of the tools are sent to the printer to generate the hard copies. Gitau says that the moderation of the final outcome is still a manual process and this is where the trainers verify the questions are in line with the curriculum.
Gitau and his two colleagues Kelly Buruku and Daniel Mwanzia have been training the system for the last three months to ensure optimal performance. Gitau says that besides cutting back on the time spent setting assessments; the quality of the tools has significantly improved.
“The beauty of using the system is that unlike in the past where we would go back to manually setting a fresh tool after it has been rejected, with AGEX you keep querying the system until you come up with a high-quality exam,” he says.
Besides streamlining the examination process for the polytechnic, Gitau says that the AGEX system would also assist the examination office in automating other processes such as generating the examination timetable, a process that he says requires a lot of precision. The polytechnic is also hoping to use the AGEX System to analyse assessment results and use the findings to inform policy decisions in the institution.
“The system can help generate in-depth performance reports that can show us details such as enrollment and completion rates and such usually take days to generate. The system will take off the load from the trainers who have to do such analyses manually and the end result will be more detailed reports,” says Gitau.
But for Dr David Mwangi, the Polytechnic’s Principal, one of the biggest advantages of adopting the AI system is that the institution will tremendously reduce its annual examination expenditure. According to Dr Mwangi, before the advent of the AGEX system, the institution would spend about Sh7 million per term in generating assessment but with the system up and running, the budget could go down to about Sh2 million per academic year.
“Assessment tools are a big business in the national polytechnics. Here at the Nyeri National polytechnic, were prepare 2800 assessments tools. For every assessment tool that we prepare, we pay Sh2500 so by the end of the term, we part with Sh7 million,” said the Principal.
“But through the AI system and with a push of a button, we will be able to reduce that expenditure from Sh21 million to less than Sh2 million per year,” he added.
By Wangari Mwangi
