One of the most common questions I get from students comes just after they submit their essays. The question is 'how soon do we get our results'? Indeed, I remember on one occasion a student who had submitted his work on a Friday made a formal complaint on the following Monday that he hadn't heard anything back yet.
Although I do try to explain to students what has to happen before they can have their results sent to them, there still seems to be a frequent bemusement about why it has to take so long. So: since we have our Assessment Boards today, I thought I would explain a little about what we get up to after you have handed in your essays.
In the past, there would normally be a week or so of grace after the submission date, while the administration office sorted out all the various scripts and logged them before distributing them to the markers. Now though, as everything is submitted electronically, it is logged and backed up seconds after submission - so markers can get on with it pretty much straight away.
Every marker will have their own method of marking, and the amount of time it takes each one will vary. My own approach involves several stages.
Overall, I would estimate it takes me on average about 40 minutes per script. Having discussed this with many colleagues, it appears that this kind of time-frame is not unusual - many agree that it takes them about the same time, although some (with centuries of experience) are able to get through things more quickly.
Having completed the marking, we move on to the paperwork. Grades need to be collated onto spreadsheets, which will be used to enter them into the College database. A module report needs to be written, in which I respond to issues raised in student module evaluations, outline any problems experienced during the delivery of the module, comment on whether the grades overall are encouraging, or whether there is anything I might suggest should be done differently next time. Finally, I need to put together a 'moderation form', listing the modules details.
All of this paperwork is then sent to an 'internal moderator' - a subject specialist in the College whose job it is to check that my grading is fair. The internal moderator will read a selection of the scripts based on a pre-calculated formula - although often they read more. If they think that everything looks fine, a moderation report is completed and sent back to me for comments. Quite often the moderator will have questions about particular grades, and would prefer to talk through the module marks - so a meeting is set. In such circumstances, the moderator needs to make sure that every script is read - rather than just the prescribed sample - in order to ensure every script is subjected to the same scrutiny. The scripts (or these days the marksheets) are usually spread out on a table among the debris of biscuits and empty red-bull cans and furious debates ensue about whether this script should have a 55% or a 58%.
I remember one meeting with a moderator where we spent 20 minutes deciding between a 62 and a 64. This may sound ridiculous, but we are always conscious of the fact that a single point could effect a students final award - so it needs to be right. If we cannot decide or discover a compromise, then everything is sent off to a third marker for arbitration - but eventually we end up with a bunch of marked scripts and a set of paperwork.
Lovely. But. Some students will have submitted work late. In our College, late work is accepted up to 7 days after the hand-in day, and subject to a penalty (which usually means it cannot get more than 40%). The process of marking and moderation up to this point has probably taken more than a week, but it is generally not until this point that we get the chance to go back and start work on marking the late scripts. Each of these needs to be marked and moderated in exactly the same way as more timely work, although obviously (the hope is) there will always be far fewer scripts this time.
For some lecturers, this is generally the end of the main process - they just have to be ready to answer any questions relating to the work should they occur. A part of my role though, is to liaise with the External Examiners, so I have a little insight into the next stage in the process. External Examiners are subject specialists from other Universities or Colleges whose job it is to review all the marked work, and determine whether the grades they have been given by us, are the same grades they would have been given in their institution.
In order to provide the External Examiners with the right information, in a form which makes it easy for them to understand, we need to collate all the paperwork and all the marked scripts into order. Since the External Examiners will not always know their way around our particular online systems, this generally involves downloading the marked scripts and the feedback from every course. These are then re-named, attached to the relevant spreadsheets, moderation reports, module reports and some other key pieces of paperwork, and either printed out or zipped into folders for emailing or sending on a CD [Ed.: Yes, all encrypted of course!].
This does take considerable time. Indeed, it is probably at least two weeks before everything is finally sent, not including the inevitable number of courses where the marking comes in late because of illness, confusion, extended moderations, etc., etc..
This is where the process tends to turn into something rather farcical - involving running back and forth to the post room with huge parcels of paperwork (enduring the sad look of despair from post officers who can see their monthly budget disappearing before their eyes), or fielding countless emails being returned because they are 'too large'. All the while you are continually harassing markers and moderators for anything which has not been completed, or for pieces of paperwork which are missing. Try as hard as you might to be organised and systematic at this stage, it is impossible to stop it being hectic.
By this stage, we are usually hugging the deadlines tightly - if they have not already slipped from our grasp - and anxiety levels are rising alarmingly. In this marginal time-frame even small problems involve a real sense of panic as we try to resolve them before the Assessment Board: The moderator thinks this module should be re-graded; the marker forgot to add feedback to this script; this late script was not marked; 'sorry, but I seem to have lost the spreadsheet'; the grades for this script are different on the feedback and the spreadsheet; this moderator has gone on holiday so we need to find another one.
While one group of us are experiencing rising blood pressures in one part of the building, in another part there is a group of administrators and technical folks working on transferring the grades into the database - and their blood pressures have already reach 'shard'-like proportions as they endure system crashes, data anomalies, corrupted spreadsheets and heaven knows what other horrors.
The Pre-Board is where the course team meet to review all the paperwork and the grades ahead of the Assessment Board - making sure that all of the data has been entered correctly and that there are no outstanding problems or issues. Generally by this stage you already have a sense of whether there are going to lots, or relatively few problems - but there will always be problems that will need resolving. Often those problems will involve needing to spend hours poring over the College regulations to find out what needs to be done about them (at which point you discover that a particular decision needs the approval of a Committee that doesn't sit for another month), but most of the time it simply involves making corrections to grades/weightings in the database and checking all the calculations.
To be honest, by the time the Assessment Board arrives I tend to have exhausted my anxiety levels and have nothing left. I hope - against hope - that no problems will arise, but at the same time you are relying on the fact that if anything has been missed it will be picked up. If the External Examiners are any good (and ours are excellent) then nothing will get approved without careful scrutiny, and lots of questions. This is one of the reasons why student grades are never released before that have been approved by the Board, because it is only at this stage that grades have been verified both internally and externally, and the data has been checked and double-checked to make sure it is right.
The Assessment Board can be a bit of a slog - so it is not without good reason that one of the first things to check is that coffee and biscuits are in good supply.
This is the point at which I retire to whimper in a corner somewhere, and others continue to ensure the results are sent out. I am utterly (and blissfully) ignorant of the inevitable problems which may arise at this stage, but the net result is that eventually students will receive a transcript with their grades.
At which point that student rolls their eyes and asks 'what took so long?'
Although I do try to explain to students what has to happen before they can have their results sent to them, there still seems to be a frequent bemusement about why it has to take so long. So: since we have our Assessment Boards today, I thought I would explain a little about what we get up to after you have handed in your essays.
Marking
In the past, there would normally be a week or so of grace after the submission date, while the administration office sorted out all the various scripts and logged them before distributing them to the markers. Now though, as everything is submitted electronically, it is logged and backed up seconds after submission - so markers can get on with it pretty much straight away.
Every marker will have their own method of marking, and the amount of time it takes each one will vary. My own approach involves several stages.
- 1) Quickly read through each essay. For essays of around 2-3,000 words this takes probably a little less than 10 minutes for each script. As I read, I will sort the scripts into order based on first impressions - with the weakest script at the top and the strongest script at the top. This used to involve piles of scripts on the floor, but this semester I have developed a method using post-it notes with each scripts number on it, which I can move around on my desk. I do not make any detailed comments at this stage, but I will add comments for obvious things (glaring spelling or referencing errors, inaccuracies, or moments of particular insight).
- 2) Next, I read through the scripts for a second time and a little more carefully. It is important for me to read through things twice, because it there is a danger in giving a grade based on a first impression, and often a second reading changes my thoughts about a script. I will add more detailed comments about the content of the work, and as I read I will start to build up my overall feedback. Once I have finished the essay I will give a tentative grade, and then revise the general feedback to ensure that it justifies that grade, gives reasons why it does not merit a higher grade, and gives explicit directions about how that grade could have been improved. This process takes about 25 minutes per script, although more problematic ones (those with problems of expression or those which are a little confused) can take a lot longer.
- 3) Having done this I will have a bunch of post-it notes with grades on them, and the final stage involves running through each script in order, reminding myself of them and checking the grade scale. This is the point where grades can change in relation to each other. For example, if one script has been given 62, another 65 and another 68, this is the point where I ask myself 'is there really the same difference between each one?' Or I might ask 'is this really 6% worse than that one?'
Overall, I would estimate it takes me on average about 40 minutes per script. Having discussed this with many colleagues, it appears that this kind of time-frame is not unusual - many agree that it takes them about the same time, although some (with centuries of experience) are able to get through things more quickly.
Having completed the marking, we move on to the paperwork. Grades need to be collated onto spreadsheets, which will be used to enter them into the College database. A module report needs to be written, in which I respond to issues raised in student module evaluations, outline any problems experienced during the delivery of the module, comment on whether the grades overall are encouraging, or whether there is anything I might suggest should be done differently next time. Finally, I need to put together a 'moderation form', listing the modules details.
Moderation
All of this paperwork is then sent to an 'internal moderator' - a subject specialist in the College whose job it is to check that my grading is fair. The internal moderator will read a selection of the scripts based on a pre-calculated formula - although often they read more. If they think that everything looks fine, a moderation report is completed and sent back to me for comments. Quite often the moderator will have questions about particular grades, and would prefer to talk through the module marks - so a meeting is set. In such circumstances, the moderator needs to make sure that every script is read - rather than just the prescribed sample - in order to ensure every script is subjected to the same scrutiny. The scripts (or these days the marksheets) are usually spread out on a table among the debris of biscuits and empty red-bull cans and furious debates ensue about whether this script should have a 55% or a 58%.
I remember one meeting with a moderator where we spent 20 minutes deciding between a 62 and a 64. This may sound ridiculous, but we are always conscious of the fact that a single point could effect a students final award - so it needs to be right. If we cannot decide or discover a compromise, then everything is sent off to a third marker for arbitration - but eventually we end up with a bunch of marked scripts and a set of paperwork.
Late Work
Lovely. But. Some students will have submitted work late. In our College, late work is accepted up to 7 days after the hand-in day, and subject to a penalty (which usually means it cannot get more than 40%). The process of marking and moderation up to this point has probably taken more than a week, but it is generally not until this point that we get the chance to go back and start work on marking the late scripts. Each of these needs to be marked and moderated in exactly the same way as more timely work, although obviously (the hope is) there will always be far fewer scripts this time.
External Examiners
For some lecturers, this is generally the end of the main process - they just have to be ready to answer any questions relating to the work should they occur. A part of my role though, is to liaise with the External Examiners, so I have a little insight into the next stage in the process. External Examiners are subject specialists from other Universities or Colleges whose job it is to review all the marked work, and determine whether the grades they have been given by us, are the same grades they would have been given in their institution.
In order to provide the External Examiners with the right information, in a form which makes it easy for them to understand, we need to collate all the paperwork and all the marked scripts into order. Since the External Examiners will not always know their way around our particular online systems, this generally involves downloading the marked scripts and the feedback from every course. These are then re-named, attached to the relevant spreadsheets, moderation reports, module reports and some other key pieces of paperwork, and either printed out or zipped into folders for emailing or sending on a CD [Ed.: Yes, all encrypted of course!].
This does take considerable time. Indeed, it is probably at least two weeks before everything is finally sent, not including the inevitable number of courses where the marking comes in late because of illness, confusion, extended moderations, etc., etc..
This is where the process tends to turn into something rather farcical - involving running back and forth to the post room with huge parcels of paperwork (enduring the sad look of despair from post officers who can see their monthly budget disappearing before their eyes), or fielding countless emails being returned because they are 'too large'. All the while you are continually harassing markers and moderators for anything which has not been completed, or for pieces of paperwork which are missing. Try as hard as you might to be organised and systematic at this stage, it is impossible to stop it being hectic.
The Panic Stage
By this stage, we are usually hugging the deadlines tightly - if they have not already slipped from our grasp - and anxiety levels are rising alarmingly. In this marginal time-frame even small problems involve a real sense of panic as we try to resolve them before the Assessment Board: The moderator thinks this module should be re-graded; the marker forgot to add feedback to this script; this late script was not marked; 'sorry, but I seem to have lost the spreadsheet'; the grades for this script are different on the feedback and the spreadsheet; this moderator has gone on holiday so we need to find another one.
While one group of us are experiencing rising blood pressures in one part of the building, in another part there is a group of administrators and technical folks working on transferring the grades into the database - and their blood pressures have already reach 'shard'-like proportions as they endure system crashes, data anomalies, corrupted spreadsheets and heaven knows what other horrors.
The Pre-Board
The Pre-Board is where the course team meet to review all the paperwork and the grades ahead of the Assessment Board - making sure that all of the data has been entered correctly and that there are no outstanding problems or issues. Generally by this stage you already have a sense of whether there are going to lots, or relatively few problems - but there will always be problems that will need resolving. Often those problems will involve needing to spend hours poring over the College regulations to find out what needs to be done about them (at which point you discover that a particular decision needs the approval of a Committee that doesn't sit for another month), but most of the time it simply involves making corrections to grades/weightings in the database and checking all the calculations.
The Assessment Board
To be honest, by the time the Assessment Board arrives I tend to have exhausted my anxiety levels and have nothing left. I hope - against hope - that no problems will arise, but at the same time you are relying on the fact that if anything has been missed it will be picked up. If the External Examiners are any good (and ours are excellent) then nothing will get approved without careful scrutiny, and lots of questions. This is one of the reasons why student grades are never released before that have been approved by the Board, because it is only at this stage that grades have been verified both internally and externally, and the data has been checked and double-checked to make sure it is right.
The Assessment Board can be a bit of a slog - so it is not without good reason that one of the first things to check is that coffee and biscuits are in good supply.
Sending the Results
This is the point at which I retire to whimper in a corner somewhere, and others continue to ensure the results are sent out. I am utterly (and blissfully) ignorant of the inevitable problems which may arise at this stage, but the net result is that eventually students will receive a transcript with their grades.
At which point that student rolls their eyes and asks 'what took so long?'
Lol, top stuff mate. as long as you take your time I'm happy with the 40 mins, the grade is above 70% then i don't have a problem with you taking your time and ensuring the bureaucratic process of marking is adhered to ;)
ReplyDeleteNoted. I'm sure that would make it easier for everyone. Will take it to the Board.
ReplyDeleteDo students realise that most of the marking is done at weekends, evenings (occasional all nighters) and holidays resulting in something close to a brain meltdown?
ReplyDelete