This should not be “fresh news” to anyone in TESOL. After all, we have all heard and read about how ETS has started to use software that automatically grades student essays here, here, and here …. It’s not just ETS, but professors at some colleges are starting to let students improve their papers by using software graders before they submit them. But of course, the final grading is still done by the professors, atleast for now. Most articles you read will tout the ability of such software and throw techno-linguistic jargon at you. Is it fact or fiction? Can computers really grade essays? What is this LSA stuff they talk about? Can students fool the computer into giving a good grade?

Statement: Yes, computers can really grade essays, it’s a fact and it is inevitable that such “intelligent” software will become more pervasive in the future.

That’s what you will read about and hear from the researchers and companies that have developed them. That’s what they want you to hear. I agree with the above statement …. I agree with the experimental results that show a high degree of agreement between human judges and the computer software …. however, I would caution readers who are not familiar with computational linguistics, that such software do not really/fully understand what is written, the current state-of-the-art that is. Most are based on LSA (Latent Semantic Analysis), which is “A mathematical/statistical technique for extracting and representing the similarity of meaning of words and passages by analysis of large bodies of text.” (http://lsa.colorado.edu/Boulder) As stated in the definition, it is at best a statistical technique that measures the similarity of words. Such a technique is then used to compare the students essay to a gold-standard, which is one or more essays written by an expert. Note that LSA measures similarity between words; it does not build a computational semantic model of the underlying meaning of the sentences etc. in the essay and hence does not fully understand what is being conveyed in the essay. Nevertheless, this technique has proven to be effective.

So, can students fool the computer? A well trained student who understands how the computer grades the paper will likely be able to hack a good grade. With systems that use LSA, one could possibly still score well by writing an essay that does not make much sense but still contains the set of “words” of ”concepts” etc. required for a given topic. More sophisticated methods are currently being explored that would handle these cases and have perhaps been implemented in some of these systems. LSA also does not take into account syntax, i.e. grammar. This leads us to the question of grading ESL essays automatically:

Turns out, taking account of syntax/grammar is actually simpler than making sure the semantics are intact. Syntactic parsers that perform fairly well exist. Such parsers could be and has been used to check for grammar in essays. LSA could still be used to make sure that the ESL student has written about a topic that is similar to the one stated in the question. The strength of the vocabulary used can also be measured by assigning levels to the words in the lexicon/dictionary.

The current generation of these software grade essays in GMAT, physics, etc. The next generation of e-Raters, computer graders, automated essay graders, … whatever you may want to call them, will most likely fill the shelves of Best Buy and find themselves into the PC’s of many students who will use them to improve their essay writing skills without the need for an ESL tutor. or will they? Will you use them if they tend to score according to your own judgement/evaluation to grade your students essays?