1 Matthieu Hermet, Stan Szpakowicz Automated Analysis of Students Free-text Answers for Computer- Assisted Assessment University of Ottawa, Canada

2 CAA for CALL To address the specificity of CALL… where student material contains syntactic and orthographic errors …with minimal pre-encoded material : –Content validation : simple –Form validation : difficult A good case for automating based on Natural Language Processing

3 Text comprehension The uOttawa project: CALL solutions for helping French-as-a-Second-Language students to enhance autonomous reading comprehension master the structure of text in order to understand the authors discursive intention guess the meaning of unknown words develop reformulation and synthesis capabilities

4 DidaLect …is a FSL tool aimed at teaching autonomous reading skill (designed for intermediate- and advanced-level students) Is an instance of eLearning Intelligent Tutoring System: –adaptation to individual students skills and agenda –access to external resources (dictionaries) –built to reflect the cognitive concerns such as matching feedback to the students behaviour

5 Intelligence in DidaLect DidaLect begins its operation with a placement test to determine a students initial level: –varying order of questions to pick up the best of a students skill –the implementation includes fuzzy logic methods A separate element of DidaLect is the processing of free-text answers: –need of a robust CAA component –a trade-off between symbolic processing and Machine Learning techniques

6 Free-text answer assessment The problem is to know in advance what material to expect in student answers. Usually implemented as a classification problem: a student answer must match reference answer(s). Size and form of reference material affects the process Here, a reference answer is the text itself A case for trying symbolic processing using techniques of Computational Linguistics

7 Expected limitations No possibility of modifying the size and form of the reference material, except by automatic processing to control reformulation. Therefore, this only works for limited forms of questions. –Strong need to ground selection of questions in a firm didactic theory. –Questions on texts for Text Comprehension (didactics offers a classification of question types).

8 Question types 1 Text-Explicit: based on a single sentence « d'habitude, l'hermaphrodisme frappe surtout les mâles, qu'elle dote de simulacres d'appareils génitaux féminins. » Q : Quarrive-t-il aux ours mâles lorsquils sont frappés dhermaphrodisme ? Ex R : « Ils ont les génitaux féminins » Text-Implicit : based on two co-referenced sentences « le détecteur de décélération situé à l'avant du véhicule génère instantanément un courant électrique, qui déclenche une amorce, qui elle-même enflamme un mélange allumeur. Ce dernier met finalement le feu à l'agent propulseur responsable du gonflement du coussin. » Q : Quelle est la réaction en chaîne qui se produit lorsque survient un impact ?

9 Question types 2 Identification, cause-effect, goal, comparison, definition, instrumental… These categories express linguistically through lexical connectors Goal : for, so that, in order to… Cause–Effect : because, therefore… So, the control of reformulation can be automated

10 Processing 1.Find lexical differences between the students answer S and reference R 2.Parse S and R, produce dependency relations 3.Process different words (using a dictionary) to detect synonyms 4.Control of syntax in S 5.Control of reformulation in S wrt R

11 Tools A robust parser that enables partial recovery from errors in students answer A dictionary of synonyms A derivational dictionary Locally derived resources: –State and action verbs –Ensemble of typical errors, set of syntactic and reformulation structures

12 Semantics and synonyms Examine word set differences and commonalities in search for: –Common words –Reformulated words –Different words Detect synonyms accross parts of speech: 1.Derive forms for a word lemma 2.Search synonyms for each form and look for a match in Word Sets

13 Syntax and reformulation Correct syntactical structures to verify syntax of students answer Lexicalized reformulation structures to verify discourse conformity Ex : pollution has increased with the rise of transportation Q : Why has pollution increased ? Ans : With the rise of transportationis partially wrong Because of the rise of transportation OR it has increased due to the rise of transportation ETC.

14 Parsing and tree-building S : Le cardio-vasculaire d'un rat s'approche à une personne humain. SUBJ(, ) OBJ(, ) VMOD_POSIT1(, ) NMOD_POSIT1(, ) PREPOBJ(, ) DETERM(, ) The above are incrementally recomposed, based on lexical selection that maximizes promise of discovering material which diverges from R. That material is processed in parallel, in a similar fashion: SUBJ(,, )>)>,, )>, )>)

15 Main types of heuristics To address syntactic correctness and/or equivalence between S and R: the same sense but different structures bank of typical errors and correct structures To address discursive variations, detected as supplementary material bank of state and action verbs: action verbs must be present, possibly reformulated To address, partially, errors in S word replacement to relaunch parsing when stopped due to lexical mistakes

16 Reformulation rules Examples : Abstraction : incidence sur le temps de gestation incidence sur la possibilité davoir une gestation écourtée (words like fact, chance, etc.) Cause-Effect : Le plasma augmente et dilue les paramètres chimiques Laugmentation du plasma dilue les paramètres chimiques Is-A : Le rat est un animal qui + S Le rat + S Attribute : Le rat possède un système cardio- vasculaire Le système C-V du rat

17 Assessment Must give student feedback on: –Agreement and orthography –Syntax: signal errors and provides correction via display of a correct structure –Semantics: signals error and provides admissible words –Completeness of content with respect to R

18 Example R: Et puisque le rat est un animal qui possède un système cardio- vasculaire très semblable à celui de lhumain, il est donc permis de tirer les mêmes conclusions pour lhumain. Q: Pourquoi peut-on tirer les mêmes conclusions pour l'humain et pour le rat ? S: Le cardio-vasculaire dun rat sapproche à une personne humain. Start by creating wordlists : 1.Words of S absent in R sapprocher, personne 2.Words of R absent in S animal, posséder, système, semblable 3.Common words rat, cardio-vasculaire, humain

19 Parse (partial output) S: SUBJ(, ) OBJ(, ) VMOD_POSIT1(, ) NMOD_POSIT1(, ) PREPOBJ(, ) DETERM(, ) R: SUBJ(, ) OBJ_SPRED(, ) OBJ(, ) COREF_REL(, ) NMOD_POSIT1(, ) ADJMOD(, ) PREPOBJ(, ) DETERM(, ) DETERM_DEF(, ) CONNECT_REL(, )

20 Comparison (partial) approche^approcherpersonne^personne cardio-vasculaire^cardio- vasculairerat^rat SUBJ(, ) >,, )>) cardio-vasculaire^cardio- vasculairesemblable^semblable humain^humain rat^rat SUBJ(,,,, ) >,,, ) >)>)>) >, ) >) >)>, ) Consider structures to assess for syntactic correctness Heuristics to put some structures into equivalence : Here, «Le SCV du rat» and «Le rat est un animal qui possède un SCV» are equivalent, but expressed in different syntactic structures

21 Synonyms - semblable D C C0065 adj. - qui ressemble à; comparable, similaire. - ….. - approchant F E0074 To retrieve a synonymy relation : 1.Produce derivations for all words in List 1 2.Find matches in a synonymy basis under entries for words of List 2 3.The search process can be repeated at most once, using lexemes semblable = approchant OR ressembler à = sapprocher de 4.In this way we can catch both synonyms and attached prepositions.

22 Conclusions and future work Automation is possible, with 2 main restrictions : –« bad faith » answers –Lexical errors based on homonymy as long as S contains elements of answer, S can be evaluated Future Work : to assemble the parts through software engineering !