Measuring Linguistic Complexity Kristopher Kyle

Who is this guy?  Interested in:  L2 Writing Quality/Development  Assessment  Natural Language Processing  Productive Vocabulary  Productive Syntax

Outline of Workshop  Why measure linguistic complexity?  How can linguistic complexity measures be conceptualized?  How do we actually measure linguistic complexity?  Hands-on workshop I: Measuring syntactic complexity  Hands-on workshop II: From raw data to findings (if time)

Why measure linguistic complexity?  In the 70’s, SLA researchers (e.g., Larsen-Freeman, 1978) wanted to measure language development  Larsen-Freeman proposed three constructs of development:  complexity  accuracy  fluency  The general hypothesis (with regard to complexity) has been: As language learners develop, their language will become more complex.  How complexity is measured has been the subject of much debate (e.g., Bulté & Housen, 2012)

How can linguistic complexity measures be conceptualized?  Wolfe-Quintero et al. (1998) provides a compendium of CAF measures up until the late 90’s  Lexical Complexity:  a variety of general and part of speech specific type/token ratio counts  Syntactic Complexity  a variety of clause, sentence, and T-unit measures that focus on clausal complexity.

How can linguistic complexity measures be conceptualized?  Most of syntactic complexity indices are ratio scores: (Structure A)/(Structure B).  The denominator (Structure B) is either:  clause: a main verb and its dependents (I eat pizza.)  T-unit: an independent clause and any attached dependent clauses (I eat pizza because it is delicious.)  sentence: A string of words that starts with a capital letter and ends with sentence-ending punctuation (I think you know what a sentence is.)

How can linguistic complexity measures be conceptualized?  The numerator (Structure A) has included many structures:  clauses  dependent clauses  adverbial clauses  T-units  complex T-units  coordinate phrases  complex nominals  verb phrases  passives

How can linguistic complexity measures be conceptualized?  Length of unit measures have also been prominent (e.g., Ortega, 2003; Lu, 2011).  Mean length of clause (MLC)  Mean length of T-unit (MLTU)  Mean length of sentence (MLS)

How can linguistic complexity measures be conceptualized?  The rise of phrasal complexity:  Biber, Poonpon, and Grey (2011) suggested that clausal subordination (i.e., what most syntactic complexity indices measure) is NOT a prominent feature of academic writing  Informal speech includes many dependent clauses, but academic writing includes many dependent phrases (and especially noun phrases.

How can linguistic complexity measures be conceptualized?  Some important issues:  Definition of measures  What counts as a clause?  Prominence of broad indices  What does MLC really tell us about development?  Often only a limited range of measures are used.

How do we actually measure linguistic complexity?  To measure linguistic complexity, we have two options.  Option #1: Count features by hand  Option #2: Count features using a computer

How do we actually measure linguistic complexity?  Advantages of Option 1:  Researcher has full control over how syntactic complexity is measured.  Human counts may be more accurate  Disadvantages of Option 1:  Expensive!  Intra-rater reliability  Inter-rater reliability – who is qualified?

How do we actually measure linguistic complexity?  Advantages of Option 2:  Very cheap  Reliable (same results every time)  Usually Accurate  Biber (e.g., 2004) and Lu (2010, 2011) report accuracies above 90%  Can analyze a broad range of indices at once.  Disadvantages of Option 2:  Research has less control (is at mercy of available programs)  Some data is not well-suited to automatic analysis  Some linguistic features cannot be reliably captured

Hands-on workshop I: Measuring syntactic complexity  Go to and download the “short_samples.zip” file.  Without talking with your neighbor(s) fill in the included excel sheet for examples 1-5.  What were your answers?  Any issues with example 5?  Now do the same for example 6…

Hands-on workshop I: Measuring syntactic complexity  Tool for the Automatic Analysis of Syntactic Complexity (TAASC)  Prototype!!!  Includes indices created by Xiaofe Lu (Syntactic Complexity Analyzer; Lu, 2011)  Also includes some replications of the Biber Tagger

Hands-on workshop I: Measuring syntactic complexity  How TAASC works:  Reads file  Splits file into sentences  Parses each sentence  uses Stanford Parser  Uses regular expressions (a way to search for patterns) to identify particular structures in the parse tree.  uses Stanford Tregex (regular expressions for parse trees)

Hands-on workshop I: Measuring syntactic complexity  Now, lets check to see if your computer is set up correctly.  First, search for Terminal (mac) or Command Prompt (Windows)  Then type: java –version  Then type: python  Go to and download the appropriate version of TAASC (windows or mac).  Extract it to your Desktop  Copy the example files to the “to_process_2” folder

Hands-on workshop I: Measuring syntactic complexity  Now, in Terminal/Command Prompt type:  cd [location of TAASC folder] (then press “return”)  python [name of the appropriate TAASC program] (“return”)  Your results should now be in a file called “results.csv”  If you want to examine the accuracy of the parse trees, look in the folder “parsed_files” using Tregex

Hands-on workshop I: Measuring syntactic complexity  Some simple patterns:  VP  VP<S  Some important patterns:  clause: S|SINV|SQ <<# MD|VBP|VBZ|VBD  T-unit: S|SBARQ|SINV|SQ > ROOT | [$-- S|SBARQ|SINV|SQ !>> SBAR|VP]

Hands-on workshop II: From raw data to findings  Go to and download the “Workshop_Data.zip” file.  58 participants, three timed essays over 1 year.  IEP Levels 3-4 (Intermediate/Advanced)  Now let’s analyze some data!  NOTE: We didn’t get to this in class…