Reasoning about Hygienic Macros David Herman Northeastern University

Programming languages research ● How do we write software? ● How do we express computation? ● How do we specify invariants about programs? ● How do we predict the behavior of programs? “I like programming languages research because you can study point-set topology and still feel the bits between your toes.” —Will Clinger (paraphrased)

1. Introduction to hygienic macros 2. Research topics in macros 3. From Grinnell to grad school

Why extensible instruction sets? Imagine languages without user-defined procedures!

Why extensible syntax? Java suffered for 18 years without for-each loops: for (Type x : expr) { body … } ≡ for (Iterator iter = expr.iterator(); iter.hasNext(); ) { Type x = iter.next(); body … }

Macros vs. program generators Macro expansion is part of the compilation process: ExpandRunCompile GenerateRunCompile

Why extensible languages? Language designers have to stop somewhere Java 688 pages C# 553 pages C pages

Macros vs. program generators ● No ad-hoc parsers or parser extensions ● No complicated makefile architecture ● No dependencies on external tools ● Automatic source-location tracking ● Compatibility with existing language tools ● Package and deploy extensions in libraries ● Linguistic support

Scheme: macros to the core Scheme takes this idea to the extreme: A for-each-loop is just a special kind of for-loop. A for-loop is just a special kind of while-loop. A while-loop is just a special kind of tail-recursive function. A tail-recursive function is just a special kind of recursive function. A recursive function is just a special kind of function.

Scheme: macros to the core Scheme takes this idea to the extreme. (for (expr 1 expr 2 expr 3 ) body …)  (begin expr 1 (while expr 2 (begin body … expr 3 )))

Scheme: macros to the core Scheme takes this idea to the extreme. (while expr body …)  (letrec ((loop (lambda () (if expr (begin body … (loop)) (void))))) (loop))

Scheme: macros to the core Core Scheme: expr ::= var | const | (lambda (var …) expr) | (expr...) | (if expr 1 expr 2 expr 3 ) | (let-syntax ((var expr) …) expr)

Macros “rewrite” programs (or expr 1 expr 2 )  (if expr 1 expr 1 expr 2 ) (or (get-next-title db) “Untitled”)  (if (get-next-title db) (get-next-title db) “Untitled”)

Macros “rewrite” programs (or expr 1 expr 2 )  (let ((tmp expr 1 )) (if tmp tmp expr 2 )) (or (get-next-title db) “Untitled”)  (let ((tmp (get-next-title db))) (if tmp tmp “Untitled”))

Scope trouble (or expr 1 expr 2 )  (let ((tmp expr 1 )) (if tmp tmp expr 2 )) (let ((tmp (get-default-title))) (or (get-next-title db) tmp))  (let ((tmp (get-default-title))) (let ((tmp (get-next-title db))) (if tmp tmp tmp)))

Scope trouble (or expr 1 expr 2 )  (let ((tmp expr 1 )) (if tmp tmp expr 2 )) (let ((if (get-default-title))) (or (get-next-title db) if))  (let ((if (get-default-title))) (let ((tmp (get-next-title db))) (if tmp tmp if)))

Hygienic macros ● Hygienic macros automatically prevent these bugs ● Expansion renames introduced bindings: (let ((tmp (get-default-title))) (or (get-next-title db) tmp))  (let ((tmp (get-default-title))) (let ((tmp.21 (get-next-title db))) (if tmp.21 tmp.21 tmp)))

Hygienic macros ● Hygienic macros automatically prevent these bugs ● Rename introduced bindings: (let ((tmp.21 expr 1 )) (if tmp.21 tmp.21 expr 2 )) ● Trickier: protect introduced references (let ((if.17 (get-default-title))) (let ((tmp.79 (get-next-title db))) (if tmp.79 tmp.79 “Untitled”)))

State of the art 1986: Hygienic macros invented 1991: Clinger/Rees macro system ● Simple macros expressed as “rewrite rules” 1993: Dybvig et al macro system ● Expressive macro system ● Macros themselves written in Scheme 2005: Van Tonder macro system ● Variation on Dybvig system

1. Introduction to hygienic macros 2. Research topics in macros 3. From Grinnell to grad school

Research topics in macros ● Expressiveness ● Module systems ● Engineering tools ● Static analysis ● Extensible grammars and parsers ● Semantics and theoretical foundations

Hygiene is hard (or expr 1 expr 2 )  (let ((tmp.8 expr 1 )) (if tmp.8 tmp.8 expr 2 ))

Hygiene is hard (mumble expr)  (quux tmp expr) If it's a binding, rename it......but not if it's a reference...but if it's both?

Hygiene is hard What is the scope structure of this expression? (quux x x)  (lambda (x) x)  (list x x)  (list x 'x)  (list 'x x)  (x (lambda (x) (list x 'x)))...

Hygiene is hard ● Macro expansion incrementally reveals scope ● Hygienic expansion renames according to scope ● Expansion algorithms are fiendishly clever

Reasoning about hygienic macros ● Most macro research about implementations rather than specifications ● What are they supposed to do? ● How do we reason about hygienic macros? ● Semantics: predictive models for behavior

Describing behavior Describing a procedure's behavior: substring(String, int, int) : String Describing a macro's behavior: (or expr expr) : expr

Describing scope (lambda (x:var...) expr  [x...]) : expr (let ((x:var expr)...) expr  [x...]) : expr

Types provide theorems! Types in procedural languages: ● If evaluation of this expression terminates, its result will be a value of type T Types for macros: ● If expansion of this form terminates, its result will be a term of type T ● (lambda (x) x) ≡ (lambda (y) y)

Alpha-equivalence Two terms are the “same” if their only syntactic difference is the choice of variable names. Names don't matter.

Alpha-equivalence and hygiene (let ((foo (get-default-title))) (or (get-next-title db) foo))  (let ((foo (get-default-title))) (let ((tmp.17 (get-next-title db))) (if tmp.17 tmp.17 foo))) Names don't matter. (let ((tmp (get-default-title))) (or (get-next-title db) tmp))  (let ((tmp (get-default-title))) (let ((tmp.4 (get-next-title db))) (if tmp.4 tmp.4 tmp))) ≡ ≡

1. Introduction to hygienic macros 2. Research topics in macros 3. From Grinnell to grad school

A Grinnellian in grad school ● Our main weakness: exposure ● Our main strength: scholarship Want to stack the deck? Google.

I wish I'd known ● Lots of CS material is freely available online. ● Citeseer, Google, Google Scholar, ACM Portal ● Follow the references! ● You don't have to understand it all. You won't. ● Interested in PL? ● Google “SICP lectures” ● lambda-the-ultimate.org

Interested? ● Contact me (Google “Dave Herman”) ● Arjun Guha, Brown University ● Summer research internships