1. Transparency: The Emerging Revolution in Qualitative Analysis Andrew Moravcsik Princeton University APSA – DA-RT Workshop (September 2014)

2. Transparency: The Emerging Revolution in Qualitative Analysis 1.Transparency brings scholars together. 2.Scholars have been thinking about what transparency means for qualitative researchers. 3.“Active citation” (digitally enabled citation) is emerging as the lead standard for qualitative transparency. 4.AC provides transparency cost-effectively. 5.Much has been done. Now journals are getting involved.

3. Transparency as an Ideal “The idea is to try to give all the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.” -- Richard P. Feynman

4. Transparency as an Ideal “History has this in common with every other science: that the historian is not allowed to claim any single piece of knowledge, except where he can justify his claim by exhibiting… the grounds upon which it is based [and] what the evidence at his disposal proves about certain events.” -- R. G. Collingwood What is History?

5. What is Research Transparency? For evidence-based knowledge claims, researchers should achieve research transparency: Data Transparency: Provide access to data. Analytic Transparency: Explicate the link from data to descriptive/causal conclusions. Process Transparency: Show procedures used to collect, generate or choose the data, theory, and methods.

6. What does research transparency mean for qualitative research? Ideal-typical qualitative research Few cases (not many) Process observations (not dataset observations) Textual evidence (not quantitative)

7.  ACTIVE CITATION: “Digitally enabled citations” linked to annotated source excerpts in an appendix  TRADITIONAL CITATION  QUALITATIVE DATA ARCHIVING  HYPERLINKS TO ON-LINE SOURCES  DATABASES Approaches to Transparency

8. MAIN TEXT Contestable Knowledge-based Claim CITATION Footnote, Endnote or In-Text TRANSPARENCY APPENDIX ENTRY 1.Citation 2.Source Excerpt 3.Annotation [ 4. Optional Scan or Link to Full Source] WHAT IS AN ACTIVE CITATION? BODY OF ARTICLE TRANSPARENCY APPENDIX (Remains Unchanged) (New)

9. Issues: data selection (“cherry picking”), robustness to theory choice and specification, precise methodological issues, relative weight of evidence. The main text, as usual. A special “methodological” appendix entry at the beginning of the transparency appendix:. Additions to individual annotations: e.g. the relative weight of evidence. WHAT IS AN ACTIVE CITATION? Process Transparency

10. Tommy Lee Jones Rep. Thaddeus Stevens on the Thirteenth Amendment “The greatest measure of the nineteenth century was passed by corruption, aided and abetted by the purest man in America.” …according to Steven Spielberg, who got it from historian Fawn Brodie (1959), who got it from popular lecturer James Scovel (1898)…

11. In Steven Spielberg’s recent film Lincoln, the abolitionist Representative Thaddeus Stevens (played by Tommy Lee Jones) returns home after the House of Representatives passed the Thirteenth Amendment (banning slavery). He utters the following striking phrase to his mulatto common-law wife, Lydia Smith, referring to Lincoln’s role in the passage of the amendment: “The greatest measure of the nineteenth century [was] passed by corruption, aided and abetted by the purest man in America.” The accuracy of this quotation has been questioned. One account, written much later, suggests that Stevens said this. Yet, if he did, it seems highly unlikely that he did so at home to his companion. FN: James Scovel, “Thaddeus Stevens,” Lippincott’s Monthly Magazine (April 1898), pp. 548-550. An Example of Active Citation The Main Text and Footnote

12. AN EXAMPLE Part 1 of 4: Full Citation James Scovel, “Thaddeus Stevens,” Lippincott’s Monthly Magazine (April 1898), pp. 548-550.

13. AN EXAMPLE Part 2 of 4: Textual Excerpt To the writer of this sketch Mr. Stevens told the story of the legislation which gave to the black man his right to vote: … [548] His favorite amusement was…to spend the evening at Hall and Pemberton's Faro Bank…and over canvas-back and Veuve Clicquot champagne woo unmolested the goddess of fortune.…Stevens was never a heavy player, although I have seen him win fourteen hundred dollars on a twenty-dollar gold-piece as his only stake… [549] Influence from the White House secured votes against a favorite measure of Mr. Stevens for an air-line railway from Washington to New York, and…these same votes helped Mr. Lincoln's great amendment for emancipation. Of this legislative bargain Stevens said, ‘The greatest measure of the nineteenth century was passed by corruption, aided and abetted by the purest man in America.’ During the last thirty years of his life its unwritten romance was the unselfish and tender devotion with which Stevens was attended by Lydia Smith, a mulatto, who in her youth had great beauty of person. [550]

14. AN EXAMPLE Part 3 of 4: Annotation The first published reference to Steven’s “corruption” statement was in 1898, by James M. Scovel, a retired New Jersey politician and lawyer. He quotes Stevens and links the reference to corruption to Lincoln’s willingness to secure votes for the Thirteenth Amendment by trading votes against a direct NY to Washington railway. Is he reliable on this point? On the positive side, Scovel’s account contains specific details, including a story Stevens allegedly told him about the passage of the Fifteenth Amendment and an anecdote about witnessing Stevens gamble. We know they Scovel and Stevens were political allies, and Stevens encouraged Scovel to run for Congress. A modest correspondence has been preserved. On the negative side, Scovel was hardly in Washington during the Lincoln Administration, and he did not write anything down for 33 years. In any case, Scovel’s account does not imply any connection between the quotation and Lydia Smith, Stevens’ common-law wife, though she is mentioned. Nor is it clear how anyone except Smith herself, who left no memoirs, would know if Stevens had said it to her alone.

15. Three Dimensions of Research Transparency A transparent decision about what Stevens said rests on: 1. Data Transparency  The Text 2. Analytic Transparency  How we interpret the text 3. Procedural Transparency  How we select the text and analytic methods

16. AN EXAMPLE Part 4 of 4: Scan or Link (Optional)

17. Transparency: What do scholars gain?  Readers can appreciate the richness and rigor of qualitative data and methods.  Scholars can display excellence, while journal editors, graduate advisors, departments and professional associations can better distinguish and promote it.  Scholars have an incentive to improve qualitative methods.  Colleagues can critique and debate empirical analysis.  Scholars (qualitative and quantitative) can reuse textual evidence for scholarly or public policy purposes.  Political scientists enjoy more opportunities for interdisciplinary cooperation with history, law, as well as public policy and journalism.

18.  Some political scientists already do much of this.  AC only applies to sources documenting “contestable knowledge-based empirical claims.”  Precise length of quotations and annotations is discretionary.  Modern technology and advance planning lighten the load.  Writers, editors and publishers maintain existing formats.  Standard practice in other fields (law, history, classics). Is Active Citation Too Costly?

19. Best Practices: Political Science

20.  ACTIVE CITATION  TRADITIONAL CITATION  QUALITATIVE DATA ARCHIVING  HYPERLINKS TO ON-LINE SOURCES  DATABASES Approaches to Transparency

21. Active Citation: What has been done?  Established a Qualitative Data Repository (QDR)  Secured NSF Funding  Sponsored Articles/Workshops on Conceptual and Practical (Human Subject, IP, and Logistical) Issues  Helped establish APSA Guidelines (with Quantitative and other communities)  Created NSF/QDR On-line Demonstration Portal: 20+ scholars retrofitting “classic” and forthcoming research  Organized Training Sessions at Qualitative/Multi-Method Institute and universities  Launched Interdisciplinary Outreach (SSRC, BITSS)  Reached Out to Journals: Special Issues and Presentations  Designed New Software

22. ACE Word Add-in Active Citation… Translate… Look Up Active Citation… Translate… Look Up OUTPUT: -Word Document -Web Document -Database (Access, Atlas…)

23. Active Citation: The Future Agenda  Work with journals on adoption, format, standards, and processes.  Convene interdisciplinary conversations and learn about common problems and “best practices” (SSRC)  Develop better guidance and classroom materials for researchers, teachers and policy analysts  Work with software providers (e.g. database software)  Reach out to journalists, policy-makers, government agencies and NGOs

24.  Not just political science…  Not just the sciences in general…  The world of information is going this way. Why is active citation inevitable?

25. Best Practices: Natural Science Example: When did Polar Bears Evolve? Article Supplementary Materials

26. Best Practices: Legal Academia (Yale On-Line Law Review)

27. Best Practices: Policy Analysis

28. Best Practices: Specialized Journalism

29. Best Practices: Classified Intelligence Reports

30. Transparency: The Emerging Revolution in Qualitative Analysis Andrew Moravcsik Princeton University APSA – DA-RT Workshop (September 2014)

32. Transparency: The Coming Revolution in Qualitative Analysis Andrew Moravcsik Princeton University IQMR – Syracuse University (June 2014)

33. Transparency: Traditional Footnotes vs. Active Citations Traditional CitationActive Citation Data Transparency De facto costs vary from moderate to prohibitive. Mandatory excerpt provides “one click” basic transparency. Optional scan/link creates option for maximum within IP, human subject and logistical constraints. Analytical Transparency Rarely possible, increasingly excluded by current formats. Mandatory. Production Transparency Only within text of article. Special section of transparency appendix plus any dedicated entries.