1. Analytic Journalism and Decoding the Political Race(s)
"Newspapers are a 'morning line' tip sheet. 2nd Session Wed, 8 Oct “Tracking the data's flow upstream. And then landing it.”
Tom Johnson Institute for Analytic Journalism Santa Fe, New Mexico “Three SF Complex”
Tuesday, 30 Sept. 2008

2. Objectives:
1st Tuesday, 30 Sept "Newspapers are a 'morning line' tip sheet.”
2nd Tuesday, 7 Oct "How to track the data's flow upstream.“
Who’s behind those sites, anyway?
What’s in online data bases and how did that data get there?
How good, i.e. clean, is the data?
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

3. Did anyone ??? What learned?
“Interrogate” the sub-pages, their links and the data in those links.
List 5 story ideas - based on the data – that could be done
Find 5 “vested interested” sites that have data pertaining to political campaigns.
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

4. Candidates Use the Internet
Web site
Fund raising
Online ads
Webcasts of events
Blogs and podcasts
Take polls and surveys
Communicate with press
Television ads on the official site
Campaign web video on other sites
Participate in social networking sites
Opposition research
Manage field operations
Text messaging
Source:
Politics and Web Strategy: Metrics of Success Sponsored by Knight Digital Media Center
April 24, 2008
Karen A.B. Jagoda, President
E-Voter Institute
Digital Politics--Weekly Internet Radio Show
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

5. Voters Use the Internet
Find out about all candidates and issues
Contribute to candidates
Organize for and against candidates and issues
Tell their friends/family about political issues
Post their own opinions in blogs
Post video and audio related to candidates
Rate posted videos
Create their own sites
Use social network sites
Source:
Politics and Web Strategy: Metrics of Success Sponsored by Knight Digital Media Center
April 24, 2008
Karen A.B. Jagoda, President
E-Voter Institute
Digital Politics--Weekly Internet Radio Show
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

6. What we’re aiming for today
Who/what is behind a web site?
How do we get data off the web into analytic tools
Why do we care?
Efficiency
Dispersed national archive
The Wayback Machine -
Accuracy
Who/what is behind a web site?
How do we get data off the web into analytic tools
Why do we care?
Efficiency
Dispersed national archive
The Wayback Machine - – One of the best ways to keep government at all levels honest is for The People to constantly capture The People’s data, not only in U.S. but globally. Proving the purity of that data can be difficult, but it can be established if there are enough duplicate documents or data sets
Accuracy If we try to re-enter the data, each keystroke carries a probability of error. And the end results magnifies just like a carpenter framing a wall: a quarter inch error at one end results in an inch of error every four feet.
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

7. Who/what is behind a web site?
DNS (Domain Name System)
Translates human-friendly computer hostnames into IP addresses. i.e. translates to = IP Address
Whois -
Betterwhois
Domain History View historical whois records
DNS: The Domain Name System (DNS) is a hierarchical naming system for computers, services, or any resource participating in the Internet. It associates various information with domain names assigned to such participants. Most importantly, it translates humanly meaningful domain names to the numerical (binary) identifiers associated with networking equipment for the purpose of locating and addressing these devices world-wide.
An often used analogy to explain the Domain Name System is that it serves as the "phone book" for the Internet by translating human-friendly computer hostnames into IP addresses. For example, translates to
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

8. But can triangulate by using phone numbers or addresses in Google
Domain walk-through
Search Google for "republican national committee" site:.org
No free sites to find all domains at address or in block of Ips, but
But can triangulate by using phone numbers or addresses in Google
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

9. Ink-on-paper (use OCR) Online (copy and paste; insert URL)
Data formats
Ink-on-paper (use OCR)
Online (copy and paste; insert URL)
Documents (file type?)
Dynamic data bases
PDF (export;OCR; c&p;
Images (.jpg .gif) [Conversion tool]
Audio and video files (text/content analysis) Less advanced tools
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

10. DB vs. Stats Apps vs. Excel
Statistical applications
SAS, R, Stata, Mathematica, SPSS
Cases or Records
Fields or variables
Alpha or numeric data
Linkage = flatfile or relational
Priority language scripts
Pros
Large data sets
Fast sorting/filtering
True, fine-tuned statistical methodology
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

11. DB vs. Stats Apps vs. Excel
Database:
Cases or Records
Fields or variables
Alpha or numeric data
Linkage = flatfile or relational
VBA Scripts
Pros
Large data sets
Fast sorting/filtering
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

12. DB vs. Stats Apps vs. Excel
Rows = records or cases
Columns = variables or fields
Alpha or numeric
Linkage to workbooks/worksheets
VBA Scripts
Pros
Counting money/cases
“Flow through” calculation
Richer calculations/statistical
Easy – if crude – graphics
Pros
Counting money/cases
“Flow through” calculation
Richer calculations/statistical
Easy – if crude – graphics
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

13. Google’s Advance Search “file type”
Finding data
Google’s Advance Search “file type”
Google cheat sheet
Bookmarks
But what is seen is not the digital reality
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

14. Getting data off web, into Excel
No silver bullet!
Try copying URL and use it to open Excel
Look at the page “source.” If you can see the actual data, good odds of c&p table
Use FireFox extensions and “right click” selections
In Excel, “Paste” and “Paste Special”
Consider “cell formats”
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

15. A few Spreadsheet Commandments
NEVER do analysis on original file.
After importing data, enter row or cell or “Comment” with Source, its URL & date
Save this file in folder with Name-the-Virgin file name
Re-save the file with a track-able file name, i.e. “OpenSource Casino v1 – Oct 8 08”
Periodically re-save with adjusted file name. (Why you want to keep a log book so you will know how far back you need to go.)
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

16. Piggy-backing on Bill Dedman
Obama leads the money chase in key counties
Challenge in getting data out of sub-tables. Look at “Source” for tips.
Use Ffox to highlight & copy (table?)
Might have to try different approaches
Scan whole web page for happy surprises.
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

17. “Copy” data and paste into MS Word Turn on Word “show/hide¶”
Different approaches???
“Copy” data and paste into MS Word
Turn on Word “show/hide¶”
Look for “delimiting” or “field separator” characters or symbols.
Use those to parse the “text to columns” in Excel.
Save Word file as “text” (.txt)
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

18. Digital Shadow Puppets
Basic, clean data and code
Source: “Liberal-Conservative Self-Identification ”
Basic, clean data and code
Source: Liberal-Conservative Self-Identification
Drop down “Source”
Look at code under the hood.
Explain how the numbers got there.
Scroll down to note Question Text
Note graphs and sub-set of breakdowns. This is a VERY helpful database/site.
Link to the ASCII text version of this table  
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

19. Links with contribution data
Opensecrets.org
Copy and paste into Excel
Opensecrets.org
Note URL and the “php” extension
Original it supposedly meant Personal Home Page. It is an open source, server-side, HTML embedded scripting language used to create dynamic Web pages. In an HTML document, PHP script (similar syntax to that of Perl or C ) is enclosed within special PHP tags. Because PHP is embedded within tags, the author can jump between HTML and PHP (similar to ASP and Cold Fusion) instead of having to rely on heavy amounts of code to output HTML. And, because PHP is executed on the server, the client cannot view the PHP code.
Demo SOURCE code
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

20. Get latest version of Acrobat Reader PDF apps alternatives
Getting data out of PDF
Get latest version of Acrobat Reader
PDF apps alternatives
Acrobat Pro – pricey but evolving (Buy one-back version, i.e.
ABBYY PDF Transformer 2.0
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

21. Getting data out of PDF
Aetna Report Page 4
Copy and paste into Excel
Check Cell Format types
Common Cause NEW MEXICO: THE CAMPAIGN CONTRIBUTIONS AND LOBBYING EXPENDITURES OF THE TOBACCO INDUSTRY AND ITS ALLIES P. 16
Common Cause NEW MEXICO: THE CAMPAIGN CONTRIBUTIONS AND LOBBYING EXPENDITURES OF THE TOBACCO INDUSTRY AND ITS ALLIES P. 16
C&P top table
What are you going to do first? Embed source info, save “Name the Virgin” file
Rename with new version file name
Demo Data-to-Text
Caution on being sure to make sure the table you see in Excel is the same as in the PDF file, I.e. perhaps handy to print out this table first
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

22. Browser’s PDF might not be PDF
GovernmentAttic.org
Office of the Governor accessions into the Alaska State Archives and Records Dispositions 01-January-2007 to September 2008
Browser’s PDF might not be PDF
GovernmentAttic.org
Office of the Governor accessions into the Alaska State Archives and Records Dispositions 01-January-2007 to September 2008
Demo that browser can “read” PDF file, but copying from what’s on THIS screen doesn’t plug into EXCEL.
Need to SAVE the file, then use ACROBAT to OPEN file.
Then copy and paste into EXCEL
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

23. Review Data In  Analysis  Info Out Need for new tools
Think about process & forms of data plus methods
Need for new tools
Document transformation tools
Objective: get to ASCII or as close to it as possible
Search for “PDF to Text” and “html to text”
“pdf to excel converter”
See PDF Online and Easy Converter Desktop
Acrobat Professional Can “export” PDF to different formats
OCR = Optical Character Recognition
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

24. For next week – 14 Oct. Pull down some data that interests you
Get it into a spreadsheet
Add a new column or row that gives you information about the topic you didn’t know before
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008

25. Objectives:
1st Tuesday, 30 Sept "Newspapers are a 'morning line' tip sheet.”
2nd Tuesday, 7 Oct "How to track the data's flow upstream.“
3nd Tuesday, 7 Oct “How to make sense of those numbers?“
More on getting the data off the web
PDFs, CSV & importing and exporting
Simple arithmetic and simple spreadsheet tips
AJ and Decoding the Political Race(s) © J.T.Johnson 2008_________________________Fall 2008