1. Part of the UK Data Archive and the Arts and Humanities Data Service. Funded by the Joint Information Systems Committee and the Arts and Humanities Research Board. Good Design - © History Data Service1 Databases of Historical Sources: Principles of Good Design Mark Merry History Data Service

2. Good Design - © History Data Service 2 History as a product History may be thought of as either product or process. As a product, a piece of history consists of a representation of a past reality based upon the interpretation of a body of known facts. Such representations of past realities are always bounded: they treat a subject chosen by the historian which might be static (the situation at point x) or dynamic (how the situation changed between points x and y). Harvey and Press, Databases in Historical Research (1996) Chapter 1

3. Good Design - © History Data Service 3 What is a database? Basically a computerized record keeping system - that is, a system whose overall purpose is to maintain information and to make that information available on demand C.J. Date, An Introduction to Database Systems. Vol I. Seventh edition (1999) A Database Management System (DBMS) is the computer application built around a database to provide flexible ways of storing, manipulating, and examining the data A DBMS on a personal computer will provide facilities for: –inputting, sharing, modifying, retrieving and deleting data –querying the data (SQL) –producing reports based on the data –building front-ends for users

4. Good Design - © History Data Service 4 What can databases do for us? Store and organise large amounts of information Provide a surrogate for the original (inaccessible, fragile) source Group physically dispersed material together at one virtual location Provide an environment for manipulating and analysing the content of the original source To search/filter/summarise complex information quickly –Analysis of large amounts of data –Analysis of complex interrelated data Downside: the time and effort needed to convert original sources into a database

5. Good Design - © History Data Service 5 Source Digitisation Resource The input channels of digitisation (keyboard, scanner etc.) are narrow and can only capture a small proportion of the sources information content identify aspects of source to digitise chose digitisation method chose data model

6. Good Design - © History Data Service 6 Organising data The field is the basic unit of data in a database. A field stores a single piece of information of a particular data type Fields are combined to form a record. A record matches an entity A set of records with the same fields are collected together in a table

7. Good Design - © History Data Service 7 Data models Data models are the abstract definitions of structures and relationships used to organise data A DBMS will implement a particular data model Data models can be characterised by how they organise the connections between different records: –flat file –relational –mark-up (hierarchical) Most DBMSs available for personal computers are either flat file or relational

8. Good Design - © History Data Service 8 Relationships One to one relationships connect one entity to one other entity One to many relationships connect one entity to one or more other entities Many to many relationships connect many entities to many other entities

9. Good Design - © History Data Service 9 Relationships example

10. Good Design - © History Data Service 10 Databases of historical sources An historical source based database is a representation of the original source, but it is not an exact replica of the original source –information may be left out –extra information may be included An historical database should: –try to reflect the source accurately and completely –improve the usability of the source –integrate the source with other data (additional sources, coding etc.) NB: these are conflicting aims! An historical source based database mixes elements of a primary source with elements of a secondary source

11. Good Design - © History Data Service 11 Building databases from historical sources Historians work with information they do not control –incomplete, poorly structured information of varying quality –sources intended for a different purpose –multiple sources not intended to be used in an integrated way Nature of historical sources –ambiguity: the meaning of material may be unclear or dependent on its context –repetition: data is often repeated in different guises –variation: the same item can be referred to using a variety of terms and spellings –variable structure: even apparently well organised sources often have margin notes and other types of random additional data

12. Good Design - © History Data Service 12 Source information content Simplify the source –Ignore unwanted information –Exclude certain types of information –Select information directly from the source or define a set of summarised information based on the source Model the information content subset –Break information content into discrete elements of information –Describe the characteristics of each information element –Describe how information elements relate to each other Successful source analysis requires a good understanding of the source and of the purpose of the database

13. Good Design - © History Data Service 13 Source database content fonts columns text spacing page size date issue fold linemarginalia headlines

14. Good Design - © History Data Service 14 Software & hardware Technical decisions are often the least important Remember that there is nearly always more than one way of doing something with a computer Define what you need to do, then seek technical advice Seek a second opinion! –Technical support staff will often suggest what is most convenient for them, not necessarily you –Commercial companies obviously have their own motives Look for software that supports common standards Avoid little-used software with proprietary features Recognise that hardware may need to be replaced in 2 or 3 years

15. Good Design - © History Data Service 15 The Three Layer Model Standardisation Layer provides a foundation for analysing the data codes and standardisation rules are applied Source Layer an accurate digital representation of the source defines level of detail captured Interpretation Layer incorporates researchers knowledge and judgement Links records and forms aggregates

16. Good Design - © History Data Service 16 Three Layer design examples

17. Good Design - © History Data Service 17 Some simple design hints The smallest unit of data should match the smallest unit of analysis –if you want to look at people by last name then have separate first and last name fields, not just a name field Dont mix data types –separate numbers and words –identify numbers being used as words (addresses) Document everything you do, either in the database or with the database –data entry, data standardisation and coding, data transformations, limits of data, issues of plausibility/probability, coping with uncertainty etc –keep information that tracks the origin and history of the database. Add information, dont delete information Have a backup procedure!

18. Good Design - © History Data Service 18 Further Information History Data Service web - http://hds.essex.ac.uk email - hds@essex.ac.uk Michael J. Hernandez, Database Design for Mere Mortals : A Hands-On Guide to Relational Database Design, Addison-Wesley, 1997 Charles Harvey & Jon Press, Databases in Historical Research, Macmillan Press, 1996 C. J. Date, An Introduction to Database Systems, Addison-Wesley, 1999 (7th ed.) SearchDatabase.Com: http://searchdatabase.techtarget.com/http://searchdatabase.techtarget.com/ Concordia University: http://www.cuaa.edu/computing/softrain/access/access15.shtml http://www.cuaa.edu/computing/softrain/access/access15.shtml University of Newcastle Database Service: http://www.ncl.ac.uk/ucs/databases/ http://www.ncl.ac.uk/ucs/databases/

19. Part of the UK Data Archive and the Arts and Humanities Data Service. Funded by the Joint Information Systems Committee and the Arts and Humanities Research Board. Good Design - © History Data Service19 web - http://hds.essex.ac.uk/ email - hds@essex.ac.uk

20. Good Design - © History Data Service 20 Source Layer Acts as the reference version of the original source. –An accurate representation of the source, including errors, omissions etc. –Contents determine the highest level of detail available about the source in the database –Includes a reference to the non-digital original source –Includes a unique identifier for each item Implementation: –as long text fields containing full text transcriptions –as blob fields containing scanned images

21. Good Design - © History Data Service 21 Standardisation Layer Organises the information into discrete units with fully defined contents –Separates information in the source into separate fields according to data type and data content –Simplifies the data by standardising and coding it –Normalises the data –Includes links back to the source layer Implementation: –Possibly as addition columns in source layer tables –Probably as separate tables with, ideally, a one-to-one relationship to records in the source layer

22. Good Design - © History Data Service 22 Interpretation Layer Creates historical entities from the data and the knowledge and expertise of the historian –Incorporates interpolations and extrapolations from the data in the standardisation layer –Selectively includes and excludes information from the standardisation layer –Links separate records to form entities such as individuals or households