1. DBA Meeting December 2006 1 Supporting the MINOS MySQL Database at FNAL Nick West

2. DBA Meeting December 2006 2 Outline Why are we asking? –We asked for, and get, ORACLE support. –Why isn’t that good enough? What are we asking for? –It terms of routine maintenance and recovery.

3. DBA Meeting December 2006 3 Why are we asking? A history lesson: A strategic plan that failed Original review – We get told: You Must use a proprietary db e.g. Oracle, not freeware e.g. MySQL as ultimate repository. Our Plan –Don't want all sites connecting to FNAL. –So need local site DBs replicated from FNAL. –Cannot afford ORACLE everywhere so use MySQL. –Need interface that works to both. –Implies ODBC type interface. –ROOT had work in progress: RDBC (ROOT's ODBC) that was to replace a primitive SQL interface. –Choose this and get community support. The plan begins –In January 2001 start work to implement early work directly to MySQL –Release of first working version in Feb 2001. –In May 2001 interface to RDBC with Valeriy Onuchin working directly to develop RDBC within MINOS.

4. DBA Meeting December 2006 4 History lesson: The plan fails! Plan failed both strategically & technically in two areas:- –Support –Running under oracle Support –To understand our support problems you have to understand the complex library structure:- TSQL Almost pure ABC implemented by... ODBC An adapter to... odbc++ That interfaces to... unixODBC The UNIX ODBC adapter to... backend ODBC client –Support failed in two areas:- Although promised, RDBC development was never accepted into ROOT and remained within our own code. odbc++ was essentially unsupported so we had to absorb it into our own code. Moreover its very heavy use of cpp macro making it hard to debug The result –We are now supporting 4 layers of code beyond our own!

5. DBA Meeting December 2006 5 History lesson: Oracle Support A long and miserable experience! –Start work in early 2004 and although eventually working at some level it has never made it into production! –Lesson: Oracle may be “industrial strength” but not when run on linux via freeware ODBC! Examples of the sorts of problems:- –Multiple server problems:- Running out of table space (multiple times) Crashing after applying security patches. Priming table Oracle wrongly reported duplicate rows (but inserted them and failed to insert others) –Multiple SQL translation issues including:- Date format Bitwise operators Table creation –Data types, Constraints, Synonyms … –Multiple problems in odbc++ layer:- Meta-data reporting problem in which char came back as varchar. Loosing error exceptions. Loosing floating point precision. Weird behaviour handling unsigned data in a cascade with MySQL.

6. DBA Meeting December 2006 6 ROOT SQL: The Second Coming! New developments in ROOT –The primitive SQL interface takes on new life! –In early 2005 ORACLE interface added. –By early 2006 major improvements across the module –It starts to look like a credible replacement for our DB layers In April 2006 –We decide to actively investigate. –Conclude: Looks feasible: –Functionality and performance look O.K. –Support looks good Requires a major re-write of our DatabaseInterface layer In July 2006 –Agreedwe will go for full reimplementation followed by migration so long as functionality and performance maintained. Will this be the end of our Oracle problem?

7. DBA Meeting December 2006 7 Oracle misery continues! Choice of 2 back-end –Oracle occi But locks us in to selected platforms. The future would be uncertain. We decide against it. –via ODBC to either freeware or Oracle supplied driver But have problems with –meta-data – in part because Oracle world view is “differently wired” –Double precision floating has limited exponent Just discovered –Oracle (against SQL standard?) treats an empty char string as a NULL –That breaks our hardware DB. Enough is enough! –MINOS is well into exploitation without ever using Oracle –Why are we still haemorrhaging effort into this project? –Why not give support to MySQL that has worked almost flawlessly?

8. DBA Meeting December 2006 8 So what do we want for MySQL support? Routine maintenance:- –Of the Software Moving the server on to new versions –So far we have moved once (or possibly twice) –So every ~ 18 months or so –Of the data All we need are regular (nightly) backups of the database files Recovery – when things go wrong –Of the Software Applying patches in cases of security incidents –Of the data Broken tables by repair if reasonable or restore of backup if not –So far Liz has had to fix tables once or twice. Our system has built-in recovery –It’s worth pointing out that we run a validated distribution system. –It sends out updates to all our critical tables in mysql1to multiple slave sites within an hour of receiving them. –So we can recover all critical tables from off-site databases if necessary.