1. And the Data Philosophy & design Therein
Pragmatic Monitoring
And the Data Philosophy & design Therein
Insert clever subtitle here: Zen and the Art of Acquiring a Mature Product? Socrates meets Howard Hughes?

2. “pragmatic monitoring”?
What is
“pragmatic monitoring”?
prag·mat·ic - praɡˈmadik/ - adjective:
dealing with things sensibly and realistically in a way that is based on practical rather than theoretical considerations.
mon·i·tor - ˈmänədər/ - verb:
observe and check the progress or quality of (something) over a period of time; keep under systematic review.

3. Charles Sanders Peirce:
The pragmatic maxim
[…] Consider what effects, that
might conceivably have practical bearings,
we conceive the object of our conception to have.
Then, our conception of these effects is
the whole of our conception of the object.

4. Vision is key
Farsighted: made or done while thinking about what will happen in the future
Shortsighted: lacking imagination or foresight : "expedient, shortsighted solutions to problems“
All of these vision problems are typically seen during any product’s development lifecycle. Before the development begins it is good to be farsighted.
How will the product be used both now and into the future (how farsighted)?
What problems are being solved and what value is being created?
What obvious choices should be made along the way / at what milestones (watch your step)?
Will stability or performance degrade over time / with increased data? (TV Dinner / Buffet)
Will stability or performance degrade over time due to user adoption? (dev in a bottle)
Regulatory concerns? Security?
At different stages along the way there will be shifts; during acquisitions, changing use cases, leadership, regulations, unseen needs, etc (trails close, streams flood, keep a map handy).
Hindsight: understanding of a situation or event only after it has happened or developed.
This is ALWAYS a meta-process. So, enter design methodologies. Developed in the furnaces of hindsight.

5. Choose your own fate Development methodology
Farsighted – hindsight inevitable
Big Design Up Front (BDUF): … the program's design is to be completed and perfected before that program's implementation is started.
Rough Design Up Front (RDUF): 'sufficient' design is completed up front to provide a framework on which to build in the design detail as the project progresses.
Sufficient Design Up Front: Joshua Kerievsky: "I'm saying that we need high design quality for stuff that is critical to our products and less design quality for stuff that isn't critical."
Waterfall: sequential (non-iterative) design process, used in software development processes, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of conception, initiation, analysis, design, construction, testing, production/implementation and maintenance.
The overall structure of waterfall is generally how you want delivering a product to market to feel like. This is like a roadmap to profitability. It looks at software as a product, with a budget and a timeline… Is that good or bad? It depends!
Nearsighted – requires (a lot of) hindsight
Agile: iterative and incremental software development methodologies
Scrum, Crystal, Dynamic Systems Development Method (DSDM), Lean Development, and Feature-Driven Development (FDD). Never ending new ideas here by the nature of the beast.
Emergent design: … delivering small pieces of working code with business value. With emergent design, a development organization starts delivering functionality and lets the design emerge.
Q: What is right? What is wrong? What is better?
A: It depends. – Socrates

6. It depends.
Depending on what “the fates” have dealt us, different methodologies can make more sense. Usually it depends on our knowledge of history, our understanding of the future, and the resources we have at the time, whether they are skills, dollars, or hours, or usually, all of these.
The problem at hand, the patient, our example – Uptime Infrastructure Monitor.
Developed over a decade by others (from Toronto!)
No existing data model – only a couple diagrams
No database dictionary – only a couple diagrams
Requirements evolved, priorities shifted, the product changed, the target audience often directing this – definite impacts of leadership
Idera purchases this company and it is incumbent on us to keep existing customers happy, develop new wanted features, stabilize performance, fix bugs, support and maintain it. Many challenges exist:
Understand the architecture – Apache, PHP, Java, SQL, DCOM / WMI, Perl, Python, Ruby, OMG!
Understand current customer issues, gain direction, set goals and milestones
Hire development staff
Hire support staff – especially difficult
Balance feature releases with bug fixes and “unseen” improvements. UX / Performance / branding…
REMAIN PROFITABLE!
Retain sanity / stay married
While a mixture of design methodologies is likely the best initial method, when you jump on the train and nobody is driving, you have to think on your feet, become a train driver, a mechanic, and keep it on the rails. While most of us aren’t the original designers, software is mutable, trains, less so.

7. Conceiving the object “[…] what effects, that might conceivably have practical bearings […]”
storage DB
app
An affect seen on any one component can have effects on the system. Just like gears in a machine, everything has to mesh properly for the machine to work smoothly.
It is crucial to build and maintain an operational understanding of our information systems.

8. BDUF – reverse engineering.
Let’s take a look at some far sighted and near sighted examples of design under the hood!
Requirement – provide business value to IT leadership for capacity planning and resource allocation (human and hardware), to technicians for troubleshooting and root-cause analysis.
Must be able to connect to a huge array of hardware and software to monitor availability and performance.
Must be able to store this information for long periods for capacity planning
Must remain usable during both of these endeavors
These basic points mean we need a database design that performs well in several ways which is a tall order.
Low latency
A “forgiving” schema to meet agile demands (emergent development is a huge thing in monitoring)
Performance over millions of rows – indexing strategy is crucial, storage engine selection, and query reusability are key (caching)
Let’s look at the database!

10. Processing data
far sighted and near sighted examples of design under the hood!
Farsighted requirements
Low latency
Flexible deployment
Performance scalability
Agile – can be adapted to meet new requirements which may vary in scope
Nearsighted requirements
Agile – can be adapted to meet new requirements which may vary in scope as technology changes
Java and PHP
object-oriented great for modular programs and reusability of code (fits Agile processes well)
Java is platform-independent and can be scaled
Free...
Apache
Stable
Scalable
Free..
MySQL – see above notes.

11. Stewardship
Each piece requires similar and different kinds of attention going forward. Like any machine.
Farsighted requirements
Ability to meet scale and scope while remaining manageable
Nearsighted requirements
Has to work (well) right now.
Java and PHP
Requires continual monitoring as environment changes, code base changes, and adoption / use morphs.
Apache
Scales “easily” – load balancing, scale out…
The DB : MySQL – Oracle – MSSQL
Require care and feeding. Monitoring and maintenance.
Backup regimes
Data integrity monitoring
Security attention if there is secure information and especially when regulations are present.

12. Trial downloads, video resources and more product info.
Trial downloads, video resources and more product info.

13. Thanks! Any questions? You can find me at: @R_Vandervoort on Twitter