Presentation is loading. Please wait.

Presentation is loading. Please wait.

Under the Covers of the data Architecture

Similar presentations

Presentation on theme: "Under the Covers of the data Architecture"— Presentation transcript:

1 Under the Covers of the data Architecture
Multi-Tenant Magic: Under the Covers of the data Architecture Craig Weissman Chief Software Architect 1

2 Safe Harbor Statement “Safe harbor” statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements including but not limited to statements concerning the potential market for our existing service offerings and future offerings. All of our forward looking statements involve risks, uncertainties and assumptions. If any such risks or uncertainties materialize or if any of the assumptions proves incorrect, our results could differ materially from the results expressed or implied by the forward-looking statements we make. The risks and uncertainties referred to above include - but are not limited to - risks associated with possible fluctuations in our operating results and cash flows, rate of growth and anticipated revenue run rate, errors, interruptions or delays in our service or our Web hosting, our new business model, our history of operating losses, the possibility that we will not remain profitable, breach of our security measures, the emerging market in which we operate, our relatively limited operating history, our ability to hire, retain and motivate our employees and manage our growth, competition, our ability to continue to release and gain customer acceptance of new and improved versions of our service, customer and partner acceptance of the AppExchange, successful customer deployment and utilization of our services, unanticipated changes in our effective tax rate, fluctuations in the number of shares outstanding, the price of such shares, foreign currency exchange rates and interest rates. Further information on these and other factors that could affect our financial results is included in the reports on Forms 10-K, 10-Q and 8-K and in other filings we make with the Securities and Exchange Commission from time to time. These documents are available on the SEC Filings section of the Investor Information section of our website at, inc. assumes no obligation and does not intend to update these forward-looking statements, except as required by law.

3 True Multi-Tenancy is our Religion
Time Technology Advances Major Architectural Shift Agenda Our Religion Virtual Database App Model Business Model Changes Agenda for this talk: Shared-everything religion and runtime composition of applications Database constructs: metadata, data, optimization Development model: declarative and procedural

4 Single tenant applications: lots of waste
App App Db Db App App Dedicated software instances: under-utilized hardware stacks for each tenant We prevent a great deal of hardware and software from being sold to our customers We do with 1000 machines what would otherwise require 100,000+ Multiple staffs to hire, train, and pay Database instances are expensive to maintain Point-in-time backup strategies are expensive Rolling upgrades are hard Multiple versions to support 3rd party vendor versions More importantly your own code base (patches) Your data schema Db Db

5 Multi-tenancy benefits are self-evident But isolation is much easier said than done…
App Db Only one software instance and hardware stack for multiple tenants Significant cost savings in hardware Provisioning is free – this is critical (we add rows to existing tables – this is the sweet spot for RBDMS) Actually we have about 10 such identical “POD’s” running 10,000’s of tenants each (avoid SPOF’s) Only one staff needed to maintain the service Surprisingly small number of humans run the entire universe of tenants Monitoring and automation are key, but much easier in a shared environment Very small number of versions to support 3rd party platform support matrix collapse Your own code base and schema are the same everywhere Critically: our developers are only working on 2 versions of our code at a time – much more innovation possible in the long term this way

6 Our religion: Not all “multi-tenant” designs are created equal
“Can’t we create a separate stack for just this one customer? I promise it’s just this one…” App App Db Db It’s a slippery slope, this is hard Upgrade timing Functionality for one tenant Performance tuning for one tenant UI for one tenant Workflow for one tenant Security concerns (real or perceived) Has any other enterprise database application resisted this temptation?

7 Introducing the metadata-driven, multi-tenant, Internet application platform
Poly- Morphic Application Separate concerns: Tenant-specific metadata Tenant business data Shared runtime engine (kernel)‏ Metadata-driven architectures facilitate multi-tenant applications Over time less and less functionally is statically built into our kernel Instead, each tenant application is composed at runtime from rich metadata

8 Key Architectural Principles
Stateless AppServers Database system of record No DDL All tables partitioned by OrgId Smart PKs, Polymorphic FKs Creative de-normalization and pivoting Use every RDBMS feature/trick All tenants are equal (AppServers scale horizontally) We can run the site with just the database and no caching (just performs/scales worse) Only cache metadata at our layer. RDBMS SGA caches data No global locks ever Primary key prefix = Object Type Id Polymorphism allows pivot tables and flex schema to work

9 Metadata, data, and pivot table structures store data corresponding to virtual data structures
A simplified look at some key metadata and data tables No DDL at runtime! Our flex schema uses clever de-normalization – many patents pending We add data across tables for reporting (the big grid-like CFDATA tables) But we also have pivoted specialized tables

10 The Objects table stores metadata about custom objects (tables)‏
Naturally we have our own multi-tenant catalog tables – like Oracle’s own system catalog, but for our own bookkeeping The definition of our custom objects is one of the most important metadata tables

11 The Fields table stores metadata about custom fields (columns)‏
And we have our own field catalog table – our field types are richer than standard RDBMS types. For instance multi-select picklists are stored in a special compact representation

12 The Data heap table stores all structured data corresponding to custom objects
Massive, massive table – wide and deep, billions of rows, partitioned, shared All flex data stored in varchar columns – canonical forms for data types ObjId distinguishes different “tables” for the same tenant The same physical column is used for different purposes across tenants and across objects within a tenant Our query engine emits the proper SQL for all readers and writers of data (e.g. the Api and reporting)

13 A single slot can store various types of data that originate from different objects
Example of a single column Note that the underlying RDBMS cannot know about this usage. Small conversion penalty for native data type processing So how do indexing and statistics work?

14 The Indexes pivot table manages tenant-specific selective indexes
An example of a critical de-normalized pivot table Data from a single object’s slot of a VAL column is copied to this tall/skinny table as well Used by the external Id feature for example… Real RDBMS btree indexes are then used on the value columns However, to use this table, SQL must lead with it 10’s of billions of rows are ok – tables are like super models

15 The UniqueFields pivot table facilitates uniqueness for custom fields
The uniqueness feature is backed by a real RDBMS unique index. Same schema as previous slide, but with unique index The platform detects the index violation and then provides additional information (the Id of the duplicating row) Other pivot tables: NameNorm, SNL, etc.

16 The Relationships pivot table facilitates referential integrity and optimizes joins
Navigation from parent to children would not be possible in the heap table This table has indexes in both directions Again, the various users of this table must know to inject this table into the SQL

17 All data & metadata structures are partitioned to improve performance and manageability
Tables hash partitioned by OrgId Separate conn pools point to physical hosts App tier is also dynamically partitioned by OrgId Distributed metadata cache w/transactional invalidation List partitioning has been suggested, but questions about DDL Local indexes all lead by OrgId Stability in the event of failover (RAC and appservers)

18 Application Framework: a whole lot for free
Native Declarative features Bulk Processing The Recycle Bin Full Text Search Smart Bulk DML Web Services APIs Our “database” is really an application server too – many higher level features built-in and tightly integrated

19’s native Application Framework provides declarative development, no coding
Example: activity tracking, field history tracking Btw, the audit data is stored in another massive pivot table

20 Validation rules and simple formulas: Business analysts can “code” these
Fairly straight-forward, like check constraints in a RDBMS

21 Not so simple: Rollup-summary fields provide for easy cross-object summaries
RSF’s must be de-normalized for grid reporting Very hard to get right all the time Done automatically by the bulk save engine Invalidated and bulk re-calculated when needed

22’s bulk processing optimizations reduce overhead for data loads
We have spent years making the save loop bulk from beginning to end Retry and partial save logic is novel Well-defined blending of declarative (RSF, validation, auditing) and procedural (Apex) logic

23 Data definition processing is optimized to avoid performance hits or concurrency limits
Examples: Sort all records by primary key before attempting DML Operate on tables in deterministic order Slot reallocation for field datatype change Deferred calculation for new rollup-summary field Background processing of mass changes Again, years of experience and database expertise Benefit from our best practices

24 The Recycle Bin: Smart Undeletes
Individual object instances (records)‏ Related object instances (parent/child records)‏ Entire fields and objects (dropped columns and tables)‏ Restore Another high-level application feature not provided by standard RDBMS Deep cascading operations are difficult to get right (deadlocks e.g.) Restoration across schema changes is novel

25’s full-text search engine
Asynchronously maintains indexes for all text fields MRU caches contain recently updated objects Optimizes ranking of search result records based on current user, modification history, and weighting preferences Another hard platform area to get right Years of iteration for massive multi-tenant performance Various point solutions available on the market – but our is integrated for free

26 Multi-tenant Query Optimization Principles
Consistent SQL generation across the application Deep awareness of pivot table structure Flex schema does impose a cost Tenant, user, object, fields statistics are crucial No runaway queries allowed Deep integration with the sharing model Core competency of our platform “Index” plans become extra joins plus indexes DML maintains some statistics synchronously; others gathered on a scheduled basis Sharing: Application layer blended into the DB layer (only possible with a platform)

27 Multi-tenant optimizer statistics’s query optimizer writes optimal queries for internal data access operations Multi-tenant optimizer statistics Multi-tenant optimization based on our flex schema with knowledge of de-normalized pivot tables and sharing model Data sharing is scalable and robust and tightly integrated into the optimizer – another high-level application feature pushed into the DB layer Statistics kept automatically for custom indexes and foreign keys Histograms for picklist values, special statistics for record owner field, etc.

28 Pre-Query Selectivity Measurements
The optimizer considers pre-query selectivity measurements when writing a query Pre-Query Selectivity Measurements … use of index related to filter. High … ordered hash join; drive using Data table. Low … nested loops join; drive using view of rows that the user can see. Write final database access query, forcing … Filter User Some statistics are cached – others require dynamic gathering Query plans differ not only in hints but in the skeleton of the query itself (extra tables joined) Ongoing research into parallelism, hashing vs nested loops, etc.

29 Apex:’s procedural frontier
Adding a procedural language to our service was not a simple decision – eventually we knew we had to do it, but safely Functionality of pl/sql Syntax and idioms of Java/C# But direct access to features for SOQL, DML, cursors, etc. Tightly integrated into our save logic Years of refactoring of our core database interaction made this possible

30 Apex code is stored as metadata, interpreted at runtime, and cached for scalability
The code itself is stored in metadata like everything else The actual compiled form is just a POJO (we’re not compiling any Java – we built our own parser (ANTLR) and runtime, written in Java) The Apex VM has a dynamic class loader that can re-use the same compiled form for multiple concurrent requests Managed code is shared across tenants

31 Apex is deeply integrated with platform features
Bulk DML and messaging Asynchronous processing (Futures) XmlStream / HTTP (RESTful) services classes Declarative exposure as new Web Services The “ADK” provides access to powerful platform features

32 governs Apex code execution
Limits on: CPU Memory # of DML statements # calculations # web service calls … and more Governance is a key concept for any multi-tenant shared system Built for safety first – controlling the runtime completely made this much easier Plus we force people to use good practices such as Bulk SQL/DML

33 Unit tests must accompany Apex code
Required 75% code coverage Profiling is built into the platform Run during application install All tests are run before each platform release by us Novel aspect of our development environment – the development and deployment model are combined Yes, we are like big brother, but we’re not apologetic The ability to run on upgrade hammer against tenant tests is invaluable for our platform server compatibility story

34 is a proven multi-tenant application platform that performs and scales
Quarterly Transactions (billions)‏ Page Response Time (ms)‏ As our workload continues to evolve with Apex and VisualForce, we continue to focus on performance and scalability Emergent property of developing and running the site We have access to all the logs and all the data and all the exceptions and all the performance information Virtuous cycle of improvement – everyone benefits 2005 2006 2007 Fiscal Year

35 Concluding Remarks PaaS is a major architectural shifts
PaaS is Application focused, high level of abstraction is the most mature, proven PaaS offering available today Optimized for fast, secure, and reliable multi-tenant application development and deployment

Download ppt "Under the Covers of the data Architecture"

Similar presentations

Ads by Google