Presentation on theme: "Under the Covers of the Force.com data Architecture"— Presentation transcript:
1 Under the Covers of the Force.com data Architecture Multi-Tenant Magic:Under the Covers of the Force.comdata ArchitectureCraig WeissmanChief Software Architectsalesforce.com1
2 Safe Harbor Statement“Safe harbor” statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements including but not limited to statements concerning the potential market for our existing service offerings and future offerings. All of our forward looking statements involve risks, uncertainties and assumptions. If any such risks or uncertainties materialize or if any of the assumptions proves incorrect, our results could differ materially from the results expressed or implied by the forward-looking statements we make.The risks and uncertainties referred to above include - but are not limited to - risks associated with possible fluctuations in our operating results and cash flows, rate of growth and anticipated revenue run rate, errors, interruptions or delays in our service or our Web hosting, our new business model, our history of operating losses, the possibility that we will not remain profitable, breach of our security measures, the emerging market in which we operate, our relatively limited operating history, our ability to hire, retain and motivate our employees and manage our growth, competition, our ability to continue to release and gain customer acceptance of new and improved versions of our service, customer and partner acceptance of the AppExchange, successful customer deployment and utilization of our services, unanticipated changes in our effective tax rate, fluctuations in the number of shares outstanding, the price of such shares, foreign currency exchange rates and interest rates.Further information on these and other factors that could affect our financial results is included in the reports on Forms 10-K, 10-Q and 8-K and in other filings we make with the Securities and Exchange Commission from time to time. These documents are available on the SEC Filings section of the Investor Information section of our website at Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements, except as required by law.
3 True Multi-Tenancy is our Religion TimeTechnologyAdvancesMajorArchitecturalShiftAgendaOur ReligionVirtual DatabaseApp ModelBusinessModelChangesAgenda for this talk:Shared-everything religion and runtime composition of applicationsDatabase constructs: metadata, data, optimizationDevelopment model: declarative and procedural
4 Single tenant applications: lots of waste AppAppDbDbAppAppDedicated software instances: under-utilized hardware stacks for each tenantWe prevent a great deal of hardware and software from being sold to our customersWe do with 1000 machines what would otherwise require 100,000+Multiple staffs to hire, train, and payDatabase instances are expensive to maintainPoint-in-time backup strategies are expensiveRolling upgrades are hardMultiple versions to support3rd party vendor versionsMore importantly your own code base (patches)Your data schemaDbDb
5 Multi-tenancy benefits are self-evident But isolation is much easier said than done… AppDbOnly one software instance and hardware stack for multiple tenantsSignificant cost savings in hardwareProvisioning is free – this is critical (we add rows to existing tables – this is the sweet spot for RBDMS)Actually we have about 10 such identical “POD’s” running 10,000’s of tenants each (avoid SPOF’s)Only one staff needed to maintain the serviceSurprisingly small number of humans run the entire universe of tenantsMonitoring and automation are key, but much easier in a shared environmentVery small number of versions to support3rd party platform support matrix collapseYour own code base and schema are the same everywhereCritically: our developers are only working on 2 versions of our code at a time – much more innovation possible in the long term this way
6 Our religion: Not all “multi-tenant” designs are created equal “Can’t we create aseparate stack for just thisone customer? I promiseit’s just this one…”AppAppDbDbIt’s a slippery slope, this is hardUpgrade timingFunctionality for one tenantPerformance tuning for one tenantUI for one tenantWorkflow for one tenantSecurity concerns (real or perceived)Has any other enterprise database application resisted this temptation?
7 Introducing the Force.com metadata-driven, multi-tenant, Internet application platform Poly-MorphicApplicationSeparate concerns:Tenant-specific metadataTenant business dataShared runtime engine (kernel)Metadata-driven architectures facilitate multi-tenant applicationsOver time less and less functionally is statically built into our kernelInstead, each tenant application is composed at runtime from rich metadata
8 Key Architectural Principles Stateless AppServersDatabase system of recordNo DDLAll tables partitioned by OrgIdSmart PKs, Polymorphic FKsCreative de-normalization and pivotingUse every RDBMS feature/trickAll tenants are equal (AppServers scale horizontally)We can run the site with just the database and no caching (just performs/scales worse)Only cache metadata at our layer. RDBMS SGA caches dataNo global locks everPrimary key prefix = Object Type IdPolymorphism allows pivot tables and flex schema to work
9 Metadata, data, and pivot table structures store data corresponding to virtual data structures A simplified look at some key metadata and data tablesNo DDL at runtime!Our flex schema uses clever de-normalization – many patents pendingWe add data across tables for reporting (the big grid-like CFDATA tables)But we also have pivoted specialized tables
10 The Objects table stores metadata about custom objects (tables) Naturally we have our own multi-tenant catalog tables – like Oracle’s own system catalog, but for our own bookkeepingThe definition of our custom objects is one of the most important metadata tables
11 The Fields table stores metadata about custom fields (columns) And we have our own field catalog table – our field types are richer than standard RDBMS types.For instance multi-select picklists are stored in a special compact representation
12 The Data heap table stores all structured data corresponding to custom objects Massive, massive table – wide and deep, billions of rows, partitioned, sharedAll flex data stored in varchar columns – canonical forms for data typesObjId distinguishes different “tables” for the same tenantThe same physical column is used for different purposes across tenants and across objects within a tenantOur query engine emits the proper SQL for all readers and writers of data (e.g. the Api and reporting)
13 A single slot can store various types of data that originate from different objects Example of a single columnNote that the underlying RDBMS cannot know about this usage.Small conversion penalty for native data type processingSo how do indexing and statistics work?
14 The Indexes pivot table manages tenant-specific selective indexes An example of a critical de-normalized pivot tableData from a single object’s slot of a VAL column is copied to this tall/skinny table as wellUsed by the external Id feature for example…Real RDBMS btree indexes are then used on the value columnsHowever, to use this table, SQL must lead with it10’s of billions of rows are ok – tables are like super models
15 The UniqueFields pivot table facilitates uniqueness for custom fields The uniqueness feature is backed by a real RDBMS unique index.Same schema as previous slide, but with unique indexThe platform detects the index violation and then provides additional information (the Id of the duplicating row)Other pivot tables: NameNorm, SNL, etc.
16 The Relationships pivot table facilitates referential integrity and optimizes joins Navigation from parent to children would not be possible in the heap tableThis table has indexes in both directionsAgain, the various users of this table must know to inject this table into the SQL
17 All data & metadata structures are partitioned to improve performance and manageability Tables hash partitioned by OrgIdSeparate conn pools point to physical hostsApp tier is also dynamically partitioned by OrgIdDistributed metadata cache w/transactional invalidationList partitioning has been suggested, but questions about DDLLocal indexes all lead by OrgIdStability in the event of failover (RAC and appservers)
18 Application Framework: a whole lot for free Native Declarative featuresBulk ProcessingThe Recycle BinFull Text SearchSmart Bulk DMLWeb Services APIsOur “database” is really an application server too – many higher level features built-in and tightly integrated
19 Force.com’s native Application Framework provides declarative development, no coding Example: activity tracking, field history trackingBtw, the audit data is stored in another massive pivot table
20 Validation rules and simple formulas: Business analysts can “code” these Fairly straight-forward, like check constraints in a RDBMS
21 Not so simple: Rollup-summary fields provide for easy cross-object summaries RSF’s must be de-normalized for grid reportingVery hard to get right all the timeDone automatically by the bulk save engineInvalidated and bulk re-calculated when needed
22 Force.com’s bulk processing optimizations reduce overhead for data loads We have spent years making the save loop bulk from beginning to endRetry and partial save logic is novelWell-defined blending of declarative (RSF, validation, auditing) and procedural (Apex) logic
23 Data definition processing is optimized to avoid performance hits or concurrency limits Examples:Sort all records by primary key before attempting DMLOperate on tables in deterministic orderSlot reallocation for field datatype changeDeferred calculation for new rollup-summary fieldBackground processing of mass changesAgain, years of experience and database expertiseBenefit from our best practices
24 The Recycle Bin: Smart Undeletes Individual object instances (records)Related object instances (parent/child records)Entire fields and objects (dropped columns and tables)RestoreAnother high-level application feature not provided by standard RDBMSDeep cascading operations are difficult to get right (deadlocks e.g.)Restoration across schema changes is novel
25 Force.com’s full-text search engine Asynchronously maintains indexes for all text fieldsMRU caches contain recently updated objectsOptimizes ranking of search result records based on current user, modification history, and weighting preferencesAnother hard platform area to get rightYears of iteration for massive multi-tenant performanceVarious point solutions available on the market – but our is integrated for free
26 Multi-tenant Query Optimization Principles Consistent SQL generation across the applicationDeep awareness of pivot table structureFlex schema does impose a costTenant, user, object, fields statistics are crucialNo runaway queries allowedDeep integration with the sharing modelCore competency of our platform“Index” plans become extra joins plus indexesDML maintains some statistics synchronously; others gathered on a scheduled basisSharing: Application layer blended into the DB layer (only possible with a platform)
27 Multi-tenant optimizer statistics Force.com’s query optimizer writes optimal queries for internal data access operationsMulti-tenant optimizer statisticsMulti-tenant optimization based on our flex schema with knowledge of de-normalized pivot tables and sharing modelData sharing is scalable and robust and tightly integrated into the optimizer – another high-level application feature pushed into the DB layerStatistics kept automatically for custom indexes and foreign keysHistograms for picklist values, special statistics for record owner field, etc.
28 Pre-Query Selectivity Measurements The optimizer considers pre-query selectivity measurements when writing a queryPre-Query Selectivity Measurements… use of index related to filter.High… ordered hash join; drive using Data table.Low… nested loops join; drive using view of rows that the user can see.Write final database access query, forcing …FilterUserSome statistics are cached – others require dynamic gatheringQuery plans differ not only in hints but in the skeleton of the query itself (extra tables joined)Ongoing research into parallelism, hashing vs nested loops, etc.
29 Apex: Force.com’s procedural frontier Adding a procedural language to our service was not a simple decision – eventually we knew we had to do it, but safelyFunctionality of pl/sqlSyntax and idioms of Java/C#But direct access to salesforce.com features for SOQL, DML, cursors, etc.Tightly integrated into our save logicYears of refactoring of our core database interaction made this possible
30 Apex code is stored as metadata, interpreted at runtime, and cached for scalability The code itself is stored in metadata like everything elseThe actual compiled form is just a POJO (we’re not compiling any Java – we built our own parser (ANTLR) and runtime, written in Java)The Apex VM has a dynamic class loader that can re-use the same compiled form for multiple concurrent requestsManaged code is shared across tenants
31 Apex is deeply integrated with platform features Bulk DMLand messagingAsynchronous processing (Futures)XmlStream / HTTP (RESTful) services classesDeclarative exposure as new Web ServicesThe “ADK” provides access to powerful platform features
32 Force.com governs Apex code execution Limits on:CPUMemory# of DML statements# calculations# web service calls… and moreGovernance is a key concept for any multi-tenant shared systemBuilt for safety first – controlling the runtime completely made this much easierPlus we force people to use good practices such as Bulk SQL/DML
33 Unit tests must accompany Apex code Required 75% code coverageProfiling is built into the platformRun during application installAll tests are run before each platform release by usNovel aspect of our development environment – the development and deployment model are combinedYes, we are like big brother, but we’re not apologeticThe ability to run on upgrade hammer against tenant tests is invaluable for our platform server compatibility story
34 Force.com is a proven multi-tenant application platform that performs and scales Quarterly Transactions (billions)Page Response Time (ms)As our workload continues to evolve with Apex and VisualForce, we continue to focus on performance and scalabilityEmergent property of developing and running the siteWe have access to all the logs and all the data and all the exceptions and all the performance informationVirtuous cycle of improvement – everyone benefits200520062007Fiscal Year
35 Concluding Remarks PaaS is a major architectural shifts PaaS is Application focused, high level of abstractionForce.com is the most mature, proven PaaS offering available todayOptimized for fast, secure, and reliable multi-tenant application development and deployment