Presentation held by Tomislav Piasevoli at the local WinDays 11 conference, Rovinj, Croatia. Monday, 16:10-17:00, Room 6. http://www.mswindays.com/

Business Intelligence Semantic Model Tomislav Piasevoli SoftPro Tetral d.o.o.

As always, a big „Thank you” goes to sponsors ...

... and you for keeping your phones muted.

Contents What is BISM? PowerPivot SQL Server „Denali” The significance of BISM Guidelines This is what we’re going to talk about today

Prerequisites interest in BI curiosity 

Few questions to start with ... how many of you worked in PowerPivot? how many used PowerPivot on SharePoint? how many of you has experience in building cubes? how many used cubes for reporting? [to get a feel of the audience and their experience, in order to be able to adjust the speed and the depth of the information presented in the presentation slides] => I estimate there were ~50 attendees, only ~15% had previous experience with OLAP/PowerPivot

1 What is BISM?

Analytical systems turn data into information goals of their architecture optimize access to source data prepare the data for analysis In order to explain what BISM is, we’ll first start with the analytical systems in general. There are two main goals in their architecture, something that every analytical system inherintly must do: taking care of data access and enriching the data for analysis.

I – Optimize access to source data store the data in structures suitable for analysis and ad-hoc querying or leave them „as is“ on the source requirements good performance flexibility scalability ... which technology to choose/use? Data can be left in the source or aquired and stored in appropriate format in another place (moved). Thereby certain requirements must be met: performance flexibility, scalability and so on. Which technology is the right one?

MOLAP, ROLAP, HOLAP row-based databases MOLAP = multidimensional ROLAP = relational HOLAP = hybrid (only for measure groups = facts) performance based on aggregations partitioning compression (MOLAP) not limited by available memory SQL Server Analysis Services is architectured to store the data in one of the three storage modes. In case of ROLAP and HOLAP, data is left in the source system; in case of MOLAP, it is aquired and compressed on a separate location. In other words, existing SSAS supports both scenarios for data access. Other characteristics are that the data can be partitioned, aggregations can be built, and that the storage is designed not to be dependent on available memory.

II – Prepare the data for analysis enable the data analysis like there’s only one source of data actually, there can be many various data sources relational databases Excel files text files RSS feeds, ... where to perform the join of the distributed systems? provide the interpretation (business meaning) to data how to make it comprehendable to the end users? From the data preparation perspective, data from various sources can be gathered and represented to end-users like it comes from a single source. One of the challenges is where to perform this virtual integration? The other is how to make it understandable to end-users, how to enrich the data with meaning?

Semantic data model a metadata layer (on top of the data) describes concepts (entities) and connections (relations) between them user-oriented shows what the tables and relations represent in the real world semantics = the study of meaning Semantic data model serves that purpose. It’s a user-oriented metadata layer that comes on top of the data and enriches it. BI developers model the data in such a way that it represents something meaningful to the end-users. The quality of the analytical solution depends on two things: the richness or the capabilities of the underlying semantic model and the knowledge BI developer posseses regarding that model.

UDM existing semantic model for BI came with SQL Server 2005 enables connecting data from various sources add relations between tables add calculations in tables – calculated columns in DSV (T-SQL) over the model – MDX script (MDX) calculations cannot be combined (different languages!) or it’s very hard to do so SQL Server Analysis Services uses UDM. UDM stands for Unified Dimensional Model. It exists from SQL Server 2005 onwards and it features two main designing capabilities: building relations and calculations.

UDM – data modeling 1/3 Here’s the example of UDM in action A DSV (data source view) and its objects is a place where the relations are made and various data sources are virtually linked. The model supports multiple key relations, parent-child relations, many-to-many relations, custom views and calculations on top of the existing objects (tables or views) and other friendly features.

UDM – data modeling 2/3 The Dimension designer is a place where we design dimensions, establish attribute relationships, build custom hierarchical structures and many other things. The designer allows for a very fine and detailed configuration of dimensions and its properties.

UDM – data modeling 3/3 The Cube designer is a place where we design cubes, establish relation between fact tables and dimensions, provide custom calculations, actions, design partitions and aggregations and many other things. User interface and exposed properties allow for a very fine tuning of the cube and its minor objects.

UDM – data analysis Finally, the model can be analyzed in the developers environment or in a separate tool. Here, we’re seeing the data through the prism of the semantic model made by a BI developer for us. The richness of the model enables fast navigation and flexible combination of objects in the model.

SQL Server 2005 Analysis Services UDM Excel 2003 + add-in Excel 2007 & Reporting Services 2005 Report Builder 1 SharePoint 2003 & SharePoint 2007 Reporting Services 3rd party SSAS clients PerformancePoint 2007 MDX MDX SQL Server Analysis Services supports the UDM as its semantic model from the version 2005. This diagram shows which tools were available to connect and query the UDM cubes and the query language they used to get the data from UDM cubes. MDX MDX MDX

SQL Server 2008 Analysis Services UDM Excel 2007 Reporting Services 2008 Report Builder 2 & SharePoint 2007 Reporting Services PerformancePoint Services Excel Services Visio Services 3rd party SSAS clients MDX MDX The support for UDM continues in SSAS 2008. Excel add-in becomes obsolete and Performance Point becomes a service inside the SharePoint. Excel Services and Visio Services emerge there too. MDX MDX

SQL Server 2008 R2 Analysis Services UDM Excel 2010 Reporting Services 2008 R2 Report Builder 3 & SharePoint 2010 Excel Services PerformancePoint Services Visio Services 3rd party SSAS clients MDX MDX Finally, the current version of the SQL Server Analysis Services supports UDM as well. No big changes, only new versions of existing applications. MDX MDX

PowerPivot add-in for Excel 2010 PP Excel 2010 But, at the same time, something else appears. Something completely different. That’s PowerPivot and its data model!

2 The story about PowerPivot ... PowerPivot

PowerPivot SQL Server 2008 R2 Analysis Services as an integrated version covers personal and team BI segments It’s SSAS undercover, a managed version of it.

Integration Excel 2010 SharePoint 2010 free add-in for Excel 2010 (www.powerpivot.com) comes as 32-bit and 64-bit application comes with SQL Server 2008 R2 x64 during the installation, you choose SQL Server PowerPivot for SharePoint It’s an analytical system that comes in two shapes: one, for Excel and the other, for SharePoint.

VertiPaq in-memory column-based database very high data compression scales linearly with the number of cores doesn’t require the process of designing and building aggregations and other tunning envisioned to support partitioning and paging on large data sizes (although it doesn’t do that in PowerPivot) Like any other analytical system, PowerPivot must also take care about storing the data and optimizing it for analysis. Instead of MOLAP, ROLAP and HOLAP, PowerPivot uses its own storage mode called VertiPaq, a vertically organized database.

BISM v1 the new semantic model for BI came with SQL Server 2008 R2 (for PowerPivot!) enables connecting data from various data sources add relations between tables add calculations, two places: in tables – calculated columns (DAX) over the whole model – calculated measures (DAX) calculations can be combined (same language) As mentioned earlier, the other characteristic of analytical systems is their semantic model. Again, PowerPivot has its own semantic model which currently doesn’t have a name, or it’s simply being called “the PowerPivot model”. And although it’s never been called BISM before, based on information about the next version of SQL Server called “Denali” it’s very reasonable to call it the version one of BISM. It came with SQL Server 2008 R2 and brought another way of defining relations and calculations. This time the calculation laguage was DAX. The good things about this model is that calculations defined on two separate places can be combined.

BISM v1 the result of merging todays analytical (UDM = SSAS) and reporting (SDML = SSRS) semantic data model with the goal that it integrates (in due time) reporting, analysis, dashboards and scorecards personal, team and corporate BI segments works in cached (VertiPaq) mode The initial version of this semantic model is, just like the UDM, based on the cached storage.

BISM v1 – data modeling The designer’s interface is simplified in terms of configuration options, yet on the other hand, it is enriched in a sense that the designer sees the data during the modeling process. This, and the relatively easy principles of relating tables and expanding them with additional columns, is very attractive and effective way of modeling the data for analysis. And the simplicity and interactivity it offers makes it appealing for much wider audience than UDM.

BISM v1 – data analysis Data analysis can be performed on spot, right in the model, right during modeling. In fact, the process of modifying the model and analyzing the data becomes a new way of getting insights about the data. Although not nearly as rich as UDM in various features, this flexible semantic model wins in terms of interactivity that one can experience while working with data.

DEMO BISM v1 in SQL Server 2008 R2 Now comes the time for a little demo. The problem is - there is no publicly available version of BISM yet. What to show in the demo? As you’re about to see, there are lots of things hidden in the current release of SQL Server, the 2008 R2 version of it, that prove that BISM is already here although we’re not aware of it at the moment. Here we go!

Short contents of the demo PowerPivot is an arhived SSAS database DAX functions are hidden among MDX functions SharePoint controls with the BISM instance of SSAS the existance of new DMVs for BISM instance on SharePoint BISM models support MDX SQL Profiler VertiPaq SE events For those of you that missed the demo, here’s a short overview of the things shown in the demo: Open Excel, start PowerPivot, create a model, save the file, close it, rename the extension of the file (xlsx to zip), find the “item1” file inside that zip, change its extension to “abf” and restore it on the SharePoint PowerPivot instance (PowerPivot model uses the existing SSAS UDM database container) Start a new MDX query in SSMS, go to the functions list, expand several groups of functions, notice empty functions (no names), drag and drop few of those empty functions in the editor and see what they turn into - those are DAX functions (EARIER, ...) hidden among regular MDX functions; the same thing can be shown using DMVs (the Origin field represents: 1 = UDM/MDX, 3 = VBA/EXCEL, 4= BISM/DAX) Connect SSMS to SSAS running in integrated mode on SharePoint, navigate the databases and notice their names - guids are a sign the SharePoint takes care of them completely The last 3 DMVs show tables for SSAS running on SharePoint, their memory footprints, etc. MDX query can be run against SSAS cube running in SharePoint integrated mode MDX queries targeting SSAS on SharePoint show VertiPaq SE events in SQL Server Profiler

3 What comes next? SQL Server „DENALI”

BISM v2 one model for all reporting, analysis, dashboards, scorecards personal, team, corporate BI has a relational and multidimensional API it was announced that it will eventually support both cached (VertiPaq) and the pass-through* (realtime) mode * only SQL Server data sources for now BISM is a semantic model envisioned to encompass reporting and analysis in a broader sense.

Pass-through no additional database data stays as is in the original structures ideal for the realtime analysis “Denali” will enable the other type of data access, the so-called “pass-through” mode. As mentioned in the introduction, one of the option is to leave the data in the source system. “Pass-through” is that option; VertiPaq means the data is aquired and stored separately.

BI semantic model and its layers Data model relational model supports both relational (DAX) and multidimensional(MDX) interface Business logic DAX calculations (columns and measures) after„Denali” maybe even MDX Data access cached (VertiPaq) in „Denali” also the realtime (pass-through mode) The slide is based on the TechNet source: http://blogs.technet.com/b/dataplatforminsider/archive/2010/11/12/analysis-services-roadmap-for-sql-server-denali-and-beyond.aspx It shows the key BISM features across three important layers. The lower one is what we called “optimizing data access” in the introduction of this presentation, the upper two are the semantic layer on top of the data: establishing relations, that is, data modeling, and providing calculations, that is, enriching the data model.

Analysis Services „Denali” - UDM Excel 2010 Reporting Services „Denali” SharePoint 2010 Excel Services Reporting Services PerformancePoint Services Visio Services 3rd party SSAS clients Project Crescent MDX MDX As before, UDM in will continue to enjoy the support of various applications in SQL Server “Denali”. In addition to that, new applications may appear as long as they support MDX as the query language. MDX MDX ? MDX

Analysis Services „Denali” - BISM Excel 2010 Reporting Services „Denali” SharePoint 2010 Excel Services Reporting Services PerformancePoint Services Visio Services 3rd party SSAS clients Project Crescent DAX ? MDX MDX BISM, on the other hand will have its standalone incarnation in “Denali”. Because of its support for both multidimensional and relational interface, existing applications will be able to use it by issueing MDX queries. However, new applications will emerge, the ones that will support DAX query language. Will some of the existing application or services that we know support DAX too? We are about to see that soon ... MDX DAX MDX DAX

PowerPivot „Denali” add-in for Excel 2010 BISM Excel 2010 Excel and its PowerPivot stay together as an integrated environment for personal BI.

Announced improvements of BISM* hierarchies, KPIs, parent-child, drillthrough, perspectives additional DAX functions (RankX, DistinctCount, GroupBy, Lookup, ...) security (role-based with Active Directory, column/row based) * doesn’t mean that „Denali“ will actually have these exact features The following features were mentioned during the PASS Summit 2010 conference

What BISM in „Denali” won’t have? some of the UDM features scripts, actions, translations, role-playing dimensions object model write-back other realtime for non-SQL Server data sources MDX query support for realtime Also, based on the PASS Summit 2010 announcements

UDM improvements in „Denali” mostly outside of GUI no more 4GB string store limit better support for PowerShell better monitoring (XEvents) various engine optimizations the goal is to make the SSAS (UDM instance) more scalable UDM will have its own improvements, although many will not be visible.

The Significance of BISM 4 Which brings us to the significants of BISM. How important is BISM, what it really means to all of us? The Significance of BISM

Advantages of BISM relatively simple model (no cubes, dimensions, measures, ...) fast response no matter the granularity of query flexible (every column is also a measure and an attribute) DAX calculations are similar to Excel formulas relational model and result format is more understandable and user-friendly to majority of people BISM brings simplicity in analytics. It lowers the entering bar and hence allows more people to do BI with their data, whatever that may be to them.

The same semantic model PowerPivot for Excel PowerPivot for SharePoint SSAS „Denali” Another important aspect is that the same semantic model will be available in all usage scenarios: personal BI, team BI and corporate BI.

Scalability Excel SharePoint „Denali” Having the same model that can scale from a personal BI all the way up to the corporate BI is a great proposition. That means the model is simple enough to be used in the small environment by non-professionals, yet it is powerful enough to support, in the beginning definitely some and later other corporate scenarios. Here, it is appropriate to mention that the model started with Excel and SharePoint, and that “Denali” will be the first stand-alone version of the model, an instance managed by IT professionals.

Other advantages of BISM faster development than in UDM prototyping by end-users they know the best what they want from the data and what the data means (semantic!) easier changes of model reduction of cost in developing the full BI solution Among other benefits BISM promises to bring it’s worth emphasizing that BISM, in personal and team BI, allows end-users to model the data the way they want which is good because they are the ones who understand the meaning of that data the best

New possibilities in data modeling Highlights from various blogs Marco Russo – ad-hoc joins, ETL? Thomas Ivarsson – master-detail design Alberto Ferrari – point-in-time, SCD-2, rank Last but not the least, BISM bring new possibilities in data modeling. Here are some of the ideas or experiments.

5 So what are the guidelines? How does the BISM positions in relation to, let’s say, UDM or other technologies? Guidelines

Positioning the BISM Scalability* ROLAP MOLAP BISM PowerPivot source: Thomas Kejser, SQLCAT ROLAP ColumnStore 50 TB Scalability= - good support for concurrent queries - not dependent on available memory - good maintainability 5 TB MOLAP MOLAP is much more complex than PowerPivot, but it offers greater scalability. Although it is everything but user-friendly, we can make great solutions with it, solutions which can scale up to 5TB. ROLAP is even more limited, but it scales above 50TB space. PowerPivot models can grow up to 2GB which is the limit set by SharePoint if they want to be shared among others. Otherwise, only the memory is the limit. BISM comes in the middle and fills the space between MOLAP and PowerPivot. It adds extra features (and hence becomes less usable by everyone), but on the other hand, it comes as a stand-alone SSAS instance which means it can scale with the memory that is available on the server. Let’s say that today the practical limit is around the 100 GB limit (it’s not just about plugging more chips, data needs to be loaded in memory and that takes time). For the space way above the 50TB there are new ColumnStore indexes (coming in relational engine). SQLCAT (http://sqlcat.com) 100 GB BISM 2 GB PowerPivot Usability

Conclusion BISM is not a replacement for UDM DAX is not a replacement for MDX every model has its advantages BI architects must decide when to apply which model To conclude ... Both models have a place dedicated for themselves, situations where they excel and situations where they are definitely not the best choice. This is how we should understand them.

Advantages and disadvantages BISM UDM scope leaf level m2m Both BISM and UDM have their advantages and disadvantages ad-hoc models complex models

BI developers most of the existing solutions should stay on UDM new solutions use UDM for complex systems (complex calculations, write-back, budgeting, forecasting, ...) try BISM as a potential solution for the rest, mostly simple systems UDM and BISM can be installed on the same server (SSAS instances) BI developers should be familiar with both models (UDM and BISM) and know that for the majority of complex solutions, UDM is still the only way to go. The good thing is that UDM and BISM should be able to co-exist on the same server?

Users, here’s your DIY kit! try and/or migrate to Office 2010 install the free PowerPivot for Excel 2010 add-in load and connect various data sources try and learn DAX analyze your data validate the completness of your model Users and power-users, what about them? They finally got their Do-It-Yourself kit! They can play with it, experiment, the learning curve is not that hard. They should be able to build at least the simple models.

Model is ok? use it for your personal BI share the model with others Excel 2010 and PowerPivot share the model with others SharePoint 2010 and soon SQL Server „Denali“ If the model they’ve built is ok, they can use it or share it with others

Model is not (yet) ok? improve it more experienced colleges give the PowerPivot model to your company guru external consultant improvement is possible? PowerPivot SQL Server „Denali“ (BISM instance) or else ... build the UDM model based on the BISM template implement it on SQL Server „Denali“ (UDM instance) If the model is to simple and needs enhancements, there are two options. Either there’s a guru inside the company who can help or the help should be asked from the external consultant. In either way, the potential solution exist. But what’s more important, the model can, as said before, end up being a PowerPivot application in Excel or SharePoint or it can appear as a new database on SQL Server. “Denali”. All options are open!

source: TechNet That’s what BISM promises to bring!

Questions?

Related content Presentations Workshops MS BI Roadmap (SQL Server Denali) Workshops Build Your First Microsoft BI Solution with SQL Server 2008 R2

Thank you!