1. Unleashing the Power of Microsoft Power Query
Microsoft Ignite 2016
5/6/ :17 PM
Unleashing the Power of Microsoft Power Query
DA326
Matt Masson
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Power Query in Excel
Power Query add-in for Excel 2010/2013, and the “Get & Transform Data” experience in Excel 2016.

3. Power Query in Power BI Desktop
Provides the Get Data experience for Power BI Desktop, providing consistent data access to 74+ data sources.

4. Power Query in SQL Server Data Tools
Now integrated in SQL Server Data Tools (SQL vNext CTP) for Analysis Services Tabular projects, closing the gap in the Self-Service to BI Pro grow up story!

5. Power Query Editor

6. What is it so Popular?
Connectivity to a wide range of data sources of various types, shapes, and sizes;
Highly interactive and intuitive experience for rapid and iterative construction of queries over any data source, any size.
Consistency of experience, and parity of query capabilities over all data sources.
Joins across different data sources; ability to create custom views over data that can then be shared with others in your organization

7. Power BI Desktop Get Data Model Visualize Publish Consume
Build your queries
Analyze and discover
Create shared dashboards
Define relationships and measures
Send to PowerBI.com

8. Demo Connectivity experience Data transformations Query folding
Microsoft Ignite 2016
5/6/ :17 PM
Demo
Connectivity experience
Data transformations
Query folding
Data mashup
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. 5/6/ :17 PM
M in Power Query
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Transformations
Formula bar
Query steps

11. M Language Flow Evaluation flows from one step to the next
5/6/2018
M Language Flow
Evaluation flows from one step to the next
Except when it doesn’t
Result of previous step typically used in the next step
Step is stored as a variable and can be reused later on
M has Lazy Evaluation - values are ignored if not used
let
Source = Web.Page(Web.Contents("
WebTable = Source{0}[Data],
RenamedColumns = Table.RenameColumns(WebTable,{{"Column1", "Rank"}, {"Column2", "2013"}}) in
RenamedColumns
let
Source = Sql.Database("localhost", "AdventureWorksDW2012"),
DimCat = Source{[Schema="dbo",Item="DimProductCategory"]}[Data],
DimSubCat = Source{[Schema="dbo",Item="DimProductSubcategory"]}[Data],
DimCustomer = Source{[Schema="dbo",Item="DimCustomer"]}[Data],
Joined = Table.NestedJoin(DimSubCat,{"CategoryKey"},DimCat,{"CategoryKey"},"Category",JoinKind.Inner) in
Joined
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. TechReady 18
5/6/2018
Query Folding
Power Query pushes work back to the source system whenever it can.
let
Source = Sql.Database("localhost", "AdventureWorksDW2012"),
dbo_DimProduct = Source{[Schema="dbo",Item="DimProduct"]}[Data],
RemovedOtherColumns = Table.SelectColumns(dbo_DimProduct,{"ProductKey", "EnglishProductName"}),
RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"EnglishProductName", "Product"}}),
FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10) in
FilteredRows
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Query Folding SELECT ProductKey, EnglishProductName as [Product]
5/6/2018
Query Folding
let
Source = Sql.Database("localhost", "AdventureWorksDW2012"),
dbo_DimProduct = Source{[Schema="dbo",Item="DimProduct"]}[Data],
RemovedOtherColumns = Table.SelectColumns(dbo_DimProduct,{"ProductKey", "EnglishProductName"}),
RenamedColumns = Table.RenameColumns(RemovedOtherColumns,{{"EnglishProductName", "Product"}}),
FilteredRows = Table.SelectRows(RenamedColumns, each [ProductKey] < 10) in
FilteredRows
SELECT ProductKey, EnglishProductName as [Product]
FROM [dbo].[DimProduct]
WHERE [ProductKey] < 10
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Type System Primitive types Complex types any, none
TechReady 18
5/6/2018
Type System
Primitive types
any, none
null, logical, number, text, binary
time, date, datetime, datetimezone, duration
Complex types
list, record, table, function
Sql.Database = (server as text, database as text, optional options as nullable record) as table
MyCoolFunction = (index as number, category as text) as nullable table
Date.StartOfDay = (dateTime as any) as any
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Ascribed Types Commonly used ascribed types
TechReady 18
5/6/2018
Ascribed Types
A value's ascribed type is the type to which a value is declared to conform. When a value is ascribed a type, only a limited conformance check occurs. M does not perform conformance checking beyond a nullable primitive type. M program authors that choose to ascribe values with type definitions more complex than a nullable primitive-type must ensure that such values conform to these types.
Commonly used ascribed types
Byte.Type, Int8.Type, Int16.Type, Int32.Type, Int64.Type
Single.Type, Double.Type, Decimal.Type, Currency.Type, Percentage.Type
Character.Type
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Null null Logical true, false Number 1, 1.2, 1.2e-3, #infinity, #nan
SIMPLE VALUE
LITERAL
Null
null
Logical
true, false
Number
1, 1.2, 1.2e-3, #infinity, #nan
Text
"hello, world!"
Date
#date(2013, 3, 8)
Time
#time(15, 10, 0)
DateTime
#datetime(2013, 3, 8, 15, 10, 0)
DateTimeZone
#datetimezone(2013, 3, 8, 15, 10, 0, -8, 0)
Duration
#duration(1, 13, 59, 12.34)

17. Complex Values - Lists

18. Complex Values - Records

19. Complex Values - Tables

20. Complex Values - Other Function Type Binary LITERAL COMPLEX VALUE
MyFunction = ( x, y, optional z ) =>
if z = null then
x + y
else
(x + y) / z
Type
type table [ n = number, #"n^2" = number ]
Binary
#binary({0x68, 0x65, 0x6C, 0x6C, 0x6F})

21. Anatomy of a Table
let
Source = Excel.Workbook(File.Contents("Sales.xlsx")),
MyTable = Source{[Item="Data",Kind="Sheet"]}[Data] in
MyTable

22. Anatomy of a Table A column in the table is a list.
MyTable[ProductKey] = { 217, 231, 485, 538, 480, 528, 480, 477 }

23. Anatomy of a Table A row in the table is a record.
MyTable{2} = [ ProductKey = 485, OrderQuantity = 1, UnitPrice = 21.98,
SalesAmount = 21.98, OrderDate = #date(2008, 1, 1) ]

24. Anatomy of a Table A table is a list of records
MyTable = {[ProductKey = 217,…],[ProductKey = 231,…],…}

25. Wait what happened to my nested tables?
Power BI, Excel, and Analysis Services only support primitive types
Complex types only exist within the M Engine and Power Query
Complex values are stripped out when the data is loaded
Navigation properties are removed entirely
Other nested values are replaced with a text value (i.e. [Table], [Record])

26. TechReady 18
5/6/2018
Symbolic Operators
Arithmetic (numbers, durations, and date-time values)
+x -x x + y x - y x * y x / y
Equality, inequality
x = y x <> y
Comparison
x < y x <= y x > y x >= y
Concatenation
(text & text, list & list, record & record, table & table, date & time)
x & y
Field access (or null)
r[f]
r[f]?
Projection (fill with null)
r[[f1],[f2]]
r[[f1],[f2]]?
Item lookup (or null)
l{i}
l{i}?
Item lookup by key (or null)
l{[k1=v1, k2=v2]}
l{[k1=v1, k2=v2]}?
Function application
f(a1, a2)
Not-implemented error …
Self-recursive reference @n
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Unary Functions Many library functions take functions as arguments
TechReady 18
5/6/2018
Unary Functions
Many library functions take functions as arguments
Often, those parameter functions are unary
A special syntactic form helps construct unary function values
An ‘each’ expression is just shorthand for a unary function
The single parameter of an ‘each’ function is named _
For conciseness and to get close to the DAX syntax, the _ can be omitted when accessing fields or columns
Table.SelectRows( table, (r) => r[Manager] = r[Buddy] )
Table.SelectRows( table, each _[Manager] = _[Buddy] )
Table.SelectRows( table, each [Manager] = [Buddy] )
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Special Syntactic Forms
TechReady 18
5/6/2018
Special Syntactic Forms
Conditional expression
Let expression
Error expression (think: “throw”)
Terminates evaluation and raises error
Try expression (think: “catch”)
Attempts evaluation, optional “otherwise”
Encodes results and errors as record values
if 1 < 2 then "hurray" else "sad face"
let x = Number.Atan(3) in x * x
error "Here I go wrong"
error [ Reason = "Expression.Error", Message = "'t' should be positive", Detail = t ]
try error "Bad" // [ HasError = true,
// Error = [ // Reason = "Expression.Error",
// Message = "Bad", // Detail = null // ]
// ]
try error "Bad" otherwise 42 // 42
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Demo Starting from a Blank Query Creating calculated columns
Writing your own functions

30. Tips and Tricks - Leverage the UI
5/6/ :17 PM
Tips and Tricks - Leverage the UI
Generate steps using the UI, then tweak the code Some things only work through the UI (“auto steps”)
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Tips and Tricks - Library Functions
5/6/ :17 PM
Tips and Tricks - Library Functions
Use #shared to see all exported functions (and keywords) Typing in the function name will display its help and prompt for parameter
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Tips and Tricks - Troubleshooting
5/6/ :17 PM
Tips and Tricks - Troubleshooting
Use try/otherwise to isolate errors Select and remove rows with errors Table.Buffer will stop folding from occurring
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. References Power Query Library Function Reference on MSDN
Power Query Language Specification

34. Continue your Ignite learning path
5/6/ :17 PM
Continue your Ignite learning path
Visit Channel 9 to access a wide range of Microsoft training and event recordings
Head to the TechNet Eval Centre to download trials of the latest Microsoft products
Visit Microsoft Virtual Academy for free online training visit
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. Win a Spark After Dark drone pilot pass by completing your session evaluation ASAP  #MSAUIGNITE

36. 5/6/ :17 PM
Thank you Chat with me in the Speaker Lounge Find |
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.