1. Advanced Tips And Tricks For Power Query
Chris Webb
@technitrain

2. Who Am I? Chris Webb UK-based consultant and trainer:
UK-based consultant and trainer:
Author of several books:
MDX Solutions
Expert Cube Development with SSAS 2008
Analysis Services 2012: The BISM Tabular Model
Power Query for Power BI and Excel
SQL Server MVP
Blogger:

3. Agenda Power Query recap Introduction to M Parameterised queries
Privacy levels and the Formula Firewall
Query folding and performance
Creating and using M functions
Combining data from multiple data sources

4. Power Query Recap Power Query is a self-service data manipulation tool
Excel add-in for Excel 2010/2013
Native feature of Excel 2016 as “Get & Transform”
The “Get Data” part of Power BI Desktop
Integrated with SSIS, SSRS and SSAS at some point?
It does three things:
Connects to multiple data sources
Applies transformations to that data
Loads data to Excel tables and/or the Data Model
New builds of Power Query released almost every month

5. M Power Query is actually a UI on a language (unofficially) called M
M is a functional language like F#
Nothing like Excel formula language, DAX or VBA 
Everything you do in the UI is translated to M
The M code is visible in the Formula Bar
You can edit the whole M script for a query in the Advanced Editor
Understanding how to write M is the key to writing more complex transformations and calculations

6. Values and Expressions
Expressions are central to the M language
Expressions can be calculated to return Values
1+1 is an expression
1+1 returns the value 2
Values can be:
Primitives like numbers or text
Tables, lists, records, functions…
Each Power Query query is a single expression that returns a single value

7. Let Expressions
Each Power Query query created by the user interface consists of a single Let expression
Let expressions contain a series of named expressions that can reference other named expressions
These become the steps in a query
The Let expression finishes with an In, where the final value of the expression is calculated
This is the output of the query
It can reference any of the named expressions

8. Parameterised Queries
For large data sources, users should be prevented from downloading all data to Excel
Takes too long
May crash Excel
Therefore you need to give users an easy way to select the data they want to download
This involves
Giving them an easy-to-use UI
Writing M code to handle the parameterization

9. Parameterised Queries
Power Query queries can reference other Power Query queries
Therefore you can use Power Query to load data on what to filter by from an Excel table or named range
…and pass that data as a parameter to a second Power Query query to load the main data set

10. Query Folding Fast data load performance is very important for users
For certain data sources (relational databases , OData and a few others) Power Query can push logic back to the data source
Eg translate your Power Query transformations to SQL
This is called Query Folding and can make a massive difference to performance

11. Query Folding
There is no indication in the UI that Query Folding is happening
You need to use tools like SQL Server Profiler
Certain operations prevent folding, such as using native SQL queries
A query may be partially folded – reordering steps can increase the amount that is folded

12. Privacy Levels And The Formula Firewall
Power Query cares about data privacy!
Sending data from one source as parameter values to another source could be dangerous
Power Query will prompt you to set data privacy levels to govern whether this can happen
Turning on ‘Fast Combine’ stops Power Query from checking this
But this will need to be turned on every time the workbook is opened on a new PC

13. M Functions
Functions are expressions that take parameters to return a value
The Standard Library contains all of the built-in functions
You can define your own functions in the Advanced Editor
They can then be invoked in
Within the query you’re writing
Other queries in the same workbook
Functions are a great way of:
Sharing logic between queries
Separating complex logic from otherwise simple queries
Controlling how end-users access data sources

14. Functions For Data Access
M functions can be an alternative to parameterised queries
Use M functions when an end user needs to create multiple queries against the same data source to get different slices of data
With more advanced M code you can provide:
Default values for parameters
A list of available values that can be selected from a dropdown box

15. Combining Data From Multiple Sources
Power Query makes it easy to combine data from multiple, identically structured csv files
However end users often need to do the same thing with other data sources, for example Excel files
The solution is to create an M function to do this

16. Thanks!