1. Data Integration with Power Query

2. Let’s talk about me
Nick Chobotar Business Intelligence (BI) Analyst, I am out of Philadelphia office. I specialize in development and implementation of business intelligence solutions using Microsoft BI stack in combination with dimensional data modelling techniques. My experience includes design and development of data warehouses and data marts, ETL and standardization of data from various conventional and unconventional data sources, design and development of tabular SSAS semantic solutions as well as dashboards and end user reporting.
Contact:

3. Let’s talk about Nihilent Technologies, Inc.
BI & Advanced Analytics
Cloud Transformation Services
Enterprise Transformation Consulting
Customer Relationship Management
Application Development
Portals & Collaboration
1,500+
Employees
14, Migrated Cloud Users & devices
Healthcare
Manufacturing
Financial Services
Retail

4. Proven Delivery
Healthcare
Manufacturing
Financial Services
Retail

5. Agenda Power Query introduction Power Query M Language Basics
Type system
Joins
Parameters
Model Overview and Design
Data extraction from various sources:
SQL Server (Product Subcategory)
Odata (Product)
OneDrive (Product Category)
List.Generate (Sales fact – mockup)
DimDate (Calendar) dimension
Data transformation
Query grouping

6. Power Query Evolution Early 2013 Data Explorer excel add-in
Late 2013 Power Query excel add-in
2016 Excel (native)
2015 part of Power BI Desktop
2017 Power Apps
2017 SQL Server
2018 Power BI Web service (Datapools in preview)

7. Power Query SDK
The Power Query SDK provides an M language service for Visual Studio, and a framework for building Data Connectors for Microsoft Power BI. This extension adds support for the .mproj project type as well as syntax highlighting and intellisense for .pq files

8. Power Query M Language Basics

9. Standard Library Functions
M function 25 Categories
1) Accessing data functions
2) Action functions
3) Binary functions
4) Combiner functions
5) Comparer functions
6) Date functions
7) DateTime functions
8) DateTimeZone functions
9) Duration functions
10) Error handling
11) Expression functions
12) Function values
13) List functions
14) Lines functions
15) Logical functions
16) Number functions
17) Record functions
18) Replacer functions
19) Splitter functions
20) Table functions
21) Text functions
22) Time functions
23) Type functions
24) Uri functions
25) Value functions
=#shared is my best friend
powerpivotpro.com
NOTE: All functions are prefixed with its category e.g. Text.Upper(), Number.Round(), List.Max(), Date.Year()

10. Power Query Types Built in Types: Complex Types:
binary, date, datetime, datetimezone, duration, logical, null, number, text, time
Complex Types:
Tables
Lists
Records
Functions

11. Types Demo

12. Joins
Merge Queries with UI
Create a Join with M

13. Joins Demo

14. Parameters
Parameter is a query that returns a single value that could be used by multiple queries.
Three ways of working with parameters:
Home -> Manage Parameters button -> Click New
New Query -> Blank Query (right click Covert To Parameter)
New Query -> Blank Query ( Parameter as a query name)
Common uses:
Elimination of hardcoding
Data filtering
Changing data sources

15. Parameters Demo

16. Model Overview Data Sources Transformations Finished Model OneDrive
Generated Fact
SQL
Server
OData
Generated Calendar
Data Sources
Transformations
Finished Model

17. Model Diagram

18. Fact Table with List.Generate
Generates a list of values given four functions that generate the initial value initial, test against a condition condition, and if successful select the result and generate the next value next.
List.Generate(
A starting condition,
When to stop,
How to make the next record,
What to put into the list )
Excellent technique for mocking data
Fact table fields:
Date
ProductID,
Quantity,
UnitCost,
UnitSalesPrice

19. Date (Calendar) Dimension
Year
QuarterNumber
QuarterIndex
Quarter
MonthNumber
MonthIndex
Month
MonthName
MonthNameShort
WeekNumber
WeekIndex
Week
DayOfYear
DayOfQuarter
DayOfMonth
DayOfWeek
DayName
DayNameShort
Weekday
WeekdayFlag
WorkdayFlag
CurrentYear
CurrentQuarter
CurrentMonth
CurrentWeek
Today
CurrentYTD
CurrentQTD
CurrentMTD
CurrentWTD
QuarterVisual

20. SQL Server DimProductSubcategory Dimension
Query Folding - pushing certain data transformations back to the data source.
List of the most common transformations Query folding
Filtering (on rows or columns)
Joins
Aggregates and GROUP BY
Pivot and unpivot
Numeric calculations
Simple transformations, such as UPPER
Do not write custom M before query folding steps !!!!
Audit steps or Groups
SQL Server Profiler
Power Query (right click applied step - “View Native Query”)

21. OneDrive DimProductCategory Dimension
let
sharingUrl = "insert URL here",
base64Value = Binary.ToText(Text.ToBinary(sharingUrl, TextEncoding.Utf8), BinaryEncoding.Base64),
encodedUrl = " & base64Value & "/root/content",
Source = Web.Contents(encodedUrl) in
Source

22. Odata DimProduct Dimension
AdventureWorks OData Feed sample
CompanySales
Documents
ManufacturingInstructions
ProductCatalog
TerritorySalesDrilldown
WorkOrderRouting
Source:

23. Questions

24. References: Power Query SDK
Power Apps
Searchable Power Query M reference