1. SharePoint Online Migration API and Improvements
Simon Bourdages
Microsoft PM

2. Introduction Overview Moving content into O365
Current tech
New Capabilities
Things to come
The Migration API and PowerShell
Using the PowerShell
Building Code for the API
Tips and Lessons Learned

3. The Migration API
A dedicated API created by Microsoft in order to provide the 3rd party vendors a dedicated way to bring large volumes of contents into SharePoint Online and OneDrive for Business.
The Migration API is live since May 2015

4. Moving Content in O365 Current Tech
The Migration API was built to bring large Amount of content into the O365 Cloud
Requires the Use of Azure
Timer Job based
Best way to avoid CSOM throttling
Scalable and in continuous improvement

5. Moving Content in O365 Current Tech
Source
Package
Azure Temporary Storage
SharePoint/ODB
Final Destination
File Share,
SharePoint On-Prem and potentially any other data Source
Create package for the API to be able to accept it
Use the Power of ingestion of Azure to bring the content faster in Microsoft network
Timer Job based import in a scalable way that will not hurt the service using back end resources

6. Moving Content in O365 Current Tech
File and Folders
Basic Metadata
Permissions
Versions

7. Moving Content in O365 New Capability
File and Folders
Basic Metadata
Permissions
Versions
Encryption at Rest
Alerts
Taxonomy

8. Moving Content in O365 Things to come
File and Folders
Basic Metadata
Permissions
Versions
Encryption at Rest
Alerts
Taxonomy
Removing the need of a customer Azure Account
TAP program available for Developers
More Based on Usage and Requests

9. PowerShell

10. The Migration API and PowerShell What is What
The PowerShell Commands
PowerShell based
Client side tool
Performs the end to end content migration
The Migration API
CSOM based Call
Requires Transformation of data
Server side Service
Requires the content to already be in Azure container

11. The Migration API and PowerShell Tooling responsibility
Source
Package
Azure Temporary Storage
SharePoint/ODB
Final Destination
File Share,
SharePoint On-Prem and potentially any other data Source
Create Package for the API to be able to accept it
Use the Power of ingestion of Azure to bring the content faster in Microsoft network
Timer Job Based import in a scalable way that will not hurt the service using back end resources
Client Tool
Azure API
SPO Migration API

12. PowerShell The Commands
New
Convert
Set
Submit
New-SPOMigrationPackage
ConvertTo-SPOMigrationTargetedPackage
Set-SPOMigrationPackageAzureSource
Submit-SPOMigrationJob

13. PowerShell commands Flow
File Share Server
Exported Content
Package
Azure
Temporary Storage
SharePoint/ODB
Final Destination
New
SharePoint On Premise
Export-SPWeb
From On-Premise
Convert
Set
Submit
Un Compressed

14. PowerShell Supported scenarios
File and Folders
Basic Metadata
Permissions
Versions
File Share
SP 2010, 2013, 2016

15. New-SPOMigrationPackage
PowerShell
New-SPOMigrationPackage
The first command will read the list of content targeted by the source path and then will generate the appropriate XML to perform migration.
Here are the required parameters to enter (others are optional)
SourcefilesPath : pointing to your content you want to migrate
OutputPackagePath : pointing to your Temporary folder
Ex:
New-SPOMigrationPackage -SourcePath C:\Users\Pictures\ -OutputPackagePath C:\Users\test\tempPackage

16. ConvertTo-SPOMigrationTargetedPackage
PowerShell
ConvertTo-SPOMigrationTargetedPackage
The Second command will simply convert the xml generated in your temporary folder into a format that the destination will understand. This will be the final package.
Here are the required parameters to enter (others are optional)
TargetwebURL : point to the Web of the destination
SourceFilesPath : pointing to your content you want to migrate
SourcePackagePath : pointing to your Temporary package folder
OutputPackagePath : pointing to your final package folder
targetDocumentLibraryPath : This is the path to your destination library
Credentials : SPO credential that has admin rights to the destination site
Ex:
ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl -SourcePackagePath C:\Users\test\tempPackage -OutputPackagePath C:\Users\test\SPOpackage -targetDocumentLibraryPath Documents -sourcefilespath C:\Users\Pictures\ -Credentials (Get-Credential

17. Set-SPOMigrationPackageAzureSource
PowerShell
Set-SPOMigrationPackageAzureSource
The third command will simply convert the XML generated into a format that the destination will understand.
Here are the required parameters to enter (others are optional)
SourceFilePath : Pointing to the content you want to migrate
SourcePackagePath : Pointing to the final package folder
AccountName : The name of the Azure storage account
Accountkey : The key to the Azure storage account
AzureQueueName : The name of the azure queue blob(optional but recommended)
We also recommend to take the output of this command and store it in a variable for the next command.
$temporaryVariable = Set-SPOMigrationPackageAzureSource….
Ex:
$azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath C:\Users\Pictures\ -SourcePackagePath C:\Users\test\SPOpackage -AccountName mystoragename -AccountKey == -AzureQueueName migrationqueue

18. Ship drive requirements
Max 6TB drives
SATA II or III
Format the drive ( not a requirement but still makes things easier)
Prepare your container
Install the Azure tool :

19. Submit-SPOMigrationJob
PowerShell
Submit-SPOMigrationJob
The last command will simply convert the XML generated into a format that the destination will understand.
Here are the required parameters to enter (others are optional)
TargetwebURL : Point to the Web of the destination
MigrationPackageAzureLocations : The temporary variable from the previous command
Credentials : SPO credential that has admin rights to the destination site
Ex:
Submit-SPOMigrationJob -TargetWebUrl -MigrationPackageAzureLocations $azureSource -Credentials (Get-Credential

20. PowerShell Remarks for PowerShell
Export-SPWeb
Target a List
Uncompressed
New-SPOMigrationPackage
Be careful of SPO limitations
ConverTo-SPOMigrationTargetedPackage
Using different output folder is recommended
Set-SPOMigrationPackageAzureSource
The Azure Containers will be required for the Submit-SPOMigrationJob
Submit-SPOMigrationJob
The destination list needs to exist

21. Migration API

22. Building code for the Migration API
Uses PRIME package formatting
Recommended to stay with what the PowerShell commands supports
SPO Migration TAP program
The Migration API is agnostic of the source

23. The Migration API calls
[ClientNS.ClientCallableMethod]
        public Guid CreateMigrationJob (
            Guid gWebId,
            string azureContainerSourceUri,
            string azureContainerManifestUri,
            string azureQueueReportUri,)
[ClientNS.ClientCallableMethod]
        public Guid CreateMigrationJobEncrypted(
            Guid gWebId,
            string azureContainerSourceUri,
            string azureContainerManifestUri,
            string azureQueueReportUri,
            MigrationOption AES256CBCKey)

24. Migration Job Status [ClientNS.ClientCallableMethod]
        public Guid SPMigrationJobState (Guid MigrationJobId)

25. Extra Requirement for Encryption
The encryption method should follow the AES CBC 256 Standard
The IV should be different for every file including the manifests in the package.
should be stored as a metadata on each files.
Name = [IV]
Value = [Base64Encoded byte array of the IV]

26. Migration API Encrypted Queue
Here is the JSON content in the queue message
{"Label", "Encrypted"},
{"JobId", "[JobId value]"},
{"IV", ”[IV value in base64format]”},
{"Content", “[encrypted message in base64string]”}
Once decrypted the messages will be the same as the API without encryption.

27. Migration API Logs
[Date]Message
[Date] [Action] message
[Date] [item type] [SPObject name] [SPObject path] [SPObject ID] [Action] message
Note that there may be other structures, and in some cases message will overlap into multiple lines, such as

28. Optimizations Parallel jobs Find the perfect package size
Number of items
Weight of metadata
Weight in GB over the wire
Single Report queue for every jobs
Keeping every threads running
No over provisioning

29. Please use Event Board to fill out a session evaluation.
Questions?
Please use Event Board to fill out a session evaluation.
Thank you!

30. Sample Code $userName = "admin@contoso.onmicrosoft.com"
$adminSite = "
 $sourceFilePath = "D:\data\documents\"
$packagePath = "d:\data\documentPackage"
$targetWebUrl = "
 $spoPackagePath = "d:\data\documentPackageForSPO"
$targetLibrary = "Shared Documents"
 $azureAccountName = "acctName"
$azureAccountKey = " =="
$azureQueueName = "contosoMigrationQueue"
 $cred = Get-Credential $userName
New-SPOMigrationPackage -SourceFilesPath $sourceFilePath -OutputPackagePath $packagePath
 ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl $targetWebUrl -SourceFilesPath $sourceFilePath -SourcePackagePath $packagePath -OutputPackagePath $spoPackagePath -TargetDocumentLibraryPath $targetLibrary -Credentials $cred
$azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFilePath -SourcePackagePath $spoPackagePath -AccountName $azureAccountName -AccountKey $azureAccountKey -AzureQueueName $azureQueueName
Submit-SPOMigrationJob -TargetWebUrl $targetWebUrl -MigrationPackageAzureLocations $azureSource -Credentials $cred