Make use of Extended IT to bolster your IT Support Offering
Extend your IT

Migrating Fileshares to SharePoint Online (PowerShell)

NB: This article is up to date as of the 6th of September 2017, SharePoint Online standards do change so it’s worth reviewing them all first.

After performing a few dozen SharePoint Online migrations all from existing windows servers (Fileshares / Network Drives) and with lots of failures along the way I now feel I have a fool proof method of completing the migrations at a decent speed, while retaining as much file meta data as possible without requiring any 3rd party tools and no additional cost.

We will be making use of the Microsoft SharePoint Online PowerShell module and I’ll take you through from beginning to end a complete fileshare to SharePoint Online migration. The basic procedure for this is;

  1. Prepare the server
  2. Analyse and Prepare the data
  3. Map fileshare to SPO
  4. Upload Data
  5. Verify Data
  6. Possibly Incremental Syncs
  7. Go Live

Fileshare to SPO – Prerequisites

  • Install SharePoint Online PowerShell module on the file server – https://www.microsoft.com/en-us/download/details.aspx?id=35588
  • A SharePoint Online site setup
  • Admin account on the file server
  • Admin account on the SharePoint site.
  • Sufficient space on your SharePoint Online site
  • A decent amount of free disk space on your file server (or available via UNC), you will need approx 10% of your total file size.

Fileshare to SPO  – File Prep / Analysis.

File Permissions

This method doesn’t copy file permissions and to be honest you’d want to re-look at how you do permissions when you move over to SPO anyway. The important thing here is to make sure that the admin account for the file server has access to all the files you want to migrate. If the account doesn’t have access the file doesn’t get copied.

If your fileshares were setup sensibly your admin account likely does have all files / folders already, if not you’ll need to take ownership of these files and assign the account access. ~(For more information about taking ownership check here – https://technet.microsoft.com/en-us/library/cc753659(v=ws.11).aspx ). Special attention should be paid to any files encrypted using EFS, you’ll have to get these un-encrypted.

SharePoint Online Pre-Upload File analysis

Now you have access to all files / folders we can perform a readiness analysis of the data. Most of the SharePoint online naming requirements are the same as Windows, however there are a couple of differences that we need to be aware of;

  • #% are not supported by default we will need to enable this support
  • & doesn’t work in some applications (Office 2010), we will remove this.
  • the maximum overall path length in SharePoint Online is 400 characters, including the protocol, site name and document library.

You should also check for the existence of the following files;

  • .aspx | ASP.NET Active server page
  • .asmx | ASP.NET web services source file
  • .ascx | ASP.NET wep user control file
  • .master | ASP.NET master web page
  • .xap | Windows phone installation
  • .swf | ShockWave Flash
  • .jar | Java archive
  • .xsf | Office InfoPath form definition file
  • .htc | HTML Component file

These are all web page files, they won’t be blocked by SharePoint Online but will be ran as a webpage if a user runs / clicks on one of them, you will likely want to review the impact this could have.

There is also a list of unsupported folder names, you can find them at the link below (It’s quite long).

(Check here for an up to date list – https://support.office.com/en-us/article/Invalid-characters-in-file-or-folder-names-or-invalid-file-types-in-OneDrive-for-Business-64883a5d-228e-48f5-b3d2-eb39e07630fa)

Now I want to start by saying that there are plenty of free & paid for tools that can do part / all of these steps, but I’ve found using Excel to be the most reliable way for me, it produces good results to send to clients and is easily updated with new standards.

SharePoint Online Pre-Upload File analysis – Excel

Our aim here is to export a list of all the files / folders you will be uploading to SharePoint online, import this list into Excel and then check for any issues.

  1. Run Command Prompt as an Administrator
  2. Change Directory to the data you will be uploading to SharePoint
  3. Type – DIR /S /B  >> Files.txt
    Type DIR /S /B >> Files.txt
  4. You will now have a list of all the files in the folder. It should look like this.
    Exported File Listing
  5. Open this data up in Excel
  6. Add a row at the top, call the data column filename
  7. Trim the C:\SP\Data bit from the paths (Find and replace will do)
  8. Now we need to add 3 Columns, Length, Special Characters & Invalid File Names. We are going to use 3 different formulas for these columns.
    Length – =LEN(A2)
    Special – =ISNUMBER(SEARCH(“&”,A2))
    FileNames – =SUMPRODUCT(–(A2={“AUX”,”PRN”,”NUL”,”CON”,”COM0″,”COM1″,”COM2″,”COM3″,”COM4″,”COM5″,”COM6″,”COM7″,”COM8″,”COM9″,”LPT0″,”LPT1″,”LPT2″,”LPT3″,”LPT4″,”LPT5″,”LPT6″,”LPT7″,”LPT8″,”LPT9″}))>0
    Data in Column A.

    Download an Example Excel file with formulas in here – ExampleAnalysisFile

  9. Now you can sort by Length, anything with a total length over 400 minus your site / library name needs to be reduced in size. For example:For this migration my SharePoint site is https://extendedittest.sharepoint.com
    My Library is called Shared Documents.
    This brings my full URL to https://extendedittest.sharepoint.com/Shared Documents/Existing File Path
    https://extendedittest.sharepoint.com/Shared Documents = 54 Characters
    So the rest of the path after this can be no longer than 346 characters.So in my Excel document I’m looking for anything with more than 346 Characters, any of these files I would then copy from here, send them to my client and ask how they want to adjust them, for me this is a big benefit of using Excel.
  10. Now that you’ve done that you can filter for any files with special characters or invalid file names using the other two columns, anything with ‘True’ in the cell is a bad file. These will all have to be fixed.
  11. Once you have confirmed with your client what they want to do you can make the changes. It’s up to you how you want to do this, typically there won’t be too many to make so I just make them manually.

NOTE: There is also a 15GB file size limit. I would say it’s fairly rare to come across such big files in normal company data, however you can search based on file size using Windows Explorer.

Uploading File Data to SharePoint online.

The Script

We will be using the following script to migrate our data.

#These are your variables
$creds = (Get-Credential admin@extendedittest.onmicrosoft.com)
$sourceFiles = 'C:\SP\DATA'
$sourcePackage = 'C:\SP\Package_source'
$targetPackage = 'C:\SP\Package_target'
$targetWeb = 'https://extendedittest.sharepoint.com'
$targetDocLib = 'Shared Documents’
$targetDocPath = 'Sub Folder'

#USE THIS OPTION for the first two commands to add a sub path
#-TargetDocumentLibrarySubFolderPath $targetDocPath

#This creates the Source Pacakage (The details of all the files you want to migrate)
New-SPOMigrationPackage -SourceFilesPath $sourceFiles -OutputPackagePath $sourcePackage -TargetWebUrl $targetWeb -TargetDocumentLibraryPath $targetDocLib -IgnoreHidden ReplaceInvalidCharacters

#This creates the target package (Same as above but targeted for SharePoint Online)
$finalPackages = ConvertTo-SPOMigrationTargetedPackage -ParallelImport -SourceFilesPath $sourceFiles -SourcePackagePath $sourcePackage -OutputPackagePath $targetPackage -Credential $creds -TargetWebUrl $targetWeb -TargetDocumentLibraryPath $targetDocLib

#This runs the job and copies the files. 
$jobs = $finalPackages | % {Invoke-SPOMigrationEncryptUploadSubmit -SourceFilesPath $_.FilesDirectory.FullName -SourcePackagePath $_.PackageDirectory.FullName -Credentials $creds -TargetWebUrl $targetWeb}

Now fill in all your variables, it should be pretty self explanatory, the only option that requires any changes to the commands is the $targetDocPath option, so if you had a folder within a document library that you wanted to upload all this data to you would fill in the $TargetDocPath variable and then also add  ‘-TargetDocumentLibrarySubFolderPath $targetDocPath’ to the end of the first two commands.

If you are copying files to OneDrive for Business then you need to adjust the following settings:

$targetWeb = ‘https://extendedittest-my.sharepoint.com/personal/admin_extendedittest_onmicrosoft_com’
(so that’s https://sitename-my.sharepoint.com/personal/username)

$targetDocLib = ‘Documents’

Migrating the data

Now that we have our script sorted it’s time to run it and get the data migrated to SharePoint Online.

  1. Run the SharePoint Online Management  Shell. Make sure to run it as Administrator with a user that has full access to all of the data!
  2. Copy all the variables into the Shell.
  3. Enter your SharePoint Online Admin details when prompted.
    SharePoint Online Powershell Connection Credential Box
    SharePoint Online Variables
  4. Now we create the source package by running the first command in the script (I’m running them all separately to check for errors)
    SharePoint Online Source Package Creation - Powershell
    (If you run into to a ‘Can’t connect to server’ error, try using -NoAzureADLookup at the end)
  5. Next we create the target package. This basically converts all the meta data that you have to a SharePoint format. This outputs lots of XML files into the $targetPackage folder.
    SharePoint Online Target Package Creation - Powershell
  6. Finally, we upload our data to SharePoint Online using the last command.
    Uploading Files to SharePoint Online

Now when you check in your document library all the files / folders will be there along with some basic metadata. time.

Verifying Data

Unfortunately it’s now time to turn to 3rd Party Tools. I haven’t yet found a good way to verify that everything has uploaded OK using a Microsoft product.  We are going to use a tool called Sync Back Pro, they offer a 30 day free trial which is all we need.

  1. Download and install SyncBack Pro – https://www.2brightsparks.com/syncback/sbpro.html
  2. Add a new profile, Call it SharePoint
  3. Select ‘Backup’
  4. Source – ‘Internal / External drive, network path, etc.’
  5. Destination – ‘SharePoint’
    SyncBack Pro Profile Direction
  6. Click on Authorize
  7. A browser window will then pop up, login to SharePoint Online with you admin details. The admin account must have a license.
  8. Click Accept on the next page to allow SyncBackPro Access.
    SyncBackPro Accept Access to SharePoint Online
  9. You will then be presented with an auth code, copy this into the SyncBackPro Window.
  10. Click done.
  11. Now configure the local location of your files, in my case C:\SP\Data
  12. Click OK and Finish.
  13. Perform a simulated run of your profile.

Now from the simulated run you can tell if you file transfer has completed or not. If SyncBackPro says no files need to be copied across then you are all good, otherwise it will list all the files that still need to be copied. Either use SyncBackPro to copy these or fix any issues and run and use the SharePoint Online script from above.

Incremental Syncing

Now if you do find yourself having to incrementally sync the data from your local storage to SharePoint there are two ways to do it. Either;

SyncBackPro – All the same steps as above and then actually run the profile.

SharePoint Online Management Shell – This doesn’t support incremental syncing however if you move any files / folders that have changed and need to be re-uploaded into a new folder and then sync this folder over the top of your SharePoint site then all the changes will get uploaded. For example if your main directory is C:\SP\Data as mine is just copy the files / folders into C:\SP\Increment and then update the $sourceFiles variable and leave everything else as is.

Right, that’s it. All your files have now successfully been uploaded to SharePoint Online. And now you have finished time to have yourself a beer – http://amzn.to/2f2ARDo

If you get stuck using this and would like extra support contact me at [email protected]


Leave a Reply