Thursday, May 12, 2016

An introduction to Microsoft Storage Spaces and Tiering

This is a hands on article explaining how to implement Microsoft Storage Spaces and in particular Tiering,  which in my case is the combination of a SSD and a spinning HDD into a single logical entity for better performance.

In 2016 I find myself (at last!) with more time to devote to working with Alternate and New Computer Operating systems.   It has already been a year of Ubuntu 16, of Microsoft Server 2016 Technical Preview 5.  Also numerous insider builds of Microsoft Windows 10 Redstone.

I already documented a move back from 8TB hard disk to 4TB disks, but the performance uplift was not enough.  I need a faster disk subsystem for use primarily under VMware so that the above OS loads, installs, upgrades, resumes, etcetera don't take an age, and can be performed concurrently.

What about pure SSD
It is tempting /just/ to argue that I could purchase a large, say 1TB SSD, one with sufficient buffer cache  (i.e. one that does not stall in real life use) and put Virtual Machines on it entirely.

Indeed this is the plan for the future, but right now the number of VM's I am testing, including Snapshots and safe copies makes this impractical.

It would be more interesting to have the OS automatically place the most often used parts of my VMs onto SSD and push the less used parts to the HDD.  That is tiering in a nutshell.

What is the Master Plan

- Use VMware as the primary virtualisation platform

- Use Local disks for VMware

- Buy SSD and combine it with a 4TB Spinning disk to form a Tiered Storage Space

- Test before committing

- Don't lose any data!

The BIG Compatibility Issue
Microsoft encourages vendors to certify their system to be sure it is compatible with Storage Spaces.  My initial tests with my Development system, which is a desktop PC,  circa 2008, with an X58 chipset , i.e. really ancient, did not work :-(  I concluded the Disk controller was deemed incompatible.

So before proceeding further employ the  Proof Of concept

- Your Test System
- Running Windows Server 2012 or Server 2016

Ideally you will have a physical system, but if not the example below uses the wonderful VMware (which is of course compatible).

Now we'll first use the GUI to create a Tier Disk
This is not a bad start, but annoyingly not all cache parameters may be specified on the GUI [NB: With Preview 5 of 2016 server, this is not true!].  So in a minute we will also use the command line

My article on how to get a free copy of 2016 Server TP5 is here 

- Master system running VMware
- Create a Virtual Machine running Windows 2016 server
- Real test PC running Windows 2016 server

- In either case you need one SSD and one HDD

Aside: Workaround Create SSD under VMware

You will need to edit the .vmx boot file and add this line in to say that your chosen .vmdk file is to be regarded as an SSD inside VMware.

The second .vmdk file will show up as an unknown Disk in VMware and there is another work around to change it to type HDD

So at this point you should have either made a basic hardware build on a Physical system OR setup a Windows 2016 TP5 server under VMware with 2 Virtual disks, one of which you have faked to look like an SSD to Windows

The following screen shots are from a VMware session that I created, running Server 2016 TP5:

Create the Storage Pool

Change Unknown Media Type
Notice the MediaType of the HDD is set to Unknown. For a VMware run.  If you are using a real machine this should be set to HDD and the following steps are therefore not needed

<For VMware> 
you will need to correct it and you can reset it to HDD but only after the Storage Pool is created.

At the admin command line find the UniqueID of the 60GB disk then use that in the set-physicaldisk command, including the curly brackets to reset the disk

Now there is a HDD and a SSD we can create a Tier.   A SSD and a Type Unknown cannot create a Tier.  But REBOOT VMachine first.  (Otherwise it will not work, I promise you, the HDD is not seen)

End </For Vmware>

Okay now you have a Physical or Virtual system and a HDD and a SSD, and you created a Storage Pool.  And the 2 disks are in the pool, and now they are seen as HDD and SSD.

Create the Tier

Actually though the read cache now shows up in the GUI Wizard I had trouble with it.  Anyway,  I'm setting it in the next command line example.

Note NTFS was not the default but I made it so

So at this point you have

- Installed Windows Server
- In a Physical or Virtual Environment
- You had one SSD and one HDD
- You aggregated them into a Tiered disk

Congratulations, you have done it!

Command Line Creation of  the Tiered Disk
On Windows Server 2012 or 2016 Server Preview 4 or earlier it is not possible to set the precise caching of the Virtual Disk.  So you are forced to use the command line.

Personally this was not trivial to me so this is a different example from my production system where I created a Tier disk from a 480GB SSD and a 4TB HDD, on a physical not virtual platform

In this worked example you are using Powershell.


FriendlyName          CanPool               OperationalStatus     HealthStatus          Usage                                  Size
------------          -------               -----------------     ------------          -----                                  ----
PhysicalDisk1         True                  OK                    Healthy               Auto-Select                         3.64 TB
PhysicalDisk7         True                  OK                    Healthy               Auto-Select                       447.13 GB

want to combine physical 1 and 7

PS C:\Windows\system32> Get-PhysicalDisk | Sort Size | FT DeviceId, FriendlyName, CanPool, Size, MediaType, UniqueId

DeviceId              FriendlyName                        CanPool                  Size MediaType             UniqueId
--------              ------------                        -------                  ---- ---------             --------

7                     PhysicalDisk7                          True          480103981056 SSD                   500A0751F0161B28
1                     PhysicalDisk1                          True         4000787030016 HDD                   5000CCA22BC1B14A

PhysicalDisk 7 is ssd cache
PhysicalDisk 1 is 4TB spinning HDisk

PS C:\Windows\system32> Get-PhysicalDisk -CanPool $true

FriendlyName          CanPool               OperationalStatus     HealthStatus          Usage                                  Size
------------          -------               -----------------     ------------          -----                                  ----
PhysicalDisk1         True                  OK                    Healthy               Auto-Select                         3.64 TB
PhysicalDisk7         True                  OK                    Healthy               Auto-Select                       447.13 GB

// create pool MusicVirtual and then add the second disk into it
New-StoragePool  -FriendlyName "MusicVirtual" –StorageSubsystemFriendlyName "Storage Spaces*"   -WriteCacheSizeDefault 64GB  -LogicalSectorSizeDefault 4096 –PhysicalDisks (Get-PhysicalDisk PhysicalDisk7 )

FriendlyName               OperationalStatus          HealthStatus              IsPrimordial              IsReadOnly
------------               -----------------          ------------              ------------              ----------
MusicVirtual               OK                         Healthy                   False                     False

// At this point disk 7 disappears from Disk Management window

// Now add in the spinning disk

Add-physicaldisk -StoragePoolFriendlyName "MusicVirtual" -PhysicalDisks (get-physicaldisk Physicaldisk1)

// create the SSD and HDD tiers 

New-StorageTier -StoragePoolFriendlyName "MusicVirtual" -FriendlyName SSDTierMV -MediaType SSD

ObjectId             : {1}\\gorilla\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageTier.ObjectId="{bd92c85b-8487-11e3-80
PassThroughClass     :
PassThroughIds       :
PassThroughNamespace :
PassThroughServer    :
UniqueId             : {cf9a8fef-01bb-11e6-97fa-001b2176755f}
Description          :
FriendlyName         : SSDTierMV
MediaType            : SSD
Size                 : 0
PSComputerName       :

New-StorageTier -StoragePoolFriendlyName "MusicVirtual" -FriendlyName HDDTierMV -MediaType HDD

ObjectId             : {1}\\gorilla\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageTier.ObjectId="{bd92c85b-8487-11e3-80
PassThroughClass     :
PassThroughIds       :
PassThroughNamespace :
PassThroughServer    :
UniqueId             : {cf9a9004-01bb-11e6-97fa-001b2176755f}
Description          :
FriendlyName         : HDDTierMV
MediaType            : HDD
Size                 : 0
PSComputerName       :

Get-StoragePool MusicVirtual | FL Size, AllocatedSize
Get-StoragePool MusicVirtual | Get-PhysicalDisk | Group MediaType, Size | Sort Name | FT -AutoSize

Count Name               Group
----- ----               -----
    1 HDD, 3999956729856 {MSFT_PhysicalDisk (ObjectId = "{1}\\gorilla\root/Microsoft/Windows/Sto...)}
    1 SSD, 479425724416  {MSFT_PhysicalDisk (ObjectId = "{1}\\gorilla\root/Microsoft/Windows/Sto...)}

// Work out how big is left space on HDD and SSD

HDD = 3725.25 GB
SSD =  446.5 GB

// create virtual disk inside the pool
$SSD = Get-StorageTier -FriendlyName SSDTierMV
$HDD = Get-StorageTier -FriendlyName HDDTierMV

New-VirtualDisk -StoragePoolFriendlyName MusicVirtual  -FriendlyName MusicVirtualDisk -ResiliencySettingName Simple  -StorageTiers $SSD, $HDD -StorageTierSizes 382GB,3725GB -WriteCacheSize 64GB

FriendlyName          ResiliencySettingName OperationalStatus     HealthStatus          IsManualAttach                         Size
------------          --------------------- -----------------     ------------          --------------                         ----
MusicVirtualDisk      Simple                OK                    Healthy               False                               4.01 TB

// show all disks

Number Friendly Name                            OperationalStatus                            Total Size Partition Style
------ -------------                            -----------------
11     Microsoft Storage Space Device           Offline                                         4.01 TB RAW

// make disk online

set-disk -Number 11 -IsReadonly 0 
set-disk -Number 11 -Isoffline 0

Initialize-Disk  -Number 11  -PartitionStyle GPT

New-Partition -DiskNumber 11 -DriveLetter "V" -UseMaximumSize
   Disk Number: 11

PartitionNumber  DriveLetter Offset                                              Size Type
---------------  ----------- ------                                              ---- ----
2                V           135266304                                        4.01 TB Basic

// format disk to NTFS
Format-Volume  -DriveLetter "V" -AllocationUnitSize 64KB -FileSystem NTFS -NewFileSystemLabel "virt music" -Confirm:$false

DriveLetter        FileSystemLabel    FileSystem         DriveType          HealthStatus            SizeRemaining              Size
-----------        ---------------    ----------         ---------          ------------            -------------              ----
V                  virt music         NTFS               Fixed              Healthy                       4.01 TB           4.01 TB

The above are the real commands.  I did not parameterise it into a script as I thought that would get real confusing, but you can!

Now some other Interesting Stuff:

Forcing a Tier Move
Optimize-Volume -DriveLetter V -TierOptimize -Verbose

Another way from Command Prompt using defrag and not Powershell

Other Things
Here is defrag working

Here is the Tiering Working

Tiering  is invoked automatically each day via a task

Potentially you could seem to replace a faulty disk in the Pool by adding its replacement in and then removing the failing disk.  I did not try it!   [My development system on which I tried tiering does not have a compatible motherboard,  if I ever get a compatible one I will try it there]

These are the secret files that enabling Tiering to figure out what to move where

I'm routinely getting 200MB/sec plus speeds now on disk operations

Manually Pinning Files
$StorageTier = Get-StorageTier -FriendlyName "vdisk_Microsoft_SSD_Template"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s001.vmdk"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s002.vmdk"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s003.vmdk"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s004.vmdk"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s005.vmdk"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s006.vmdk"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s007.vmdk"
Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s008.vmdk"

Set-FileStorageTier -DesiredStorageTier $StorageTier -FilePath "V:\development\win10\windows 10  build-s009.vmdk"

#list files in a tier

Get-FileStorageTier -VolumeDriveLetter V

#should move files now

Get-ScheduledTask -TaskName "Storage Tiers Optimization" | Start-ScheduledTask

# remove files example

Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s001.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s002.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s003.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s004.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s005.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s006.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s007.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s008.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s009.vmdk"
Clear-FileStorageTier -FilePath "V:\development\win10\windows 10  build-s010.vmdk"

After 2 Months
2 months later and I have lost no data!

Furthermore the disk speeds are subjectively faster than just using spinning disk.   My main use as stated above is for Virtual Machines.

I hope that storing frequently used (hot) files in the SSD cache tier is resulting in an easier life for the spinning disk.

Some Confessions
Storage Spaces have been provided in Windows server for over 2 years now.   If I had realised that I would have tried this earlier. Doh!

Apple OSX
Just because I can I also investigated the Apple equivalent that is the Fusion Drive.  I attempted to create an OSX Fusion drive from a RAM disk combined with a Spinning Disk.  This needs careful use of the diskutil command.  BUT: Though OSX said it had created the drive, and did form an aggregate, it did not appear to be doing auto-tiering, or in fact any tiering at all.  To be revisited.

(Creation of an aggregate, which I did use, but it does not regard it as a Fusion Drive)


- Storage Spaces is a worthy feature of Windows Server
- Tiering allows for SSD and HDD combination into an aggregate disk
- Daily tasks move hot file blocks into the SSD and colder file  blocks to the HDD
- You can run a Tieroptimise at the command line anytime to promote a tier move
- You can pin files into the SSD that you always want the fastest access to
- You should be able to remove a failing Disk and have Windows migrate the data to another equivalent disk in the pool  (not tested)
- Windows Server is somewhat picky on hardware so not all Disk Controllers or Motherboards can be used by Storage Spaces, leading to frustration if your hardware is so.


Storage Spaces Overview Microsoft
Storage Spaces at Command Line reference
What is Storage Spaces Youtube
Monitoring Storage Tiers
Best Practices Storage Spaces