Windows2016

Windows 2016 server - Storage spaces Direct - Hyper converged - Hardware used

Compute

• 2028TP-HC1R
• 4x BPN-ADP-6SATA3P (changed out the 3108 BPN`s since storage spaces direct will NOT work with RAID cards)
• Intel Xeon E5-2680 V3
• DDR4 2133MHz
• 4x NICs: Mellanox Connect-x 3 Single port
• 4x OS drive: Samsung SSD 256GB SATA
• 4x Cache drive: Sandisk 128GB SATA
• 16x Hard drives: 250GB Seagate SATA hard drive

Networking

• Supermicro SSE-G24-TG4 (1Gbe RJ-45 24 port)
• Supermicro (10Gbe SFP+ 24 port)

Windows 2016 Server - Storage Space Direct - Issues/Requirements

Software

• Storage Spaces Direct is a licensed feature set ONLY available in Datacenter edition. So Standard, storage editions CANNOT be used.
• You can setup storage spaces either with GUI desktop or with server CORE.

Compute Hardware

• Do NOT use RAID controllers since the storage spaces configuration checks this and will stop setup on a RAID array. It has to detect media type (HDD, SSD, NVME, SATA, SAS) itself to function correctly. You can use standard onboard SATA or SAS HBA. I learnt this the hard way and had to change hardware used.
• minimum disk requirements per node will be at least: 1 OS disk, 1 Caching disk and 1 capacity disk.

Networking

• Requires both 1Gbe and 10Gbe.
• Will require 10Gbe that supports RDMA. This cannot be worked around if your cards do not support RDMA then find some that do.
• The demo system used 1x 1Gbe and 10Gbe switches. For production i would suggest 2 switches each for 10 and 1 for HA.
• The demo system used Mellanox Connect x -3 single port network cards. Solely down to RDMA being support and availability in the lab. Again i would use a dual port network card for production to allow for redundant links/load balancing.
• 10Gbe switch must support tagged VLAN if using one switch. If using two then LCAP support will be required to allow for ISL and port channels of ports A/B.

Windows 2016 server - Storage spaces Direct - Hyper converged install

Switching configuration

• Tagged vlan will be required for RDMA. For my example i set a VLAN of 12 on the super micro switch on every port that its required and this is set to tagged. You can do this in the GUI or CLI of the swtich.

Basic windows 2016 setup

• Install windows server 2016 Datacenter either with GUI or without. In this example i did with GUI but apparently you can use server core without issue. Not sure if you can use nano server. (Storage spaces direct is a Datacenter feature, this will not work or be licensed in anything less. For example Standard)

• Enable RDP to allow easy access.

• Install hardware drivers and fully update OS. Make sure every node has the same update level otherwise cluster manager will complain. Plus network drivers will be required to get RDMA working (Storage spaces direct cannot function without RDMA)

Install roles and add to active directory

• Every node will need certain services. these can be installed using:

Install-WindowsFeature -Name File-Services
Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools
Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart

• Add all the nodes you wish to use for storage spaces direct to your active directory server. In my example i made a new 2016 AD server with a new forest

"win16.local" since i could never get adding to the cluster working on "bostonlabs.co.uk" This could be down to bostonlabs labs being 2008 R2 AD.

Check visible storage drives in each node

• Verify that the internal drives are online, by going to Server Manager > Tools > Computer Management > Disk Management. If any are offline, select the drive, right-click it, and click Online.
• If any drive has old data or partitions, clear these with diskpart.

Networking setup

• For the Mellanox NICs used in this solution, we need to enable Data Center Bridging (DCB),

which is required for RDMA. Then we create a policy to establish network Quality of Service (QoS) to ensure that the Software Defined Storage system has enough bandwidth to communicate between the nodes, ensuring resiliency and performance. We also need to disable regular Flow Control (Global Pause) on the Mellanox adapters, since Priority Flow Control (PFC) and Global Pause cannot operate together on the same interface.

Enable Data Center Bridging (required for RDMA)

Install-WindowsFeature -Name Data-Center-Bridging

Configure a QoS policy for SMB-Direct

New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3

Turn on Flow Control for SMB

Enable-NetQosFlowControl -Priority 3

Make sure flow control is off for other traffic

Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7

Apply a Quality of Service (QoS) policy to the target adapters. In my demo i only used one NIC named in windows as "mellanox1", so to use these commands successfully make sure the name your stating is actually the one you have applied to the NIC in windows.

Enable-NetAdapterQos -Name "Mellanox1"(,"examplenic2")

Give SMB Direct a minimum bandwidth of 50%

New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 50 -Algorithm ETS

Disable Flow Control on physical adapters. If you have two or more ports then just replicate this line and change the name of the NIC targeted.

Set-NetAdapterAdvancedProperty -Name "Mellanox1" -RegistryKeyword "*FlowControl" -RegistryValue 0

Virtual networking setup specifically for hyperconverged S2D

For an S2D hyperconverged solution, we deploy a SET-enabled Hyper-V switch and add RDMA-enabled host virtual NICs to it for use by Hyper-V. Since many switches won't pass traffic class information on untagged vLAN traffic, we need to make sure that the vNICs using RDMA are on vLANs.

To keep this hyperconverged solution as simple as possible and since we are using single port 10GbE NIC, we will pass all traffic on vLAN 12. If you need to segment your network traffic more, for example to isolate VM Live Migration traffic, you can use additional vLANs. Example 10 shows the PowerShell script that can be used to perform the SET configuration, enable RDMA, and assign vLANs to the vNICs. These steps are necessary only for configuring a hyperconverged solution.

For a dis aggregated solution these steps can be skipped since Hyper-V is not enabled on the S2D storage nodes.

Create a SET-enabled vSwitch supporting multiple uplinks provided by the Mellanox adapter

New-VMSwitch -Name S2DSwitch -NetAdapterName "Mellanox 1", "Mellanox 2" -EnableEmbeddedTeaming $true -AllowManagementOS $false

Add host vNICs to the vSwitch just created. Add another line here for anymore SMB names created. You would normally have

Add-VMNetworkAdapter -SwitchName S2DSwitch -Name SMB1 -ManagementOS

Enable RDMA on the vNICs just created. The second SMB2 is in brackets since i only have one 10Gbe nic in my demo, so removed the second SMB from my command when applied. But if you have more than one SMB then you will need to add these as shown in the brackets below.

Enable-NetAdapterRDMA -Name "vEthernet (SMB1)"(,"vEthernet (SMB2)")

Assign the vNICs to a vLAN. You will need to assign per SMB, so if as above you have applied more than one then copy the below and change the adapter name.

Set-VMNetworkAdapterVlan -VMNetworkAdapterName SMB1 -VlanId 12 -Access –ManagementOS

Networking: IP setting and DNS consideration

Now setup networking IP`s for the created Vethernet "SMB1" and team the 1Gbe network ports if you have multiple. Each has to have a unique DNS name.

Dns names used in my demo (win16.local is the active directory name):

Win16-nodea.win16.local
Win16-nodeb.win16.local 
Win16-nodec.win16.local 
Win16-noded.win16.local

In my example i had 4 nodes, 4 10Gbe links and 8 1Gbe links. The DNS and AD server in my example is a VM i created for this specifically.

The 10Gbe does not need to be externally rout able, as long as each windows storage server can see the other.

1Gbe Example:

Win16-nodea: 1Gbe (2x 1Gbe teamed): 10.7.7.35, 255.0.0.0, Gateway: 10.0.0.3, DNS: 10.7.7.7
Win16-nodeb: 1Gbe (2x 1Gbe teamed): 10.7.7.36, 255.0.0.0, Gateway: 10.0.0.3, DNS: 10.7.7.7
Win16-nodeb: 1Gbe (2x 1Gbe teamed): 10.7.7.37, 255.0.0.0, Gateway: 10.0.0.3, DNS: 10.7.7.7
Win16-nodeb: 1Gbe (2x 1Gbe teamed): 10.7.7.38, 255.0.0.0, Gateway: 10.0.0.3, DNS: 10.7.7.7

10Gbe Example:

Win16-nodea: 10Gbe: 192.168.0.5, 255.255.255.0
Win16-nodeb: 10Gbe: 192.168.0.6, 255.255.255.0 
Win16-nodeb: 10Gbe: 192.168.0.7, 255.255.255.0
Win16-nodeb: 10Gbe: 192.168.0.8, 255.255.255.0

Networking: RDMA check

Since RDMA is so critical to the performance of the final solution, it’s a good idea to make sure each piece of the configuration is correct as we move through the steps.

Get-NetAdapterRdma | ? Name -Like *SMB* | ft Name, Enabled

In Powershell you should see true next to enabled. If not then something has gone wrong.

VMQ: Virtual Machine Queue configuration guide

For the 10GbE Mellanox adapters in our solution, the operating system automatically enables dynamic VMQ and RSS, which improve network performance and throughput to the VMs. VMQ is a scaling network technology for Hyper-V switch that improves network throughput by distributing processing of network traffic for multiple VMs among multiple processors. When VMQ is enabled, a dedicated queue is established on the physical NIC for each vNIC that has requested a queue. As packets arrive for a vNIC, the physical NIC places them in that vNIC's queue. These queues are managed by the system's processors. Although not strictly necessary, it is a best practice to assign base and maximum processors for VMQ queues on each server in order to ensure maximum efficiency of queue management. Although the concept is straight forward, there are a few things to keep in mind when determining proper processor assignment. First, only physical processors are used to manage VMQ queues. Therefore, if Hyper-Threading (HT) Technology is enabled, only the even-numbered processors are considered viable. Next, since processor 0 is assigned to many internal tasks, it is best not to assign queues to this particular processor. Before configuring VMQ queue management, execute a couple of PowerShell commands to gather in-formation. We need to know if HT is enabled and how many processors are available. You can issue a WMI query for this, comparing the “NumberOfCores” field to the “NumberOfLogicalProcessors” field. As an alternative, issue the Get-NetAdapterRSS command to see a list of viable processors (remember not to use Processor 0:0/0).

Check for Hyper-Threading (if there are twice as many logical procs as number of cores, HT is enabled)

Get-WmiObject -Class win32_processor | ft -Property NumberOfCores, NumberOfLogicalProcessors -AutoSize

Check procs available for queues (check the RssProcessorArray field)

Get-NetAdapterRSS

Once you have this information, it's a simple math problem. We have a pair of 14-core CPUs in each host, providing 28 processors total, or 56 logical processors, including Hyper-Threading. Excluding processor 0 and eliminating all odd-numbered processors leaves us with 27 processors to assign. Given the dual-port Mellanox adapter, this means we can assign 13 processors to one port and 14 processors to the other. This results in the following processor assignment: Mellanox 1: procs 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28 Mellanox 2: procs 30, 32, 34, 46, 38, 40, 42, 44, 46, 48, 50, 52, 54 Use the following PowerShell script to define the base (starting) processor as well as how many processors to use for managing VMQ queues on each physical NIC consumed by the vSwitch (in our solution, the two Mellanox ports.)

Configure the base and maximum processors to use for VMQ queues

Set-NetAdapterVmq -Name "Mellanox 1" -BaseProcessorNumber 2 -MaxProcessors 14
20 Microsoft Storage Spaces Direct (S2D) Deployment Guide
Set-NetAdapterVmq -Name "Mellanox 2" -BaseProcessorNumber 30 -MaxProcessors 13

Check VMQ queues

Get-NetAdapterVmqQueue

Create Failover Cluster

To create the cluster either use this command line to validate your cluster setup:

Test-Cluster -Node win16-nodea,win16-nodeb,win16-nodec,win16-noded -Include "Storage Spaces Direct",Inventory,Network,"System Configuration"

or just navigate in GUI to Server manager. Then look top right see "tools". In this drop down you should see "Failover Cluster Manager". Within this you should see the option to validate your cluster.

if you get failures on validated, the best advice is check that everything is uniform between nodes. so updates are the same, gateways are the same, subnet masks etc. This validation will pick up and fail on all these things, sometimes the error codes are not all that obvious but normally its this kind of issue.

This is easy to use various commandline but does not validate for storage spaces direct, so once you tick off any issues with normal cluster valdiation. Run the cli above.

Once validate create cluster, the no storage tag is important, you will provision storage later (the process for S2D is different to storage for standard failover cluster):

New-Cluster -Name S2DCluster -Node win16-nodea,win16-nodeb,win16-nodec,win16-noded -NoStorage

10Gb networking failover cluster config

If using all 10Gbe interfaces then set all of them to Cluster and Client traffic. By default the cluster will set one inteface to cluster only.

Which would result in half the performance!

Cluster file witness

Failover cluster has since 2012 required a witness disk to keep a heart beat between nodes in the cluster.

However this has now been updated to allow for File share witness AND cloud witness.

Cloud witness uses un surprisely Azure, so we will not use this in our install.

File share witness, i just recreated a SMB share on our storage1 box with open guest permissions. this worked without issue.

for more info: https://blogs.msdn.microsoft.com/clustering/2014/03/31/configuring-a-file-share-witness-on-a-scale-out-file-server/

Enable and configure Storage spaces Direct

Once the failover cluster has been created, run the PowerShell command.

I kept the default naming here but of course, you can change the name as you wish.

Enable-ClusterStorageSpacesDirect –CimSession S2DCluster -PoolFriendlyName S2DPool

Completion of the command may take abit of time, in my demo this was 10-15 mins but it just depends on the setup really and how many nodes in the cluster i would imagine.

The above command will:

• Create a single storage pool that has a name as specified by the -PoolFriendlyName

parameter

• Configure S2D cache tier using the highest performance storage devices available, such

as NVMe or SSD

• Create two storage tiers, one called “Capacity” and the other called “Performance.”

Check the results of this command with what storage tiers have been created:

Get-StorageTier | ft FriendlyName, ResiliencySettingName

I would also check with:

Get-PhysicalDisk

Since i found the first time i deployed S2D it go my caching tier wrong. out of 4 caching SSD`s it showed two as journal (cache in MS speak) and two as auto-select. All of them at least in my demo should have been "journal". So i deleted the entire storage spaces cluster and applied again.

For some unknown reason, on the second run it picked out all 4 for caching automatically.

last RDMA check

Using

netstat

You can check if all your 10Gb are recieving traffic and if so what port. You should see the port 445 being used since this is a port we specific`d earlier.

Here is a example output of what you should see:

TCP    192.168.0.5:445        Win16-nodeb:62933      ESTABLISHED
 TCP    192.168.0.5:445        Win16-nodec:62352      ESTABLISHED
 TCP    192.168.0.5:445        Win16-noded:62352      ESTABLISHED

That should be the storage space direct cluster up and running!

S2D user guide

Display storage pools and tiers

Display all known Storage pools and there health

Get-StoragePool S2D* | FT FriendlyName, FaultDomainAwarenessDefault, OperationalStatus, HealthStatus -autosize

Find out what storage tiers have been created with this command:

Get-StorageTier | FT FriendlyName, ResiliencySettingName, MediaType, PhysicalDiskRedundancy -autosize

Volumes

There are 3 types of volume you can create within the storage pool: Mirror, parity and multi resilient.

	Mirror	Parity	Multi-resilient
optimized for	Performance	Best use of disk	mix of the previous two
Use case	All data is hot	All data is cold	Mix of hot and cold
Storage efficiency	Least (33%)	Most (50+%)	Medium (~50%)
FS	NTFS or ReFS	NTFS or ReFS	ReFS
Minimum nodes	3	4	4

Further information can be found:

https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/plan-volumes

References

• https://lenovopress.com/lp0064.pdf
• https://argonsys.com/learn-microsoft-cloud/articles/storage-spaces-direct-configuration/
• https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/