Solarflare NVMeoF over TCP Linux kernel drivers - Revision history

Matt hole at 15:50, 8 August 2018

2018-08-08T15:50:11Z

Matt hole at 15:49, 8 August 2018

2018-08-08T15:49:32Z

Matt hole at 15:48, 8 August 2018

2018-08-08T15:48:00Z

Matt hole at 15:46, 8 August 2018

2018-08-08T15:46:34Z

Matt hole at 15:44, 8 August 2018

2018-08-08T15:44:09Z

Matt hole at 15:37, 8 August 2018

2018-08-08T15:37:47Z

Matt hole at 11:01, 8 August 2018

2018-08-08T11:01:53Z

Matt hole at 15:46, 3 August 2018

2018-08-03T15:46:41Z

Matt hole at 15:39, 3 August 2018

2018-08-03T15:39:16Z

Matt hole at 15:30, 3 August 2018

2018-08-03T15:30:35Z

@@ Line 97: / Line 97: @@
    # Associate subsystem with port
    ln -s /sys/kernel/config/nvmet/subsystems/ramdisk /sys/kernel/config/nvmet/ports/1/subsystems/ramdisk
-=== My version of the configuration when setting for the NVMe drive if  showing in /dev/ as "nvme0n1" ===
+My version of the configuration when setting for the NVMe drive if  showing in /dev/ as "nvme0n1"
 <syntaxhighlight>
 touch /var/lock/subsys/local

@@ Line 65: / Line 65: @@
 == Configuration ==
 On the *target* system, perform the steps below.  The example commands assume
@@ Line 114: / Line 116: @@
 Once installed setup the host and clients with static IPv4 addresses. In my example i made the target (storage server) 192.168.0.10 and the client 192.168.0.11 since it was a closed network.
 On the *host* system, perform the steps below.  The example

← Older revision		Revision as of 15:48, 8 August 2018
Line 130:		Line 130:
	lsblk \| grep nvme		lsblk \| grep nvme

		+	Once confirmed you can then add file system as normal:
		+	<syntaxhighlight>
		+	mkfs.xfs /dev/nvme0n1
		+	</syntaxhighlight>
	== Make everything above stateful ==		== Make everything above stateful ==

@@ Line 226: / Line 226: @@
 chmod +x /usr/local/sbin/nvme-tcp.sh
 </syntaxhighlight>
-Insert the following into the above script
+Insert the following into nvme-tcp.sh
 <syntaxhighlight>

← Older revision		Revision as of 15:44, 8 August 2018
Line 195:		Line 195:
	systemctl enable nvme-tcp.service		systemctl enable nvme-tcp.service
	</syntaxhighlight>		</syntaxhighlight>
		+
		+	Host System:
		+
		+	Add Kernel module to module-load so centos will load the kernel module on boot:
		+	<syntaxhighlight>
		+	vim /etc/modules-load.d/nvme-tcp.conf
		+	</syntaxhighlight>
		+	Insert the following line into nvme-tcp.conf
		+	<syntaxhighlight>
		+	nvme-tcp
		+	</syntaxhighlight>
		+	Create a new service unit file at /etc/systemd/system/nvme-tcp.service:
		+	<syntaxhighlight>
		+	[Unit]
		+	Description=Service unit file for NVMe over Fabrics TCP target system
		+	After=network.target
		+	[Service]
		+	Type=simple
		+	ExecStart=/usr/local/sbin/nvm-tcp.sh
		+	TimeoutStartSec=0
		+	[Install]
		+	WantedBy=default.target
		+	</syntaxhighlight>
		+	Create the ExecStart script outlined above:
		+	<syntaxhighlight>
		+	vim /usr/local/sbin/nvme-tcp.sh
		+	</syntaxhighlight>
		+	Make script executable:
		+	<syntaxhighlight>
		+	chmod +x /usr/local/sbin/nvme-tcp.sh
		+	</syntaxhighlight>
		+	Insert the following into the above script
		+	<syntaxhighlight>

@@ Line 136: / Line 136: @@
 Target System:
 Create a new service unit file at /etc/systemd/system/nvmet-tcp.service:
 <syntaxhighlight>
@@ Line 155: / Line 163: @@
 Make script executable:
 <syntaxhighlight>
-chmod a+x /usr/local/sbin/nvmet-tcp.sh
+chmod +x /usr/local/sbin/nvmet-tcp.sh
 </syntaxhighlight>
-To get around this all of the configuration shown for nvme0n1 above in /etc/rc.d/rc.local then enabled/started the rc-local service.
+Insert the following into the above script
+<syntaxhighlight>
-However this failed since the nvmet kernel module thats required does not get loaded either and this will be need to be imported into the kernel before rc.local is run on boot.
+#!/bin/bash
+# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
-In comes the following (this will load the module in on every boot):
+# It is highly advisable to create own systemd services or udev rules
 <syntaxhighlight>
-echo modprobe nvmet-tcp > /etc/rc.modules
+systemctl enable nvme-tcp.service
 </syntaxhighlight>

@@ Line 130: / Line 130: @@
    lsblk | grep nvme
-== Findings and tweaks outside of what Solarflare provided ==
+== Make everything above stateful ==
-Everything set above for the kernel is stateless having been set in /sys/ meaning once you reboot then your nvme drive will go missing, the modules wont be loaded and none of the configuration above will be set anymore.
+Everything set above for the kernel is stateless having been set in /sys/ meaning once you reboot then your NVMe drive will go missing, the modules wont be loaded and none of the configuration above will be set anymore.
 To get around this all of the configuration shown for nvme0n1 above in /etc/rc.d/rc.local then enabled/started the rc-local service.

@@ Line 110: / Line 110: @@
 ln -s /sys/kernel/config/nvmet/subsystems/nvme0n1 /sys/kernel/config/nvmet/ports/1/subsystems/nvme0n1
 </syntaxhighlight>
 On the *host* system, perform the steps below.  The example
-commands assume a target IP of 10.0.0.1 and port 11345.
+commands assume a target IP of 192.168.0.10and port 11345.
    # Install a version of the nvme-cli utility that supports NVMeoF over TCP.
@@ Line 121: / Line 125: @@
    # Connect to the target
-   nvme connect -t tcp -a 10.0.0.1 -s 11345 -n ramdisk
+   nvme connect -t tcp -a 192.168.0.10 -s 11345 -n nvme0n1
    # Confirm NVMe device is present
    lsblk | grep nvme
-== findings ==
+== Findings and tweaks outside of what Solarflare provided ==
 Everything set above for the kernel is stateless having been set in /sys/ meaning once you reboot then your nvme drive will go missing, the modules wont be loaded and none of the configuration above will be set anymore.

@@ Line 95: / Line 95: @@
    # Associate subsystem with port
    ln -s /sys/kernel/config/nvmet/subsystems/ramdisk /sys/kernel/config/nvmet/ports/1/subsystems/ramdisk
+=== My version of the configuration when setting for the NVMe drive if  showing in /dev/ as "nvme0n1" ===
 <syntaxhighlight>
 touch /var/lock/subsys/local
-mkdir /sys/kernel/config/nvmet/subsystems/'''nvme0n1'''
+mkdir /sys/kernel/config/nvmet/subsystems/nvme0n1
-echo 1 > /sys/kernel/config/nvmet/subsystems/'''nvme0n1'''/attr_allow_any_host
+echo 1 > /sys/kernel/config/nvmet/subsystems/nvme0n1/attr_allow_any_host
-mkdir /sys/kernel/config/nvmet/subsystems/'''nvme0n1'''/namespaces/1
+mkdir /sys/kernel/config/nvmet/subsystems/nvme0n1/namespaces/1
-echo -n /dev/'''nvme0n1''' > /sys/kernel/config/nvmet/subsystems/'''nvme0n1'''/namespaces/1/device_path
+echo -n /dev/nvme0n1 > /sys/kernel/config/nvmet/subsystems/nvme0n1/namespaces/1/device_path
 echo 1 > /sys/kernel/config/nvmet/subsystems/nvme0n1/namespaces/1/enable
 mkdir /sys/kernel/config/nvmet/ports/1
@@ Line 109: / Line 108: @@
 echo "11345" > /sys/kernel/config/nvmet/ports/1/addr_trsvcid
 echo "192.168.0.10" > /sys/kernel/config/nvmet/ports/1/addr_traddr
-ln -s /sys/kernel/config/nvmet/subsystems/'''nvme0n1''' /sys/kernel/config/nvmet/ports/1/subsystems/'''nvme0n1'''
+ln -s /sys/kernel/config/nvmet/subsystems/nvme0n1 /sys/kernel/config/nvmet/ports/1/subsystems/nvme0n1
 </syntaxhighlight>
@@ Line 126: / Line 125: @@
    # Confirm NVMe device is present
    lsblk | grep nvme