Lustre intel: Setting up a Windows client

The following instructions are for one (or multiple) linux system(s), which will serve as the middle-layer cluster, which "translates" Lustre to Windows, using Samba.

There are two components that need to be installed:

1. CTDB
2. Samba

Pre-Requisites

• Install Lustre Client packages
• Lustre Client File System mounted with full cluster-coherent locking support on all CTDB nodes:

mkdir -m 0755 -p /mnt/lfs
mount -t lustre -oflock 172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs/

• Networking

  Each CTDB node will require three network interfaces on three separate networks inorder to operate correctly:

  1. a high performance data network interface, used to mount the Lustre client file system. This is typically an InfiniBand or 10Gb Ethernet fabric
  2. a private network interface over which all CTDB nodes communicate, sending heartbeats, exchanging data and cluster commands
  3. a public network interface over which clients will connect to the services running on the CTDB cluster, such as Samba

  In our case, networks 1 and 2 are the same (pxe). Here is an example of a CTDB Framework on Lustre:

Error creating thumbnail: File missing

Install CTDB

Install CTDB on all nodes as follows:

sudo yum -y install ctdb

Configure CTDB

On RHEL 6, the global configuration file is /etc/sysconfig/ctdb. At a minimum, CTDB needs the following information:

• The list of IP addresses of every member of the CTDB cluster on the private network. This is stored in a plain text file referred to as the node list, the location of which is governed by the variable CTDB_NODES in the global configuration.

  In our example:

[root@ctdb-node ~] cat /etc/ctdb/nodes 
172.28.0.131
172.28.1.17

• The list of public IP addresses, along with the subnet mask and the network interface, that will be used for hosting public services. This information is kept in a plain text file referenced by the global configuration variable CTDB_PUBLIC_ADDRESSES. The format of the IPs should be <IP Address>/<NetMask> <Network Device>.

  In our example:

[root@ctdb-node ~] cat /etc/ctdb/public_addresses 
10.0.0.129/8 eth0
10.0.0.128/8 eth0

• The network interface for the public network should not be configured by the operating system with any of the CTDB public IP addresses. It is possible for the interface to be configured and active with an address that lies outside the range used by CTDB, but it is far simpler to prevent the operating system from attempting to manage the network interface device in the first place. In the configuration file /etc/sysconfig/network-scripts/ifcfg-<device>, set the following variables as shown:

ONBOOT=no
NM_CONTROLLED=no
BOOTPROTO=none

• The location of the recovery lock file on the shared file system (in this case, the recovery lock is kept on Lustre). The recovery lock location is kept in the global configuration variable CTDB_RECOVERY_LOCK.

  For example:

# Lustre is mounted at /lustre/scratch.
# Create a directory to contain the CTDB recovery lock:
[root@ctdb-node ~] mkdir -p -m 0700 /mnt/lfs/.ctdb
[root@ctdb-node ~] ls -ld /mnt/lfs/.ctdb
drwx------ 2 root root 4096 Oct 10 17:56 /mnt/lfs/.ctdb

CTDB Configuration example

[root@ctdb-node] cat /etc/sysconfig/ctdb
CTDB_RECOVERY_LOCK="/mnt/lfs/.ctdb/recovery_lock"
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_NODES=/etc/ctdb/nodes
CTDB_LOGFILE=/var/log/log.ctdb
CTDB_DEBUGLEVEL=ERR
CTDB_SYSLOG=no

Multiple CTDB Nodes

Copy the configuration files created (nodes, public_addresses, etc/sysconfig/ctdb) on all nodes, on the same location.

Starting the CTDB Cluster

Start the ctdb service on all CTDB nodes and validate that it is working.

Note: Disable iptables first, on all nodes!

[root@ctdb-node ~] sudo service iptables stop
[root@ctdb-node ~] sudo service ctdb start
[root@ctdb-node ~] sudo ctdb status
Number of nodes:2
pnn:0 172.28.0.131     OK (THIS NODE)
pnn:1 172.28.1.17      OK
Generation:1501954827
Size:2
hash:0 lmaster:0
hash:1 lmaster:1
Recovery mode:NORMAL (0)
Recovery master:0

Install Samba

Install Samba packages on all CTDB nodes:

sudo yum -y install samba samba-client samba-common samba-doc samba-winbind samba-winbind-clients

Configure and Start Samba

Edit the Samba configuration file

[root@ctdb-node ~] mv /etc/samba/smb.conf /etc/samba/smb.conf-backup
[root@ctdb-node ~] vim /etc/samba/smb.conf #create a new config file with the following info
[root@ctdb-node ~] cat /etc/samba/smb.conf
[global]
netbios name = usedForWindows
workgroup = bostonLustre
server string = [%h] Samba %v Server (Lustre)
security = user
encrypt passwords = yes
passdb backend = tdbsam
clustering = yes

[root@ctdb-node ~] testparm -s #test samba
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Loaded services file OK.
Server role: ROLE_STANDALONE
[global]
	workgroup = BOSTONLUSTRE
	netbios name = USEDFORWINDOWS
	server string = [%h] Samba %v Server (Lustre)
	clustering = Yes
	idmap config * : backend = tdb

Add a User in Samba DataBase

[root@ctdb-node ~] id ajax 
uid=500(ajax) gid=500(ajax) groups=500(ajax)
[root@ctdb-node ~] sudo smbpasswd -a ajax #add an EXISTING user in samba database
New SMB password:
Retype new SMB password:
Added user ajax

Check it is working

[root@ctdb-node ~] sudo service nmb start
[root@ctdb-node ~] sudo service smb start

[root@ctdb-node ~] service iptables stop #stop iptables

[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -N
Anonymous login successful
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Sharename       Type      Comment
	---------       ----      -------
	lfs             Disk      
	IPC$            IPC       IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
Anonymous login successful
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Server               Comment
	---------            -------
	USEDFORWINDOWS       [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre

	Workgroup            Master
	---------            -------
	BOSTONLUSTRE         USEDFORWINDOWS

[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -U ajax #test the config for the user
Enter ajaxs password:
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Sharename       Type      Comment
	---------       ----      -------
	lfs             Disk      
	IPC$            IPC       IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Server               Comment
	---------            -------
	USEDFORWINDOWS       [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre

	Workgroup            Master
	---------            -------
	BOSTONLUSTRE         USEDFORWINDOWS

Add Samba to CTDB

Preparation

sudo service ctdb stop
sudo service nmb stop
sudo service smb stop

Update the Samba Configuration

To turn on clustering support, edit the Samba configuration file (/etc/samba/smb.conf) created in the previous section and add the following to the [global] section:

clustering = yes

Update the CTDB Configuration

Edit the CTDB configuration file (default: /etc/sysconfig/ctdb) and add the following entries:

CTDB_MANAGES_SAMBA=yes
CTDB_SERVICE_NMB=nmb

Disable SMB and NMB Services from System Startup

sudo chkconfig nmb off
sudo chkconfig smb off
sudo chkconfig winbind off

Verify:

sudo chkconfig --list nmb
sudo chkconfig --list smb
sudo chkconfig --list winbind

Start the Samba CTDB Cluster on One Node

1. Stop all CTDB services on the node:

sudo service ctdb stop

2. Stop all Samba services on the node:

sudo service nmb stop
sudo service smb stop
sudo service winbind stop

3. (Optional, but recommended.) Stop the CTDB services on all other cluster nodes:

sudo onnode all service ctdb stop

4. Check that the Samba and CTDB configuration files have been distributed to all CTDB nodes.

5. Check that the Lustre file system is mounted and has global lock (flock) support enabled:

mount -t lustre

6. Start the CTDB cluster service:

sudo service ctdb start

7. Verify that the CTDB service is running:

sudo ctdb status

8. Verify that the Samba NMB and SMB services are running:

service nmb status
service smb status

9. Attempt a connection to the SMB service running in the cluster. Try the NetBios service name, as well as each individual IP addresses of each CTDB public address:

smbclient -L //USEDFORWINDOWS -N

Samba Authentication with CTDB

In order to add a User in the Samba Database, you must first create the User in Unix and then add it like this:

id <username>
sudo smbpasswd -a <username>
#or
sudo pdbedit -a -u <username>

To query the database for a list of Samba users, run the pdbedit command:

sudo pdbedit -L

Once the account has been added to the CTDB password database and is listed in the output of pdbedit -L, one can attempt an authenticated connection to the Samba cluster:

smbclient -L //USEDFORWINDOWS -U ajax

Create a Samba Share for Lustre

The Samba Share must be added in the /etc/samba/smb.conf:

[root@ctdb-node ~] cat /etc/samba/smb.conf
[global]
netbios name = usedForWindows
workgroup = bostonLustre
server string = [%h] Samba %v Server (Lustre)
security = user
encrypt passwords = yes
passdb backend = tdbsam
clustering = yes

[lfs]
path = /mnt/lfs
valid users = root samba
available = yes
read only = no
browseable = yes
public = yes

Note: Be sure to add the users, you want to access the Samba Share, in the valid users value.

Multiple CTDB nodes

When using more than one CTDB node, make sure you copy the Samba configuration file on the same location on all nodes. Then start Samba, according to the above procedure, on all nodes.

After that you will be able to mount the share (lfs) from a Windows client, using any of the CTDB nodes IPs (see below).

Configuration on Windows Client

1. Create a new User with the same username and password as one of the existing users in the Samba database. (In our example ajax)

2. Open a File Explorer window

3. Right click on "This PC" on the left side panel and click on "Map Network Drive":

Error creating thumbnail: File missing

4. In the "Folder" field, add the public ip of a CTDB node and then the name of the Samba Share. If you are not logged in as the created User, tick the box for "Connect with different credentials":

Error creating thumbnail: File missing

5. Fill in the credentials of the user you created:

Error creating thumbnail: File missing

6. Lustre File System is now mounted:

Error creating thumbnail: File missing