Lustre intel: Setting up a Windows client

The following instructions are for one (or multiple) linux system(s), which will serve as the middle-layer cluster, which "translates" Lustre to Windows, using Samba.

There are two components that need to be installed:

1. CTDB
2. Samba

Pre-Requisites

• Install Lustre Client packages
• Lustre Client File System mounted with full cluster-coherent locking support on all CTDB nodes:

mkdir -m 0755 -p /mnt/lfs
mount -t lustre -oflock 172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs/

• Networking

  Each CTDB node will require three network interfaces on three separate networks inorder to operate correctly:

  1. a high performance data network interface, used to mount the Lustre client file system. This is typically an InfiniBand or 10Gb Ethernet fabric
  2. a private network interface over which all CTDB nodes communicate, sending heartbeats, exchanging data and cluster commands
  3. a public network interface over which clients will connect to the services running on the CTDB cluster, such as Samba

  In our case, networks 1 and 2 are the same (pxe). Here is an example of a CTDB Framework on Lustre:

Error creating thumbnail: File missing

Install CTDB

Install CTDB as follows:

sudo yum -y install ctdb

Configure CTDB

On RHEL 6, the global configuration file is /etc/sysconfig/ctdb. At a minimum, CTDB needs the following information:

• The list of IP addresses of every member of the CTDB cluster on the private network. This is stored in a plain text file referred to as the node list, the location of which is governed by the variable CTDB_NODES in the global configuration.

  In our one node example:

[root@ctdb-node ~] cat /etc/ctdb/nodes 
172.28.0.131

• The list of public IP addresses, along with the subnet mask and the network interface, that will be used for hosting public services. This information is kept in a plain text file referenced by the global configuration variable CTDB_PUBLIC_ADDRESSES. The format of the IPs should be <IP Address>/<NetMask> <Network Device>.

  In our one node example:

[root@ctdb-node ~] cat /etc/ctdb/public_addresses 
10.0.0.129/8 eth0

• The network interface for the public network should not be configured by the operating system with any of the CTDB public IP addresses. It is possible for the interface to be configured and active with an address that lies outside the range used by CTDB, but it is far simpler to prevent the operating system from attempting to manage the network interface device in the first place. In the configuration file /etc/sysconfig/network-scripts/ifcfg-<device>, set the following variables as shown:

ONBOOT=no
NM_CONTROLLED=no
BOOTPROTO=none

• The location of the recovery lock file on the shared file system (in this case, the recovery lock is kept on Lustre). The recovery lock location is kept in the global configuration variable CTDB_RECOVERY_LOCK.

  For example:

# Lustre is mounted at /lustre/scratch.
# Create a directory to contain the CTDB recovery lock:
[root@ctdb-node ~] mkdir -p -m 0700 /mnt/lfs/.ctdb
[root@ctdb-node ~] ls -ld /mnt/lfs/.ctdb
drwx------ 2 root root 4096 Oct 10 17:56 /mnt/lfs/.ctdb

CTDB Configuration example

CTDB_RECOVERY_LOCK="/mnt/lfs/.ctdb/recovery_lock"
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_NODES=/etc/ctdb/nodes
CTDB_LOGFILE=/var/log/log.ctdb
CTDB_DEBUGLEVEL=ERR
CTDB_SYSLOG=no

Starting the CTDB Cluster

[root@ctdb-node ~] sudo service ctdb start
[root@ctdb-node ~] sudo ctdb status
Number of nodes:1
pnn:0 10.0.0.129 OK (THIS NODE)
Generation:1160838828
Size:1
hash:0 lmaster:0
Recovery mode:NORMAL (0)
Recovery master:0

Install Samba

sudo yum -y install samba samba-client samba-common samba-doc samba-winbind samba-winbind-clients

Configure and Start Samba

[root@ctdb-node ~] mv /etc/samba/smb.conf /etc/samba/smb.conf-backup
[root@ctdb-node ~] vim /etc/samba/smb.conf #create a new config file with the following info
[root@ctdb-node ~] cat /etc/samba/smb.conf
[global]
netbios name = SERVICENAME
workgroup = SMBDOMAINNAME
server string = [%h] Samba %v Server (Lustre)
security = user
encrypt passwords = yes
passdb backend = tdbsam

[root@ctdb-node ~] testparm -s #test samba
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit
(16384)
Loaded services file OK.
Server role: ROLE_STANDALONE
[global]
workgroup = SMBDOMAINNAME
netbios name = SERVICENAME
server string = [%h] Samba %v Server for Lustre
idmap config * : backend = tdb2

[root@ctdb-node ~] id ajax 
uid=500(ajax) gid=500(ajax) groups=500(ajax)
[root@ctdb-node ~] sudo smbpasswd -a ajax #add an EXISTING user in samba database
New SMB password:
Retype new SMB password:
Added user ajax

[root@ctdb-node ~] sudo service nmb start
[root@ctdb-node ~] sudo service smb start

[root@ctdb-node ~] service iptables stop #stop iptables

[root@ctdb-node ~] smbclient -L //SERVICENAME -N #test the configuration
Anonymous login successful
Domain=[SMBDOMAINNAME] OS=[Unix] Server=[Samba 3.6.9-151.el6_4.1]
Sharename Type Comment
--------- ---- -------
IPC$
IPC
IPC Service ([c64-1] Samba 3.6.9-
151.el6_4.1 Server (Lustre))
Anonymous login successful
Domain=[SMBDOMAINNAME] OS=[Unix] Server=[Samba 3.6.9-151.el6_4.1]
Server Comment
--------- -------
SERVICENAME
(Lustre)
[c64-1] Samba 3.6.9-151.el6_4.1 Server
Workgroup Master
--------- -------
SMBDOMAINNAME

[root@ctdb-node ~] smbclient -L //SERVICENAME -U ajax #test the config for the user
Enter ajax's password:
Domain=[SMBDOMAINNAME] OS=[Unix] Server=[Samba 3.6.9-151.el6_4.1]
Sharename Type Comment
--------- ---- -------
IPC$
IPC
IPC Service ([c64-1] Samba 3.6.9-
151.el6_4.1 Server (Lustre))
Domain=[SMBDOMAINNAME] OS=[Unix] Server=[Samba 3.6.9-151.el6_4.1]
Server Comment
--------- -------
SERVICENAME
(Lustre)
[c64-1] Samba 3.6.9-151.el6_4.1 Server
Workgroup Master
--------- -------
SMBDOMAINNAME
SERVICENAME