View source for Lustre intel: Setting up a Windows client

The following instructions are for one (or multiple) linux system(s), which will serve as the middle-layer cluster, which "translates" Lustre to Windows, using Samba.

There are two components that need to be installed:
#CTDB
#Samba

== Pre-Requisites ==

*Install Lustre Client packages
*Lustre Client File System mounted with full cluster-coherent locking support on all CTDB nodes:
<syntaxhighlight>
mkdir -m 0755 -p /mnt/lfs
mount -t lustre -oflock 172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs/
</syntaxhighlight>
Also add this line in the <tt>/etc/fstab</tt> file:
<syntaxhighlight>
172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs  lustre   rw,flock     0 0
</syntaxhighlight>

*Networking
*:Each CTDB node will require three network interfaces on three separate networks inorder to operate correctly:
*#a high performance data network interface, used to mount the Lustre client file system. This is typically an InfiniBand or 10Gb Ethernet fabric
*#a private network interface over which all CTDB nodes communicate, sending heartbeats, exchanging data and cluster commands
*#a public network interface over which clients will connect to the services running on the CTDB cluster, such as Samba
*:In our case, networks 1 and 2 are the same (pxe). Here is an example of a CTDB Framework on Lustre:

[[File:Ctdb-nodes.png | 450px]]

=== <tt>iptables</tt> Setup ===

Make sure to add these rules in the iptables setup, before the reject all rule (if there is one):

<syntaxhighlight>
iptables -I INPUT 5 -p udp --dport 137:138 -j ACCEPT #samba ports
iptables -I INPUT 6 -p tcp --dport 139 -j ACCEPT     #samba port
iptables -I INPUT 7 -p tcp --dport 445 -j ACCEPT     #samba port
iptables -I INPUT 9 -p tcp --dport 4379 -j ACCEPT    #ctdb port
iptables -I INPUT 8 -p tcp --dport 2049 -j ACCEPT    #nfs ports (optional)
</syntaxhighlight>

== Install CTDB ==

Install CTDB on all nodes as follows:
<syntaxhighlight>
sudo yum -y install ctdb
</syntaxhighlight>

== Configure CTDB ==

On RHEL 6, the global configuration file is <tt>/etc/sysconfig/ctdb</tt>. At a minimum, CTDB needs the following information:

*The list of IP addresses of every member of the CTDB cluster on the private network. This is stored in a plain text file referred to as the node list, the location of which is governed by the variable CTDB_NODES in the global configuration.
*:In our example:
<syntaxhighlight>
[root@ctdb-node ~] cat /etc/ctdb/nodes 
172.28.0.131
172.28.1.17
</syntaxhighlight>

*The list of public IP addresses, along with the subnet mask and the network interface, that will be used for hosting public services. This information is kept in a plain text file referenced by the global configuration variable CTDB_PUBLIC_ADDRESSES. The format of the IPs should be <IP Address>/<NetMask> <Network Device>.
*:In our example:
<syntaxhighlight>
[root@ctdb-node ~] cat /etc/ctdb/public_addresses 
10.0.0.129/8 eth0
10.0.0.128/8 eth0
</syntaxhighlight>

*:The network interface for the public network should not be configured by the operating system with any of the CTDB public IP addresses. It is possible for the interface to be configured and active with an address that lies outside the range used by CTDB, but it is far simpler to prevent the operating system from attempting to manage the network interface device in the first place. In the configuration file <tt>/etc/sysconfig/network-scripts/ifcfg-<device></tt>, set the following variables as shown:
<syntaxhighlight>
ONBOOT=no
NM_CONTROLLED=no
BOOTPROTO=none
</syntaxhighlight>

*The location of the recovery lock file on the shared file system (in this case, the recovery lock is kept on Lustre). The recovery lock location is kept in the global configuration variable CTDB_RECOVERY_LOCK.
*:For example:
<syntaxhighlight>
# Lustre is mounted at /lustre/scratch.
# Create a directory to contain the CTDB recovery lock:
[root@ctdb-node ~] mkdir -p -m 0700 /mnt/lfs/.ctdb
[root@ctdb-node ~] ls -ld /mnt/lfs/.ctdb
drwx------ 2 root root 4096 Oct 10 17:56 /mnt/lfs/.ctdb
</syntaxhighlight>

=== CTDB Configuration example ===

<syntaxhighlight>
[root@ctdb-node] cat /etc/sysconfig/ctdb
CTDB_RECOVERY_LOCK="/mnt/lfs/.ctdb/recovery_lock"
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_NODES=/etc/ctdb/nodes
CTDB_LOGFILE=/var/log/log.ctdb
CTDB_DEBUGLEVEL=ERR
CTDB_SYSLOG=no
</syntaxhighlight>

=== Multiple CTDB Nodes ===

Copy the configuration files created (nodes, public_addresses, etc/sysconfig/ctdb) on all nodes, on the same location.

== Starting the CTDB Cluster ==

Start the ctdb service on all CTDB nodes and validate that it is working.

=== Start the service ===

<syntaxhighlight>
[root@ctdb-node ~] sudo service ctdb start
[root@ctdb-node ~] sudo ctdb status
Number of nodes:2
pnn:0 172.28.0.131     OK (THIS NODE)
pnn:1 172.28.1.17      OK
Generation:1501954827
Size:2
hash:0 lmaster:0
hash:1 lmaster:1
Recovery mode:NORMAL (0)
Recovery master:0
</syntaxhighlight>

== Install Samba ==

Install Samba packages on all CTDB nodes:

<syntaxhighlight>
sudo yum -y install samba samba-client samba-common samba-doc samba-winbind samba-winbind-clients
</syntaxhighlight>

== Configure and Start Samba ==

=== Edit the Samba configuration file ===

<syntaxhighlight>
[root@ctdb-node ~] mv /etc/samba/smb.conf /etc/samba/smb.conf-backup
[root@ctdb-node ~] vim /etc/samba/smb.conf #create a new config file with the following info
[root@ctdb-node ~] cat /etc/samba/smb.conf
[global]
netbios name = usedForWindows
workgroup = bostonLustre
server string = [%h] Samba %v Server (Lustre)
security = user
encrypt passwords = yes
passdb backend = tdbsam
clustering = yes

[root@ctdb-node ~] testparm -s #test samba
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Loaded services file OK.
Server role: ROLE_STANDALONE
[global]
	workgroup = BOSTONLUSTRE
	netbios name = USEDFORWINDOWS
	server string = [%h] Samba %v Server (Lustre)
	clustering = Yes
	idmap config * : backend = tdb
</syntaxhighlight>

=== Add a User in Samba DataBase ===

<syntaxhighlight>
[root@ctdb-node ~] id ajax 
uid=500(ajax) gid=500(ajax) groups=500(ajax)
[root@ctdb-node ~] sudo smbpasswd -a ajax #add an EXISTING user in samba database
New SMB password:
Retype new SMB password:
Added user ajax
</syntaxhighlight>

=== Check it is working ===

<syntaxhighlight>

[root@ctdb-node ~] sudo service nmb start
[root@ctdb-node ~] sudo service smb start

[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -N
Anonymous login successful
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Sharename       Type      Comment
	---------       ----      -------
	lfs             Disk      
	IPC$            IPC       IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
Anonymous login successful
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Server               Comment
	---------            -------
	USEDFORWINDOWS       [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre

	Workgroup            Master
	---------            -------
	BOSTONLUSTRE         USEDFORWINDOWS

[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -U ajax #test the config for the user
Enter ajaxs password:
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Sharename       Type      Comment
	---------       ----      -------
	lfs             Disk      
	IPC$            IPC       IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Server               Comment
	---------            -------
	USEDFORWINDOWS       [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre

	Workgroup            Master
	---------            -------
	BOSTONLUSTRE         USEDFORWINDOWS
</syntaxhighlight>

== Add Samba to CTDB ==

=== Preparation ===

<syntaxhighlight>
sudo service ctdb stop
sudo service nmb stop
sudo service smb stop
</syntaxhighlight>

=== Update the Samba Configuration ===

To turn on clustering support, edit the Samba configuration file (/etc/samba/smb.conf) created in the previous section and add the following to the [global] section:
<syntaxhighlight>
clustering = yes
</syntaxhighlight>

=== Update the CTDB Configuration ===

Edit the CTDB configuration file (default: /etc/sysconfig/ctdb) and add the following entries:
<syntaxhighlight>
CTDB_MANAGES_SAMBA=yes
CTDB_SERVICE_NMB=nmb
</syntaxhighlight>

=== Disable SMB and NMB Services from System Startup ===

<syntaxhighlight>
sudo chkconfig nmb off
sudo chkconfig smb off
sudo chkconfig winbind off
</syntaxhighlight>

Verify:
<syntaxhighlight>
sudo chkconfig --list nmb
sudo chkconfig --list smb
sudo chkconfig --list winbind
</syntaxhighlight>

=== Start the Samba CTDB Cluster on One Node ===

1. Stop all CTDB services on the node:
<syntaxhighlight>
sudo service ctdb stop
</syntaxhighlight>

2. Stop all Samba services on the node:
<syntaxhighlight>
sudo service nmb stop
sudo service smb stop
sudo service winbind stop
</syntaxhighlight>

3. (Optional, but recommended.) Stop the CTDB services on all other cluster nodes:
<syntaxhighlight>
sudo onnode all service ctdb stop
</syntaxhighlight>

4. Check that the Samba and CTDB configuration files have been distributed to all CTDB nodes.

5. Check that the Lustre file system is mounted and has global lock (flock) support enabled:
<syntaxhighlight>
mount -t lustre
</syntaxhighlight>

6. Start the CTDB cluster service:
<syntaxhighlight>
sudo service ctdb start
</syntaxhighlight>

7. Verify that the CTDB service is running:
<syntaxhighlight>
sudo ctdb status
</syntaxhighlight>

8. Verify that the Samba NMB and SMB services are running:
<syntaxhighlight>
service nmb status
service smb status
</syntaxhighlight>

9. Attempt a connection to the SMB service running in the cluster. Try the NetBios service name, as well as each individual IP addresses of each CTDB public address:
<syntaxhighlight>
smbclient -L //USEDFORWINDOWS -N
</syntaxhighlight>

=== Samba Authentication with CTDB ===

In order to add a User in the Samba Database, you must first create the User in Unix and then add it like this:
<syntaxhighlight>
id <username>
sudo smbpasswd -a <username>
#or
sudo pdbedit -a -u <username>
</syntaxhighlight>

To query the database for a list of Samba users, run the pdbedit command:
<syntaxhighlight>
sudo pdbedit -L
</syntaxhighlight>

Once the account has been added to the CTDB password database and is listed in the output of pdbedit -L, one can attempt an authenticated connection to the Samba cluster:
<syntaxhighlight>
smbclient -L //USEDFORWINDOWS -U ajax
</syntaxhighlight>

== Create a Samba Share for Lustre ==

The Samba Share must be added in the <tt>/etc/samba/smb.conf</tt>:

<syntaxhighlight>
[root@ctdb-node ~] cat /etc/samba/smb.conf
[global]
netbios name = usedForWindows
workgroup = bostonLustre
server string = [%h] Samba %v Server (Lustre)
security = user
encrypt passwords = yes
passdb backend = tdbsam
clustering = yes

[lfs]
path = /mnt/lfs
valid users = root samba
available = yes
writable = yes
read only = no
browseable = yes
public = yes
</syntaxhighlight>

'''Note:''' Be sure to add the users, you want to access the Samba Share, in the <tt>"valid users"</tt> value.

=== Multiple CTDB nodes ===

When using more than one CTDB node, make sure you copy the Samba configuration file on the same location on all nodes. Then start Samba, according to the above procedure, on all nodes.

After that you will be able to mount the share (lfs) from a Windows client, using any of the CTDB nodes IPs (see below).

== Samba and Active Directory Integration Using Winbind ==

=== Update Samba and Configure Kerberos ===

'''1.''' Make the following changes to the [global] section of the Samba configuration file, smb.conf:
<syntaxhighlight>
security = ads
workgroup = <AD DOMAIN>
realm = <KRB REALM>
</syntaxhighlight>

The Kerberos realm is the DNS domain of the Active Directory network and must be recorded in uppercase. The workgroup is the AD domain, also in upper–case, and is the first section of the Kerberos realm: the text up to but not including the first separator character, "." (period). For example:
<syntaxhighlight>
security = ads
workgroup = BOSTONLABS
realm = BOSTONLABS.LFS.INT
</syntaxhighlight>

'''2.''' The DNS resolver must also be updated for each participating Samba server so that the Active Directory domain controller is the name server. If there is more than one domain controller, they can be added to the list. Edit etc/resolv.conf to make the changes, for example:
<syntaxhighlight>
domain bostonlabs.co.uk
search bostonlabs.co.uk
nameserver 10.0.0.1
nameserver 172.28.0.2
</syntaxhighlight>

'''3.''' Install the bind-utils package;it contains tools to interrogate DNS records an assist with verification and debugging:
<syntaxhighlight>
sudo yum -y install bind-utils
</syntaxhighlight>

'''4.''' However, make sure that the Kerberos tools are installed, to aid debugging. On RHEL and CentOS systems, install the krb5-workstation package:
<syntaxhighlight>
sudo yum -y install krb5-workstation
</syntaxhighlight>

'''5.''' The Kerberos configuration on RHEL systems is recorded in /etc/krb5.conf. On installation, the krb5 configuration file contains mostly example information that is effectively ignored. The following basic configuration is recommended:
<syntaxhighlight>
[libdefaults]
default_realm = <KRB REALM>
dns_lookup_kdc = true
dns_lookup_realm = false
</syntaxhighlight>

The default_realm variable should match the realm supplied in Samba's smb.conf. The variable dns_lookup_kdc tells the Kerberos library to use DNS SRV records to locate a Key Distribution Centre (KDC). This variable should be set to true; this is why DNS configuration is so important. Finally, the variable dns_lookup_realm tries to use DNS TXT records to determine the Kerberos realm of a host. Set dns_lookup_realm to false.

'''6.'''
Run the following commands on all of the Samba servers in the CTDB cluster to ensure that DNS is working correctly:
<syntaxhighlight>
host -t SRV _ldap._tcp.<DOMAIN>
host -t SRV _kerberos._tcp.<DOMAIN>
host -t SRV _kerberos._udp.<DOMAIN>
</syntaxhighlight>

For example:
<syntaxhighlight>
[root@ctdb-node2 ~] host -t SRV _ldap._tcp.bostonlabs.co.uk
_ldap._tcp.bostonlabs.co.uk has SRV record 0 100 389 srv2.bostonlabs.co.uk.
[root@ctdb-node2 ~] host -t SRV _kerberos._udp.bostonlabs.co.uk
_kerberos._udp.bostonlabs.co.uk has SRV record 0 100 88 srv2.bostonlabs.co.uk.
[root@ctdb-node2 ~] host -t SRV _kerberos._tcp.bostonlabs.co.uk
_kerberos._tcp.bostonlabs.co.uk has SRV record 0 100 88 srv2.bostonlabs.co.uk.
</syntaxhighlight>

If these commands fail, there may be something wrong with the DNS setup, or the DNS servers don’t have the requisite information.

'''7.''' To verify the Kerberos connection, run the kinit command:
<syntaxhighlight>
sudo kinit Administrator@<REALM>
</syntaxhighlight>
The realm must be supplied in uppercase. If dns_lookup_kdc is false, then the kinit command may return an error:
<syntaxhighlight>
kinit: Cannot find KDC for requested realm while getting initial credentials
</syntaxhighlight>

'''8.''' On success, you can use the klist command to list the granted Kerberos ticket. The following example shows the complete transaction for acquiring what Kerberos calls a "ticket granting ticket" for the realm BOSTONLABS.CO.UK:
<syntaxhighlight>
[root@ctdb-node2 ~] kinit mouza@BOSTONLABS.CO.UK
Password for mouza@BOSTONLABS.CO.UK:
[root@ctdb-node2 ~] klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: mouza@BOSTONLABS.CO.UK

Valid starting     Expires            Service principal
06/29/15 10:42:01  06/29/15 20:39:44  krbtgt/BOSTONLABS.CO.UK@BOSTONLABS.CO.UK
	renew until 07/06/15 10:42:01
</syntaxhighlight>

=== Join Samba to the Active Directory Domain ===

'''1.''' Start or restart the CTDB cluster with the updated Samba configuration. CTDB must be running so that the machine trust account credentials are saved into Samba's private configuration:
<syntaxhighlight>
service ctdb restart
</syntaxhighlight>

'''2.''' Join the Samba server to the Active Directory domain and create the machine trust account on the Active Directory server:
<syntaxhighlight>
sudo net join -U Administrator
</syntaxhighlight>

The machine trust account is a special user account used to authenticate computers that wish to participate in an Active Directory network. Substitute Administrator for any valid account with the required authorization to add machines to the directory. The following example demonstrates a successful join to the AD domain SMBDOM:
<syntaxhighlight>
$ sudo net join -U Administrator
Enter Administrator's password:
Using short domain name -- BOSTONLABS
Joined 'UsedForWindows' to dns domain 'bostonlabs.co.uk'
Not doing automatic DNS update in a clustered setup.
</syntaxhighlight>

This step need only be carried out once on a single node in the CTDB Samba cluster, provided that each cluster member has the same Samba global configuration (i.e.: the same NetBIOS name, Active Directory domain and Kerberos realm).

'''3.''' The machine account status can be checked at any time as follows:
<syntaxhighlight>
[root@ctdb-node2 ~] net ads testjoin
Join is OK
</syntaxhighlight>

=== Integrate Winbind ===

== Configuration on Windows Client ==

1. Create a new User with the same username and password as one of the existing users in the Samba database. (In our example ajax)

2. Open a File Explorer window

3. Right click on "This PC" on the left side panel and click on "Map Network Drive":

[[File:Right-click.png | 500px]]

4. In the "Folder" field, add the public ip of a CTDB node and then the name of the Samba Share. If you are not logged in as the created User, tick the box for "Connect with different credentials":

[[File:Ip-share name.png | 500px]]

5. Fill in the credentials of the user you created:

[[File:Credentials.png | 500px]]

6. Lustre File System is now mounted:

[[File:Mounted!.png | 500px]]

== Troubleshooting ==

Check the <tt>/var/log/log.ctdb</tt> file for details.