Difference between revisions of "Lustre intel: Setting up a Windows client"

From Define Wiki
Jump to navigation Jump to search
 
(59 intermediate revisions by the same user not shown)
Line 5: Line 5:
 
#Samba
 
#Samba
  
−
=== Pre-Requisites ===
+
== Pre-Requisites ==
  
 +
*Install Lustre Client packages
 
*Lustre Client File System mounted with full cluster-coherent locking support on all CTDB nodes:
 
*Lustre Client File System mounted with full cluster-coherent locking support on all CTDB nodes:
 
<syntaxhighlight>
 
<syntaxhighlight>
 
mkdir -m 0755 -p /mnt/lfs
 
mkdir -m 0755 -p /mnt/lfs
 
mount -t lustre -oflock 172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs/
 
mount -t lustre -oflock 172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs/
 +
</syntaxhighlight>
 +
Also add this line in the <tt>/etc/fstab</tt> file:
 +
<syntaxhighlight>
 +
172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs  lustre  rw,flock    0 0
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 21: Line 26:
  
 
[[File:Ctdb-nodes.png | 450px]]
 
[[File:Ctdb-nodes.png | 450px]]
 +
 +
=== <tt>iptables</tt> Setup ===
 +
 +
Make sure to add these rules in the iptables setup, before the reject all rule (if there is one):
 +
 +
<syntaxhighlight>
 +
iptables -I INPUT 5 -p udp --dport 137:138 -j ACCEPT #samba ports
 +
iptables -I INPUT 6 -p tcp --dport 139 -j ACCEPT    #samba port
 +
iptables -I INPUT 7 -p tcp --dport 445 -j ACCEPT    #samba port
 +
iptables -I INPUT 9 -p tcp --dport 4379 -j ACCEPT    #ctdb port
 +
iptables -I INPUT 8 -p tcp --dport 2049 -j ACCEPT    #nfs ports (optional)
 +
</syntaxhighlight>
 +
 +
== Install CTDB ==
 +
 +
Install CTDB on all nodes as follows:
 +
<syntaxhighlight>
 +
sudo yum -y install ctdb
 +
</syntaxhighlight>
 +
 +
== Configure CTDB ==
 +
 +
On RHEL 6, the global configuration file is <tt>/etc/sysconfig/ctdb</tt>. At a minimum, CTDB needs the following information:
 +
 +
*The list of IP addresses of every member of the CTDB cluster on the private network. This is stored in a plain text file referred to as the node list, the location of which is governed by the variable CTDB_NODES in the global configuration.
 +
*:In our example:
 +
<syntaxhighlight>
 +
[root@ctdb-node ~] cat /etc/ctdb/nodes
 +
172.28.0.131
 +
172.28.1.17
 +
</syntaxhighlight>
 +
 +
*The list of public IP addresses, along with the subnet mask and the network interface, that will be used for hosting public services. This information is kept in a plain text file referenced by the global configuration variable CTDB_PUBLIC_ADDRESSES. The format of the IPs should be <IP Address>/<NetMask> <Network Device>.
 +
*:In our example:
 +
<syntaxhighlight>
 +
[root@ctdb-node ~] cat /etc/ctdb/public_addresses
 +
10.0.0.129/8 eth0
 +
10.0.0.128/8 eth0
 +
</syntaxhighlight>
 +
 +
*:The network interface for the public network should not be configured by the operating system with any of the CTDB public IP addresses. It is possible for the interface to be configured and active with an address that lies outside the range used by CTDB, but it is far simpler to prevent the operating system from attempting to manage the network interface device in the first place. In the configuration file <tt>/etc/sysconfig/network-scripts/ifcfg-<device></tt>, set the following variables as shown:
 +
<syntaxhighlight>
 +
ONBOOT=no
 +
NM_CONTROLLED=no
 +
BOOTPROTO=none
 +
</syntaxhighlight>
 +
 +
*The location of the recovery lock file on the shared file system (in this case, the recovery lock is kept on Lustre). The recovery lock location is kept in the global configuration variable CTDB_RECOVERY_LOCK.
 +
*:For example:
 +
<syntaxhighlight>
 +
# Lustre is mounted at /lustre/scratch.
 +
# Create a directory to contain the CTDB recovery lock:
 +
[root@ctdb-node ~] mkdir -p -m 0700 /mnt/lfs/.ctdb
 +
[root@ctdb-node ~] ls -ld /mnt/lfs/.ctdb
 +
drwx------ 2 root root 4096 Oct 10 17:56 /mnt/lfs/.ctdb
 +
</syntaxhighlight>
 +
 +
=== CTDB Configuration example ===
 +
 +
<syntaxhighlight>
 +
[root@ctdb-node] cat /etc/sysconfig/ctdb
 +
CTDB_RECOVERY_LOCK="/mnt/lfs/.ctdb/recovery_lock"
 +
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
 +
CTDB_NODES=/etc/ctdb/nodes
 +
CTDB_LOGFILE=/var/log/log.ctdb
 +
CTDB_DEBUGLEVEL=ERR
 +
CTDB_SYSLOG=no
 +
</syntaxhighlight>
 +
 +
=== Multiple CTDB Nodes ===
 +
 +
Copy the configuration files created (nodes, public_addresses, etc/sysconfig/ctdb) on all nodes, on the same location.
 +
 +
== Starting the CTDB Cluster ==
 +
 +
Start the ctdb service on all CTDB nodes and validate that it is working.
 +
 +
=== Start the service ===
 +
 +
<syntaxhighlight>
 +
[root@ctdb-node ~] sudo service ctdb start
 +
[root@ctdb-node ~] sudo ctdb status
 +
Number of nodes:2
 +
pnn:0 172.28.0.131    OK (THIS NODE)
 +
pnn:1 172.28.1.17      OK
 +
Generation:1501954827
 +
Size:2
 +
hash:0 lmaster:0
 +
hash:1 lmaster:1
 +
Recovery mode:NORMAL (0)
 +
Recovery master:0
 +
</syntaxhighlight>
 +
 +
== Install Samba ==
 +
 +
Install Samba packages on all CTDB nodes:
 +
 +
<syntaxhighlight>
 +
sudo yum -y install samba samba-client samba-common samba-doc samba-winbind samba-winbind-clients
 +
</syntaxhighlight>
 +
 +
== Configure and Start Samba ==
 +
 +
=== Edit the Samba configuration file ===
 +
 +
<syntaxhighlight>
 +
[root@ctdb-node ~] mv /etc/samba/smb.conf /etc/samba/smb.conf-backup
 +
[root@ctdb-node ~] vim /etc/samba/smb.conf #create a new config file with the following info
 +
[root@ctdb-node ~] cat /etc/samba/smb.conf
 +
[global]
 +
netbios name = usedForWindows
 +
workgroup = bostonLustre
 +
server string = [%h] Samba %v Server (Lustre)
 +
security = user
 +
encrypt passwords = yes
 +
passdb backend = tdbsam
 +
clustering = yes
 +
 +
[root@ctdb-node ~] testparm -s #test samba
 +
Load smb config files from /etc/samba/smb.conf
 +
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
 +
Loaded services file OK.
 +
Server role: ROLE_STANDALONE
 +
[global]
 +
workgroup = BOSTONLUSTRE
 +
netbios name = USEDFORWINDOWS
 +
server string = [%h] Samba %v Server (Lustre)
 +
clustering = Yes
 +
idmap config * : backend = tdb
 +
</syntaxhighlight>
 +
 +
=== Add a User in Samba DataBase ===
 +
 +
<syntaxhighlight>
 +
[root@ctdb-node ~] id ajax
 +
uid=500(ajax) gid=500(ajax) groups=500(ajax)
 +
[root@ctdb-node ~] sudo smbpasswd -a ajax #add an EXISTING user in samba database
 +
New SMB password:
 +
Retype new SMB password:
 +
Added user ajax
 +
</syntaxhighlight>
 +
 +
=== Check it is working ===
 +
 +
<syntaxhighlight>
 +
 +
[root@ctdb-node ~] sudo service nmb start
 +
[root@ctdb-node ~] sudo service smb start
 +
 +
[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -N
 +
Anonymous login successful
 +
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]
 +
 +
Sharename      Type      Comment
 +
---------      ----      -------
 +
lfs            Disk     
 +
IPC$            IPC      IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
 +
Anonymous login successful
 +
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]
 +
 +
Server              Comment
 +
---------            -------
 +
USEDFORWINDOWS      [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre
 +
 +
Workgroup            Master
 +
---------            -------
 +
BOSTONLUSTRE        USEDFORWINDOWS
 +
 +
[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -U ajax #test the config for the user
 +
Enter ajaxs password:
 +
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]
 +
 +
Sharename      Type      Comment
 +
---------      ----      -------
 +
lfs            Disk     
 +
IPC$            IPC      IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
 +
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]
 +
 +
Server              Comment
 +
---------            -------
 +
USEDFORWINDOWS      [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre
 +
 +
Workgroup            Master
 +
---------            -------
 +
BOSTONLUSTRE        USEDFORWINDOWS
 +
</syntaxhighlight>
 +
 +
== Add Samba to CTDB ==
 +
 +
=== Preparation ===
 +
 +
<syntaxhighlight>
 +
sudo service ctdb stop
 +
sudo service nmb stop
 +
sudo service smb stop
 +
</syntaxhighlight>
 +
 +
=== Update the Samba Configuration ===
 +
 +
To turn on clustering support, edit the Samba configuration file (/etc/samba/smb.conf) created in the previous section and add the following to the [global] section:
 +
<syntaxhighlight>
 +
clustering = yes
 +
</syntaxhighlight>
 +
 +
=== Update the CTDB Configuration ===
 +
 +
Edit the CTDB configuration file (default: /etc/sysconfig/ctdb) and add the following entries:
 +
<syntaxhighlight>
 +
CTDB_MANAGES_SAMBA=yes
 +
CTDB_SERVICE_NMB=nmb
 +
</syntaxhighlight>
 +
 +
=== Disable SMB and NMB Services from System Startup ===
 +
 +
<syntaxhighlight>
 +
sudo chkconfig nmb off
 +
sudo chkconfig smb off
 +
sudo chkconfig winbind off
 +
</syntaxhighlight>
 +
 +
Verify:
 +
<syntaxhighlight>
 +
sudo chkconfig --list nmb
 +
sudo chkconfig --list smb
 +
sudo chkconfig --list winbind
 +
</syntaxhighlight>
 +
 +
=== Start the Samba CTDB Cluster on One Node ===
 +
 +
1. Stop all CTDB services on the node:
 +
<syntaxhighlight>
 +
sudo service ctdb stop
 +
</syntaxhighlight>
 +
 +
2. Stop all Samba services on the node:
 +
<syntaxhighlight>
 +
sudo service nmb stop
 +
sudo service smb stop
 +
sudo service winbind stop
 +
</syntaxhighlight>
 +
 +
3. (Optional, but recommended.) Stop the CTDB services on all other cluster nodes:
 +
<syntaxhighlight>
 +
sudo onnode all service ctdb stop
 +
</syntaxhighlight>
 +
 +
4. Check that the Samba and CTDB configuration files have been distributed to all CTDB nodes.
 +
 +
5. Check that the Lustre file system is mounted and has global lock (flock) support enabled:
 +
<syntaxhighlight>
 +
mount -t lustre
 +
</syntaxhighlight>
 +
 +
6. Start the CTDB cluster service:
 +
<syntaxhighlight>
 +
sudo service ctdb start
 +
</syntaxhighlight>
 +
 +
7. Verify that the CTDB service is running:
 +
<syntaxhighlight>
 +
sudo ctdb status
 +
</syntaxhighlight>
 +
 +
8. Verify that the Samba NMB and SMB services are running:
 +
<syntaxhighlight>
 +
service nmb status
 +
service smb status
 +
</syntaxhighlight>
 +
 +
9. Attempt a connection to the SMB service running in the cluster. Try the NetBios service name, as well as each individual IP addresses of each CTDB public address:
 +
<syntaxhighlight>
 +
smbclient -L //USEDFORWINDOWS -N
 +
</syntaxhighlight>
 +
 +
=== Samba Authentication with CTDB ===
 +
 +
In order to add a User in the Samba Database, you must first create the User in Unix and then add it like this:
 +
<syntaxhighlight>
 +
id <username>
 +
sudo smbpasswd -a <username>
 +
#or
 +
sudo pdbedit -a -u <username>
 +
</syntaxhighlight>
 +
 +
To query the database for a list of Samba users, run the pdbedit command:
 +
<syntaxhighlight>
 +
sudo pdbedit -L
 +
</syntaxhighlight>
 +
 +
Once the account has been added to the CTDB password database and is listed in the output of pdbedit -L, one can attempt an authenticated connection to the Samba cluster:
 +
<syntaxhighlight>
 +
smbclient -L //USEDFORWINDOWS -U ajax
 +
</syntaxhighlight>
 +
 +
== Create a Samba Share for Lustre ==
 +
 +
The Samba Share must be added in the <tt>/etc/samba/smb.conf</tt>:
 +
 +
<syntaxhighlight>
 +
[root@ctdb-node ~] cat /etc/samba/smb.conf
 +
[global]
 +
netbios name = usedForWindows
 +
workgroup = bostonLustre
 +
server string = [%h] Samba %v Server (Lustre)
 +
security = user
 +
encrypt passwords = yes
 +
passdb backend = tdbsam
 +
clustering = yes
 +
 +
[lfs]
 +
path = /mnt/lfs
 +
valid users = root samba
 +
available = yes
 +
writable = yes
 +
read only = no
 +
browseable = yes
 +
public = yes
 +
</syntaxhighlight>
 +
 +
'''Note:''' Be sure to add the users, you want to access the Samba Share, in the <tt>"valid users"</tt> value.
 +
 +
=== Multiple CTDB nodes ===
 +
 +
When using more than one CTDB node, make sure you copy the Samba configuration file on the same location on all nodes. Then start Samba, according to the above procedure, on all nodes.
 +
 +
After that you will be able to mount the share (lfs) from a Windows client, using any of the CTDB nodes IPs (see below).
 +
 +
== Samba and Active Directory Integration Using Winbind ==
 +
 +
=== Update Samba and Configure Kerberos ===
 +
 +
'''1.''' Make the following changes to the [global] section of the Samba configuration file, smb.conf:
 +
<syntaxhighlight>
 +
security = ads
 +
workgroup = <AD DOMAIN>
 +
realm = <KRB REALM>
 +
</syntaxhighlight>
 +
 +
The Kerberos realm is the DNS domain of the Active Directory network and must be recorded in uppercase. The workgroup is the AD domain, also in upper–case, and is the first section of the Kerberos realm: the text up to but not including the first separator character, "." (period). For example:
 +
<syntaxhighlight>
 +
security = ads
 +
workgroup = BOSTONLABS
 +
realm = BOSTONLABS.LFS.INT
 +
</syntaxhighlight>
 +
 +
'''2.''' The DNS resolver must also be updated for each participating Samba server so that the Active Directory domain controller is the name server. If there is more than one domain controller, they can be added to the list. Edit etc/resolv.conf to make the changes, for example:
 +
<syntaxhighlight>
 +
domain bostonlabs.co.uk
 +
search bostonlabs.co.uk
 +
nameserver 10.0.0.1
 +
nameserver 172.28.0.2
 +
</syntaxhighlight>
 +
 +
'''3.''' Install the bind-utils package;it contains tools to interrogate DNS records an assist with verification and debugging:
 +
<syntaxhighlight>
 +
sudo yum -y install bind-utils
 +
</syntaxhighlight>
 +
 +
'''4.''' However, make sure that the Kerberos tools are installed, to aid debugging. On RHEL and CentOS systems, install the krb5-workstation package:
 +
<syntaxhighlight>
 +
sudo yum -y install krb5-workstation
 +
</syntaxhighlight>
 +
 +
'''5.''' The Kerberos configuration on RHEL systems is recorded in /etc/krb5.conf. On installation, the krb5 configuration file contains mostly example information that is effectively ignored. The following basic configuration is recommended:
 +
<syntaxhighlight>
 +
[libdefaults]
 +
default_realm = <KRB REALM>
 +
dns_lookup_kdc = true
 +
dns_lookup_realm = false
 +
</syntaxhighlight>
 +
 +
The default_realm variable should match the realm supplied in Samba's smb.conf. The variable dns_lookup_kdc tells the Kerberos library to use DNS SRV records to locate a Key Distribution Centre (KDC). This variable should be set to true; this is why DNS configuration is so important. Finally, the variable dns_lookup_realm tries to use DNS TXT records to determine the Kerberos realm of a host. Set dns_lookup_realm to false.
 +
 +
'''6.'''
 +
Run the following commands on all of the Samba servers in the CTDB cluster to ensure that DNS is working correctly:
 +
<syntaxhighlight>
 +
host -t SRV _ldap._tcp.<DOMAIN>
 +
host -t SRV _kerberos._tcp.<DOMAIN>
 +
host -t SRV _kerberos._udp.<DOMAIN>
 +
</syntaxhighlight>
 +
 +
For example:
 +
<syntaxhighlight>
 +
[root@ctdb-node2 ~] host -t SRV _ldap._tcp.bostonlabs.co.uk
 +
_ldap._tcp.bostonlabs.co.uk has SRV record 0 100 389 srv2.bostonlabs.co.uk.
 +
[root@ctdb-node2 ~] host -t SRV _kerberos._udp.bostonlabs.co.uk
 +
_kerberos._udp.bostonlabs.co.uk has SRV record 0 100 88 srv2.bostonlabs.co.uk.
 +
[root@ctdb-node2 ~] host -t SRV _kerberos._tcp.bostonlabs.co.uk
 +
_kerberos._tcp.bostonlabs.co.uk has SRV record 0 100 88 srv2.bostonlabs.co.uk.
 +
</syntaxhighlight>
 +
 +
If these commands fail, there may be something wrong with the DNS setup, or the DNS servers don’t have the requisite information.
 +
 +
'''7.''' To verify the Kerberos connection, run the kinit command:
 +
<syntaxhighlight>
 +
sudo kinit Administrator@<REALM>
 +
</syntaxhighlight>
 +
The realm must be supplied in uppercase. If dns_lookup_kdc is false, then the kinit command may return an error:
 +
<syntaxhighlight>
 +
kinit: Cannot find KDC for requested realm while getting initial credentials
 +
</syntaxhighlight>
 +
 +
'''8.''' On success, you can use the klist command to list the granted Kerberos ticket. The following example shows the complete transaction for acquiring what Kerberos calls a "ticket granting ticket" for the realm BOSTONLABS.CO.UK:
 +
<syntaxhighlight>
 +
[root@ctdb-node2 ~] kinit mouza@BOSTONLABS.CO.UK
 +
Password for mouza@BOSTONLABS.CO.UK:
 +
[root@ctdb-node2 ~] klist
 +
Ticket cache: FILE:/tmp/krb5cc_0
 +
Default principal: mouza@BOSTONLABS.CO.UK
 +
 +
Valid starting    Expires            Service principal
 +
06/29/15 10:42:01  06/29/15 20:39:44  krbtgt/BOSTONLABS.CO.UK@BOSTONLABS.CO.UK
 +
renew until 07/06/15 10:42:01
 +
</syntaxhighlight>
 +
 +
=== Join Samba to the Active Directory Domain ===
 +
 +
'''1.''' Start or restart the CTDB cluster with the updated Samba configuration. CTDB must be running so that the machine trust account credentials are saved into Samba's private configuration:
 +
<syntaxhighlight>
 +
service ctdb restart
 +
</syntaxhighlight>
 +
 +
'''2.''' Join the Samba server to the Active Directory domain and create the machine trust account on the Active Directory server:
 +
<syntaxhighlight>
 +
sudo net join -U Administrator
 +
</syntaxhighlight>
 +
 +
The machine trust account is a special user account used to authenticate computers that wish to participate in an Active Directory network. Substitute Administrator for any valid account with the required authorization to add machines to the directory. The following example demonstrates a successful join to the AD domain SMBDOM:
 +
<syntaxhighlight>
 +
$ sudo net join -U Administrator
 +
Enter Administrator's password:
 +
Using short domain name -- BOSTONLABS
 +
Joined 'UsedForWindows' to dns domain 'bostonlabs.co.uk'
 +
Not doing automatic DNS update in a clustered setup.
 +
</syntaxhighlight>
 +
 +
This step need only be carried out once on a single node in the CTDB Samba cluster, provided that each cluster member has the same Samba global configuration (i.e.: the same NetBIOS name, Active Directory domain and Kerberos realm).
 +
 +
'''3.''' The machine account status can be checked at any time as follows:
 +
<syntaxhighlight>
 +
[root@ctdb-node2 ~] net ads testjoin
 +
Join is OK
 +
</syntaxhighlight>
 +
 +
=== Integrate Winbind ===
 +
 +
'''1.''' Edit smb.conf and add the following in the [global] section:
 +
<syntaxhighlight>
 +
idmap config *:backend = tdb2
 +
idmap config *:range = 200000 - 300000
 +
</syntaxhighlight>
 +
 +
'''2.''' Make the Winbind service to be managed by CTDB by editing <tt>/etc/sysconfig/ctdb</tt> and setting the following variable:
 +
<syntaxhighlight>
 +
CTDB_MANAGES_WINBIND=yes
 +
</syntaxhighlight>
 +
 +
'''3.''' Make the following changes to the NSS configuration file, <tt>/etc/nsswitch.conf</tt>:
 +
<syntaxhighlight>
 +
passwd: files winbind
 +
group: files winbind
 +
</syntaxhighlight>
 +
 +
With this configuration, user and group lookups will be executed against the local /etc/passwd and /etc/group files, followed by the Winbind service if no match is found.
 +
 +
'''4.''' Edit the PAM files:
 +
*Edit the files /etc/pam.d/system-auth-ac and /etc/pam.d/password-auth-ac (the two files have identical content on CentOS 6). There are effectively four sections: auth, account, password, and session.
 +
 +
*In the auth section, add the following line directly after the entry for pam_unix.so and before the pam_deny.so line:
 +
<syntaxhighlight>
 +
auth sufficient pam_winbind.so use_first_pass
 +
</syntaxhighlight>
 +
 +
*In the account section, add the following line directly after the entry for am_succeed_if.so and before pam_permit.so:
 +
<syntaxhighlight>
 +
account sufficient pam_winbind.so use_first_pass
 +
</syntaxhighlight>
 +
 +
*In the password section, add the following line directly after the entry for pam_unix.so and before pam_deny.so:
 +
<syntaxhighlight>
 +
password sufficient pam_winbind.so use_first_pass
 +
</syntaxhighlight>
 +
 +
*In the session section, add the following directly after the entry for pam_unix.so (this should be the last line in the file):
 +
<syntaxhighlight>
 +
session required pam_winbind.so use_first_pass
 +
</syntaxhighlight>
 +
 +
'''Remember to edit both files.'''
 +
 +
'''5.''' In order to verify Winbind:
 +
<syntaxhighlight>
 +
[root@ctdb-node2 ~]# wbinfo -p
 +
Ping to winbindd succeeded
 +
 +
[root@ctdb-node2 ~]# wbinfo -t
 +
checking the trust secret for domain BOSTONLABS0 via RPC calls succeeded
 +
 +
[root@ctdb-node2 ~]# wbinfo -u
 +
USEDFORWINDOWS\root
 +
BOSTONLABS0\administrator
 +
BOSTONLABS0\guest
 +
BOSTONLABS0\krbtgt
 +
BOSTONLABS0\mh
 +
BOSTONLABS0\dj
 +
BOSTONLABS0\mhol
 +
BOSTONLABS0\md
 +
BOSTONLABS0\tm
 +
BOSTONLABS0\mouza
 +
BOSTONLABS0\jh
 +
BOSTONLABS0\bs
 +
 +
[root@ctdb-node2 ~]# wbinfo -g
 +
BOSTONLABS0\dhcp users
 +
BOSTONLABS0\dhcp administrators
 +
BOSTONLABS0\domain computers
 +
BOSTONLABS0\domain controllers
 +
BOSTONLABS0\schema admins
 +
BOSTONLABS0\enterprise admins
 +
BOSTONLABS0\cert publishers
 +
BOSTONLABS0\domain admins
 +
BOSTONLABS0\domain users
 +
BOSTONLABS0\domain guests
 +
BOSTONLABS0\group policy creator owners
 +
BOSTONLABS0\ras and ias servers
 +
BOSTONLABS0\allowed rodc password replication group
 +
BOSTONLABS0\denied rodc password replication group
 +
BOSTONLABS0\read-only domain controllers
 +
BOSTONLABS0\enterprise read-only domain controllers
 +
BOSTONLABS0\dnsadmins
 +
BOSTONLABS0\dnsupdateproxy
 +
BOSTONLABS0\bostonlabs-administrators
 +
</syntaxhighlight>
 +
 +
== Configuration on Windows Client ==
 +
 +
1. Create a new User with the same username and password as one of the existing users in the Samba database. (In our example ajax)
 +
 +
2. Open a File Explorer window
 +
 +
3. Right click on "This PC" on the left side panel and click on "Map Network Drive":
 +
 +
[[File:Right-click.png | 500px]]
 +
 +
4. In the "Folder" field, add the public ip of a CTDB node and then the name of the Samba Share. If you are not logged in as the created User, tick the box for "Connect with different credentials":
 +
 +
[[File:Ip-share name.png | 500px]]
 +
 +
5. Fill in the credentials of the user you created:
 +
 +
[[File:Credentials.png | 500px]]
 +
 +
6. Lustre File System is now mounted:
 +
 +
[[File:Mounted!.png | 500px]]
 +
 +
== Troubleshooting ==
 +
 +
Check the <tt>/var/log/log.ctdb</tt> file for details.

Latest revision as of 16:30, 6 July 2015

The following instructions are for one (or multiple) linux system(s), which will serve as the middle-layer cluster, which "translates" Lustre to Windows, using Samba.

There are two components that need to be installed:

  1. CTDB
  2. Samba

Pre-Requisites

  • Install Lustre Client packages
  • Lustre Client File System mounted with full cluster-coherent locking support on all CTDB nodes:
mkdir -m 0755 -p /mnt/lfs
mount -t lustre -oflock 172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs/

Also add this line in the /etc/fstab file: 

172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs  lustre   rw,flock     0 0
  • Networking
    Each CTDB node will require three network interfaces on three separate networks inorder to operate correctly:
    1. a high performance data network interface, used to mount the Lustre client file system. This is typically an InfiniBand or 10Gb Ethernet fabric
    2. a private network interface over which all CTDB nodes communicate, sending heartbeats, exchanging data and cluster commands
    3. a public network interface over which clients will connect to the services running on the CTDB cluster, such as Samba
    In our case, networks 1 and 2 are the same (pxe). Here is an example of a CTDB Framework on Lustre:
Error creating thumbnail: File missing

iptables Setup

Make sure to add these rules in the iptables setup, before the reject all rule (if there is one): 

iptables -I INPUT 5 -p udp --dport 137:138 -j ACCEPT #samba ports
iptables -I INPUT 6 -p tcp --dport 139 -j ACCEPT     #samba port
iptables -I INPUT 7 -p tcp --dport 445 -j ACCEPT     #samba port
iptables -I INPUT 9 -p tcp --dport 4379 -j ACCEPT    #ctdb port
iptables -I INPUT 8 -p tcp --dport 2049 -j ACCEPT    #nfs ports (optional)

Install CTDB

Install CTDB on all nodes as follows: 

sudo yum -y install ctdb

Configure CTDB

On RHEL 6, the global configuration file is /etc/sysconfig/ctdb. At a minimum, CTDB needs the following information:

  • The list of IP addresses of every member of the CTDB cluster on the private network. This is stored in a plain text file referred to as the node list, the location of which is governed by the variable CTDB_NODES in the global configuration.
    In our example:
[root@ctdb-node ~] cat /etc/ctdb/nodes 
172.28.0.131
172.28.1.17
  • The list of public IP addresses, along with the subnet mask and the network interface, that will be used for hosting public services. This information is kept in a plain text file referenced by the global configuration variable CTDB_PUBLIC_ADDRESSES. The format of the IPs should be <IP Address>/<NetMask> <Network Device>.
    In our example:
[root@ctdb-node ~] cat /etc/ctdb/public_addresses 
10.0.0.129/8 eth0
10.0.0.128/8 eth0
  • The network interface for the public network should not be configured by the operating system with any of the CTDB public IP addresses. It is possible for the interface to be configured and active with an address that lies outside the range used by CTDB, but it is far simpler to prevent the operating system from attempting to manage the network interface device in the first place. In the configuration file /etc/sysconfig/network-scripts/ifcfg-<device>, set the following variables as shown:
ONBOOT=no
NM_CONTROLLED=no
BOOTPROTO=none
  • The location of the recovery lock file on the shared file system (in this case, the recovery lock is kept on Lustre). The recovery lock location is kept in the global configuration variable CTDB_RECOVERY_LOCK.
    For example:
# Lustre is mounted at /lustre/scratch.
# Create a directory to contain the CTDB recovery lock:
[root@ctdb-node ~] mkdir -p -m 0700 /mnt/lfs/.ctdb
[root@ctdb-node ~] ls -ld /mnt/lfs/.ctdb
drwx------ 2 root root 4096 Oct 10 17:56 /mnt/lfs/.ctdb

CTDB Configuration example

[root@ctdb-node] cat /etc/sysconfig/ctdb
CTDB_RECOVERY_LOCK="/mnt/lfs/.ctdb/recovery_lock"
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_NODES=/etc/ctdb/nodes
CTDB_LOGFILE=/var/log/log.ctdb
CTDB_DEBUGLEVEL=ERR
CTDB_SYSLOG=no

Multiple CTDB Nodes

Copy the configuration files created (nodes, public_addresses, etc/sysconfig/ctdb) on all nodes, on the same location.

Starting the CTDB Cluster

Start the ctdb service on all CTDB nodes and validate that it is working.

Start the service

[root@ctdb-node ~] sudo service ctdb start
[root@ctdb-node ~] sudo ctdb status
Number of nodes:2
pnn:0 172.28.0.131     OK (THIS NODE)
pnn:1 172.28.1.17      OK
Generation:1501954827
Size:2
hash:0 lmaster:0
hash:1 lmaster:1
Recovery mode:NORMAL (0)
Recovery master:0

Install Samba

Install Samba packages on all CTDB nodes: 

sudo yum -y install samba samba-client samba-common samba-doc samba-winbind samba-winbind-clients

Configure and Start Samba

Edit the Samba configuration file

[root@ctdb-node ~] mv /etc/samba/smb.conf /etc/samba/smb.conf-backup
[root@ctdb-node ~] vim /etc/samba/smb.conf #create a new config file with the following info
[root@ctdb-node ~] cat /etc/samba/smb.conf
[global]
netbios name = usedForWindows
workgroup = bostonLustre
server string = [%h] Samba %v Server (Lustre)
security = user
encrypt passwords = yes
passdb backend = tdbsam
clustering = yes

[root@ctdb-node ~] testparm -s #test samba
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Loaded services file OK.
Server role: ROLE_STANDALONE
[global]
	workgroup = BOSTONLUSTRE
	netbios name = USEDFORWINDOWS
	server string = [%h] Samba %v Server (Lustre)
	clustering = Yes
	idmap config * : backend = tdb

Add a User in Samba DataBase

[root@ctdb-node ~] id ajax 
uid=500(ajax) gid=500(ajax) groups=500(ajax)
[root@ctdb-node ~] sudo smbpasswd -a ajax #add an EXISTING user in samba database
New SMB password:
Retype new SMB password:
Added user ajax

Check it is working

[root@ctdb-node ~] sudo service nmb start
[root@ctdb-node ~] sudo service smb start

[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -N
Anonymous login successful
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Sharename       Type      Comment
	---------       ----      -------
	lfs             Disk      
	IPC$            IPC       IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
Anonymous login successful
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Server               Comment
	---------            -------
	USEDFORWINDOWS       [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre

	Workgroup            Master
	---------            -------
	BOSTONLUSTRE         USEDFORWINDOWS

[root@ctdb-node ~] smbclient -L //USEDFORWINDOWS -U ajax #test the config for the user
Enter ajaxs password:
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Sharename       Type      Comment
	---------       ----      -------
	lfs             Disk      
	IPC$            IPC       IPC Service ([ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre))
Domain=[BOSTONLUSTRE] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

	Server               Comment
	---------            -------
	USEDFORWINDOWS       [ctdb-node] Samba 3.6.23-14.el6_6 Server (Lustre

	Workgroup            Master
	---------            -------
	BOSTONLUSTRE         USEDFORWINDOWS

Add Samba to CTDB

Preparation

sudo service ctdb stop
sudo service nmb stop
sudo service smb stop

Update the Samba Configuration

To turn on clustering support, edit the Samba configuration file (/etc/samba/smb.conf) created in the previous section and add the following to the [global] section: 

clustering = yes

Update the CTDB Configuration

Edit the CTDB configuration file (default: /etc/sysconfig/ctdb) and add the following entries: 

CTDB_MANAGES_SAMBA=yes
CTDB_SERVICE_NMB=nmb

Disable SMB and NMB Services from System Startup

sudo chkconfig nmb off
sudo chkconfig smb off
sudo chkconfig winbind off

Verify: 

sudo chkconfig --list nmb
sudo chkconfig --list smb
sudo chkconfig --list winbind

Start the Samba CTDB Cluster on One Node

1. Stop all CTDB services on the node: 

sudo service ctdb stop

2. Stop all Samba services on the node: 

sudo service nmb stop
sudo service smb stop
sudo service winbind stop

3. (Optional, but recommended.) Stop the CTDB services on all other cluster nodes: 

sudo onnode all service ctdb stop

4. Check that the Samba and CTDB configuration files have been distributed to all CTDB nodes.

5. Check that the Lustre file system is mounted and has global lock (flock) support enabled: 

mount -t lustre

6. Start the CTDB cluster service: 

sudo service ctdb start

7. Verify that the CTDB service is running: 

sudo ctdb status

8. Verify that the Samba NMB and SMB services are running: 

service nmb status
service smb status

9. Attempt a connection to the SMB service running in the cluster. Try the NetBios service name, as well as each individual IP addresses of each CTDB public address: 

smbclient -L //USEDFORWINDOWS -N

Samba Authentication with CTDB

In order to add a User in the Samba Database, you must first create the User in Unix and then add it like this: 

id <username>
sudo smbpasswd -a <username>
#or
sudo pdbedit -a -u <username>

To query the database for a list of Samba users, run the pdbedit command: 

sudo pdbedit -L

Once the account has been added to the CTDB password database and is listed in the output of pdbedit -L, one can attempt an authenticated connection to the Samba cluster: 

smbclient -L //USEDFORWINDOWS -U ajax

Create a Samba Share for Lustre

The Samba Share must be added in the /etc/samba/smb.conf: 

[root@ctdb-node ~] cat /etc/samba/smb.conf
[global]
netbios name = usedForWindows
workgroup = bostonLustre
server string = [%h] Samba %v Server (Lustre)
security = user
encrypt passwords = yes
passdb backend = tdbsam
clustering = yes

[lfs]
path = /mnt/lfs
valid users = root samba
available = yes
writable = yes
read only = no
browseable = yes
public = yes

Note: Be sure to add the users, you want to access the Samba Share, in the "valid users" value.

Multiple CTDB nodes

When using more than one CTDB node, make sure you copy the Samba configuration file on the same location on all nodes. Then start Samba, according to the above procedure, on all nodes.

After that you will be able to mount the share (lfs) from a Windows client, using any of the CTDB nodes IPs (see below).

Samba and Active Directory Integration Using Winbind

Update Samba and Configure Kerberos

1. Make the following changes to the [global] section of the Samba configuration file, smb.conf: 

security = ads
workgroup = <AD DOMAIN>
realm = <KRB REALM>

The Kerberos realm is the DNS domain of the Active Directory network and must be recorded in uppercase. The workgroup is the AD domain, also in upper–case, and is the first section of the Kerberos realm: the text up to but not including the first separator character, "." (period). For example: 

security = ads
workgroup = BOSTONLABS
realm = BOSTONLABS.LFS.INT

2. The DNS resolver must also be updated for each participating Samba server so that the Active Directory domain controller is the name server. If there is more than one domain controller, they can be added to the list. Edit etc/resolv.conf to make the changes, for example: 

domain bostonlabs.co.uk
search bostonlabs.co.uk
nameserver 10.0.0.1
nameserver 172.28.0.2

3. Install the bind-utils package;it contains tools to interrogate DNS records an assist with verification and debugging: 

sudo yum -y install bind-utils

4. However, make sure that the Kerberos tools are installed, to aid debugging. On RHEL and CentOS systems, install the krb5-workstation package: 

sudo yum -y install krb5-workstation

5. The Kerberos configuration on RHEL systems is recorded in /etc/krb5.conf. On installation, the krb5 configuration file contains mostly example information that is effectively ignored. The following basic configuration is recommended: 

[libdefaults]
default_realm = <KRB REALM>
dns_lookup_kdc = true
dns_lookup_realm = false

The default_realm variable should match the realm supplied in Samba's smb.conf. The variable dns_lookup_kdc tells the Kerberos library to use DNS SRV records to locate a Key Distribution Centre (KDC). This variable should be set to true; this is why DNS configuration is so important. Finally, the variable dns_lookup_realm tries to use DNS TXT records to determine the Kerberos realm of a host. Set dns_lookup_realm to false.

6. Run the following commands on all of the Samba servers in the CTDB cluster to ensure that DNS is working correctly: 

host -t SRV _ldap._tcp.<DOMAIN>
host -t SRV _kerberos._tcp.<DOMAIN>
host -t SRV _kerberos._udp.<DOMAIN>

For example: 

[root@ctdb-node2 ~] host -t SRV _ldap._tcp.bostonlabs.co.uk
_ldap._tcp.bostonlabs.co.uk has SRV record 0 100 389 srv2.bostonlabs.co.uk.
[root@ctdb-node2 ~] host -t SRV _kerberos._udp.bostonlabs.co.uk
_kerberos._udp.bostonlabs.co.uk has SRV record 0 100 88 srv2.bostonlabs.co.uk.
[root@ctdb-node2 ~] host -t SRV _kerberos._tcp.bostonlabs.co.uk
_kerberos._tcp.bostonlabs.co.uk has SRV record 0 100 88 srv2.bostonlabs.co.uk.

If these commands fail, there may be something wrong with the DNS setup, or the DNS servers don’t have the requisite information.

7. To verify the Kerberos connection, run the kinit command: 

sudo kinit Administrator@<REALM>

The realm must be supplied in uppercase. If dns_lookup_kdc is false, then the kinit command may return an error: 

kinit: Cannot find KDC for requested realm while getting initial credentials

8. On success, you can use the klist command to list the granted Kerberos ticket. The following example shows the complete transaction for acquiring what Kerberos calls a "ticket granting ticket" for the realm BOSTONLABS.CO.UK: 

[root@ctdb-node2 ~] kinit mouza@BOSTONLABS.CO.UK
Password for mouza@BOSTONLABS.CO.UK:
[root@ctdb-node2 ~] klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: mouza@BOSTONLABS.CO.UK

Valid starting     Expires            Service principal
06/29/15 10:42:01  06/29/15 20:39:44  krbtgt/BOSTONLABS.CO.UK@BOSTONLABS.CO.UK
	renew until 07/06/15 10:42:01

Join Samba to the Active Directory Domain

1. Start or restart the CTDB cluster with the updated Samba configuration. CTDB must be running so that the machine trust account credentials are saved into Samba's private configuration: 

service ctdb restart

2. Join the Samba server to the Active Directory domain and create the machine trust account on the Active Directory server: 

sudo net join -U Administrator

The machine trust account is a special user account used to authenticate computers that wish to participate in an Active Directory network. Substitute Administrator for any valid account with the required authorization to add machines to the directory. The following example demonstrates a successful join to the AD domain SMBDOM: 

$ sudo net join -U Administrator
Enter Administrator's password:
Using short domain name -- BOSTONLABS
Joined 'UsedForWindows' to dns domain 'bostonlabs.co.uk'
Not doing automatic DNS update in a clustered setup.

This step need only be carried out once on a single node in the CTDB Samba cluster, provided that each cluster member has the same Samba global configuration (i.e.: the same NetBIOS name, Active Directory domain and Kerberos realm).

3. The machine account status can be checked at any time as follows: 

[root@ctdb-node2 ~] net ads testjoin
Join is OK

Integrate Winbind

1. Edit smb.conf and add the following in the [global] section: 

idmap config *:backend = tdb2
idmap config *:range = 200000 - 300000

2. Make the Winbind service to be managed by CTDB by editing /etc/sysconfig/ctdb and setting the following variable: 

CTDB_MANAGES_WINBIND=yes

3. Make the following changes to the NSS configuration file, /etc/nsswitch.conf: 

passwd: files winbind
group: files winbind

With this configuration, user and group lookups will be executed against the local /etc/passwd and /etc/group files, followed by the Winbind service if no match is found.

4. Edit the PAM files:

  • Edit the files /etc/pam.d/system-auth-ac and /etc/pam.d/password-auth-ac (the two files have identical content on CentOS 6). There are effectively four sections: auth, account, password, and session.
  • In the auth section, add the following line directly after the entry for pam_unix.so and before the pam_deny.so line:
auth sufficient pam_winbind.so use_first_pass
  • In the account section, add the following line directly after the entry for am_succeed_if.so and before pam_permit.so:
account sufficient pam_winbind.so use_first_pass
  • In the password section, add the following line directly after the entry for pam_unix.so and before pam_deny.so:
password sufficient pam_winbind.so use_first_pass
  • In the session section, add the following directly after the entry for pam_unix.so (this should be the last line in the file):
session required pam_winbind.so use_first_pass

Remember to edit both files.

5. In order to verify Winbind: 

[root@ctdb-node2 ~]# wbinfo -p
Ping to winbindd succeeded

[root@ctdb-node2 ~]# wbinfo -t
checking the trust secret for domain BOSTONLABS0 via RPC calls succeeded

[root@ctdb-node2 ~]# wbinfo -u
USEDFORWINDOWS\root
BOSTONLABS0\administrator
BOSTONLABS0\guest
BOSTONLABS0\krbtgt
BOSTONLABS0\mh
BOSTONLABS0\dj
BOSTONLABS0\mhol
BOSTONLABS0\md
BOSTONLABS0\tm
BOSTONLABS0\mouza
BOSTONLABS0\jh
BOSTONLABS0\bs

[root@ctdb-node2 ~]# wbinfo -g
BOSTONLABS0\dhcp users
BOSTONLABS0\dhcp administrators
BOSTONLABS0\domain computers
BOSTONLABS0\domain controllers
BOSTONLABS0\schema admins
BOSTONLABS0\enterprise admins
BOSTONLABS0\cert publishers
BOSTONLABS0\domain admins
BOSTONLABS0\domain users
BOSTONLABS0\domain guests
BOSTONLABS0\group policy creator owners
BOSTONLABS0\ras and ias servers
BOSTONLABS0\allowed rodc password replication group
BOSTONLABS0\denied rodc password replication group
BOSTONLABS0\read-only domain controllers
BOSTONLABS0\enterprise read-only domain controllers
BOSTONLABS0\dnsadmins
BOSTONLABS0\dnsupdateproxy
BOSTONLABS0\bostonlabs-administrators

Configuration on Windows Client

1. Create a new User with the same username and password as one of the existing users in the Samba database. (In our example ajax)

2. Open a File Explorer window

3. Right click on "This PC" on the left side panel and click on "Map Network Drive":

Error creating thumbnail: File missing

4. In the "Folder" field, add the public ip of a CTDB node and then the name of the Samba Share. If you are not logged in as the created User, tick the box for "Connect with different credentials":

Error creating thumbnail: File missing

5. Fill in the credentials of the user you created:

Error creating thumbnail: File missing

6. Lustre File System is now mounted:

Error creating thumbnail: File missing

Troubleshooting

Check the /var/log/log.ctdb file for details.

Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Lustre_intel:_Setting_up_a_Windows_client&oldid=8441"