Lustre intel: Setting up a Windows client

The following instructions are for one (or multiple) linux system(s), which will serve as the middle-layer cluster, which "translates" Lustre to Windows, using Samba.

There are two components that need to be installed:

1. CTDB
2. Samba

Pre-Requisites

• Lustre Client File System mounted with full cluster-coherent locking support on all CTDB nodes:

mkdir -m 0755 -p /mnt/lfs
mount -t lustre -oflock 172.28.50.2@tcp0:172.28.50.3@tcp0:/lfs1 /mnt/lfs/

• Networking

  Each CTDB node will require three network interfaces on three separate networks inorder to operate correctly:

  1. a high performance data network interface, used to mount the Lustre client file system. This is typically an InfiniBand or 10Gb Ethernet fabric
  2. a private network interface over which all CTDB nodes communicate, sending heartbeats, exchanging data and cluster commands
  3. a public network interface over which clients will connect to the services running on the CTDB cluster, such as Samba

  In our case, networks 1 and 2 are the same (pxe). Here is an example of a CTDB Framework on Lustre:

Error creating thumbnail: File missing

Install CTDB

Install CTDB as follows:

sudo yum -y install ctdb

Configure CTDB

On RHEL 6, the global configuration file is /etc/sysconfig/ctdb. At a minimum, CTDB needs the following information:

• The list of IP addresses of every member of the CTDB cluster on the private network. This is stored in a plain text file referred to as the node list, the location of which is governed by the variable CTDB_NODES in the global configuration.

  In our one node example:

[root@ctdb-node ~] cat /etc/ctdb/nodes 
172.28.0.131

• The list of public IP addresses, along with the subnet mask and the network interface, that will be used for hosting public services. This information is kept in a plain text file referenced by the global configuration variable CTDB_PUBLIC_ADDRESSES. The format of the IPs should be <IP Address>/<NetMask> <Network Device>.

  In our one node example:

[root@ctdb-node ~] cat /etc/ctdb/public_addresses 
10.0.0.129/8 eth0

• The network interface for the public network should not be configured by the operating system with any of the CTDB public IP addresses. It is possible for the interface to be configured and active with an address that lies outside the range used by CTDB, but it is far simpler to prevent the operating system from attempting to manage the network interface device in the first place. In the configuration file /etc/sysconfig/network-scripts/ifcfg-<device>, set the following variables as shown:

ONBOOT=no
NM_CONTROLLED=no
BOOTPROTO=none

• The location of the recovery lock file on the shared file system (in this case, the recovery lock is kept on Lustre). The recovery lock location is kept in the global configuration variable CTDB_RECOVERY_LOCK.

  For example:

# Lustre is mounted at /lustre/scratch.
# Create a directory to contain the CTDB recovery lock:
[root@c64-1 ~] mkdir -p -m 0700 /mnt/lfs/.ctdb
[root@c64-1 ~] ls -ld /mnt/lfs/.ctdb
drwx------ 2 root root 4096 Oct 10 17:56 /mnt/lfs/.ctdb

CTDB Configuration example

CTDB_RECOVERY_LOCK="/mnt/lfs/.ctdb/recovery_lock"
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_NODES=/etc/ctdb/nodes
CTDB_LOGFILE=/var/log/log.ctdb
CTDB_DEBUGLEVEL=ERR
CTDB_SYSLOG=no