Lustre: Installing and setting up a RobinHood server (RBH)

Enabling the HSM Coordinator for an Existing File System

The HSM Coordinator can be enabled at any time after a file system has been created, provided that the file system is based on Lustre 2.5.0 or newer, the Metadata service is online, and the MDT and MGT are mounted.

1. Log into Intel® Manager for Lustre dashboard as a superuser.
2. Navigate to the File Systems window: Click Configuration > File Systems.
3. Select the file system for which HSM support is to be enabled from the list of file systems.
4. Under Metadata Target, click the name of the metadata target.
5. Click the Advanced tab and look for the property hsm_control.
6. Enter the string enabled into the text field.
7. Click the Apply button. The background will briefly flash to grey then back to white to indicate that the setting has been applied. Click Close when done.

Verifying the Current Status of the Coordinator

The most reliable way to verify the status of the HSM Coordinator process is to log into the metadata server that currently has the MDT mounted. Then enter the following command:

 lctl get_param mdt.*.hsm_control

For example:

  [root@mds2 ~]#  lctl get_param mdt.*.hsm_control
  mdt.lfs-MDT0000.hsm_control=stopped
  # Go perform the step above
  [root@mds2 ~]#  lctl get_param mdt.*.hsm_control
  mdt.lfs-MDT0000.hsm_control=enabled

Configure MDS Changelogs

Robinhood relies on the MDT Changelogs feature in Lustre in order to track changes to the file system. The Changelog records changes to the file system's metadata, such as creating and deleting files, modifying a file's content, ownership, permissions, and other attributes. Robinhood captures this data and stores it in a MySQL database for processing by its policy engine. Applications that want to consume the Changelogs content need to register with the MDT, which will return a unique identifier, referred to as a userid. This is not the same as a UNIX user account identifier; it is internal to Lustre and is a token used to distinguish registrations from multiple consumers. Each registered userid has its own view of the Changelog register. Changelog entries are kept until each registered user has acknowledged that the entry has been consumed.

# on server with MGT mounted / also assumes only one FSname (used a * for fsname below)
[root@mds2 ~]# lctl get_param  mdd.*-MDT*.changelog_mask 
mdd.lfs-MDT0000.changelog_mask=    
MARK CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RENME RNMTO OPEN LYOUT TRUNC SATTR XATTR HSM MTIME CTIME 
[root@mds2 ~]# lctl set_param  -Pmdd.*-MDT*.changelog_mask="all-XATTR-MARK-ATIME"
mdd.lfs-MDT0000.changelog_mask=all-XATTR-MARK-ATIME 
[root@mds2 ~]# lctl get_param  mdd.*-MDT*.changelog_mask
mdd.lfs-MDT0000.changelog_mask= 
CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RENME RNMTO OPEN CLOSE LYOUT TRUNC SATTR HSM MTIME CTIME 
# so the word MARK seems to be removed and XATTR 
# Note; If you run without the -P, the changes wont be persistent (and without the -P you can run on the MDS)

Register a Changelog user identifier on the MDS and keep a record of the name of the userid that is returned (normally this will be "cl1", unless there is more than one Changelog registration):

# lctl --device <fsname>-<MDT index> changelog_register
[root@mds2 ~]# lctl --device lfs-MDT0000 changelog_register 
lfs-MDT0000: Registered changelog userid 'cl1'
# Note; This is not a UNIX user account; it is an identifier used to track Lustre changelog events.

Setup the RobinHood Server (RBH)

Provision the server through the IML Interface; - Configuration -> Servers -> Add Servers -> Enter details -> Add -> Select the profile to Deploy (Robinhood policy engine)

Quick note; I got an error about a file initramfs-.img missing so created a symlink to get around this. Its as if the installer isnt picking the kernel version installed.

Verify the RBH has been setup correctly

[root@x10-ft-02 boot]# rpm -ql robinhood-adm
/usr/sbin/rbh-config

Setup the Robinhood Database

# Basic MySQL setup 
/etc/init.d/mysqld start
/usr/bin/mysql_secure_installation
chkconfig mysqld on

# Once the above is sorted, check that the sql DB is 
[root@x10-ft-02 boot]# rbh-config precheck_db
Checking system configuration...
mysqladmin command OK.
mysql_config command OK.
MySQL version is 5.
mysqld service OK.
mysql command OK.

Create the RBH Database. Without any additional arguments, rbh-config create_db will enter an interactive mode and will ask for the following information:

1. A unique identifier for the database instance (use the file system name).
2. A list of hosts that can access the database (this uses SQL formatting, including wildcards, and forms part of the SQL GRANT command; SQL syntax rules apply). For systems where Robinhood and MySQL are co-located, set to localhost. Set to % to allow any host access to the database.
3. A password for the database user. Robinhood will connect to the database as user robinhood with the password you enter here.
4. The root user password for the MySQL database (required in order to be able to create the new database and grant privileges).

# create the rbh dbs
[root@x10-ft-02 boot]# rbh-config create_db
Checking system configuration...
mysqladmin command OK.
mysql_config command OK.
MySQL version is 5.
mysqld service OK.
mysql command OK.

Enter a custom identifier for your filesystem. E.g. lustre
fsname (max 8 chars): lfs

Enter hosts where robinhood commands will run. E.g. localhost
You can use '%' as wildcard: "%" for all hosts, "cluster%" for nodes starting with 'cluster'...
hosts: localhost

Choose a password for connecting to the database (user 'robinhood'). 
password: 
confirm password: 
Write this password to /etc/robinhood.d/.dbpassword file

Configuration summary:
- Database name: 'robinhood_lfs'
- Client hosts: 'localhost'
- Database user name: 'robinhood'

Do you agree? [y/N]y

Enter password for root's database account (leave blank if none is set):
root's DB password: 

Creating database 'robinhood_lfs'...
done

Setting access right for user 'robinhood'@'localhost'...
(notice: user robinhood must have SUPER privilege to create triggers)
Grants for robinhood@localhost
GRANT SUPER ON *.* TO 'robinhood'@'localhost' IDENTIFIED BY PASSWORD '*A3D1A5853B73542DD7CFBF300F4873BAD075D819'
GRANT ALL PRIVILEGES ON `robinhood_lfs`.* TO 'robinhood'@'localhost'

Testing connection to 'robinhood_lfs'...

Database sucessfully created!

Verify the database is setup correctly

[root@x10-ft-02 boot]# mysql -p -h localhost -u robinhood
Enter password: 
mysql> show grants; 
mysql> use robinhood_lfs;
mysql> \q

# worth keeping the pass in order;
[root@x10-ft-02 boot]# vi /etc/robinhood.d/.dbpassword
[root@x10-ft-02 boot]# cat /etc/robinhood.d/.dbpassword 
robinhood