Lustre: Installation of lustre 1.8.6-wc1 whamcloud

• Downloaded prebuilt RPMs from http://downloads.whamcloud.com/public/lustre/lustre-1.8.6-wc1/el5/

Install pre-built RPMs

• First install a lustre patched kernel

[root@comp00 ~]$ rpm -ivh kernel-2.6.18-238.12.1.el5_lustre.<buildid>.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:kernel                 ########################################### [100%]
Modulefile is /etc/modprobe.conf
[root@comp00 ~]$ vi /boot/grub/menu.lst 
# verify kernel will be booted up

• Reboot
• Verify kernel has booted correctly

[root@comp00 ~]$ uname -a
Linux comp00 2.6.18-238.12.1.el5_lustre.g266a955 1 SMP Fri Jun 10 16:39:27 PDT 2011 x86_64 x86_64 x86_64 GNU/Linux

• Install the rest of the lustre RPMs: lustre, ldiskfs, lustre-modules, lustre-tests

[root@comp00 x86_64]$ rpm -ivh lustre-1.8.6-wc1* lustre-ldiskfs-3.1.50-wc1* lustre-tests-1.8.6-wc1* lustre-modules-1.8.6-wc1*
Preparing...                ########################################### [100%]
   1:lustre-modules         ########################################### [ 25%]
Modulefile is /etc/modprobe.conf
   2:lustre                 ########################################### [ 50%]
   3:lustre-ldiskfs         ########################################### [ 75%]
   4:lustre-tests           ########################################### [100%]

• Install e2fsprogs which is needed for the lustre tests, from http://downloads.whamcloud.com/public/e2fsprogs/
• Note, did an update as it was already installed as part of platform hpc

[root@comp00 x86_64]$ rpm -Uvh e2fsprogs-1.41.90.wc2-0redhat.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:e2fsprogs              ########################################### [100%]

• Repeat procedure on all lustre server nodes (mds / oss)

Configuring the MDS

• In this example we're using a single system for both the MDS (MetaData Server) and MDT (MetaData Target)
• Requires a blanks partition (in this example /dev/sda6)
• Create the lustre MDT filesystem:

[root@comp00 x86_64]$ mkfs.lustre --fsname=lustrewt --mgs --mdt /dev/sda6

   Permanent disk data:
Target:     lustrewt-MDTffff
Index:      unassigned
Lustre FS:  lustrewt
Mount type: ldiskfs
Flags:      0x75
              (MDT MGS needs_index first_time update )
Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups

checking for existing Lustre data: not found
device size = 223317MB
WARNING: The e2fsprogs package currently installed on your system does not support "uninit_bg" feature.
Please install the latest version of e2fsprogs from
http://downloads.whamcloud.com/public/e2fsprogs/latest/
to enable this feature.
Feature will not be enabled until e2fsprogsis updated and 'tune2fs -O uninit_bg %{device}' is run.

2 6 18
formatting backing filesystem ldiskfs on /dev/sda6
	target name  lustrewt-MDTffff
	4k blocks     57169302
	options        -J size=400 -i 4096 -I 512 -q -O dir_index -F
mkfs_cmd = mke2fs -j -b 4096 -L lustrewt-MDTffff  -J size=400 -i 4096 -I 512 -q -O dir_index -F /dev/sda6 57169302
Writing CONFIGS/mountdata

• Mount the MDT

mkdir /mnt/mdt
mount -t lustre /dev/sda6 /mnt/mdt/

• Setup the lnet module

[root@comp00 x86_64]$ modprobe lnet
[root@comp00 x86_64]$ lctl network up 
LNET configured
[root@comp00 x86_64]$ lctl list_nids
172.20.0.4@tcp
[root@comp00 x86_64]$ lsmod | grep lnet 
lnet                  274856  4 lustre,ksocklnd,ptlrpc,obdclass
libcfs                200708  14 mds,fsfilt_ldiskfs,mgs,mgc,lustre,lov,mdc,lquota,osc,ksocklnd,ptlrpc,obdclass,lnet,lvfs

Configure the OSS/OST

• Again, in this example we will be using a single server for the OST and OSS
• Create lustre filesystem and mount, mgs argument is from the output of lctl

[root@comp01 ~]$ mkfs.lustre --ost --fsname=lustrewt --mgsnode=172.20.0.4@tcp /dev/sda6

   Permanent disk data:
Target:     lustrewt-OSTffff
Index:      unassigned
Lustre FS:  lustrewt
Mount type: ldiskfs
Flags:      0x72
              (OST needs_index first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=172.20.0.4@tcp

checking for existing Lustre data: not found
device size = 223317MB
2 6 18
formatting backing filesystem ldiskfs on /dev/sda6
	target name  lustrewt-OSTffff
	4k blocks     57169302
	options        -J size=400 -i 16384 -I 256 -q -O dir_index,extents,uninit_groups -F
mkfs_cmd = mke2fs -j -b 4096 -L lustrewt-OSTffff  -J size=400 -i 16384 -I 256 -q -O dir_index,extents,uninit_groups -F /dev/sda6 57169302
Writing CONFIGS/mountdata
[root@comp01 ~]$ mkdir /mnt/ostoss_mnt
[root@comp01 ~]$ mount -t lustre /dev/sda6 /mnt/ostoss_mnt/

To extend the lustre filesystem, issue the same commands on a new OSS system. Client should automatically become aware of the additional storage capacity available.

Configure the Client

• Installed the same RPMs as listed at the start.
• Load the module and mount the lustre fs.

[root@comp07 ~]$ mkdir /mnt/lustre
[root@comp07 ~]$ mount -t lustre 172.20.0.4@tcp:/lustrewt /mnt/lustre/