BeeGFS Installation and Configuration (Hyperconverged compute/storage)

Note; Further notes and documentation are available on the BeeGFS wiki

http://www.beegfs.com/wiki/UserGuide

Hardware Setup

A couple of notes on setting up the hardware. Apply the steps below to all servers/partitions.

Setup Optimal Partitioning

[root@haswell07 ~]# parted -a optimal /dev/sdc 
GNU Parted 2.1 
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos    
Warning: The existing disk label on /dev/sdc will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes    
(parted) mkpart primary 0% 10% 
(parted) mkpart primary 10% 100%    
(parted) print    
Model: ATA SAMSUNG MZ7PC256 (scsi)
Disk /dev/sdc: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  25.6GB  25.6GB  primary
 2      25.6GB  256GB   230GB   primary

(parted) quit    
Information: You may need to update /etc/fstab.

Setup deadline scheduler

echo deadline > /sys/block/sdX/queue/scheduler

Set Scaling Governor to Performance

pdsh -w haswell[05-08] 'echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null'

Create the Filesystems

FS for the MetaData Server

pdsh -w haswell[05-08] mkfs.ext4 -i 2048 -I 512 -J size=400 -Odir_index,filetype /dev/sdc1 
pdsh -w haswell[05-08] tune2fs -o user_xattr /dev/sdc1
pdsh -w haswell[05-08] mount -onoatime,nodiratime,nobarrier /dev/sdc1 /data/beegfs/meta

Create filesystems for Storage Servers

pdsh -w haswell[05-08] mkfs.xfs -f /dev/sdc2
pdsh -w haswell[05-08] mkfs.xfs -f /dev/sda6
[root@head-Boston ~]# pdsh -w haswell[05-08] mount -onoatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sda6 /data/beegfs/storage1/
[root@head-Boston ~]# pdsh -w haswell[05-08] mount -onoatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sdc2 /data/beegfs/storage2/

Install the BeeGFS Software

Setup the BeeGFS repo

[root@haswell05 yum.repos.d]# wget http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel6.repo

OR

wget -O /etc/yum.repos.d/beegfs-rhel6.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel6.repo

Ensure we have OFED setup for RDMA

[root@haswell05 yum.repos.d]# rpm --import http://www.beegfs.com/release/latest-stable/gpg/RPM-GPG-KEY-beegfs
[root@haswell05 ~]# yum groupinstall 'Development tools'
[root@haswell05 ~]# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-3.1-1.0.3/MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.tgz
[root@haswell05 ~]# tar zxvf MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.tgz 
[root@haswell05 ~]# cd MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64
[root@haswell05 MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64]# ./mlnxofedinstall

Build the Client Kernel

On all clients

# BeeGFS Client IB; 
# in file: [root@haswell05 beegfs]# vi /etc/beegfs/beegfs-client-autobuild.conf (NOTE THE PATH FOR OFED INCLUDE; BASED ON MLXS OFED) 
# Change the buildArgs line
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include/

(THE OFA-2.4 DIDNT WORK!)

Install the RPMS

On all servers

yum install beegfs-storage beegfs-admon beegfs-client beegfs-helperd beegfs-meta beegfs-mgmtd beegfs-utils

Setup the services / Create conf files

Setup the Management Service

The management service needs to know where it can store its data. It will only store some node information like connectivity data, so it will not require much storage space and its data access is not performance critical. Thus, this service is typically not running on a dedicated machine.

[root@haswell05 data]# /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd
Preparing storage directory: /data/beegfs/beegfs_mgmtd
 * Creating format.conf file...
Updating config file: /etc/beegfs/beegfs-mgmtd.conf
 * Setting storage directory in config file...
 * Disabling usage of uninitialized storage directory in config file...
All done.

OR 

/opt/beegfs/sbin/beegfs-setup-mgmtd -p /local/beegfs/beegfs_mgmtd