BeeGFS Installation and Configuration (Hyperconverged compute/storage)
Note; Further notes and documentation are available on the BeeGFS wiki
http://www.beegfs.com/wiki/UserGuide
Hardware Setup
A couple of notes on setting up the hardware. Apply the steps below to all servers/partitions.
Setup Optimal Partitioning
[root@haswell07 ~]# parted -a optimal /dev/sdc
GNU Parted 2.1
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdc will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart primary 0% 10%
(parted) mkpart primary 10% 100%
(parted) print
Model: ATA SAMSUNG MZ7PC256 (scsi)
Disk /dev/sdc: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 1049kB 25.6GB 25.6GB primary
2 25.6GB 256GB 230GB primary
(parted) quit
Information: You may need to update /etc/fstab.Setup deadline scheduler
echo deadline > /sys/block/sdX/queue/schedulerSet Scaling Governor to Performance
pdsh -w haswell[05-08] 'echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null'Create the Filesystems
FS for the MetaData Server
pdsh -w haswell[05-08] mkfs.ext4 -i 2048 -I 512 -J size=400 -Odir_index,filetype /dev/sdc1
pdsh -w haswell[05-08] tune2fs -o user_xattr /dev/sdc1
pdsh -w haswell[05-08] mount -onoatime,nodiratime,nobarrier /dev/sdc1 /data/beegfs/metaCreate filesystems for Storage Servers
pdsh -w haswell[05-08] mkfs.xfs -f /dev/sdc2
pdsh -w haswell[05-08] mkfs.xfs -f /dev/sda6
[root@head-Boston ~]# pdsh -w haswell[05-08] mount -onoatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sda6 /data/beegfs/storage1/
[root@head-Boston ~]# pdsh -w haswell[05-08] mount -onoatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sdc2 /data/beegfs/storage2/Install the BeeGFS Software
Setup the BeeGFS repo
[root@haswell05 yum.repos.d]# wget http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel6.repo
OR
wget -O /etc/yum.repos.d/beegfs-rhel6.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel6.repoEnsure we have OFED setup for RDMA
[root@haswell05 yum.repos.d]# rpm --import http://www.beegfs.com/release/latest-stable/gpg/RPM-GPG-KEY-beegfs
[root@haswell05 ~]# yum groupinstall 'Development tools'
[root@haswell05 ~]# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-3.1-1.0.3/MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.tgz
[root@haswell05 ~]# tar zxvf MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.tgz
[root@haswell05 ~]# cd MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64
[root@haswell05 MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64]# ./mlnxofedinstallBuild the Client Kernel
On all clients
# BeeGFS Client IB;
# in file: [root@haswell05 beegfs]# vi /etc/beegfs/beegfs-client-autobuild.conf (NOTE THE PATH FOR OFED INCLUDE; BASED ON MLXS OFED)
# Change the buildArgs line
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include/
(THE OFA-2.4 DIDNT WORK!)Install the RPMS
On all servers
yum install beegfs-storage beegfs-admon beegfs-client beegfs-helperd beegfs-meta beegfs-mgmtd beegfs-utilsSetup the services / Create conf files
Setup the Management Service
The management service needs to know where it can store its data. It will only store some node information like connectivity data, so it will not require much storage space and its data access is not performance critical. Thus, this service is typically not running on a dedicated machine.
[root@haswell05 data]# /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd
Preparing storage directory: /data/beegfs/beegfs_mgmtd
* Creating format.conf file...
Updating config file: /etc/beegfs/beegfs-mgmtd.conf
* Setting storage directory in config file...
* Disabling usage of uninitialized storage directory in config file...
All done.
OR
/opt/beegfs/sbin/beegfs-setup-mgmtd -p /local/beegfs/beegfs_mgmtdSetup the Meta Data Service
The metadata service needs to know where it can store its data and where the management service is running. Typically, you will have multiple metadata services running on different machines.
Optionally, you can also define a custom numeric metadata service ID (range 1..65535). As this service is running on a server with name "node02" in our example, we will also pick number "2" as metadata service ID here.
[root@haswell05 data]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m haswell05
Preparing storage directory: /data/beegfs/beegfs_meta
* Creating format.conf file...
* Creating server numeric ID file: /data/beegfs/beegfs_meta/nodeNumID
Updating config file: /etc/beegfs/beegfs-meta.conf
* Setting management host: haswell05
* Setting storage directory in config file...
* Disabling usage of uninitialized storage directory in config file...
* Setting usage of extended attributes to: true
All done.
OR
/opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m haswell05.ib.cluster
ON ALL NODES
[root@haswell05 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 2 -m haswell05.ib.cluster -f
[root@haswell06 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 3 -m haswell05.ib.cluster -f
[root@haswell07 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 4 -m haswell05.ib.cluster -f
[root@haswell08 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 5 -m haswell05.ib.cluster -f
(START ALL THE MDS SERVICES)Setup the Storage Service
The storage service needs to know where it can store its data and how to reach the management server. Typically, you will have multiple storage services running on different machines and/or multiple storage targets (e.g. multiple RAID volumes) per storage service.
Optionally, you can also define a custom numeric storage service ID and numeric storage target ID (both in range 1..65535). As this service is running on a server with name "node03" in our example, we will pick number "3" as ID for this storage service and we will use "301" as storage target ID to show that this is the first target ("01") of storage service "3".
[root@haswell05 data]# /opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 3 -i 301 -m haswell05
Preparing storage target directory: /data/beegfs/beegfs_storage
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /data/beegfs/beegfs_storage/targetNumID
* Creating server numeric ID file: /data/beegfs/beegfs_storage/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: haswell05
* Appending to target directory list in config file...
* Disabling usage of uninitialized storage targets in config file...
All done.
# Example commands
# on haswell05;
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 3 -i 301 -m haswell05.ib.cluster
# on haswell06;
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 4 -i 401 -m haswell05.ib.cluster
# on haswell07;
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 5 -i 501 -m haswell05.ib.cluster
# Example command with 2 storage blocks on a storage server.
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/storage1 -s 17 -i 1701 -m haswell05.ib.cluster
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/storage2 -s 17 -i 1702Setting up the Client
The client mount directory is defined in a separate configuration file. This file will be used by the /etc/init.d/beegfs-client script. By default, BeeGFS will be mounted to /mnt/beegfs. Thus, you need to perform this step only if you want to mount the file system to a different location.
$ ssh root@node04
$ vim /etc/beegfs/beegfs-mounts.conf
# The first entry defines the mount directory. The second entry refers to the corresponding config file for this mountpoint.Client Software Setup
yum install beegfs-helperd beegfs-client
/opt/beegfs/sbin/beegfs-setup-client -m haswell05
[root@haswell07 yum.repos.d]# /opt/beegfs/sbin/beegfs-setup-client -m haswell05
Updating config file: /etc/beegfs/beegfs-client.conf
* Setting management host: haswell05
All done.Admon Service
The optional admon service needs to know how to reach the management server. (Use your favorite text file editor, we will use vim in this example.)
$ ssh root@node05
$ vim /etc/beegfs/beegfs-admon.conf
# Find the option "sysMgmtdHost" and set it to "sysMgmtdHost=node01".Start BeeGFS Services
[root@haswell05 data]# /etc/init.d/beegfs-mgmtd start
Starting BeeGFS Management daemon: [ OK ]
[root@haswell05 data]# /etc/init.d/beegfs-meta start
Starting BeeGFS Meta Server: [ OK ]
[root@haswell05 data]# /etc/init.d/beegfs-storage start
Starting BeeGFS Storage Server: [ OK ]
[root@haswell05 data]# /etc/init.d/beegfs-helperd start
Starting BeeGFS Client Helper Daemon: [ OK ]
[root@haswell05 data]# /etc/init.d/beegfs-admon start
Starting BeeGFS Admon: [ OK ]
# Then mount the clients