BeeGFS Installation and Configuration (Hyperconverged compute/storage)

From Define Wiki
Revision as of 16:28, 22 December 2015 by David (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Note; Further notes and documentation are available on the BeeGFS wiki

http://www.beegfs.com/wiki/UserGuide


Hardware Setup

A couple of notes on setting up the hardware. Apply the steps below to all servers/partitions.

Setup Optimal Partitioning

[root@haswell07 ~]# parted -a optimal /dev/sdc 
GNU Parted 2.1 
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos    
Warning: The existing disk label on /dev/sdc will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes    
(parted) mkpart primary 0% 10% 
(parted) mkpart primary 10% 100%    
(parted) print    
Model: ATA SAMSUNG MZ7PC256 (scsi)
Disk /dev/sdc: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  25.6GB  25.6GB  primary
 2      25.6GB  256GB   230GB   primary

(parted) quit    
Information: You may need to update /etc/fstab.

Setup deadline scheduler

echo deadline > /sys/block/sdX/queue/scheduler

Set Scaling Governor to Performance

pdsh -w haswell[05-08] 'echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null'

Create the Filesystems

FS for the MetaData Server

pdsh -w haswell[05-08] mkfs.ext4 -i 2048 -I 512 -J size=400 -Odir_index,filetype /dev/sdc1 
pdsh -w haswell[05-08] tune2fs -o user_xattr /dev/sdc1
pdsh -w haswell[05-08] mount -onoatime,nodiratime,nobarrier /dev/sdc1 /data/beegfs/meta

Create filesystems for Storage Servers

pdsh -w haswell[05-08] mkfs.xfs -f /dev/sdc2
pdsh -w haswell[05-08] mkfs.xfs -f /dev/sda6
[root@head-Boston ~]# pdsh -w haswell[05-08] mount -onoatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sda6 /data/beegfs/storage1/
[root@head-Boston ~]# pdsh -w haswell[05-08] mount -onoatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sdc2 /data/beegfs/storage2/

Install the BeeGFS Software

Setup the BeeGFS repo

[root@haswell05 yum.repos.d]# wget http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel6.repo

OR

wget -O /etc/yum.repos.d/beegfs-rhel6.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel6.repo

Ensure we have OFED setup for RDMA

[root@haswell05 yum.repos.d]# rpm --import http://www.beegfs.com/release/latest-stable/gpg/RPM-GPG-KEY-beegfs
[root@haswell05 ~]# yum groupinstall 'Development tools'
[root@haswell05 ~]# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-3.1-1.0.3/MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.tgz
[root@haswell05 ~]# tar zxvf MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.tgz 
[root@haswell05 ~]# cd MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64
[root@haswell05 MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64]# ./mlnxofedinstall

Build the Client Kernel

On all clients 

# BeeGFS Client IB; 
# in file: [root@haswell05 beegfs]# vi /etc/beegfs/beegfs-client-autobuild.conf (NOTE THE PATH FOR OFED INCLUDE; BASED ON MLXS OFED) 
# Change the buildArgs line
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include/

(THE OFA-2.4 DIDNT WORK!)

Install the RPMS

On all servers 

yum install beegfs-storage beegfs-admon beegfs-client beegfs-helperd beegfs-meta beegfs-mgmtd beegfs-utils

Setup the services / Create conf files

Setup the Management Service

The management service needs to know where it can store its data. It will only store some node information like connectivity data, so it will not require much storage space and its data access is not performance critical. Thus, this service is typically not running on a dedicated machine. 

[root@haswell05 data]# /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd
Preparing storage directory: /data/beegfs/beegfs_mgmtd
 * Creating format.conf file...
Updating config file: /etc/beegfs/beegfs-mgmtd.conf
 * Setting storage directory in config file...
 * Disabling usage of uninitialized storage directory in config file...
All done.

OR 

/opt/beegfs/sbin/beegfs-setup-mgmtd -p /local/beegfs/beegfs_mgmtd

Setup the Meta Data Service

The metadata service needs to know where it can store its data and where the management service is running. Typically, you will have multiple metadata services running on different machines.

Optionally, you can also define a custom numeric metadata service ID (range 1..65535). As this service is running on a server with name "node02" in our example, we will also pick number "2" as metadata service ID here. 

[root@haswell05 data]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m haswell05
Preparing storage directory: /data/beegfs/beegfs_meta
 * Creating format.conf file...
 * Creating server numeric ID file: /data/beegfs/beegfs_meta/nodeNumID
Updating config file: /etc/beegfs/beegfs-meta.conf
 * Setting management host: haswell05
 * Setting storage directory in config file...
 * Disabling usage of uninitialized storage directory in config file...
 * Setting usage of extended attributes to: true
All done.

OR 

/opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m haswell05.ib.cluster

ON ALL NODES
[root@haswell05 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 2 -m haswell05.ib.cluster -f
[root@haswell06 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 3 -m haswell05.ib.cluster -f
[root@haswell07 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 4 -m haswell05.ib.cluster -f
[root@haswell08 beegfs]# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 5 -m haswell05.ib.cluster -f

(START ALL THE MDS SERVICES)

Setup the Storage Service

The storage service needs to know where it can store its data and how to reach the management server. Typically, you will have multiple storage services running on different machines and/or multiple storage targets (e.g. multiple RAID volumes) per storage service.

Optionally, you can also define a custom numeric storage service ID and numeric storage target ID (both in range 1..65535). As this service is running on a server with name "node03" in our example, we will pick number "3" as ID for this storage service and we will use "301" as storage target ID to show that this is the first target ("01") of storage service "3". 

[root@haswell05 data]# /opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 3 -i 301 -m haswell05
Preparing storage target directory: /data/beegfs/beegfs_storage
 * Creating format.conf file...
 * Creating chunks directory...
 * Creating buddymir directory...
 * Creating target numeric ID file: /data/beegfs/beegfs_storage/targetNumID
 * Creating server numeric ID file: /data/beegfs/beegfs_storage/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
 * Setting management host: haswell05
 * Appending to target directory list in config file...
 * Disabling usage of uninitialized storage targets in config file...
All done.

# Example commands 
# on haswell05;
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 3 -i 301 -m haswell05.ib.cluster
# on haswell06;
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 4 -i 401 -m haswell05.ib.cluster
# on haswell07;
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/beegfs_storage -s 5 -i 501 -m haswell05.ib.cluster

# Example command with 2 storage blocks on a storage server. 
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/storage1 -s 17 -i 1701 -m haswell05.ib.cluster
/opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/storage2 -s 17 -i 1702

Setting up the Client

The client mount directory is defined in a separate configuration file. This file will be used by the /etc/init.d/beegfs-client script. By default, BeeGFS will be mounted to /mnt/beegfs. Thus, you need to perform this step only if you want to mount the file system to a different location. 

$ ssh root@node04
$ vim /etc/beegfs/beegfs-mounts.conf

# The first entry defines the mount directory. The second entry refers to the corresponding config file for this mountpoint.

Client Software Setup

yum install beegfs-helperd beegfs-client
/opt/beegfs/sbin/beegfs-setup-client -m haswell05

[root@haswell07 yum.repos.d]# /opt/beegfs/sbin/beegfs-setup-client -m haswell05
Updating config file: /etc/beegfs/beegfs-client.conf
 * Setting management host: haswell05
All done.

Admon Service

The optional admon service needs to know how to reach the management server. (Use your favorite text file editor, we will use vim in this example.) 

$ ssh root@node05
$ vim /etc/beegfs/beegfs-admon.conf

# Find the option "sysMgmtdHost" and set it to "sysMgmtdHost=node01".

Start BeeGFS Services

[root@haswell05 data]# /etc/init.d/beegfs-mgmtd start
Starting BeeGFS Management daemon:                         [  OK  ]
[root@haswell05 data]# /etc/init.d/beegfs-meta start
Starting BeeGFS Meta Server:                               [  OK  ]
[root@haswell05 data]# /etc/init.d/beegfs-storage start
Starting BeeGFS Storage Server:                            [  OK  ]
[root@haswell05 data]# /etc/init.d/beegfs-helperd start
Starting BeeGFS Client Helper Daemon:                      [  OK  ]
[root@haswell05 data]# /etc/init.d/beegfs-admon start
Starting BeeGFS Admon:                                     [  OK  ]

# Then mount the clients
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=BeeGFS_Installation_and_Configuration_(Hyperconverged_compute/storage)&oldid=11591"