FhGFS Installation and Configuration

• Steps taken from: http://www.fhgfs.com/wiki/wikka.php?wakka=DownloadInstallationPackages
• Installation on Centos 6 servers with IB

System Architecture

http://www.fhgfs.com/wiki/images/sysarch2.png

Prepare Hosts

• Add the FhGFS repo to your environment

  cd /etc/yum.repos.d/
  wget http://www.fhgfs.com/release/latest-stable/dists/fhgfs-rhel6.repo

• Repository Signature Keys

  rpm --import http://www.fhgfs.com/release/latest-stable/gpg/RPM-GPG-KEY-fhgfs

Setup the Hosts

• In this configuration we will be running an 8 node setup
  • Node1, Management
  • Node2, Metadata
  • Node3, Storage Server
  • Node4, Storage Server
  • Nodes5-8, Clients

Management Server

• Install the management server

  yum install fhgfs-mgmtd

MetaData Server

• Install the metadata server package

  yum install fhgfs-meta

Storage Server(s)

• On each storage server

  yum install fhgfs-storage

Clients

• On each client

  yum install fhgfs-client

• For each client, there will need to be a kernel built.
• This happens automatically when the service starts
• Ensure 'Development tools' and 'kernel-devel' are installed
• Verify the kernel in /usr/src/kernels matches the output of uname -r

  $ pwd
  /usr/src/kernels
  $ uname -r
  2.6.32-358.el6.x86_64
  $ ls
  2.6.32-358.11.1.el6.x86_64
  ln -s 2.6.32-358.11.1.el6.x86_64/ `uname -r`
  $ ls
  2.6.32-358.11.1.el6.x86_64  2.6.32-358.el6.x86_64

• Then either build the kernel manually or start the service

  $ /etc/init.d/fhgfs-client rebuild
  - FhGFS module autobuild
  Building fhgfs-client-opentk module
  Building fhgfs client module
  $ modinfo fhgfs
  filename:       /lib/modules/2.6.32-358.el6.x86_64/updates/fs/fhgfs_autobuild/fhgfs.ko
  author:         Fraunhofer ITWM, CC-HPC
  description:    FhGFS parallel file system client (http://www.fhgfs.com)
  license:        GPL v2
  srcversion:     D15ED391FD2704BA5D65B86
  depends:        fhgfs-client-opentk
  vermagic:       2.6.32-358.11.1.el6.x86_64 SMP mod_unload modversions

• Note: The standard module only build for ethernet only, no IB support
• Rebuild client for IB (OFED)