Difference between revisions of "FhGFS Installation and Configuration"

From Define Wiki
Jump to navigation Jump to search
Line 5: Line 5:
 
http://www.fhgfs.com/wiki/images/sysarch2.png
 
http://www.fhgfs.com/wiki/images/sysarch2.png
  
−
== Prepare Hosts ==
+
= File Systems Tips =
 +
* Taken from: http://www.fhgfs.com/wiki/wikka.php?wakka=PartitionAlignment
 +
 
 +
== Partition Alignment ==
 +
* Avoid problems with partition alignment by not using partitions
 +
<syntaxhighlight>
 +
  mkfs.[ext4|xfs] /dev/sdX
 +
</syntaxhighlight>
 +
 
 +
== Raid Optimised FS ==
 +
* Create a RAID optimised filesystem
 +
* Assuming we are using a 10drive RAID5 (or 9 usable drives less parity) and a raid stripe/chunk size of 64k
 +
<syntaxhighlight>
 +
  $ mkfs.[xfs|ext4] -d su=64k,sw=9 -l version=2,su=64k /dev/sdc1
 +
</syntaxhighlight>
 +
 
 +
== MetaData FS Optimisation ==
 +
* Use EA (Extended Attributes) and Ext4 for the metadata FS
 +
<syntaxhighlight>
 +
  $ mkfs.ext4 -i 2048 -I 512 -J size=400 -Odir_index,filetype,^extents /dev/sdX
 +
</syntaxhighlight>
 +
 
 +
* Mount using <tt>user_xattr</tt>
 +
<syntaxhighlight>
 +
  $ tune2fs -o user_xattr /dev/sdX
 +
</syntaxhighlight>
 +
 
 +
* Other mount options
 +
<syntaxhighlight>
 +
  $ mount -onoatime,nodiratime,nobarrier /dev/sdX <mountpoint>
 +
</syntaxhighlight>
 +
 
 +
* Other MetaData Server Suggestions
 +
<syntaxhighlight>
 +
  # The deadline scheduler typically yields best results for metadata access.
 +
  $ echo deadline > /sys/block/sdX/queue/scheduler
 +
 
 +
  # In order to avoid latencies, the number of schedulable requests should not be too high, the linux default of 128 is a good value.
 +
  $ echo 128 > /sys/block/sdX/queue/nr_requests
 +
</syntaxhighlight>
 +
 
 +
* '''Hint''': When tuning your metadata servers, keep in mind that this is often not so much about throughput, but rather about latency and also some amount of fairness: There are probably also some interactive users on your cluster, who want to see the results of their <tt>ls</tt> and other commands in an acceptable time, so you should try to work on that. This means, for instance, that you probably don't want to set a high value for <tt>/sys/block/sdX/iosched/read_expire</tt> on the metadata servers to make sure that users won't be waiting too long for their operations to complete.
 +
 
 +
= Prepare Hosts for Installation =
 
* Add the FhGFS repo to your environment
 
* Add the FhGFS repo to your environment
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 16: Line 59:
 
   rpm --import http://www.fhgfs.com/release/latest-stable/gpg/RPM-GPG-KEY-fhgfs
 
   rpm --import http://www.fhgfs.com/release/latest-stable/gpg/RPM-GPG-KEY-fhgfs
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
= Host Installation =
  
 
== Setup the Hosts ==
 
== Setup the Hosts ==
Line 35: Line 80:
 
<syntaxhighlight>
 
<syntaxhighlight>
 
   yum install fhgfs-meta
 
   yum install fhgfs-meta
 +
</syntaxhighlight>
 +
 +
* Enable IB
 +
<syntaxhighlight>
 +
  $ fhgfs-opentk-lib-update-ib
 +
  Running Infiniband auto-detection...
 +
 +
  Setting symlink in /opt/fhgfs/lib: libfhgfs-opentk.so -> libfhgfs-opentk-enabledIB.so
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 41: Line 94:
 
<syntaxhighlight>
 
<syntaxhighlight>
 
   yum install fhgfs-storage
 
   yum install fhgfs-storage
 +
</syntaxhighlight>
 +
 +
* Enable IB
 +
<syntaxhighlight>
 +
  $ fhgfs-opentk-lib-update-ib
 +
  Running Infiniband auto-detection...
 +
 +
  Setting symlink in /opt/fhgfs/lib: libfhgfs-opentk.so -> libfhgfs-opentk-enabledIB.so
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 46: Line 107:
 
* On each client
 
* On each client
 
<syntaxhighlight>
 
<syntaxhighlight>
−
   yum install fhgfs-client
+
   yum install fhgfs-client fhgfs-helperd fhgfs-utils
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 83: Line 144:
 
* '''Note: The standard module only build for ethernet only, no IB support'''
 
* '''Note: The standard module only build for ethernet only, no IB support'''
 
* Rebuild client for IB (OFED)
 
* Rebuild client for IB (OFED)
 +
<syntaxhighlight>
 +
  # file: /etc/fhgfs/fhgfs-client-autobuild.conf
 +
  # change:
 +
  buildArgs=-j8
 +
  # to: (if using the kernel ofed source)
 +
  buildArgs=-j8 FHGFS_OPENTK_IBVERBS=1
 +
  # or: (if using own OFED)
 +
  buildArgs=-j8 FHGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/openib/include
 +
  # then rebuild
 +
</syntaxhighlight>
 +
 +
= Host Configuration =
 +
 +
== Management Server Configuration ==
 +
<syntaxhighlight>
 +
  mkdir /data/fhgfs/fhgfs_mgmt
 +
  # edit: /etc/fhgfs/fhgfs-mgmtd.conf
 +
  storeMgmtdDirectory=/data/fhgfs/fhgfs_mgmtd
 +
  /etc/init.d/fhgfs-mgmtd restart
 +
</syntaxhighlight>
 +
 +
== MetaData Server ==
 +
* Recommended to use ext4, see: http://www.fhgfs.com/wiki/wikka.php?wakka=MetaServerTuning for more details on the advantages of ext4 over XFS for metadata
 +
<syntaxhighlight>
 +
 +
</syntaxhighlight>

Revision as of 15:36, 30 June 2013

System Architecture

http://www.fhgfs.com/wiki/images/sysarch2.png

File Systems Tips

Partition Alignment

  • Avoid problems with partition alignment by not using partitions
  mkfs.[ext4|xfs] /dev/sdX

Raid Optimised FS

  • Create a RAID optimised filesystem
  • Assuming we are using a 10drive RAID5 (or 9 usable drives less parity) and a raid stripe/chunk size of 64k
  $ mkfs.[xfs|ext4] -d su=64k,sw=9 -l version=2,su=64k /dev/sdc1

MetaData FS Optimisation

  • Use EA (Extended Attributes) and Ext4 for the metadata FS
  $ mkfs.ext4 -i 2048 -I 512 -J size=400 -Odir_index,filetype,^extents /dev/sdX
  • Mount using user_xattr
  $ tune2fs -o user_xattr /dev/sdX
  • Other mount options
  $ mount -onoatime,nodiratime,nobarrier /dev/sdX <mountpoint>
  • Other MetaData Server Suggestions
  # The deadline scheduler typically yields best results for metadata access.
  $ echo deadline > /sys/block/sdX/queue/scheduler

  # In order to avoid latencies, the number of schedulable requests should not be too high, the linux default of 128 is a good value.
  $ echo 128 > /sys/block/sdX/queue/nr_requests
  • Hint: When tuning your metadata servers, keep in mind that this is often not so much about throughput, but rather about latency and also some amount of fairness: There are probably also some interactive users on your cluster, who want to see the results of their ls and other commands in an acceptable time, so you should try to work on that. This means, for instance, that you probably don't want to set a high value for /sys/block/sdX/iosched/read_expire on the metadata servers to make sure that users won't be waiting too long for their operations to complete.

Prepare Hosts for Installation

  • Add the FhGFS repo to your environment
  cd /etc/yum.repos.d/
  wget http://www.fhgfs.com/release/latest-stable/dists/fhgfs-rhel6.repo
  • Repository Signature Keys
  rpm --import http://www.fhgfs.com/release/latest-stable/gpg/RPM-GPG-KEY-fhgfs

Host Installation

Setup the Hosts

  • In this configuration we will be running an 8 node setup
    • Node1, Management
    • Node2, Metadata
    • Node3, Storage Server
    • Node4, Storage Server
    • Nodes5-8, Clients

Management Server

  • Install the management server
  yum install fhgfs-mgmtd

MetaData Server

  • Install the metadata server package
  yum install fhgfs-meta
  • Enable IB
  $ fhgfs-opentk-lib-update-ib 
  Running Infiniband auto-detection...

  Setting symlink in /opt/fhgfs/lib: libfhgfs-opentk.so -> libfhgfs-opentk-enabledIB.so

Storage Server(s)

  • On each storage server
  yum install fhgfs-storage
  • Enable IB
  $ fhgfs-opentk-lib-update-ib 
  Running Infiniband auto-detection...

  Setting symlink in /opt/fhgfs/lib: libfhgfs-opentk.so -> libfhgfs-opentk-enabledIB.so

Clients

  • On each client
  yum install fhgfs-client fhgfs-helperd fhgfs-utils
  • For each client, there will need to be a kernel built.
  • This happens automatically when the service starts
  • Ensure 'Development tools' and 'kernel-devel' are installed
  • Verify the kernel in /usr/src/kernels matches the output of uname -r
  $ pwd
  /usr/src/kernels
  $ uname -r
  2.6.32-358.el6.x86_64
  $ ls
  2.6.32-358.11.1.el6.x86_64
  ln -s 2.6.32-358.11.1.el6.x86_64/ `uname -r`
  $ ls
  2.6.32-358.11.1.el6.x86_64  2.6.32-358.el6.x86_64
  • Then either build the kernel manually or start the service
  $ /etc/init.d/fhgfs-client rebuild
  - FhGFS module autobuild
  Building fhgfs-client-opentk module
  Building fhgfs client module
  $ modinfo fhgfs
  filename:       /lib/modules/2.6.32-358.el6.x86_64/updates/fs/fhgfs_autobuild/fhgfs.ko
  author:         Fraunhofer ITWM, CC-HPC
  description:    FhGFS parallel file system client (http://www.fhgfs.com)
  license:        GPL v2
  srcversion:     D15ED391FD2704BA5D65B86
  depends:        fhgfs-client-opentk
  vermagic:       2.6.32-358.11.1.el6.x86_64 SMP mod_unload modversions
  • Note: The standard module only build for ethernet only, no IB support
  • Rebuild client for IB (OFED)
  # file: /etc/fhgfs/fhgfs-client-autobuild.conf
  # change:
  buildArgs=-j8
  # to: (if using the kernel ofed source)
  buildArgs=-j8 FHGFS_OPENTK_IBVERBS=1 
  # or: (if using own OFED)
  buildArgs=-j8 FHGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/openib/include 
  # then rebuild

Host Configuration

Management Server Configuration

  mkdir /data/fhgfs/fhgfs_mgmt
  # edit: /etc/fhgfs/fhgfs-mgmtd.conf
  storeMgmtdDirectory=/data/fhgfs/fhgfs_mgmtd
  /etc/init.d/fhgfs-mgmtd restart

MetaData Server

Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=FhGFS_Installation_and_Configuration&oldid=2771"