Revision as of 15:03, 1 March 2018

Error creating thumbnail: File missing

Server Installation

Before installing Quobyte server software there are some tasks that need to be performed on each server to ensure correct functionality.

Configure NTP

Each server needs to have the same time, or some services will not start. Ensure that NTP is configured and running on all servers, and check all clocks are synced.

Disable Swap

Disable swap on each storage server. Running swapoff -a will disable all swap devices found in /proc/swaps and /etc/fstab. Also comment out/remove any swap lines in /etc/fstab to prevent swap being activated if a server is rebooted.

Install dependencies

yum -y install java-1.8.0-openjdk-headless wget

Download the Quobyte yum repo file

cd /etc/yum.repos.d
wget https://packages.quobyte.com/repo/9/<YOUR_REPO_ID>/rpm/CentOS_7/quobyte.repo

Install Quobyte packages

yum -y install quobyte-server quobyte-client

Server Configuration

Prepare Drives

Any drives being used by Quobyte need to be formatted and mounted before Quobyte can use them. Currently only ext4 and XFS are supported. Each server in our testbed has 3 available drives. 2 SSDs (/dev/sdb and /dev/sdc) and 1 HHD (/dev/sdd). To prepare each drive do the following

# Create a filesystem on each drive and mount them.
# Note it is recommended to use the full drive and not partitions.

mkfs.xfs /dev/sdX
mount /dev/sdX /some/mount/point

# The testbed was configured as per below, where /dev/sda was the OS drive
[root@q01 ~]# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 238.5G  0 disk
├─sda1   8:1    0     1M  0 part
├─sda2   8:2    0   512M  0 part /boot
├─sda3   8:3    0  15.6G  0 part
├─sda4   8:4    0     1K  0 part
└─sda5   8:5    0 222.4G  0 part /
sdb      8:16   0 372.6G  0 disk /mnt/quobyte/metadata0
sdc      8:32   0 894.3G  0 disk /mnt/quobyte/data0
sdd      8:48   0 931.5G  0 disk /mnt/quobyte/data1

# The same procedure was performed on each storage server

Define Registry Servers

Edit /etc/quobyte/host.cfg on each server to state which servers are running the registry servers. In our setup all 4 are so it was updated to read

registry=q01:7861,q02:7861,q03:7861,q04:7861

If name resolution isn't configured on the servers the hostnames can be replaced with IP addresses.

Create registry devices

To create the first registry device do the following on one server only.

qbootstrap /mnt/quobyte/metadata0

Then start services on this server.

systemctl start quobyte-registry
systemctl start quobyte-webconsole
systemctl start quobyte-api

To confirm that the registry services is running and the device is available run the following command.

[root@q01 ~]# qmgmt device list
Id  Host                                      Mode                   Disk Used  Disk Avail  Services    LED Mode
 1  q01                                       ONLINE                     34 MB      400 GB  REGISTRY    OFF

Note it make take a minute for the device to initially register.

To create other registry services, do the following on each server

qmkdev -t REGISTRY /mnt/quobyte/metadata0
systemctl start quobyte-registry

Once this is completed on each server you can list and check availability of each registry from the first server.

[root@q01 ~]# qmgmt device list
 Id  Host                                      Mode                   Disk Used  Disk Avail  Services    LED Mode
  1  q01                                       ONLINE                     34 MB      400 GB  REGISTRY    OFF
  2  q02                                       ONLINE                     34 MB      400 GB  REGISTRY    OFF
  3  q03                                       ONLINE                     34 MB      400 GB  REGISTRY    OFF
  4  q04                                       ONLINE                     34 MB      400 GB  REGISTRY    OFF

Add Metadata Devices

From the first server run the following command to add Metadata to the registry devices

qmgmt device update add-type <id> METADATA
# id for each registry is listed in the output of 'qmgmt device list'

SSH to each host with a metadata device and start the metadata service by running

systemctl start quobyte-metadata

Confirm that metadata devices are running

[root@q01 ~]# qmgmt device list
 Id  Host                                      Mode                   Disk Used  Disk Avail  Services    LED Mode
  1  q01                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF
  2  q02                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF
  3  q03                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF
  4  q04                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF

Add Data Devices

To add data devices perform the following on each server.

# Define data devices
qmkdev -t DATA /mnt/quobyte/data0
qmkdev -t DATA /mnt/quobyte/data1

# Start Quobyte Data service
systemctl start quobyte-data

Once completed on each server check all devices are registered and available.

[root@q01 ~]# qmgmt device list
 Id  Host                                      Mode                   Disk Used  Disk Avail  Services    LED Mode
  1  q01                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF
  5  q01                                       ONLINE                     22 GB      960 GB  DATA        OFF
  6  q01                                       ONLINE                     34 GB     1000 GB  DATA        OFF
  2  q02                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF
  7  q02                                       ONLINE                     36 MB      960 GB  DATA        OFF
  8  q02                                       ONLINE                     46 GB     1000 GB  DATA        OFF
  3  q03                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF
  9  q03                                       ONLINE                     36 MB      960 GB  DATA        OFF
 10  q03                                       ONLINE                     46 GB     1000 GB  DATA        OFF
  4  q04                                       ONLINE                     34 MB      400 GB  METADATA REGISTRY  OFF
 11  q04                                       ONLINE                     36 MB      400 GB  DATA        OFF
 12  q04                                       ONLINE                     46 GB     1000 GB  DATA        OFF

Volume Management

By default Quobyte creates one volume configuration called BASE, which

Viewing Volume Configurations

Configurations can be viewed through the API or from the webconsole.

API

[root@q01 ~]# qmgmt volume config export BASE
configuration_name: "BASE"
volume_metadata_configuration {
  placement_settings {
    required_device_tags {
    }
    forbidden_device_tags {
    }
    prefer_client_local_device: false
    optimize_for_mapreduce: false
  }
  replication_factor: 1
}

default_config {
  file_layout {
    stripe_width: 1
    replication_factor: 1
    block_size_bytes: 524288
    object_size_bytes: 8388608
    segment_size_bytes: 10737418240
    crc_method: CRC_32_ISCSI
  }
  placement {
    required_device_tags {
    }
    forbidden_device_tags {
    }
    prefer_client_local_device: false
    optimize_for_mapreduce: false
  }
  io_policy {
    cache_size_in_objects: 10
    enable_async_writebacks: true
    enable_client_checksum_verification: true
    enable_client_checksum_computation: true
    sync_writes: AS_REQUESTED
    direct_io: AS_REQUESTED
    OBSOLETE_implicit_locking: false
    lost_lock_behavior: IO_ERROR
    OBSOLETE_keep_page_cache: false
    implicit_locking_mode: NO_LOCKING
    enable_direct_writebacks: false
    notify_dataservice_on_close: false
    keep_page_cache_mode: USE_HEURISTIC
    rpc_retry_mode: RETRY_FOREVER
    lock_scope: GLOBAL
  }
}
snapshot_configuration {
  snapshot_interval_s: 0
  snapshot_lifetime_s: 0
}
metadata_cache_configuration {
  cache_ttl_ms: 10000
  negative_cache_ttl_ms: 10000
  enable_write_back_cache: false
}

Webconsole

Login to webconsole and navigate to "Volume Configuration". Select BASE to view the configuration.

Error creating thumbnail: File missing

Editing Volume Configuration

API

qmgmt volume config edit BASE

This will open in your default editor (or use the value of the EDITOR environment variable if it is set).

Webconsole

Navigate to 'Volume Configurations' and tick the box beside BASE. Then select 'edit' from the drop down menu.

Creating Volume Configurations

@@ Line 158: / Line 158: @@
 =Volume Management=
-===Volume Configurations===
+By default Quobyte creates one volume configuration called BASE, which
-By default Quobyte creates one volume configuration called BASE, which  This configuration can be viewed through the API or from the webconsole.
+===Viewing Volume Configurations===
-* Using API
+Configurations can be viewed through the API or from the webconsole.
+* API
 <syntaxhighlight>
@@ Line 178: / Line 180: @@
    replication_factor: 1
 }
 default_config {
    file_layout {
@@ Line 223: / Line 226: @@
 }
 </syntaxhighlight>
- * From webconsole
+* Webconsole
 Login to webconsole and navigate to "Volume Configuration". Select BASE to view the configuration.
 [[File:Quobyte base vol config webconsole.png|551px]]
+===Editing Volume Configuration===
+* API
+<syntaxhighlight>
+qmgmt volume config edit BASE
+</syntaxhighlight>
+This will open in your default editor (or use the value of the EDITOR environment variable if it is set).
+* Webconsole
+Navigate to 'Volume Configurations' and tick the box beside BASE. Then select 'edit' from the drop down menu.
+===Creating Volume Configurations===

Difference between revisions of "Quobyte"