Remove an OST from an existing Lustre system

Notes from Gabriele on JIRA: https://jira.hpdd.intel.com/browse/BOS-8?jql=project%20%3D%20BOS

this is a procedure to fix the sequence issue.
0. If it is possible backup all the file system
1. Unmount all the clients
2. Mount only one client and verify that no files are belonging to 0c 0d 0e 0f using the following command:
# lfs find /mnt/lustrefs --ost 0xc --ost 0xd --ost 0xe --ost 0xf
you can use the #lfs_migrate to move those files to other OSTs.
3. Unmount all the clients again
4. Remove 0c 0d 0e 0f from IML using the web interface
5. Umount all the OSTs, MDT and MGT
6. Run tunefs.lustre --writeconf on the MDT (not the MGS) 
6. Run tunefs.lustre --writeconf on OST00[00-09] scsi devices
7. Mount the file system using IML or manually following the following sequence: MGT, MDT, OSTs
8. Mount the file system on 1 client and verify the sequence using the command #lfs df -h 
9. Add the new 4 OST using IML again
10. Verify the sequence again on the client an

We had to manually ammend the IML database

[root@iml1 ~]# su -l postgres
-bash-4.1$ psql chroma -U chroma 
psql (8.4.20)
Type "help" for help.

chroma=>  select * from chroma_core_managedfilesystem;
 id |       state_modified_at       |   state   | immutable_state |   name   | mgs_id | mdt_next_index | ost_next_index | not_deleted | content_type_id 
----+-------------------------------+-----------+-----------------+----------+--------+----------------+----------------+-------------+-----------------
  1 | 2015-10-05 10:52:34.961875+01 | available | f               | lustrefs |      3 |              1 |             17 | t           |              80
(1 row)

chroma=> update chroma_core_managedfilesystem set ost_next_index=10 where id=1; 
UPDATE 1
chroma=>  select * from chroma_core_managedfilesystem;  
 id |       state_modified_at       |   state   | immutable_state |   name   | mgs_id | mdt_next_index | ost_next_index | not_deleted | content_type_id 
----+-------------------------------+-----------+-----------------+----------+--------+----------------+----------------+-------------+-----------------
  1 | 2015-10-05 10:52:34.961875+01 | available | f               | lustrefs |      3 |              1 |             10 | t           |              80
(1 row)

chroma=> quit
chroma-> exit
chroma-> \q

Sample outputs:

[root@oss1 ~]# tunefs.lustre --writeconf /dev/sda
checking for existing Lustre data: found
Reading CONFIGS/mountdata

   Read previous values:
Target:     lustrefs-OST0000
Index:      0
Lustre FS:  lustrefs
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: errors=remount-ro
Parameters: mgsnode=172.23.19.42@tcp1 mgsnode=172.23.19.41@tcp1 failover.node=172.23.19.38@tcp1


   Permanent disk data:
Target:     lustrefs=OST0000
Index:      0
Lustre FS:  lustrefs
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: errors=remount-ro
Parameters: mgsnode=172.23.19.42@tcp1 mgsnode=172.23.19.41@tcp1 failover.node=172.23.19.38@tcp1

Writing CONFIGS/mountdata

Problem stopping lustre;

lustre_rmmod
lctl network down
lustre_rmmod