DRBD: How to replace drbd backend devices

In this tutorial, I will explain how to replace the drbd backend devices (for data and metadata) without destroying the information. Actually, during this process, the information will be safely kept on the other node. This should be done in case of one node's disk failure or if you simply just want to move the date to other disk.

1. disconnect the drbd device:

root@candy:~# drbdadm disconnect nfs
root@candy:~# /etc/init.d/drbd status | grep nfs
0:nfs      StandAlone  Secondary/Unknown  Inconsistent/DUnknown  r--u--

2. Detach the drbd device:

root@candy:~# drbdadm detach nfs
root@candy:~# drbdadm dstate nfs
Unconfigured

3. Edit the configuration file for this specific device and replace the both device and metadata on the node where you want to replace the backend drive.

4. recreate the metadata information on the newly added device:

root@candy:~# drbdadm create-md nfs
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success

5. Bring up the drbd device and the synchronization should start automatically:

root@candy:~# drbdadm up nfs
root@candy:~# drbdadm dstate nfs
Inconsistent/UpToDate
root@candy:~# /etc/init.d/drbd status | egrep "nfs|sync"
...        sync'ed:    1.1%               (70956/71680)Mfinish:  2:11:09  9,224    (7,956)   want:  30,720  K/sec
0:nfs      SyncTarget  Secondary/Primary  Inconsistent/UpToDate  C

Once the synchronisation is done, you can replace the device on the other node if needed.