Multipath on Linux

My notes on how to setup a TrueNAS server and a RedHat Linux server to do Multipath testing

Part 1 - Setup a TrueNAS

Part 2 - Setup a RHEL server

Part 3 - Adding new disk (what this whole testing is really about)


Part 1 - Setting up the TrueNAS server

1) Build a basic TrueNas system

    You will need to have:
    a) a storage pool to store the iscsi disk(s)
    b) two network cards configured and working
    c) enable the iscsi service under services


2) In the actions context menu for your pool (the three buttons on the right), select Add Zvol

3) Give the zvol a name, Size, compression level and set deduplication. then click submit

4) Under Sharing, Select Block Sharing (ISCSI)

5) Click the WIZARD button

6) Enter a name for the device (this will be the name of the disk that is visible to the RHEL server)

7) Set Type to Device

8) Under device, you should be able to select the zvol you created above (steps 3 and 4)

9) Click Next

10) Create a new portal if you don't already have one, and select the IP address of the server

11) On the initiators page, click Next

12) Click Submit

13) Add second NIC to the system, configure and test it.

14) On the Block Shares (iSCSI) page, select Portals

15) On the acitons menu for your portal group select Edit

16) Click the Add button

17) On the new IP address line, select the address of the second NIC you added (step 13)

18) click Save

 


Part 2 - Setting up the RHEL 8 Server

1) Build a basic RHEL 8 server

2) Install and configure two NICs with different IP and check they both work.

 

3) Setup iscsi

3.1) Change to the root user

$ sudo -i

3.2) Tell iscsiadm to discover what target are available. Note Change the IP to match the address of your TrueNAS server. Your output will differ depending on IP address, system name and disk name.

# iscsiadm --mode discovery -t sendtargets -p 192.168.1.110
192.168.1.110:3260,1 iqn.2005-10.org.freenas.ctl:disk1
192.168.1.210:3260,1 iqn.2005-10.org.freenas.ctl:disk1

3.3) login to share

# iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk1 --portal 192.168.1.110:3260 --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.110,3260]
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.110,3260] successful.

3.4) Check disk is available

# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0   32G  0 disk 
├─sda1        8:1    0    1G  0 part /boot
└─sda2        8:2    0   31G  0 part 
  ├─cs-root 253:0    0   29G  0 lvm  /
  └─cs-swap 253:1    0  2.1G  0 lvm  [SWAP]
sdb           8:16   0   50G  0 disk 
sr0          11:0    1 1024M  0 rom  

3.5) login via second interface

# iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk1 --portal 192.168.1.210:3260 --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.210,3260]
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.210,3260] successful.

3.6) Check 'second disk' is visible (its not really a second disk, but a copy/mirror of the first disk)
It will appear as a completely seperate disk until multipath is setup.

# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0   32G  0 disk 
├─sda1        8:1    0    1G  0 part /boot
└─sda2        8:2    0   31G  0 part 
  ├─cs-root 253:0    0   29G  0 lvm  /
  └─cs-swap 253:1    0  2.1G  0 lvm  [SWAP]
sdb           8:16   0   50G  0 disk 
sdc           8:32   0   50G  0 disk 
sr0          11:0    1 1024M  0 rom 

3.7) Edit iscsi.conf file to adjust timeout (so it works better?)

# vi /etc/iscsi/iscsid.conf
node.session.timeo.replacement_timeout = 5

 

4) Setting up multipath

4.1) Create multipath.conf and turn on user friendly name

# /sbin/mpathconf --enable --user_friendly_names y

4.2) Start multipath

# systemctl enable --now multipathd

4.3) check what is visible via multipath (you'll need the long ID number in later steps)

# multipath -ll
mpatha (36589cfc000000c37dc0b02aed21d154f) dm-2 TrueNAS,iSCSI Disk
size=50G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 9:0:0:0 sdc 8:32 active ready running

4.x) Check what is visiable via lsblk

# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda           8:0    0   32G  0 disk  
├─sda1        8:1    0    1G  0 part  /boot
└─sda2        8:2    0   31G  0 part  
  ├─cs-root 253:0    0   29G  0 lvm   /
  └─cs-swap 253:1    0  2.1G  0 lvm   [SWAP]
sdb           8:16   0   50G  0 disk  
└─mpatha    253:2    0   50G  0 mpath 
sdc           8:32   0   50G  0 disk  
└─mpatha    253:2    0   50G  0 mpath 
sr0          11:0    1 1024M  0 rom    

4.4) Add the following section to the bottom of multipath.conf 

# vi /etc/multipath.conf

        multipaths {
                multipath {
                        wwid 36589cfc000000c37dc0b02aed21d154f
                        alias truenasdisk1
                }
        }

4.5) Restart service, so it uses new names

# systemctl restart multipathd

4.6) Check new name

# lsblk
NAME           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda              8:0    0   32G  0 disk  
├─sda1           8:1    0    1G  0 part  /boot
└─sda2           8:2    0   31G  0 part  
  ├─cs-root    253:0    0   29G  0 lvm   /
  └─cs-swap    253:1    0  2.1G  0 lvm   [SWAP]
sdb              8:16   0   50G  0 disk  
└─truenasdisk1 253:2    0   50G  0 mpath 
sdc              8:32   0   50G  0 disk  
└─truenasdisk1 253:2    0   50G  0 mpath 
sr0             11:0    1 1024M  0 rom   

5) use the new disk(s)

5.1) make new root folder

# mkdir /truenas

5.2) Check name of device(s)

# ls -l /dev/mapper
total 0
crw-------. 1 root root 10, 236 Jan 16 20:43 control
lrwxrwxrwx. 1 root root       7 Jan 16 20:43 cs-root -> ../dm-0
lrwxrwxrwx. 1 root root       7 Jan 16 20:43 cs-swap -> ../dm-1
lrwxrwxrwx. 1 root root       7 Jan 17 09:26 truenasdisk1 -> ../dm-2

5.3) setup disks

# pvcreate /dev/mapper/truenasdisk1
Physical volume "/dev/mapper/truenasdisk1" successfully created.
# vgcreate truenasvg /dev/mapper/truenasdisk1
Volume group "truenasvg" successfully created
# lvcreate -l +100%FREE --name truenaslv truenasvg
Logical volume "truenaslv" created.
# mkfs.ext4 /dev/mapper/truenasvg-truenaslv
mke2fs 1.45.6 (20-Mar-2020)
Discarding device blocks: done
Creating filesystem with 13106176 4k blocks and 3276800 inodes
Filesystem UUID: 105e6cfd-4e6f-4d63-b364-7b82b742e206
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done
# lsblk

NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                       8:0    0   32G  0 disk  
├─sda1                    8:1    0    1G  0 part  /boot
└─sda2                    8:2    0   31G  0 part  
  ├─cs-root             253:0    0   29G  0 lvm   /
  └─cs-swap             253:1    0  2.1G  0 lvm   [SWAP]
sdb                       8:16   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   50G  0 lvm   
sdc                       8:32   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   50G  0 lvm   
sr0                      11:0    1 1024M  0 rom   

5.4) Mount the new disk to the directory you created earlier

# mount /dev/mapper/truenasvg-truenaslv /truenas

5.5) Check it is mounted

# df -h

Filesystem                       Size  Used Avail Use% Mounted on
devtmpfs                         888M     0  888M   0% /dev
tmpfs                            907M     0  907M   0% /dev/shm
tmpfs                            907M  8.6M  898M   1% /run
tmpfs                            907M     0  907M   0% /sys/fs/cgroup
/dev/mapper/cs-root               29G  2.6G   27G   9% /
/dev/sda1                       1014M  345M  670M  34% /boot
tmpfs                            182M     0  182M   0% /run/user/0
tmpfs                            182M     0  182M   0% /run/user/1000
/dev/mapper/truenasvg-truenaslv   49G   53M   47G   1% /truenas

 

6) If you wanted to test multipath is working, turn off or disconnect one of the NICs on either the RHEL server or the TrueNAS server and see if you can still access the /truenas path


 

Part 3 - Adding second disk

 

1. Add another disk to your NAS

1.1) Go to Storage, pools, ADD

1.2) On the actions menu for your pool, select Add Zvol

1.3) Create a new Zvol similar to the first one you created

1.4) Go to Sharing, Block Shares (iSCSI), Extents

1.5) Click WIZARD

1.6) Specify a Name, select Device as the Type and the new disk/Zvol above as the Device

1.7) Select the existing portal then click Next

1.8) Click Next on the initiator page

1.9) Click Submit

 

2. Add the new disk to the RHEL server

2.1) Tell iscsi to rediscover available targets

# iscsiadm --mode discovery --type sendtargets --portal 192.168.1.110 
192.168.1.110:3260,1 iqn.2005-10.org.freenas.ctl:disk1
192.168.1.210:3260,1 iqn.2005-10.org.freenas.ctl:disk1
192.168.1.110:3260,1 iqn.2005-10.org.freenas.ctl:disk2
192.168.1.210:3260,1 iqn.2005-10.org.freenas.ctl:disk2

2.2) Login to the new disk(s)

# iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk2 --portal 192.168.1.110:3260 --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.110,3260]
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.110,3260] successful.

# iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk2 --portal 192.168.1.210:3260 --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.210,3260]
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.210,3260] successful.

 

2.3) Check if new disk is visible

# lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                       8:0    0   32G  0 disk  
├─sda1                    8:1    0    1G  0 part  /boot
└─sda2                    8:2    0   31G  0 part  
  ├─cs-root             253:0    0   29G  0 lvm   /
  └─cs-swap             253:1    0  2.1G  0 lvm   [SWAP]
sdb                       8:16   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   50G  0 lvm   /truenas
sdc                       8:32   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   50G  0 lvm   /truenas
sdd                       8:48   0   50G  0 disk  
└─mpathb                253:4    0   50G  0 mpath 
sde                       8:64   0   50G  0 disk  
└─mpathb                253:4    0   50G  0 mpath 
sr0                      11:0    1 1024M  0 rom 

2.4) Get multipath to reload the device maps

# multipath -r
# multipath -ll

mpathb (36589cfc0000005542eeba3e9ff402915) dm-4 TrueNAS,iSCSI Disk
size=50G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:0 sdd 8:48 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 11:0:0:0 sde 8:64 active ready running
truenasdisk1 (36589cfc000000c37dc0b02aed21d154f) dm-2 TrueNAS,iSCSI Disk
size=50G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:0:0  sdb 8:16 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 9:0:0:0  sdc 8:32 active ready running

Disk is visible, but with standard name.

2.5) Add the new disk ID to multipath.conf

# vi /etc/multipath.conf 
multipaths {
     multipath {
         wwid 36589cfc000000c37dc0b02aed21d154f
         alias truenasdisk1
     }
     multipath {
         wwid 36589cfc0000005542eeba3e9ff402915
         alias truenasdisk2
     }
}

 

2.6) Reload multipath and check it is now using the user friendly name

# multipath -r
# multipath -ll
truenasdisk2 (36589cfc0000005542eeba3e9ff402915) dm-4 TrueNAS,iSCSI Disk
size=50G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:0 sdd 8:48 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 11:0:0:0 sde 8:64 active ready running
truenasdisk1 (36589cfc000000c37dc0b02aed21d154f) dm-2 TrueNAS,iSCSI Disk
size=50G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:0:0  sdb 8:16 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 9:0:0:0  sdc 8:32 active ready running

2.7) Check disk is visible to system and as a multipath device

# lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                       8:0    0   32G  0 disk  
├─sda1                    8:1    0    1G  0 part  /boot
└─sda2                    8:2    0   31G  0 part  
  ├─cs-root             253:0    0   29G  0 lvm   /
  └─cs-swap             253:1    0  2.1G  0 lvm   [SWAP]
sdb                       8:16   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   50G  0 lvm   /truenas
sdc                       8:32   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   50G  0 lvm   /truenas
sdd                       8:48   0   50G  0 disk  
└─truenasdisk2          253:4    0   50G  0 mpath 
sde                       8:64   0   50G  0 disk  
└─truenasdisk2          253:4    0   50G  0 mpath 
sr0                      11:0    1 1024M  0 rom

2.8) Add disk to system

# ls -l /dev/mapper
total 0
crw-------. 1 root root 10, 236 Jan 16 20:43 control
lrwxrwxrwx. 1 root root       7 Jan 16 20:43 cs-root -> ../dm-0
lrwxrwxrwx. 1 root root       7 Jan 16 20:43 cs-swap -> ../dm-1
lrwxrwxrwx. 1 root root       7 Jan 17 15:27 truenasdisk1 -> ../dm-2
lrwxrwxrwx. 1 root root       7 Jan 17 15:27 truenasdisk2 -> ../dm-4
lrwxrwxrwx. 1 root root       7 Jan 17 09:31 truenasvg-truenaslv -> ../dm-3

# pvcreate /dev/mapper/truenasdisk2 
  Physical volume "/dev/mapper/truenasdisk2" successfully created.

# vgextend truenasvg /dev/mapper/truenasdisk2 
  Volume group "truenasvg" successfully extended

# vgs
  VG        #PV #LV #SN Attr   VSize   VFree
  cs          1   2   0 wz--n- <31.00g      0
  truenasvg   2   1   0 wz--n-  99.99g <50.00g

# lvextend --resizefs -L +10G /dev/mapper/truenasvg-truenaslv
  Size of logical volume truenasvg/truenaslv changed from <50.00 GiB (12799 extents) to <60.00 GiB (15359 extents).
  Logical volume truenasvg/truenaslv successfully resized.
resize2fs 1.45.6 (20-Mar-2020)
Filesystem at /dev/mapper/truenasvg-truenaslv is mounted on /truenas; on-line resizing required
old_desc_blocks = 7, new_desc_blocks = 8
The filesystem on /dev/mapper/truenasvg-truenaslv is now 15727616 (4k) blocks long.
# lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                       8:0    0   32G  0 disk  
├─sda1                    8:1    0    1G  0 part  /boot
└─sda2                    8:2    0   31G  0 part  
  ├─cs-root             253:0    0   29G  0 lvm   /
  └─cs-swap             253:1    0  2.1G  0 lvm   [SWAP]
sdb                       8:16   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   60G  0 lvm   /truenas
sdc                       8:32   0   50G  0 disk  
└─truenasdisk1          253:2    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   60G  0 lvm   /truenas
sdd                       8:48   0   50G  0 disk  
└─truenasdisk2          253:4    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   60G  0 lvm   /truenas
sde                       8:64   0   50G  0 disk  
└─truenasdisk2          253:4    0   50G  0 mpath 
  └─truenasvg-truenaslv 253:3    0   60G  0 lvm   /truenas
sr0                      11:0    1 1024M  0 rom   

2.9) Check system is now using the correct amount of space

# df -h
Filesystem                       Size  Used Avail Use% Mounted on
devtmpfs                         888M     0  888M   0% /dev
tmpfs                            907M     0  907M   0% /dev/shm
tmpfs                            907M  8.6M  898M   1% /run
tmpfs                            907M     0  907M   0% /sys/fs/cgroup
/dev/mapper/cs-root               29G  2.6G   27G   9% /
/dev/sda1                       1014M  345M  670M  34% /boot
tmpfs                            182M     0  182M   0% /run/user/0
/dev/mapper/truenasvg-truenaslv   59G   52M   56G   1% /truenas
tmpfs                            182M     0  182M   0% /run/user/1000

10) both disks are now visible to system via two paths.

WARNING : Before adding the disk to the fstab file, you need to change when the multipathd.service is loaded.

11) Add NetworkManager.service to the After line of /etc/systemd/system/sysinit.target.wants/multipath.service

After=multipathd.socket systemd-udev-trigger.service systemd-udev-settle.service NetworkManager.service

12) Then move multipathd.service to /etc/systemd/system/multi-user.target.wants

      This will cause an ordering cycle message in dmesg, but the system boots and mounts the path.

 

 

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer