My notes on how to setup a TrueNAS server and a RedHat Linux server to do Multipath testing
Part 1 - Setup a TrueNAS
Part 2 - Setup a RHEL server
Part 3 - Adding new disk (what this whole testing is really about)
Part 1 - Setting up the TrueNAS server
1) Build a basic TrueNas system
You will need to have:
a) a storage pool to store the iscsi disk(s)
b) two network cards configured and working
c) enable the iscsi service under services
2) In the actions context menu for your pool (the three buttons on the right), select Add Zvol
3) Give the zvol a name, Size, compression level and set deduplication. then click submit
4) Under Sharing, Select Block Sharing (ISCSI)
5) Click the WIZARD button
6) Enter a name for the device (this will be the name of the disk that is visible to the RHEL server)
7) Set Type to Device
8) Under device, you should be able to select the zvol you created above (steps 3 and 4)
9) Click Next
10) Create a new portal if you don't already have one, and select the IP address of the server
11) On the initiators page, click Next
12) Click Submit
13) Add second NIC to the system, configure and test it.
14) On the Block Shares (iSCSI) page, select Portals
15) On the acitons menu for your portal group select Edit
16) Click the Add button
17) On the new IP address line, select the address of the second NIC you added (step 13)
18) click Save
Part 2 - Setting up the RHEL 8 Server
1) Build a basic RHEL 8 server
2) Install and configure two NICs with different IP and check they both work.
3) Setup iscsi
3.1) Change to the root user
$ sudo -i
3.2) Tell iscsiadm to discover what target are available. Note Change the IP to match the address of your TrueNAS server. Your output will differ depending on IP address, system name and disk name.
# iscsiadm --mode discovery -t sendtargets -p 192.168.1.110 192.168.1.110:3260,1 iqn.2005-10.org.freenas.ctl:disk1 192.168.1.210:3260,1 iqn.2005-10.org.freenas.ctl:disk1
3.3) login to share
# iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk1 --portal 192.168.1.110:3260 --login Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.110,3260] Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.110,3260] successful.
3.4) Check disk is available
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk sr0 11:0 1 1024M 0 rom
3.5) login via second interface
# iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk1 --portal 192.168.1.210:3260 --login Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.210,3260] Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk1, portal: 192.168.1.210,3260] successful.
3.6) Check 'second disk' is visible (its not really a second disk, but a copy/mirror of the first disk)
It will appear as a completely seperate disk until multipath is setup.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk sdc 8:32 0 50G 0 disk sr0 11:0 1 1024M 0 rom
3.7) Edit iscsi.conf file to adjust timeout (so it works better?)
# vi /etc/iscsi/iscsid.conf node.session.timeo.replacement_timeout = 5
4) Setting up multipath
4.1) Create multipath.conf and turn on user friendly name
# /sbin/mpathconf --enable --user_friendly_names y
4.2) Start multipath
# systemctl enable --now multipathd
4.3) check what is visible via multipath (you'll need the long ID number in later steps)
# multipath -ll mpatha (36589cfc000000c37dc0b02aed21d154f) dm-2 TrueNAS,iSCSI Disk size=50G features='0' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 8:0:0:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=50 status=enabled `- 9:0:0:0 sdc 8:32 active ready running
4.x) Check what is visiable via lsblk
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk └─mpatha 253:2 0 50G 0 mpath sdc 8:32 0 50G 0 disk └─mpatha 253:2 0 50G 0 mpath sr0 11:0 1 1024M 0 rom
4.4) Add the following section to the bottom of multipath.conf
# vi /etc/multipath.conf
multipaths {
multipath {
wwid 36589cfc000000c37dc0b02aed21d154f
alias truenasdisk1
}
}
4.5) Restart service, so it uses new names
# systemctl restart multipathd
4.6) Check new name
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath sdc 8:32 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath sr0 11:0 1 1024M 0 rom
5) use the new disk(s)
5.1) make new root folder
# mkdir /truenas
5.2) Check name of device(s)
# ls -l /dev/mapper total 0 crw-------. 1 root root 10, 236 Jan 16 20:43 control lrwxrwxrwx. 1 root root 7 Jan 16 20:43 cs-root -> ../dm-0 lrwxrwxrwx. 1 root root 7 Jan 16 20:43 cs-swap -> ../dm-1 lrwxrwxrwx. 1 root root 7 Jan 17 09:26 truenasdisk1 -> ../dm-2
5.3) setup disks
# pvcreate /dev/mapper/truenasdisk1 Physical volume "/dev/mapper/truenasdisk1" successfully created. # vgcreate truenasvg /dev/mapper/truenasdisk1 Volume group "truenasvg" successfully created # lvcreate -l +100%FREE --name truenaslv truenasvg Logical volume "truenaslv" created. # mkfs.ext4 /dev/mapper/truenasvg-truenaslv mke2fs 1.45.6 (20-Mar-2020) Discarding device blocks: done Creating filesystem with 13106176 4k blocks and 3276800 inodes Filesystem UUID: 105e6cfd-4e6f-4d63-b364-7b82b742e206 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (65536 blocks): done Writing superblocks and filesystem accounting information: done
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 50G 0 lvm sdc 8:32 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 50G 0 lvm sr0 11:0 1 1024M 0 rom
5.4) Mount the new disk to the directory you created earlier
# mount /dev/mapper/truenasvg-truenaslv /truenas
5.5) Check it is mounted
# df -h
Filesystem Size Used Avail Use% Mounted on devtmpfs 888M 0 888M 0% /dev tmpfs 907M 0 907M 0% /dev/shm tmpfs 907M 8.6M 898M 1% /run tmpfs 907M 0 907M 0% /sys/fs/cgroup /dev/mapper/cs-root 29G 2.6G 27G 9% / /dev/sda1 1014M 345M 670M 34% /boot tmpfs 182M 0 182M 0% /run/user/0 tmpfs 182M 0 182M 0% /run/user/1000 /dev/mapper/truenasvg-truenaslv 49G 53M 47G 1% /truenas
6) If you wanted to test multipath is working, turn off or disconnect one of the NICs on either the RHEL server or the TrueNAS server and see if you can still access the /truenas path
Part 3 - Adding second disk
1. Add another disk to your NAS
1.1) Go to Storage, pools, ADD
1.2) On the actions menu for your pool, select Add Zvol
1.3) Create a new Zvol similar to the first one you created
1.4) Go to Sharing, Block Shares (iSCSI), Extents
1.5) Click WIZARD
1.6) Specify a Name, select Device as the Type and the new disk/Zvol above as the Device
1.7) Select the existing portal then click Next
1.8) Click Next on the initiator page
1.9) Click Submit
2. Add the new disk to the RHEL server
2.1) Tell iscsi to rediscover available targets
# iscsiadm --mode discovery --type sendtargets --portal 192.168.1.110 192.168.1.110:3260,1 iqn.2005-10.org.freenas.ctl:disk1 192.168.1.210:3260,1 iqn.2005-10.org.freenas.ctl:disk1 192.168.1.110:3260,1 iqn.2005-10.org.freenas.ctl:disk2 192.168.1.210:3260,1 iqn.2005-10.org.freenas.ctl:disk2
2.2) Login to the new disk(s)
# iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk2 --portal 192.168.1.110:3260 --login Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.110,3260] Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.110,3260] successful. # iscsiadm --mode node --targetname iqn.2005-10.org.freenas.ctl:disk2 --portal 192.168.1.210:3260 --login Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.210,3260] Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:disk2, portal: 192.168.1.210,3260] successful.
2.3) Check if new disk is visible
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 50G 0 lvm /truenas sdc 8:32 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 50G 0 lvm /truenas sdd 8:48 0 50G 0 disk └─mpathb 253:4 0 50G 0 mpath sde 8:64 0 50G 0 disk └─mpathb 253:4 0 50G 0 mpath sr0 11:0 1 1024M 0 rom
2.4) Get multipath to reload the device maps
# multipath -r # multipath -ll mpathb (36589cfc0000005542eeba3e9ff402915) dm-4 TrueNAS,iSCSI Disk size=50G features='0' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 10:0:0:0 sdd 8:48 active ready running `-+- policy='service-time 0' prio=50 status=enabled `- 11:0:0:0 sde 8:64 active ready running truenasdisk1 (36589cfc000000c37dc0b02aed21d154f) dm-2 TrueNAS,iSCSI Disk size=50G features='0' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 8:0:0:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=50 status=enabled `- 9:0:0:0 sdc 8:32 active ready running
Disk is visible, but with standard name.
2.5) Add the new disk ID to multipath.conf
# vi /etc/multipath.conf
multipaths {
multipath {
wwid 36589cfc000000c37dc0b02aed21d154f
alias truenasdisk1
}
multipath {
wwid 36589cfc0000005542eeba3e9ff402915
alias truenasdisk2
}
}
2.6) Reload multipath and check it is now using the user friendly name
# multipath -r # multipath -ll truenasdisk2 (36589cfc0000005542eeba3e9ff402915) dm-4 TrueNAS,iSCSI Disk size=50G features='0' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 10:0:0:0 sdd 8:48 active ready running `-+- policy='service-time 0' prio=50 status=enabled `- 11:0:0:0 sde 8:64 active ready running truenasdisk1 (36589cfc000000c37dc0b02aed21d154f) dm-2 TrueNAS,iSCSI Disk size=50G features='0' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 8:0:0:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=50 status=enabled `- 9:0:0:0 sdc 8:32 active ready running
2.7) Check disk is visible to system and as a multipath device
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 50G 0 lvm /truenas sdc 8:32 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 50G 0 lvm /truenas sdd 8:48 0 50G 0 disk └─truenasdisk2 253:4 0 50G 0 mpath sde 8:64 0 50G 0 disk └─truenasdisk2 253:4 0 50G 0 mpath sr0 11:0 1 1024M 0 rom
2.8) Add disk to system
# ls -l /dev/mapper total 0 crw-------. 1 root root 10, 236 Jan 16 20:43 control lrwxrwxrwx. 1 root root 7 Jan 16 20:43 cs-root -> ../dm-0 lrwxrwxrwx. 1 root root 7 Jan 16 20:43 cs-swap -> ../dm-1 lrwxrwxrwx. 1 root root 7 Jan 17 15:27 truenasdisk1 -> ../dm-2 lrwxrwxrwx. 1 root root 7 Jan 17 15:27 truenasdisk2 -> ../dm-4 lrwxrwxrwx. 1 root root 7 Jan 17 09:31 truenasvg-truenaslv -> ../dm-3 # pvcreate /dev/mapper/truenasdisk2 Physical volume "/dev/mapper/truenasdisk2" successfully created. # vgextend truenasvg /dev/mapper/truenasdisk2 Volume group "truenasvg" successfully extended # vgs VG #PV #LV #SN Attr VSize VFree cs 1 2 0 wz--n- <31.00g 0 truenasvg 2 1 0 wz--n- 99.99g <50.00g # lvextend --resizefs -L +10G /dev/mapper/truenasvg-truenaslv Size of logical volume truenasvg/truenaslv changed from <50.00 GiB (12799 extents) to <60.00 GiB (15359 extents). Logical volume truenasvg/truenaslv successfully resized. resize2fs 1.45.6 (20-Mar-2020) Filesystem at /dev/mapper/truenasvg-truenaslv is mounted on /truenas; on-line resizing required old_desc_blocks = 7, new_desc_blocks = 8 The filesystem on /dev/mapper/truenasvg-truenaslv is now 15727616 (4k) blocks long.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 32G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 31G 0 part ├─cs-root 253:0 0 29G 0 lvm / └─cs-swap 253:1 0 2.1G 0 lvm [SWAP] sdb 8:16 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 60G 0 lvm /truenas sdc 8:32 0 50G 0 disk └─truenasdisk1 253:2 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 60G 0 lvm /truenas sdd 8:48 0 50G 0 disk └─truenasdisk2 253:4 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 60G 0 lvm /truenas sde 8:64 0 50G 0 disk └─truenasdisk2 253:4 0 50G 0 mpath └─truenasvg-truenaslv 253:3 0 60G 0 lvm /truenas sr0 11:0 1 1024M 0 rom
2.9) Check system is now using the correct amount of space
# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 888M 0 888M 0% /dev tmpfs 907M 0 907M 0% /dev/shm tmpfs 907M 8.6M 898M 1% /run tmpfs 907M 0 907M 0% /sys/fs/cgroup /dev/mapper/cs-root 29G 2.6G 27G 9% / /dev/sda1 1014M 345M 670M 34% /boot tmpfs 182M 0 182M 0% /run/user/0 /dev/mapper/truenasvg-truenaslv 59G 52M 56G 1% /truenas tmpfs 182M 0 182M 0% /run/user/1000
10) both disks are now visible to system via two paths.
WARNING : Before adding the disk to the fstab file, you need to change when the multipathd.service is loaded.
11) Add NetworkManager.service to the After line of /etc/systemd/system/sysinit.target.wants/multipath.service
After=multipathd.socket systemd-udev-trigger.service systemd-udev-settle.service NetworkManager.service
12) Then move multipathd.service to /etc/systemd/system/multi-user.target.wants
This will cause an ordering cycle message in dmesg, but the system boots and mounts the path.