• iSCSI target node (iqn.date.domain:number or eui.xxxxxxxxxxxxxxxx) ≡ "disk controller", has multiple LUNs inside,
  published to one or more iSCSI target ports (port has the same eui/iqn id as node, but also tag#)

• iSCSI target port is published to target portal group

• portal ≡ TCP address + port number (default: 3260)

• portal is published on HBA/NIC,
  HBA/NIC can host multiple portals

• portal group ≡ set of portals (portal may belong to multiple TPGs) with the same LUN view,
  that can be used in multiple-connection session (MC/S) and can coordinate MC/S session with multiple connections spanning member portals

• TPGT (TPG tag ≡ target portal group tag) is a number distinguishing TPG from other TPGs  


http://linux-iscsi.org/images/Portal-Groups.jpg


http://linux-iscsi.org/images/LIO_iSCSI_Architecture.png


Client / Initiator

Install the open-iscsi package. The package is also known as the Linux Open-iSCSI Initiator.

Ubuntu:

apt-get install open-iscsi lsscsi
gedit /etc/iscsi/iscsid.conf
/etc/init.d/open-iscsi restart

Fedora:

dnf install -y iscsi-initiator-utils lsscsi
systemctl enable iscsid.service
systemctl start iscsid.service

SUSE/openSUSE

zypper install -y open-iscsi yast2-iscsi-client
systemctl enable iscsid.service
systemctl start iscsid.service

Display/edit initiator name, ensure it is unique in the landscape (especially if cloned the system)

cat /etc/iscsi/initiatorname.iscsi

e.g.

InitiatorName=iqn.1994-05.com.redhat:cbf2ba2dff2 => iqn.1994-05.com.redhat:mynode1
InitiatorName=iqn.1993-08.org.debian:01:16c1be18eee8 => iqn.1993-08.org.debian:01:myhost2

Optional: edit configuration

gedit /etc/iscsi/iscsid.conf

restart the service

SUSE/openSUSE iSCSI initiator GUI:

YaST -> iSCSI Initiator

iscsiadm reference

http://www.open-iscsi.org/docs/README

Discover the iSCSI targets on a specific host

iscsiadm -m discovery -t sendtargets -p qnap1:3260
    
Discovery mode can be sendtargets, isns or fw

On first discovery (when no db exists) set the values in /etc/iscsid.conf

discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = jdoe
discovery.sendtargets.auth.password = mypass
    
--name discovery.sendtargets.auth.authmethod --value CHAP
--name discovery.sendtargets.auth.username --value jdoe
--name discovery.sendtargets.auth.password --value YourSecurePwd1
 
Theoretically, the following sequence should suffice:

alias xxd="iscsiadm -m discoverydb --portal nas1:3260 -t sendtargets"
xxd --op delete
xxd --op new
xxd --op update --name discovery.sendtargets.auth.authmethod --value CHAP
xxd --op update --name discovery.sendtargets.auth.username --value myuserid
xxd --op update --name discovery.sendtargets.auth.password --value mypassword
xxd --discover

In practice, it may or may not work (some distros of iscsiadm are broken, e.g. openSUSE 12.3).
iscsiadm is supposed to update /etc/iscsi/send_targets/nas1,3260/st_config with above key values, but it may not.
If it does not:

Execute:

alias xxd="iscsiadm -m discoverydb --portal nas1:3260 -t sendtargets"
# delete/new is optional
xxd --op delete
xxd --op new

Then edit /etc/iscsi/send_targets/nas1,3260/st_config manually.
Add keys:

# delete previous discovery.sendtargets.auth.authmethod !!
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = myuserid
discovery.sendtargets.auth.password = mypassword
Execute

xxd --discover

Check the available iSCSI node(s) to connect to.

iscsiadm -m node

Delete node(s) you don’t want to connect to when the service is on with the following command:

iscsiadm -m node --op delete --targetname iqn.2004-04.com.qnap:ts-569l:iscsi.s1.e4cd7c
iscsiadm -m node --op delete --targetname iqn.2004-04.com.qnap:ts-569l:iscsi.s3.e4cd7c

Set authorization on every needed target:

Theoretically, the following sequence should suffice:

alias xxn="iscsiadm -m node --targetname iqn.2004-04.com.qnap:ts-569l:iscsi.s2.e4cd7c -p qnap1:3260"
xxn --op=update --name node.session.auth.authmethod --value=CHAP
xxn --op=update --name node.session.auth.username --value=myuserid
xxn --op=update --name node.session.auth.password --value=mypassword
xxn --login

In practice, it may or may not work (some distros of iscsiadm are broken, e.g. openSUSE 12.3).
iscsiadm is supposed to update /etc/iscsi/send_targets/qnap1,3260/iqn..../default with above key values, but it may not.
(Mentioned directory is cross-linked to /etc/iscsi/nodes/iqn.../ipaddr,port,n)
If it does not, edit the mentioned file manually and add keys to it.
Then:

alias xxn="iscsiadm -m node --targetname iqn.2004-04.com.qnap:ts-569l:iscsi.s2.e4cd7c -p qnap1:3260"
xxn --login

You should be able to see the login message as below:

Login session [iface: default, target: iqn.2004-04.com:NAS:iSCSI.ForUbuntu.B9281B, portal: 10.8.12.31,3260] [ OK ]

Optionally restart open-iscsi to login to all of the available nodes.
/etc/init.d/open-iscsi restart

Check the device status with dmesg.

dmesg | tail

List available devices:

lsscsi
lsscsi -s
lsscsi -dg
lsscsi -c
lsscsi -Lvl

iscsiadm -m session [-P 3] [-o show]

Create a partition, /dev/sde is the device name.

fdisk /dev/sde  (p, n, w, p)
blockdev --rereadpt /dev/sde

Format the partition, mount the file system.

mkfs.ext3 /dev/sde1
mount /dev/sde1 /mnt

Test the I/O speed using the following command:

hdparm -tT /dev/sde1

Below are some “iscsiadm” related commands.

Discover the Targets on the host:

iscsiadm -m discovery --type sendtargets --portal HOST_IP

Login to a Target:
iscsiadm –m node --targetname THE_TARGET_IQN --login

Login to all available targets

iscsiadm -m node -L all

Logout from a Target:

iscsiadm –m node --targetname THE_TARGET_IQN --logout
iscsiadm -m node --targetname "iqn.2004-04.com.qnap:ts-569l:iscsi.s2.e4cd7c" --logout

Logout from all availavle targets
iscsiadm -m node -U all

Delete a Target:
iscsiadm –m node --op delete --targetname THE_TARGET_IQN

Show connected sessions/devices
iscsiadm -m session [-P 3] [-o show]

iscsiadm -m session

tcp: [1] 192.168.73.2:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs1.e4cd7c (non-flash)
tcp: [2] 192.168.73.2:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs3.e4cd7c (non-flash)
tcp: [3] 192.168.73.2:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c (non-flash)
tcp: [4] 192.168.1.160:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c (non-flash)

[1], [2], [3], [4] – session id's
1 – target portal group tag

Display information about a target

iscsiadm -m node -T targetname -p host:port

Display statistics about a target

iscsiadm -m node -s -T targetname -p host:port

(do not specify -p to display information or statistics about all targets)

View iSCSI database regarding discovery

iscsiadm -m discovery -o show

View iSCSI database regarding targets to log into
iscsiadm -m node -o show

If iSCSI target server was down when SCSI daemon was started, session is not created.
In this case, re-login:

iscsiadm -m node --login
or:

iscsiadm -m node -L all

To discover a new LUN on the target or targets:

rescan all sessions iscsiadm -m session --rescan
rescan a specific session iscsiadm -m session --rescan --sid=N
rescan all buses /usr/bin/rescan-scsi-bus.sh

To remove a device from system:
echo 1 > /sys/block/sdx/device/delete

or

echo 1 > /sys/class/scsi_device/hba:ch:trg:lun/device/delete

Explicit use of multiple network interfaces (binding a node to a particular interface).

Not neeed if interfaces are regular Ethernet (not offload) and are on different subnets.
However if interfaces are on the same subnet, may want to select specific interface to talk to the node.

May also want to bind a node to a specific interface when using iSCSI offload cards (e.g. Chelsio, QLogic or Broadcom).

During session log-in into the target, a session is tied to a particular interface.

Documentation: RHEL

sysctl -w net.ipv4.conf.default.rp_filter=2   # or 0, see here

Also update /etc/sysctl.conf:

net.ipv4.conf.default.rp_filter = 2  # or 0

# iscsiadm -m iface -I scsi-iface0 -o new
# iscsiadm -m iface -I scsi-iface0 --op=update -n iface.net_ifacename -v eth0.210
# iscsiadm -m iface -I scsi-iface1 -o new
# iscsiadm -m iface -I scsi-iface1 --op=update -n iface.net_ifacename -v eth1.210

# iscsiadm -m iface -I scsi-iface2 -o new
# iscsiadm -m iface -I scsi-iface2 --op=update -n iface.net_ifacename -v eth0.210

# iscsiadm -m iface -I scsi-iface3 -o new
# iscsiadm -m iface -I scsi-iface3 --op=update -n iface.net_ifacename -v eth1.210

// note: instead of identifying interface by name, may rather identify it by MAC and IP addresses:

# iscsiadm -m iface -I scsi-iface3 --op=update -n iface.hwaddress -v 00:0F:1F:92:6B:6F
# iscsiadm -m iface -I scsi-iface3 --op=update -n iface.ipaddress -v 10.0.0.103

# iscsiadm -m iface
default tcp,default,default,unknown
iser iser,default,default,unknown
bnx2i bnx2i,default,default,unknown
iface0 tcp,default,eth0.210,unknown
iface2 tcp,default,eth0.210,unknown
iface1 tcp,default,eth1.210,unknown
iface3 tcp,default,eth1.210,unknown

# iscsiadm -m discovery -t st -p QNAP1:3260 -P 1
    Target: iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c
        Portal: QNAP1:3260,1
           Iface Name: iface0
           Iface Name: iface2
           Iface Name: default
           Iface Name: iface3
           Iface Name: iface1

// create (login) sessions, tying them to particular interfaces
# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c -p QNAP1:3260 -I scsi-iface0 --login
# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c -p QNAP1:3260 -I scsi-iface1 --login
# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c -p QNAP1:3260 -I scsi-iface2 --login
# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c -p QNAP1:3260 -I scsi-iface3 --login

# iscsiadm  -m session
tcp: [1] QNAP1:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c
tcp: [2] QNAP1:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c
tcp: [3] QNAP1:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c
tcp: [4] QNAP1:3260,1 iqn.2004-04.com.qnap:ts-569l:iscsi.xs2.e4cd7c

SUSE/openSUSE  also has GUI:

YaST -> iSCSI initiator

https://www.suse.com/documentation/sles-12/stor_admin/data/sec_iscsi_initiator.html

iSNS (initiator side)

alias xsns ="iscsiadm -m discoverydb -t isns -p sns1:3205"
xsns --discover

xsns -o new
xsns -o update -n discovery.isns.use_discoveryd -v Yes
xsns -o update -n discovery.isns.discoveryd_poll_inval -v 30

http://linux-iscsi.org/wiki/Internet_Storage_Name_Service
http://www.open-iscsi.org/docs/README



Server / Target

RHEL7 Documentation: click 25.1

apt-get install lio-utils
chmod 755 /usr/lib/python/dist-packages/tcm_*py
/etc/init.d/target start
/etc/init.d/target status

see /var/log messages

Ubuntu:    apt-get install targetcli
SUSE:      zypper install -y lio-utils yast2-iscsi-lio-server

Ubuntu:    update-rc.d target defaults
RH:          chkconfig target on

http://blog.jamiebaldanza.org/2012/08/14/ubuntu-12-lio-iscsi-target-configuration

delete all current config's (backing storges, portals etc.)

targetcli>  
clearconfig confirm=true                # command no longer exists?
ls

create backing store

/backstores/iblock  create name=mydev1 dev=/dev/sdx10    # block devices
/backstores/fileio  create name=mydev2 file_or_dev=/home/mydir/disk.vd size=2G
/baskstores/pscsi   create name=mydev3 dev=/dev/sr0      # passthrough SCSI
/backstores/rd_mcp  create name=mydev4 size=1GB          # RAM disk

disable all authorization checks

/iscsi set discovery_auth enable="0"                # note the quotes

enable discovery security

/iscsi set discovery_auth userid=myuser            # no quotes
/iscsi set discovery_auth password=mypass        # no quotes
/iscsi set discovery_auth enable=1                    # no quotes

create target wwn (target = host)

/iscsi create
/iscsi ls

create a LUN

cd /iscsi/my-wwn-id/tpg1/luns
create /backstores/block/mydev1
ls

create ACL

cd /iscsi/my-wwn-id/tpg1/acls
create initiator1-iqn-id
create initiator2-iqn-id

# for example
#create iqn.1991-05.com.microsoft:myhost
#create iqn.1991-05.com.microsoft:myhost.mydomain
#create iqn.1993-08.org.debian:01:7e8a7e1de7

ls

create a portal

cd ../portals
create
# or with explicit IP address and port:
# create 192.168.1.139 3260

set login authorization for tpg (shared by all initiators, provides defaults for tpg/acls/xxx)

set ..                             # to tpgt<n>
set attribute authentication=1     # default (or 0)
set parameter AuthMethod=CHAP      # default (or None)
set auth userid=myuser
set auth password=mypass

optionally, configure the target to use a single userid and password for all initiators, including unlisted ones
(generate_node_acls=1 makes acls to be generated on the fly for every initiator trying to connect to the target):

set attribute generate_node_acls=1
set attribute cache_dynamic_acls=1

or set login authorization for each specific tpg/acls/initiator-id

cd /iscsi/my-wwn-id/tpg1/acls/initiator-id
set auth userid=myuser
set auth password=mypass

display help

/iscsi get
/iscsi get discovery_auth

"get" in any context => displays available groups
then "get <groupname>"

save configuration to /etc/target/*.sh

/ saveconfig

check /var/log messages for errors

"targetcli" is a browser/editor of /sys/kernel/config/target
more info (files) in /var/target


http://linux-iscsi.org/wiki/ISCSI

http://linux-iscsi.org/wiki/Persistent_Reservations
SCSI-3 Persistent Reservations on Fedora Core 20 with targetcli over iSCSI and Red Hat Cluster

http://linux-iscsi.org/wiki/Asymmetric_Logical_Unit_Assignment


http://www.open-iscsi.org
http://linux-iscsi.org
http://linux-iscsi.org/wiki/Targetcli
http://linux-iscsi.org/wiki/LIO
http://www.linux-iscsi.org/Doc/RTS OS Admin Manual.pdf
http://blog.jamiebaldanza.org/2012/08/14/ubuntu-12-lio-iscsi-target-configuration

http://sg.danny.cz/sg/sg3_utils.html
http://sg.danny.cz/sg/sdparm.html

apt-get install sg3-utils sdparm lsscsi [ddpt]

cat /proc/scsi/scsi
sg_map -i -x
lsscsi [-s] [-t] [-c] [-g] [--hosts] [--list] [--transport] [-v] [-lll]
sg_scan [-x] [-i] [-v]
sg_inq [-d] [-e] /dev/sda
sg_vpd [-ilv] /dev/sda
sg_vpd /dev/sda --page=0xb2 -l
sginfo /dev/sda
sdparm /dev/sda [--all]
sg_modes /dev/sda [--all]
sg_readcap /dev/sda
....
sg_persist ....

libsgutils


Setup a basic iSCSI target and export a mapped LUN:

#!/usr/bin/python
# iSCSI setup script example with RTSlib
from rtslib import *

# Setup an IBLOCK backstore
backstore = IBlockBackstore(3, mode='create')
try:
    so = IBlockStorageObject(backstore, "sdb", "/dev/sdb", gen_wwn=True)
except:
    backstore.delete()
    raise

# Create an iSCSI target endpoint using an iSCSI IQN
fabric = FabricModule('iscsi')
target = Target(fabric, "iqn.2003-01.org.linux-iscsi.x.x8664:sn.d3d8b0500fde")
tpg = TPG(target, 1)

# Setup a network portal in the iSCSI TPG
# The IP address must already be active on the system
portal = NetworkPortal(tpg, "192.168.1.128", "5060")

# Export LUN 0 via the 'so' StorageObject class
lun0 = tpg.lun(0, so, "my_lun")

# Setup the NodeACL for an iSCSI initiator, and create MappedLUN 0
node_acl = tpg.node_acl("iqn.2003-01.org.linux-iscsi.y.x8664:sn.abcdefghijkl")
mapped_lun = node_acl.mapped_lun(0, 0, False)

Setup a basic Fibre Channel target and export a mapped LUN:

#!/usr/bin/python
# Fibre Channel setup script example with RTSlib
from rtslib import *

# Setup an IBLOCK backstore
backstore = IBlockBackstore(3, mode='create')
try:
    so = IBlockStorageObject(backstore, "fioa", "/dev/fioa", gen_wwn=True)
except:
    backstore.delete()
    raise

# Create an FC target endpoint using a qla2xxx WWPN
fabric = FabricModule('qla2xxx')
target = Target(fabric, '21:00:00:24:ff:31:4c:48')
tpg = TPG(target, 1)

# Export LUN 0 via the 'so' StorageObject class
lun0 = tpg.lun(0, so, "my_lun")

# Setup the NodeACL for an FC initiator, and create MappedLUN 0
node_acl = tpg.node_acl('21:00:00:24:ff:31:4c:4c')
mapped_lun = node_acl.mapped_lun(0, 0, False)

Setup a basic SRP target and export a mapped LUN:

#!/usr/bin/python
# InfiniBand setup script example with RTSlib
from rtslib import *

# Setup an IBLOCK backstore
backstore = IBlockBackstore(3, mode='create')
try:
    so = IBlockStorageObject(backstore, "fioa", "/dev/fioa", gen_wwn=True)
except:
    backstore.delete()
    raise

# Create an IB target endpoint using an ib_srpt WWPN
fabric = FabricModule('ib_srpt')
target = Target(fabric, '0x00000000000000000002c903000e8acd')
tpg = TPG(target, 1)

# Export LUN 0 via the 'so' StorageObject class
lun0 = tpg.lun(0, so, "my_lun")

# Setup the NodeACL for an IB initiator, and create MappedLUN 0
node_acl = tpg.node_acl('0x00000000000000000002c903000e8acd')
mapped_lun = node_acl.mapped_lun(0, 0, False)