PowerVM I/O Balancing Part I: VSCSI Traffic

jmsearcy — Wed, 30 Nov 2011 04:06:57 +0000

This is the first part in a series of posts concerning balancing the I/O traffic of client LPARS across multiple VIOS. This part is focused on VSCSI traffic in a client LPAR configured with MPIO with paths coming from two VIOS. I’m not going to spend a lot of time describing what’s involved to get to this point just so I can keep these short and sweet. So let’s go ahead and dive in.

View the current priority for an individual path:

$ lspath -l  -p  -E

Example with output:

$ lspath -l hdisk0 -p vscsi0 -E
priority 1 Priority True

To set priority for an individual path:

$ chpath -l  -p  -a priority=

Example:

$ chpath -l hdisk0 -p vscsi1 -a priority=2

Simple right?! There’s really not much to it at all. The hard part is establishing your standard for how you balance traffic across your entire environment. Do you balance all odd number LPARS to VIOS1 and all even to VIOS2, do you split each managed system in half? That answer I can’t give you as it will depend on your specific scenario, as well as your needs. In ending this first part, here’s a short script you can use to set priority for all disks in an LPAR.

for i in `lspv | awk '{ print $1 }'`
	do
		chpath -l $i -p vscsi1 -a priority=2
		echo "$i's path for vscsi1 is now set to Priority 2"
done

[UPDATE]

Here’s a little more advanced script for setting the path priority based on an even/odd hdisk numbering scheme (ie. hdisk2, hdisk3, etc.)

#!/usr/bin/ksh

#
# PowerVM vSCSI Path Balancing Script
# by: Joe M. Searcy
# email: jmsearcy@thewebroot.com
#
# 
# This script is used to direct vSCSI path priority to "$VSCSI0" for all even numbered disks, and to
# "$VSCSI1" for all odd number disks. This is to help balance the CPU/Memory load attributed to vSCSI 
# traffic across 2 VIO Servers.
#
#       v1.0 -> 08-16-2012
#               Initial script
#
#

# Variables ######
EVENS="0"
ODDS="1"
TOTDISKS=`lspv | awk '{ print $1 }' | wc -l | sed 's/^ *//'`
TOTDISKS2="$TOTDISKS"
VSCSI0="vscsi0"
VSCSI1="vscsi1"

# Print total number of disks
echo
echo "Total disks in this system: $TOTDISKS"
echo

# Determine even or odd number of disks
let x="$TOTDISKS % 2"

if [ $x -gt 0 ]

		then

			ISODD=true

		else

			ISODD=false

fi

# Uncomment for DEBUG
#echo "This is an odd number of disks: $ISODD"
#echo

# Main Routine
if [ $ISODD == true ]

		then

			let LASTEVEN="$TOTDISKS - 1"
			let LASTODD="$TOTDISKS"

			echo "#### EVEN DISKS ############"
			echo

			while [ $EVENS -le $LASTEVEN ]

				do

					chpath -l hdisk$EVENS -p $VSCSI0 -a priority=1
					chpath -l hdisk$EVENS -p $VSCSI1 -a priority=2

					echo "hdisk$EVENS's path for $VSCSI0 is now set to Priority 1"

					((EVENS+=2))

					if [ $EVENS -gt $LASTEVEN ]

						then

							echo "No more even numbered disks."
							echo
							echo

						else

							echo "The next even numbered disk is \"hdisk$EVENS\""
							echo

					fi

			done

			echo "#### ODD DISKS ############"
			echo

			while [ $ODDS -le $LASTODD ]

				do

					chpath -l hdisk$ODDS -p $VSCSI0 -a priority=2
					chpath -l hdisk$ODDS -p $VSCSI1 -a priority=1

					echo "hdisk$ODDS's path for $VSCSI1 is now set to Priority 2"

					((ODDS+=2))

					if [ $ODDS -gt $LASTODD ]

						then

							echo "No more odd numbered disks."
							echo
							echo

						else

							echo "The next odd numbered disk is \"hdisk$ODDS\""
							echo							

					fi

			done

		else

			let LASTEVEN="$TOTDISKS"
			let LASTODD="$TOTDISKS - 1"

			echo "#### EVEN DISKS ############"
			echo

			while [ $EVENS -le $LASTEVEN ]

				do

					chpath -l hdisk$EVENS -p $VSCSI0 -a priority=1
					chpath -l hdisk$EVENS -p $VSCSI1 -a priority=2

					echo "hdisk$EVENS's path for $VSCSI0 is now set to Priority 1"

					((EVENS+=2))

					if [ $EVENS -gt $LASTEVEN ]

						then

							echo "No more even numbered disks."
							echo
							echo

						else

							echo "The next even numbered disk is \"hdisk$EVENS\""
							echo

					fi

			done

			echo "#### ODD DISKS ############"
			echo

			while [ $ODDS -le $LASTODD ]

				do

					chpath -l hdisk$ODDS -p $VSCSI0 -a priority=2
					chpath -l hdisk$ODDS -p $VSCSI1 -a priority=1

					echo "hdisk$ODDS's path for $VSCSI1 is now set to Priority 2"

					((ODDS+=2))

					if [ $ODDS -gt $LASTODD ]

						then

							echo "No more odd numbered disks."
							echo
							echo

						else

							echo "The next odd numbered disk is \"hdisk$ODDS\""
							echo	

					fi

			done

fi

exit

[/UPDATE]

[UPDATE2]

Here’s yet another update. After helping a customer increase performance for an LPAR using vSCSI storage I developed this script to balance path priority in a situation where you may have multiple vSCSI server adapters. For example:

The customer has 200 LUN’s mapped in a round robin format across 8 different vSCSI server adapters (4 per VIOS using MPIO in the client).

#!/usr/bin/ksh93
# set -x
#
# PowerVM Multi-vSCSI Path Balancing Script
# by: Joe M. Searcy
# email: jmsearcy@us.ibm.com
#
# 
# This script is used to dynamically balance hdisk path priority across multiple vSCSI adapters
# for all disks within a system.
#
# This was developed for customers who have vSCSI disks federated across multiple vSCSI adapters
# for improved performance.
#
#       v1.0 -> 
#               01-18-2013 - Joe Searcy -> Initial script
#               01-18-2013 - Joe Searcy -> Added filter for non-vSCSI disks
#               01-18-2013 - Joe Searcy -> Added logic for single path vSCSI disks
#  	        01-25-2013 - Joe Searcy -> Fixed vSCSI count array arithmetic
#               01-27-2013 - Joe Searcy -> Added some additional comments and configured to 
#                                          perform changes instead of only echoing to stdout
#  	        03-07-2014 - Joe Searcy -> Added debug mode and fixed variable values (stuff never removed from testing)
#

#### Variables ####

DEBUG=echo
LSPATH_OUT=`lspath`
DISKLIST=`echo ${LSPATH_OUT} | awk '{print $2}' | sort -u  | grep "vscsi"`
VSCSILIST=`echo ${LSPATH_OUT} | awk '{print $3}' | sort -u | grep "vscsi" | sed 's/vscsi//'`


#### Create Arrays ####

typeset -a vscsiCountArray
typeset -A path1Array
typeset -A path2Array


#### Build vscsiCountArray ####

        for vscsi in $VSCSILIST

                do
        
                        vscsiCountArray[$vscsi]=0
                
        done


#### Build path1Array ####

        for hdisk in $DISKLIST

                do
        
                        PATH1=`echo ${LSPATH_OUT} | grep "${hdisk} " | head -1 | awk '{ print $3 }' | sed s/vscsi//`
                        path1Array[$hdisk]=$PATH1
                
        done


#### Build path2Array ####

        for hdisk in $DISKLIST

                do
                        
                        PATH2=`echo ${LSPATH_OUT} | grep  "$hdisk " | tail -1 | awk '{ print $3 }' | sed s/vscsi//`
                        path2Array[$hdisk]=$PATH2
                
        done

        
#### Main Loop to set path priority per hdisk ####

        for hdisk in $DISKLIST

                do
        
                        PATH1=${path1Array[$hdisk]}
                        PATH2=${path2Array[$hdisk]}
                        
                        if [ $PATH1 == $PATH2 ]
                        
                                then
                                
                                        echo
                                        echo "Disk \"$hdisk\" does not have multiple paths"
                                        
                                        echo
                                        echo "#######################################################"
                                        echo
                                        
                        else
                
                                        # Compare Path 1 count to Path 2 count and set priority
                
                                        if [ ${vscsiCountArray[$PATH1]} == ${vscsiCountArray[$PATH2]} ]
                
                                                then
                                        
                                                        echo "$hdisk uses \"vscsi${path1Array[$hdisk]}\" & \"vscsi${path2Array[$hdisk]}\""
                                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #echo
                                                        #### End DEBUG ####

                                                        echo
                                                        
                                                        # Actually change path priority (comment out for DEBUG)

                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH1 -a priority=1
                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH2 -a priority=2
                                
                                                        ((vscsiCountArray[$PATH1]+=1))
                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #### End DEBUG ####

                                        
                                                        echo                    
                                                        echo "#######################################################"
                                                        echo
                                
                                        else
                
                                                        echo "$hdisk uses \"vscsi${path1Array[$hdisk]}\" & \"vscsi${path2Array[$hdisk]}\""
                                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #### End DEBUG ####

                                                        echo

                                                        # Actually change path priority (comment out for DEBUG)

                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH1 -a priority=2
                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH2 -a priority=1
                                
                                                        ((vscsiCountArray[$PATH2]+=1))
                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #### End DEBUG ####
                                        
                                                        echo
                                                        echo "#######################################################"
                                                        echo
                                
                                        fi
										
                                        
                        fi
                
        done

Joe Searcy is a UNIX Systems Engineer living in Atlanta, GA. He’s an avid Power Systems enthusiast with a very diverse IT background. He’s a Certified IBM Systems Administrator for AIX 6.1, and is the founder of this site. In addition to promoting the Power Systems community, Joe enjoys working with Open Source software and currently contributes on a few projects.

priority – A Daily Dose of Power

PowerVM I/O Balancing Part I: VSCSI Traffic