A Daily Dose of Power

AIX RPM Dependency Hell

jmsearcy — Tue, 27 May 2014 20:40:41 +0000

Anyone that’s ever worked in a mixed OS environment has surely encountered a time when they’ve needed to install rpm’s on AIX. Whether is was to get the latest GNU version of utilities for more flexible scripting, to support 3rd party monitoring tools, or because you’re developers were too lazy to properly compile installp bundles, we’ve almost all been there. It does, however, seem as if IBM added the rpm support and then has been on vacation ever since. The single most annoying limitation is the lack of dependency resolution. RPM “a” requires rpm’s “a-l” and each of those rpm’s require others, and it goes on and on and on. A simple port of YUM would be stupendous, but alas, we are left without. One man, Michael Perzl, of “http://www.perzl.org/” has graciously supported maintaining various rpm’s compiled for AIX and has even taken some great strides to setting up an infrastructure to support rpm dependency trees for AIX. Michael has setup the rpm distribution and has programmatically generated some dependency lists for those rpm’s. The last remaining step of the rpm dependency hell resolution is to have dependency lists for all rpm’s in the tree sorted and downloaded automatically. I think I’ve fixed the final piece with a script that recursively calls a function to build a list and download all unique rpm’s in the tree.. I’ve tested locally to much success, but as always, feel free to try it out and let me know if anything breaks. My tools are meant to be free, open, and a product of the larger Power community, so I’ve added this out to GitHub to make distribution easier.

Let me know if you have any questions or additions/corrections.

AIX RPM Dependency Script at GitHub

Joe Searcy is a UNIX Systems Engineer living in Atlanta, GA. He’s an avid Power Systems enthusiast with a very diverse IT background. He’s a Certified IBM Systems Administrator for AIX 6.1, and is the founder of this site. In addition to promoting the Power Systems community, Joe enjoys working with Open Source software and currently contributes on a few projects.

Scripts to the Rescue!

jmsearcy — Sat, 08 Mar 2014 02:22:05 +0000

It’s been a while since I’ve added anything new to the site, but hopefully that will change soon. I’m no longer traveling every week and I’m back in a Production Shop so I should be able to generate some juicy content! While you all wait for that, I’m uploaded some scripts I’ve written for various things here and there to GitHub. Some are polished and have a good amount of logic….some are NOT! Feel free to peruse and hopefully they can be of use to someone out there.

https://github.com/jmsearcy/scripts/tree/master/AIX

I will eventually move these to a dedicated DDPWR GitHUb account, but they will be under my personal account for now. If you have any personal scripts you’re fond of, feel free to shoot them this way and I can add them to the collection (or add you to the repo on GitHub).

Enjoy!

Migrating an LV from one VG to another

jmsearcy — Thu, 09 Feb 2012 05:16:09 +0000

In a perfect world you define a standard build (Golden Image), deploy it, and then let it be, never touching it again…..right? If you can answer “yes” to this, I’d like to shake your hand. In most of the environments I’ve worked in, even the term “standard” is relative. You have a patches that are released, configs that change, and technology that gets updated or made obsolete. There are many forces that would cause your beloved golden image to be made “null and void”. That being said, you sometimes have to revisit older machines deployed in the beginning and bring them up to new standards.

In a few environments, you may even be faced with picking up the pieces where other, less skilled folk, may have been in charge. This can often leave you scratching your head asking, “What in the heck were they thinking…” One common scenario I’ve encountered is when someone built the system and included application/database volumes within “rootvg”. Although not a complete wreck, I highly recommend not doing this, for more reasons than it’s worth to explain. This leads me to the topic if this post, how to migrate a logical volume from one volume group to another. Let’s dive in:

NOTE: This example shows how to migrate the “optopenv” LV from “rootvg’ to “nbuvg” and renames the LV to “nbu_optopenv_lv”

Create new VG (or use an existing VG)

$ mkvg -S -y [vgname] -s [pp size in MB] [pv1] [pv2] ...

EXAMPLE:

$ mkvg -S -y nbuvg -s 512 hdisk3

Copy the LV to the new VG

NOTE: The LV being copied needs to be inactive for this, please make sure the associated FS is unmounted

$ cplv -v [new vg] [lv name]

EXAMPLE:

$ cplv -v nbuvg optopenv

NOTE: The new LV will have a generic name similar to “lv00”

Setup LV log

– For inline log FS’s

$ logform /dev/[lvname]

EXAMPLE:

$ logform /dev/lv00

– For external log

$ mklv -t jfs2log [vg name] 1

EXAMPLE:

$ mklv -t jfs2log nbuvg 1

NOTE: this will create a new log LV similar to “loglv00”

$ logform /dev/loglv00

Change the FS to reference the new log

– For inline log FS’s

$ chfs -a /dev/[lv name] -a log=INLINE /filesystem

EXAMPLE:

$ chfs -a /dev/lv00 -a log=INLINE /opt/openv

– For external log FS’s

$ chfs -a /dev/[lv name] -a log=/dev/[log lv name] /filesystem

EXAMPLE:

$ chfs -a /dev/lv00 -a log=/dev/loglv00 /opt/openv

Perform a FS integrity check

$ fsck -p /dev/[lv name]

EXAMPLE:

$ fsck -p /dev/lv00

NOTE: It is common to receive errors after running fsck -p /dev/lvname prior to mounting the filesystem.
These errors are due to a known bug that development is currently aware of and which will be resolved in a future release of AIX.
Once the filesystem is mounted, a future fsck with the filesystem unmounted should no longer produce an error.

Mount the FS

$ mount /[fs name]

EXAMPLE:

$ mount /opt/openv

Remove old LV

$ rmlv [old lv name]

EXAMPLE:

$ rmlv optopenv

Rename new LV

$ chlv -n [new name] [old name]

EXAMPLE:

$ chlv -n nbu_optopenv_lv lv00

DDPWR is now on Twitter!

sysadmin — Thu, 01 Dec 2011 02:34:46 +0000

You can now follow DDPWR on Twitter: @ddpwr

We’ll tweet a link for all new articles posted to give you even more ways to access our content.

PowerVM I/O Balancing Part I: VSCSI Traffic

jmsearcy — Wed, 30 Nov 2011 04:06:57 +0000

This is the first part in a series of posts concerning balancing the I/O traffic of client LPARS across multiple VIOS. This part is focused on VSCSI traffic in a client LPAR configured with MPIO with paths coming from two VIOS. I’m not going to spend a lot of time describing what’s involved to get to this point just so I can keep these short and sweet. So let’s go ahead and dive in.

View the current priority for an individual path:

$ lspath -l  -p  -E

Example with output:

$ lspath -l hdisk0 -p vscsi0 -E
priority 1 Priority True

To set priority for an individual path:

$ chpath -l  -p  -a priority=

Example:

$ chpath -l hdisk0 -p vscsi1 -a priority=2

Simple right?! There’s really not much to it at all. The hard part is establishing your standard for how you balance traffic across your entire environment. Do you balance all odd number LPARS to VIOS1 and all even to VIOS2, do you split each managed system in half? That answer I can’t give you as it will depend on your specific scenario, as well as your needs. In ending this first part, here’s a short script you can use to set priority for all disks in an LPAR.

for i in `lspv | awk '{ print $1 }'`
	do
		chpath -l $i -p vscsi1 -a priority=2
		echo "$i's path for vscsi1 is now set to Priority 2"
done

[UPDATE]

Here’s a little more advanced script for setting the path priority based on an even/odd hdisk numbering scheme (ie. hdisk2, hdisk3, etc.)

#!/usr/bin/ksh

#
# PowerVM vSCSI Path Balancing Script
# by: Joe M. Searcy
# email: jmsearcy@thewebroot.com
#
# 
# This script is used to direct vSCSI path priority to "$VSCSI0" for all even numbered disks, and to
# "$VSCSI1" for all odd number disks. This is to help balance the CPU/Memory load attributed to vSCSI 
# traffic across 2 VIO Servers.
#
#       v1.0 -> 08-16-2012
#               Initial script
#
#

# Variables ######
EVENS="0"
ODDS="1"
TOTDISKS=`lspv | awk '{ print $1 }' | wc -l | sed 's/^ *//'`
TOTDISKS2="$TOTDISKS"
VSCSI0="vscsi0"
VSCSI1="vscsi1"

# Print total number of disks
echo
echo "Total disks in this system: $TOTDISKS"
echo

# Determine even or odd number of disks
let x="$TOTDISKS % 2"

if [ $x -gt 0 ]

		then

			ISODD=true

		else

			ISODD=false

fi

# Uncomment for DEBUG
#echo "This is an odd number of disks: $ISODD"
#echo

# Main Routine
if [ $ISODD == true ]

		then

			let LASTEVEN="$TOTDISKS - 1"
			let LASTODD="$TOTDISKS"

			echo "#### EVEN DISKS ############"
			echo

			while [ $EVENS -le $LASTEVEN ]

				do

					chpath -l hdisk$EVENS -p $VSCSI0 -a priority=1
					chpath -l hdisk$EVENS -p $VSCSI1 -a priority=2

					echo "hdisk$EVENS's path for $VSCSI0 is now set to Priority 1"

					((EVENS+=2))

					if [ $EVENS -gt $LASTEVEN ]

						then

							echo "No more even numbered disks."
							echo
							echo

						else

							echo "The next even numbered disk is \"hdisk$EVENS\""
							echo

					fi

			done

			echo "#### ODD DISKS ############"
			echo

			while [ $ODDS -le $LASTODD ]

				do

					chpath -l hdisk$ODDS -p $VSCSI0 -a priority=2
					chpath -l hdisk$ODDS -p $VSCSI1 -a priority=1

					echo "hdisk$ODDS's path for $VSCSI1 is now set to Priority 2"

					((ODDS+=2))

					if [ $ODDS -gt $LASTODD ]

						then

							echo "No more odd numbered disks."
							echo
							echo

						else

							echo "The next odd numbered disk is \"hdisk$ODDS\""
							echo							

					fi

			done

		else

			let LASTEVEN="$TOTDISKS"
			let LASTODD="$TOTDISKS - 1"

			echo "#### EVEN DISKS ############"
			echo

			while [ $EVENS -le $LASTEVEN ]

				do

					chpath -l hdisk$EVENS -p $VSCSI0 -a priority=1
					chpath -l hdisk$EVENS -p $VSCSI1 -a priority=2

					echo "hdisk$EVENS's path for $VSCSI0 is now set to Priority 1"

					((EVENS+=2))

					if [ $EVENS -gt $LASTEVEN ]

						then

							echo "No more even numbered disks."
							echo
							echo

						else

							echo "The next even numbered disk is \"hdisk$EVENS\""
							echo

					fi

			done

			echo "#### ODD DISKS ############"
			echo

			while [ $ODDS -le $LASTODD ]

				do

					chpath -l hdisk$ODDS -p $VSCSI0 -a priority=2
					chpath -l hdisk$ODDS -p $VSCSI1 -a priority=1

					echo "hdisk$ODDS's path for $VSCSI1 is now set to Priority 2"

					((ODDS+=2))

					if [ $ODDS -gt $LASTODD ]

						then

							echo "No more odd numbered disks."
							echo
							echo

						else

							echo "The next odd numbered disk is \"hdisk$ODDS\""
							echo	

					fi

			done

fi

exit

[/UPDATE]

[UPDATE2]

Here’s yet another update. After helping a customer increase performance for an LPAR using vSCSI storage I developed this script to balance path priority in a situation where you may have multiple vSCSI server adapters. For example:

The customer has 200 LUN’s mapped in a round robin format across 8 different vSCSI server adapters (4 per VIOS using MPIO in the client).

#!/usr/bin/ksh93
# set -x
#
# PowerVM Multi-vSCSI Path Balancing Script
# by: Joe M. Searcy
# email: jmsearcy@us.ibm.com
#
# 
# This script is used to dynamically balance hdisk path priority across multiple vSCSI adapters
# for all disks within a system.
#
# This was developed for customers who have vSCSI disks federated across multiple vSCSI adapters
# for improved performance.
#
#       v1.0 -> 
#               01-18-2013 - Joe Searcy -> Initial script
#               01-18-2013 - Joe Searcy -> Added filter for non-vSCSI disks
#               01-18-2013 - Joe Searcy -> Added logic for single path vSCSI disks
#  	        01-25-2013 - Joe Searcy -> Fixed vSCSI count array arithmetic
#               01-27-2013 - Joe Searcy -> Added some additional comments and configured to 
#                                          perform changes instead of only echoing to stdout
#  	        03-07-2014 - Joe Searcy -> Added debug mode and fixed variable values (stuff never removed from testing)
#

#### Variables ####

DEBUG=echo
LSPATH_OUT=`lspath`
DISKLIST=`echo ${LSPATH_OUT} | awk '{print $2}' | sort -u  | grep "vscsi"`
VSCSILIST=`echo ${LSPATH_OUT} | awk '{print $3}' | sort -u | grep "vscsi" | sed 's/vscsi//'`


#### Create Arrays ####

typeset -a vscsiCountArray
typeset -A path1Array
typeset -A path2Array


#### Build vscsiCountArray ####

        for vscsi in $VSCSILIST

                do
        
                        vscsiCountArray[$vscsi]=0
                
        done


#### Build path1Array ####

        for hdisk in $DISKLIST

                do
        
                        PATH1=`echo ${LSPATH_OUT} | grep "${hdisk} " | head -1 | awk '{ print $3 }' | sed s/vscsi//`
                        path1Array[$hdisk]=$PATH1
                
        done


#### Build path2Array ####

        for hdisk in $DISKLIST

                do
                        
                        PATH2=`echo ${LSPATH_OUT} | grep  "$hdisk " | tail -1 | awk '{ print $3 }' | sed s/vscsi//`
                        path2Array[$hdisk]=$PATH2
                
        done

        
#### Main Loop to set path priority per hdisk ####

        for hdisk in $DISKLIST

                do
        
                        PATH1=${path1Array[$hdisk]}
                        PATH2=${path2Array[$hdisk]}
                        
                        if [ $PATH1 == $PATH2 ]
                        
                                then
                                
                                        echo
                                        echo "Disk \"$hdisk\" does not have multiple paths"
                                        
                                        echo
                                        echo "#######################################################"
                                        echo
                                        
                        else
                
                                        # Compare Path 1 count to Path 2 count and set priority
                
                                        if [ ${vscsiCountArray[$PATH1]} == ${vscsiCountArray[$PATH2]} ]
                
                                                then
                                        
                                                        echo "$hdisk uses \"vscsi${path1Array[$hdisk]}\" & \"vscsi${path2Array[$hdisk]}\""
                                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #echo
                                                        #### End DEBUG ####

                                                        echo
                                                        
                                                        # Actually change path priority (comment out for DEBUG)

                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH1 -a priority=1
                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH2 -a priority=2
                                
                                                        ((vscsiCountArray[$PATH1]+=1))
                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #### End DEBUG ####

                                        
                                                        echo                    
                                                        echo "#######################################################"
                                                        echo
                                
                                        else
                
                                                        echo "$hdisk uses \"vscsi${path1Array[$hdisk]}\" & \"vscsi${path2Array[$hdisk]}\""
                                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #### End DEBUG ####

                                                        echo

                                                        # Actually change path priority (comment out for DEBUG)

                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH1 -a priority=2
                                                        ${DEBUG} chpath -l $hdisk -p vscsi$PATH2 -a priority=1
                                
                                                        ((vscsiCountArray[$PATH2]+=1))
                                        
                                                        #### Begin DEBUG ####
                                                        #echo "Path 1 count: ${vscsiCountArray[$PATH1]}"
                                                        #echo "Path 2 count: ${vscsiCountArray[$PATH2]}"
                                                        #### End DEBUG ####
                                        
                                                        echo
                                                        echo "#######################################################"
                                                        echo
                                
                                        fi
										
                                        
                        fi
                
        done

Calling all Power Fanatics…

jmsearcy — Mon, 28 Nov 2011 04:07:59 +0000

The desire I have for this site is to provide a place for the Power systems community to contribute practical knowledge that could be useful to an admin beginning his first day of work, all the way up to the seasoned grey beard. I’ve often scavenged the web in the heat of an outage for an article to help with the current situation, only to find that there aren’t many sites that provide brief or concise information to help. Typically I end up having to read through a ton of information that’s just not useful before I get to the meat of the solution. All of that being said, if you feel you have information that the viewers of this site could find useful, please send your article to submissions@ddpwr.com. Please only submit original articles.