First prepared for KYSYS( DISCOVER FAILURE). For VERIFY and MOVE change the KYSYS command.
Raj Patel: August 28, 2020 - First Edition. 
                           - Revised Dec 28, 2020 - Minor changes and VIOs requirements.

Steps includes: VM for AIX, Linux and IBMi and npiv_val 
.
 NOTE: VIOs 2.2.X reached end of support as of 2020. Please be sure to have supported VIO 3.X version and same VIO versions on HOME and BACKUP sites 
.
This the latest copy of the instructions:
These steps gather data twice which may be needed in some cases. 
Only data is needed to be run once here called "before" and in certain cases to be rerun as "after" to confirm some of the results.
Hence, for more indepth, continue to follow all steps which also names the data collection as "after" if requested.
.
.
The data collection and test should include these machines:
a.  4 VIOS: vio1_source and vio2_source and vio3_destination and vio4_destination
b.  Use only one managed LPAR - pzaeng9
c.  KSYS
.
The steps are
Step1:  Download from ecurep get_dr.sh ( this is an updated script ) and place it on each VIOS in
	/tmp and change permission to 755 ./tmp/get_dr.sh
	{on each VIOS}
	{save /tmp/get_dr.sh}
	# chmod 777 /tmp/get_dr.sh
        .
Step 2. Collect a base line more detailed devscan and snap on each VIOS
	{on each VIOS}
	$ oem_setup_env
	.
	{on each VIOS}
	# snap -r
 	# ctctrl memtraceon -r -c efcdd,efscsidd      (for 8 Gb Fiber Channel Adapter)
        # ctctrl memtracedetail -r -c efcdd,efscsidd  (for 8 Gb Fiber Channel Adapter)
        # ctctrl memtracedetail -r -c emfcdd,emfscsidd  (for 16 or 32 Gb Fiber Channel Adapter) 
        # ctctrl memtracedetail -r -c emfcdd,emfscsidd  (for 16 or 32 Gb Fiber Channel Adapter) 
	# mkdir -p  /tmp/ibmsupt/testcase/before/devscan
       	# /usr/local/bin/devscan -Fc 7 >  /tmp/ibmsupt/testcase/before/devscan/devscan-fc7.$(hostname).$(date +%Y.%m%d.%Hh%M).before.test.out
       	# /usr/local/bin/devscan -v9 >  /tmp/ibmsupt/testcase/before/devscan/devscan-v9.$(hostname).$(date +%Y.%m%d.%Hh%M).before.test.out
        # mkdir -p  /tmp/ibmsupt/testcase/before/NPIVVAL
        # cp -R /home/ios/logs/LPM/npiv_val /tmp/ibmsupt/testcase/before/NPIVVAL
       	# cd /tmp ; ./get_dr.sh                     # get_dr.sh will create /tmp/ibmsupt/testcase/DR directory
    	# snap -ac ( Type Y when prompted )
        .
	{save copy of each snap as a "before"}
        .	

        Link to devscan tool:  devscan 

Step 3: Collect a base line more detailed devscan and snap on each VM Client ( Only for AIX VM )
   	{as root user}
	# snap -r
	# mkdir -p  /tmp/ibmsupt/testcase/before
        # devscan -Fc 7 >  /tmp/ibmsupt/testcase/before/devscan/devscan-fc7.$(hostname).$(date +%Y.%m%d.%Hh%M).before.test.out
        # devscan -v9 >  /tmp/ibmsupt/testcase/before/devscan/devscan-v9.$(hostname).$(date +%Y.%m%d.%Hh%M).before.test.out
	# snap -ac ( type Y when prompted )
        .
	{save copy of each snap as a "before"}
        .
        .
        .
            If this is a Linux VM use below: 
            Refer to: https://www.ibm.com/support/pages/how-generate-sosreport
            # 'multipath -l" > multipath.before.out
            #  sosreports

            If this is an IBMi Initiate a Resource Dump from the HMC for Classic HMC 
            Link to IBMi Initiate a Resource Dump:  IBMi Initiate a Resource Dump using Classical HMC 

            If this is an IBMi Initiate a Resource Dump from the HMC for Enhanced HMC 
            Link to IBMi Initiate a Resource Dump:  IBMi Initiate a Resource Dump using Enhanced HMC 
	    {save copy of each snap as a "before"}
        .
Step 4: Collect a baseline HMC mapping.
        # lshmc -v
        # lshmc -V
        # lssyscfg -r sys -F name  ( to get system_name )
        # lshwres -r virtualio --rsubtype fc -m [system_name] --level lpar > CASE_NUM.fc.before.out
        # lshwres -r virtualio --rsubtype fc --level lpar -m [system_name] -F lpar_name,lpar_id,slot_num,adapter_type,state,is_required,remote_lpar_id,remote_lpar_name,remote_slot_num,wwpns,topology > CASE_NUM.fc.before.topology.out
        .
Step 5. For KSYS
	# snap -r
	# rm -fr /tmp/ctsupt /tmp/ibmsupt,
	# mkdir -p /tmp/ibmsupt/testcase/
	# date && time  ksysmgr q vm 2>&1 | tee /tmp/ibmsupt/testcase/$(hostname).ksysmgr.before
        .
Step 6. Recreate the issue
	# date >  /tmp/ibmsupt/testcase/$(hostname).ksysmgr.during
	# /opt/IBM/ksys/ksysmgr -t -v discover site PDC
 	1.  If the discover fails, proceed to data collection below.
	2.  If the discover succeeds, then attempt a verify:
        . 	
	# /opt/IBM/ksys/ksysmgr -t -v verify site PDC
        .
Step 7. Collect new more indepth devscan on each VIOS
	{on each VIOS}
	$ oem_setup_env
	{on each VIOS and the client LPAR}
        # snap -r ***
        # ctctrl memtraceon -r -c efcdd,efscsidd      (for 8 Gb Fiber Channel Adapter)
        # ctctrl memtraceon -r -c efcdd,efscsidd      (for 8 Gb Fiber Channel Adapter)
        # ctctrl memtracedetail -r -c emfcdd,emfscsidd  (for 16 or 32 Gb Fiber Channel Adapter) 
        # ctctrl memtracedetail -r -c emfcdd,emfscsidd  (for 16 or 32 Gb Fiber Channel Adapter) 
	# mkdir -p  /tmp/ibmsupt/testcase/after/devscan
        # mkdir -p  /tmp/ibmsupt/testcase/after/NPIVVAL
        # cp -R /home/ios/logs/LPM/npiv_val /tmp/ibmsupt/testcase//after/NPIVVAL
        # devscan -Fc 7 >  /tmp/ibmsupt/testcase/after/devscan/devscan-fc7.$(hostname).$(date +%Y.%m%d.%Hh%M).after.test.out
        # devscan -v9 >  /tmp/ibmsupt/testcase/after/devscan/devscan-v9.$(hostname).$(date +%Y.%m%d.%Hh%M).after.test.out
        # cd /tmp ; ./get_dr.sh   # get_dr.sh will create /tmp/ibmsupt/testcase/DR directory
	# snap -ac
        .
	{save copy of each snap as an "after"}
        .
        Link to devscan tool:  devscan 

Step 8. Collect a new more detailed devscan and snap on each VM Client ( Only for AIX VM )
	# snap -r
	# mkdir -p  /tmp/ibmsupt/testcase/after/devscan
        # devscan -Fc 7 >  /tmp/ibmsupt/testcase/after/devscan/devscan-fc7.$(hostname).$(date +%Y.%m%d.%Hh%M)after.test.out
        # devscan -v9 >  /tmp/ibmsupt/testcase/after/devscan/devscan-v9.$(hostname).$(date +%Y.%m%d.%Hh%M).after.test.out
	# snap -ac ( Type Y when prompted )
        .
	{save copy of each snap as a "after"}
        .
        Link to devscan tool:  devscan 

            If this is a Linux VM use below: 
            Refer to:  sosreport 
            # 'multipath -l" > multipath.after.out
            #  sosreports
            .
            If this is an IBMi Initiate a Resource Dump from the HMC for Classic HMC 
            Link to IBMi Initiate a Resource Dump:  IBMi Initiate a Resource Dump Using Classical HMC 
            .
            If this is an IBMi Initiate a Resource Dump from the HMC for Enhanced HMC 
            Link to IBMi Initiate a Resource Dump:  IBMi Initiate a Resource Dum
p Using  Enhanced HMC 
            {save copy of each snap as a "after"}

Step 9. Stop trace on each VIOS
        $ oem_setup_env
        # ctctrl -D -r -c efcdd,efscsidd
        .
        .
        .
Step 10. Collect KSYS snap
	# snap vmsnap
        .
There should be /tmp/ibmsupt/snap.pax.Z and /tmp/ibmsupt/ksys.pax.Z and /tmp/ibmsupt/testcase/*.ksysmgr.* files
In the end, there should be 12-15 total files to be collected.
1.  2 snap files (before and after) for each VIOS and the client LPAR
2.  For KSYS, then snap.pax.Z, ksys.pax.Z, and the files in /tmp/ibmsupt/testcase and /tmp/ibmsupt/testcase/DR