Raj Patel: August 28, 2020 - First Edition. 
                           - Revised Dec 28, 2020 - Minor changes and VIOs requirements.
                           - Revised Jun 04, 2021 - Breakup for VMRM HA and VMRM target (This will cover VMRM HA)
                           - Revised August 16, 2021 - devscan details with flags.   
                           - Revised April 04, 2023 - lshwres output ( remove >)
              
Steps includes: VM for AIX, Linux and IBMi and npiv_val 
.
 NOTE: VIOs 2.2.X reached end of support as of 2020. Please be sure to have supported VIO 3.X version and same VIO versions on Source and Target sites 
 
The data collection and test should include these machines:
a)  2 VIOS from Source site 
b)  2 VIOS from Target site 
c)  1 AIX VM snap for failed VM. For Linux (sosreports). For IBMi (Initiate a Resource Dump)
d)  KSYS
e)  pedbg from HMC
f)  devscan output
g)  details on the storage and fabric switches with make and model and any ISL setup.
 
The steps are:

1) Confirm that KSYS, VM, and VIOs are on same time zone and if not what timezone they are on and if the times are in sync.
  
2) For AIX Failed VM:
   devscan and VM snap from failed VM using:
   Download devscan tool from below link and install on VM that failed 
   Link to devscan tool:  devscan 
.
   Collect snap and devscan from failed VM and run it in order provided.
    a) snap -r
    b) devscan -tf -n  [Inactive_wwpns] -o  > /tmp/ibmsupt/testcase/devscan.[inactive].wwpns.vm.out (NOTE: Only run on Inactive WWPNs)
     snap -ac ( data in /tmp/ibmsupt/snap.pax.Z)
       rename to CASE_NUMBER.VM.snap.pax.Z and upload CASE_NUMBER.VM.snap.pax.Z

    NOTE: On AIX you can get listing of all Active WWPNs using details from below link:
           Active WWPN of Virtual Fiber Channel Adapter 
   
    To simplify, use below script: devscan_vmrmha_aix.client.sh
    NOTE:  AIX devscan script 
.
3)   If the failed VM is Linux use below:    
        Refer to link to sosreport:  sosreport 
        # 'multipath -l" > multipath.after.out
        # sosreports

4)   If this is an IBMi Initiate a Resource Dump from the HMC for Classic HMC 
        Link to IBMi Resource Dump using Classic HMC:  Resource Dump using Classic HMC 
         
     If this is an IBMi Initiate a Resource Dump from the HMC for Enhanced HMC 
        Link to IBMi Resource Dump using Enhaced HMC:  Resource Dump using Enhanced HMC 
 
5) Both VIO snaps from Source site using:
   $ snap ( data in /source/padmin/snap.pax.Z)
     rename both snaps to:
     CASE_NUMBER.VIO1.source.snap.pax.Z
     CASE_NUMBER.VIO2.source.snap.pax.Z

6) Both VIO snaps from destination (target) site using:
   $ snap ( data in /source/padmin/snap.pax.Z)
     rename both snaps to:
      CASE_NUMBER.VIO1.target.snap.pax.Z
      CASE_NUMBER.VIO2.target.snap.pax.Z

7) devscan from both destibation (target) using:
   Download devscan tool from below link and install on both VIO on target site. Note: target=destination.
   Link to devscan tool:  devscan 

    devscan -tf -n  [Inactive_wwpns] CASE_NUMBER.devscan.inactive_wwpn.VIO1.target.out
    devscan -tf -n  [Inactive_wwpns] CASE_NUMBER.devscan.inactive_wwpn.VIO2.target.out

    To simplify, use below script: devscan_vmrmha_vio.sh
    NOTE:  VIO devscan script 

9) Collect a baseline HMC mapping.
        # lshmc -v
        # lshmc -V
        # lssyscfg -r sys -F name  ( to get system_name )
        # lshwres -r virtualio --rsubtype fc -m [system_name] --level lpar 
        # lshwres -r virtualio --rsubtype fc --level lpar -m [system_name] -F lpar_name,lpar_id,slot_num,adapter_type,state,is_required,remote_lpar_id,remote_lpar_name,remote_slot_num,wwpns,topology 
        .
10) For KSYS
	# snap -r
	# rm -fr /tmp/ctsupt /tmp/ibmsupt,
	# mkdir -p /tmp/ibmsupt/testcase/
	# date && time  ksysmgr q vm 2>&1 | tee /tmp/ibmsupt/testcase/$(hostname).ksysmgr.before

11) Provide toplogy of the config including switches.

12) pedbg from HMC if requested
      # pedbg -c -q 4;
        - Say, YES when prompted to collect ctsnap.
          (if there are RSCT problems. ctsnap can hang)
        - File created is
          "HsClogsXXzz2007zzzzzz.zip", created in /dump directory.
        - Copy file to the common directory for FTP to IBM.
          # scp /dump/HSClogsXXzz2007zzzzzz.zip
            @:.
         (NOTE: the  ":."  means keep the same name on remote,
            do not forget the :.  At the end of the line )
     - Rename the file to include the PMR number
        # mv HSClogsXXzz2007zzzzzz.zip CASE_NUMBER.pedbg.zip

13) Provide  logs from the switches if requested
     Switch Logs for Cisco & Brocade:
      - Switch logs ("show tech-support details" for Cisco)
      - 'show tech detail' if more then 1 switch.
      - 'supportshow' collected via CLI using either
         HyperTerm or Putty to collect the output for Brocade.
      - 'supportsave'  as that has additional debug information.
      - How the RSCN ( Registered State Change Notification )
        events are sent when a zoning change is done on the switch?

   Switch Logs for McData:
      - "data collection" from the switch management console, EFCM.
      - any other related to ones described above under Cisco / Brocade.

   Switch Logs using iSCSI TOE:
      - igroup show
      - lun show -m