Raj Patel: August 28, 2020 - First Edition.
- Revised December 28, 2020 - Minor changes and VIOs requirements.
- Revised June 04, 2021 - Breakup for VMRM HA and VMRM DR (This will cover VMRM DR)
- Revised July 02, 2021 - Include SVC storage logs
- Revised August 03, 2021 - devscan checks for VMRMDR
- Revised August 18, 2021 - added script to simplify devscan collection on AIX VMs and VIOs
- Revised April 04, 2023 - remove pipe to lshwres command
Steps includes: VM for AIX, Linux and IBMi and npiv_val
NOTE: VIOs 2.2.X reached end of support as of 2020. Please be sure to have supported VIO 3.X version and same VIO versions on HOME and BACKUP sites
The data collection and test should include these machines:
a) 2 VIOS from HOME site
b) 2 VIOS from BACKUP site
c) 1 AIX VM snap for failed VM. For Linux (sosreports). For IBMi (Initiate a Resource Dump)
d) KSYS
e) pedbg from HMC
f) devscan output
g) details on the storage and fabric switches with make and model and any ISL setup.
h) SVC storage logs. For other storage contact storage vendor.
The steps are:
1) Confirm that KSYS, VM, and VIOs are on same time zone and if not what timezone they are on and if the times are in sync.
2) For AIX Failed VM:
devscan and VM snap from failed VM using:
Download devscan tool from below link and install on VM that failed
Link to devscan tool: devscan
Collect snap and devscan from failed VM and run it in order provided.
a) snap -r
c) devscan -tf -n [active_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.vm.active_wwpn.out (For all active_wwpn)
d) devscan -tf -n [inactive_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.vm.inactive_wwpn.out (For all inactive_wwpn)
e) snap -ac ( data in /tmp/ibmsupt/snap.pax.Z)
rename to CASE_NUMBER.VM.snap.pax.Z and upload CASE_NUMBER.VM.snap.pax.Z
NOTE: On AIX you can get listing of all Active WWPNs using details from below link:
Active WWPN of Virtual Fiber Channel Adapter
NOTE: STEPS ABOVE b) to d) can be replaces if using script: script for devscan on AIX VM
3) If the failed VM is Linux use below:
Refer to sosreport: sosreport
# 'multipath -l" > multipath.after.out
# sosreports
4) If this is an IBMi Initiate a Resource Dump from the HMC for Classic HMC
Link to resource dump Classical HMC: Resource Dump using Classic HMC
.
If this is an IBMi Initiate a Resource Dump from the HMC for Enhanced HMC
Link to resource dump Enhanced HMC: Resource Dump using Enhanced HMC
5) Both VIO snaps from HOME site using:
$ snap ( data in /home/padmin/snap.pax.Z)
rename both snaps to:
CASE_NUMBER.VIO1.home.snap.pax.Z
CASE_NUMBER.VIO2.home.snap.pax.Z
6) devscan from both VIO on HOME site using:
Download devscan tool from below link and install on both VIO on HOME site
Link to devscan tool: devscan
a) devscan -tf -n [active_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO1.home.active_wwpn.out (For all active_wwpn)
b) devscan -tf -n [inactive_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO1.home.inactive_wwpn.out (For all inactive_wwpn)
c) devscan -tf -n [active_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO2.home.active_wwpn.out (For all active_wwpn)
d) devscan -tf -n [inactive_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO2.home.inactive_wwpn.out (For all inactive_wwpn)
NOTE: STEPS ABOVE a) to d) can be replaces if using script: script for devscan on VIOs
7) Both VIO snaps from DR site using:
$ snap ( data in /home/padmin/snap.pax.Z)
rename both snaps to:
CASE_NUMBER.VIO1.DR.snap.pax.Z
CASE_NUMBER.VIO2.DR.snap.pax.Z
8) devscan from both DR site using:
Download devscan tool from below link and install on both VIO on DR site
Link to devscan tool: devscan
a) devscan -tf -n [active_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO1.DR.active_wwpn.out (For all active_wwpn)
b) devscan -tf -n [inactive_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO1.DR.inactive_wwpn.out (For all inactive_wwpn)
c) devscan -tf -n [active_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO2.DR.active_wwpn.out (For all active_wwpn)
d) devscan -tf -n [inactive_wwpn] -o /tmp/ibmsupt/testcase/devscan.tfn.VIO2.DR.inactive_wwpn.out (For all inactive_wwpn)
NOTE: STEPS ABOVE a) to d) can be replaces if using script: script for devscan on VIOs
9) Collect a baseline HMC mapping.
# lshmc -v
# lshmc -V
# lssyscfg -r sys -F name ( to get system_name )
# lshwres -r virtualio --rsubtype fc -m [system_name] --level lpar
# lshwres -r virtualio --rsubtype fc --level lpar -m [system_name] -F lpar_name,lpar_id,slot_num,adapter_type,state,is_required,remote_lpar_id,remote_lpar_name,remote_slot_num,wwpns,topology
.
10) For KSYS
# snap -r
# rm -fr /tmp/ctsupt /tmp/ibmsupt,
# mkdir -p /tmp/ibmsupt/testcase/
# date && time ksysmgr q vm 2>&1 | tee /tmp/ibmsupt/testcase/$(hostname).ksysmgr.before
11) Provide toplogy of the config including switches.
12) pedbg from HMC if requested
# pedbg -c -q 4;
- Say, YES when prompted to collect ctsnap.
(if there are RSCT problems. ctsnap can hang)
- File created is
"HsClogsXXzz2007zzzzzz.zip", created in /dump directory.
- Copy file to the common directory for FTP to IBM.
# scp /dump/HSClogsXXzz2007zzzzzz.zip
@:.
(NOTE: the ":." means keep the same name on remote,
do not forget the :. At the end of the line )
- Rename the file to include the PMR number
# mv HSClogsXXzz2007zzzzzz.zip CASE_NUMBER.pedbg.zip
13) Provide logs from the switches if requested
Switch Logs for Cisco & Brocade:
- Switch logs ("show tech-support details" for Cisco)
- 'show tech detail' if more then 1 switch.
- 'supportshow' collected via CLI using either
HyperTerm or Putty to collect the output for Brocade.
- 'supportsave' as that has additional debug information.
- How the RSCN ( Registered State Change Notification )
events are sent when a zoning change is done on the switch?
Switch Logs for McData:
- "data collection" from the switch management console, EFCM.
- any other related to ones described above under Cisco / Brocade.
Switch Logs using iSCSI TOE:
- igroup show
- lun show -m
14) SVC storage logs;
- Typically, storage support requests an "option 4" support log collection (also known as Live Dumps).
- For VMR troubleshooting, really an "option 1" (base) logs I'd expect would be more than sufficient.
- In the SVC/Flash GUI, the option to collect support logs has a radio button to choose the option #
CLI for a level 1: svc_snap -gui1
CLI for a level 4 is multi-step. Need to engage SVC storage support if needed.