First Written: Raj Patel: Nov 12, 2014
Approval: Dave Dilling , Paul Stimets, Steve Fullerton
Rev15: May 05, 2019
////////////////////////////////////////////////////////////////////////
// LATEST DOCUMENT FOR POWERVC AND HMC DATA COLLECTION //
///////////////////////////////////////////////////////////////////////
>href ="http://www-01.ibm.com/support/docview.wss?uid=nas8N1020618"> Latest PowerVC & HMC Data Collection
/////////////////////////////////////////////////////////////////////////
/// Minimum Requirement Link ( PowerVC 1.2.2 ) - As of Feb 02, 2015 ///
/// Please try to keep current or n-1 level. ///
/////////////////////////////////////////////////////////////////////////
Please check with PowerVC team for changes related to PowerVC level
or HMC levels.
SSVSPA_1.4.1
SSVSPA_1.4.2
/////////////////////////////////////////////////////////////////////////
/// Minimum Requirement Link ( VIOS) - As of Feb 02, 2015 ///
/// Please try to keep current or n-1 level. ///
/////////////////////////////////////////////////////////////////////////
Please check with VIO team. As of May 05, 2019 below are the minimum
requirements.
2.2.6.32 or higher
3.1 or higher
////////////////////////////////////////////////////////////////////
/// Initial Problem Description ///
////////////////////////////////////////////////////////////////////
1) Please provide:
a) Gather time difference between the PowerVC linux server,
HMC, and VIOS server(s) (use "date" command)."
- Are these syncronised using NTP ?
b) Error messsage on:
1) PowerVC ( From GUI )
c) Current levels and fixes on:
1) PowerVC levels and fixes.
2) HMC levels and fixes.
3) VIOs levels and fixes.
////////////////////////////////////////////////////////////////////
/// Initial data collection from PowerVC - Powervc-diag etc ///
////////////////////////////////////////////////////////////////////
1) Run "/usr/bin/powervc-diag" to get powervc logs.
- run as root user
- cd /usr/bin
- powervc-diag
- zip file is created in /tmp ( default ).
- Rename to PMR#.lwi.date.zip
////////////////////////////////////////////////////////////////////
/// Initial data collection from HMC - pedbg ///
////////////////////////////////////////////////////////////////////
1) login into HMC as hscpe
2) run "pedbg -c -q 4"
3) data created.
"HsClogsXXzz2007zzzzzz.zip", created in /dump directory.
- Copy file to the common directory for FTP to IBM.
# scp /dump/HSClogsXXzz2007zzzzzz.zip
@:.
(NOTE: the ":." means keep the same name on remote,
do not forget the :. At the end of the line )
- Rename the file to include the PMR number
# mv HSClogsXXzz2007zzzzzz.zip 17343.b057.c649.pedbg.date.zip
////////////////////////////////////////////////////////////////////
/// Initial data collection from ALL VIOS ///
////////////////////////////////////////////////////////////////////
1. snap from padmin of each node.
Incase of SSP,snap from the primary DBN node and failing node.
$ snap ( creates /home/padmin/snap.pax.Z)
2. Basic CM Database checks on VIOs:
If query fails see if there is any change in status:
$ oem_setup_env
# ps -ef|grep CM/DB
# /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc
# lssrc -ls ctrmc|grep running
# lssrc -g rsct_rm
////////////////////////////////////////////////////////////////////
/// Upload Data ///
////////////////////////////////////////////////////////////////////
1) Upload testcase:
FTP the file to ibm:
ftp testcase.software.ibm.com,
login: anonymous,
passwd: your email address,
ftp> cd /toibm/aix
ftp> bin
ftp> put pmr#.powervc.pax.gz), pmr#.hmc.pax.gz and pmr#.vios.pax.gz
ftp> quit
////////////////////////////////////////////////////////////////////
/// END INITIAL DATA COLLECECTION ///
/// NEXT SECTION FOR RE-CREATE and INCRESING LOGS ///
////////////////////////////////////////////////////////////////////
===================================================================
===================================================================
===================================================================
////////////////////////////////////////////////////////////////////
/// DATA COLLECTION FOR RECREATE ///
////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////
/// VIOS ///
////////////////////////////////////////////////////////////////////
1) Remove or rename /home/ios/logs/viod_CM.log.FATAL
- mv /home/ios/logs/viod_CM.log.FATAL to
/home/ios/logs/viod_CM.log.FATAL.old
*** You can use below script to cleanup - run in oem_setup_env ***
================= CUT ==================
#!/usr/bin/ksh
DATE=$(date +"%m_%d_%Y-%H.%M.%S")
mv viosvc.log.err viosvc.log.err.$DATE
mv viosvc.log viosvc.log.$DATE
mv viod_CM.log viod_CM.log.$DATE
mv viod_CM.log.FATAL viod_CM.log.FATAL.$DATE
mv viod.log viod.log.$DATE
mv vioCmd.log vioCmd.log.$DATE
touch viod_CM.log
touch viod.log
touch viosvc.log
touch vioCmd.log
chown root:system viod_CM.log
chown root:system viod.log
chown root:system viosvc.log
chown root:system vioCmd.log
chmod 644 viod_CM.log
chmod 644 viod.log
chmod 644 viosvc.log
chmod 644 vioCmd.log
================= CUT ==================
////////////////////////////////////////////////////////////////////
/// PowerVC - Powervc-diag ///
////////////////////////////////////////////////////////////////////
A) Please provide:
1) Gather time difference between the PowerVC linux server,
HMC, and VIOS server(s) (use "date" command)."
- Are these syncronised using NTP ?
2) Error messsage on:
a) PowerVC ( GUI )
3) Current / New levels or fixes.
a) PowerVC levels and fixes.
b) HMC levels and fixes.
c) VIOs levels and fixes.
4) VIOs lpar ID, lpar name, and host name the failure occured on.
5) Run "/usr/bin/powervc-diag" to get powervc logs.
- run as root user
- cd /usr/bin
- powervc-diag
- zip file is created in /tmp ( default ).
- Rename to PMR#.lwi.date.zip
////////////////////////////////////////////////////////////////////
/// Data collection from HMC - pedbg ///
////////////////////////////////////////////////////////////////////
1) login into HMC as hscpe
2) run "pedbg -c -q 4"
3) data created.
"HsClogsXXzz2007zzzzzz.zip", created in /dump directory.
- Copy file to the common directory for FTP to IBM.
# scp /dump/HSClogsXXzz2007zzzzzz.zip
@:.
(NOTE: the ":." means keep the same name on remote,
do not forget the :. At the end of the line )
- Rename the file to include the PMR number
# mv HSClogsXXzz2007zzzzzz.zip 17343.b057.c649.pedbg.date.zip
////////////////////////////////////////////////////////////////////
/// Data collection from ALL VIOS ///
////////////////////////////////////////////////////////////////////
1. snap from padmin of each node.
Incase of SSP,snap from the primary DBN node and failing node.
$ snap ( creates /home/padmin/snap.pax.Z)
2. Basic CM Database checks on VIOs:
If query fails see if there is any change in status:
$ oem_setup_env
# ps -ef|grep CM/DB
# /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc
# lssrc -ls ctrmc|grep running
# lssrc -g rsct_rm
////////////////////////////////////////////////////////////////////
/// Data collection from ALL VIOS incase of RSCT problems. ///
/// snap collect rsct data, however, for escalation RSCT team ///
/// prefers below which collects data and formats report. ///
////////////////////////////////////////////////////////////////////
$ oem_setup_env
# ctsnap -x runrpttr
This will create => /tmp/ctsupt/ctsnap*.tar.gz
rename pmr#.vio.ctsnap.pax.Z
////////////////////////////////////////////////////////////////////
/// Upload Data ///
////////////////////////////////////////////////////////////////////
1) Upload testcase:
FTP the file to ibm:
ftp testcase.software.ibm.com,
login: anonymous,
passwd: your email address,
ftp> cd /toibm/aix
ftp> bin
ftp> put pmr#.powervc.pax.gz), pmr#.hmc.pax.gz and pmr#.vios.pax.gz
ftp> quit
//////////////////////////////////////////////////////////////
// Debugging Tips For Support Teams //
//////////////////////////////////////////////////////////////
PowerVC:
1) Provide VIOs lpar ID, lpar name, and host name the failure occured on.
2) Initial log files to start debugging.
- Inside powervc_diag
- /var/log/nova/* first
- compute logs,
- scheduler log, etc
HMC:
1) Provide VIOs lpar ID, lpar name, and host name the failure occured on.
2) Initial HMC log files to start debugging.
FFDC.log ( unzip /dump/pmclogs13288.zip - /dump/var/hsc/log/wlp_pedbg/FFDC.log )
3) ./dump/var/hsc/log/wlp_pedbg/Audit.log
VIOS: Log Files used for debugging.
1) viosvc.log.err
2) viosvc.log
3) viosp.log
4) viod_CM.log
5) vioCmd.log
6) viod_CM.log.FATAL
/////////////////////////////////////////////////////////////////////////
/// IBM Internal Wiki for current ifixes etc
/// If you do not haveaccess please email me with your intranet and GSA id.
/////////////////////////////////////////////////////////////////////////
Internal Wiki
Link to techdoc
Must Gather: PowerVC Data Collecion used by PowerVC Support
Troubleshooting a PowerVC region