........................................................................
By Raj Patel: First Created: May 26, 2016
              Rev4:          Jan 01, 2017
              Rev5:          Apr 01, 2017
              Rev6:          May 26, 2017
              Rev7:          Jun 15, 2017

////////////////////////////////////////////////////////
Below for VIOs 2.2.3.0 and higher and HMC 7 and higher.
////////////////////////////////////////////////////////
                                           

Please check using below on the VIO lpar as user oem_setup_env
and this should list the HMC connection indicating RMC state OK.

CAUTION !! WITH ALL COMMANDS IN RED. Rest are safe to run.

From VIO:
=========
$ oem_setup_env
1) # lsrsrc IBM.MCP ( If HMC is not listed, possible RMC problem )
   Resource Persistent Attributes for IBM.MCP
   resource 1:
        HMCName           = "7042CR6*068C51C"
        HMCIPAddr         = "9.3.4.35"

2) #  /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc  

   This should be similar to below with Ipaddress of lpar.
   Management Domain Status: Management Control Points
   I A  0xd6fbc2ad6d84493a  0001  9.3.4.35

From HMC:
=========         
To confirm, also check from HMC using below as user hscroot:
State ACtive<1> with ipaddress indicates you have access to 
VIO and RMC is working.

1) hmc@myhmc:~> lspartition -dlpar
2) hmc@myhmc:~> lssyscfg -r sys -F name <= To get [managed_system]
3) hmc@myhmc:~> lssyscfg -r lpar -m [managed-system] 
                -F name,lpar_id,state,rmc_state,rmc_ipaddr



   ONLY AND ONLY FOR non-SSP nodes you can use below to check RMC 
   and recycle on HMC and VIO lpar.

   # /usr/sbin/rsct/bin/rmcctrl -z    
   # /usr/sbin/rsct/bin/rmcctrl -A    
   # /usr/sbin/rsct/bin/rmcctrl -p    

   If you need to run above on VIOs with SSP cluster, stop the
   SSP node effected first using:  
   "clstartstop -stop  -n clustername -m node.name.com "




** NEVER RUN BELOW ON VIOs LPAR / node that has CAA/SSP cluster **
** SINCE THIS WILL CREATE A NEW NODE ID and no longer match with SSP Cluster **

# /usr/sbin/rsct/install/bin/recfgct 

NOTE: a)  Rebooting the VIO Server node is a safer option.
          This way, RMC will be restarted correctly. 
      b)  Make sure /var is not full and has at least 50% free space.

** Contact IBM Support for assistance if reboot does not resolve the problem.
   




In cases including SSP cluster, where the existing node id needs to 
be retained, you can use "-s" flag.


Flags

-i  Node_ID
Specifies the node ID that must be used. 
The node ID must contain 9 - 16 hexadecimal characters.

-n
Generates a new node ID. It is the default behavior if no option is specified.

-s
Saves the node ID.


So "recfgct -s" is basically the same as saving the node ID file somewhere else, 
reinstalling the node, then copying the old node ID file back onto the 
reinstalled node, so that it remains the same as before it was reinstalled.

The "-i" option is useful if something happened and you didn't save the 
old node ID file (like if the system crashed and you really had to reinstall), 
but you have a record from other nodes as to what they expect that node ID to be, 
so that you can set it manually based on what you see reported from other nodes.


REFERENCE:
https://www.ibm.com/support/knowledgecenter/SGVKBA_3.2.2/com.ibm.rsct.aixcmds/recfgct.htm




a) Please re-check again from below from both HMC and VIOs 
   if the problem is corrected. 
   
   If not, please provide ctsnap below from VIOs and HMC:

b) From HMC as user root:

   # /usr/sbin/rsct/bin/ctsnap -xrunrpttr

    If you do not have root access, collect pedbg: 


c) From VIOS - snap/ctsnap.
   NOTE: We recommand running $snap from padmin which will also gather ctsnap.
         ( $ snap data will be located in /home/padmin/snap.pax.Z )

   Other option is to run below in case of speed and time.
    - $ oem_setup_env
    - # ctsnap -x runrpttr ( note: $ snap with no flags run from padmin will also gather ctsnap )
    - This will create =>  /tmp/ctsupt/ctsnap*.tar.gz
    - Extract ctsnap and locate file ctsupt/ctsnap_out/rmcdomainstatus.out
 
d) Provide ctsnap from AIX lpar. Data in /tmp/ctsupt
   # /usr/sbin/rsct/bin/ctsnap -xrunrpttr
   
  REF: This covers CAA / SSP condition.  
 
  Diagnostic procedures  
  RMC Details  
  RMC Details - Current Infcenter  
  RMC Test From HMC