Last updated: July 06, 2012 - Raj Patel ( rajpat@us.ibm.com )

Recovering nodes When "cluster -status" shows nodes are  "DOWN"
before using addnode or rmnode can be used sucessfully.

OUTPUT WHEN one of the NODE is "DOWN" - Example below in which vio1 is DOWN.
============================================================================

    $ cluster -status -clustername [cluster-name]
    Cluster Name         State
    ssp_cluster1        DEGRADED

      Node Name        MTM           Partition Num  State  Pool State
      vio1                               0  DOWN
      vio2     8202-E4B02067FECP         1  OK     OK

Problem:
========

    $ cluster -addnode -clustername [cluster-name]  -hostname vio1.sspgroup.com
    Node is already a cluster member
    vio1.sspgroup.com

    Command did not complete.


Solution: 
=========

    Please try below Steps:

    a) On vio2: export VIO_API_DEBUG=7

    b) On vio2: cluster -list

    c) Check if vio_daemon will list the DBN 
      ( it should since VIO2 node is UP )
      $ lssrc -ls vio_daemon 

    d) On vio2: cluster -status -clustername [cluster-name]

    e) On vio2: cluster -sync -clustername [cluster-name]

    f) On vio1: check caa daemons & ctrmc
      $ lssrc -s ctrmc; lssrc -g caa

      1) Pool daemons were not started/running
        1a) On vio1: clstartstop to stop pool
            $ clstartstop -stop  -n  [cluster-name] -m vio1
            May report:
            clmain.c cl_startstop 2955 Local node has not been STOPPED.

        1b) On vio1: clstartstop to start pool
            $ clstartstop -start -n  [cluster-name] -m vio1
            May report:
            clmain.c cl_startstop 2955 Local node has not been STOPPED.

        1b) On vio1: Wait up to 10 minutes for pool to be started.

        1c) On vio2: cluster -sync -clustername [cluster-name] ( just to be sure )

      2) Pool daemons were running, but pool didn't start
        2a) On vio2: remove vio1 node from vio2
            $ cluster -rmnode -clustername [cluster-name] -hostnane
            vio1.sspgroup.com
            May report:
            Partition vio1 has been removed from the [cluster-name] cluster

        2b) On vio2: check cluster removed vio1
            $ cluster -status -clustername [cluster-name]
            Cluster Name         State
            [cluster-name]        OK

            Node Name        MTM           Partition Num  State  Pool State
            vio2     8202-E4B02067FECP         1  OK     OK

        2c) On vio2: add vio1 back to cluster
            $ cluster -addnode -clustername [cluster-name] -hostname
            vio1.sspgroup.com

            Partition vio1 has been added to the [cluster-name] cluster.

       REF: 
       http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hcgl/clstartstop.htm

Side Notes:
Run "startsrc -g rsct" , "startsrc -g caa" if after node is rebooted and
necessary to re-start these services.