APAR status |
Closed as program error.
| Error description
InvalidTokenException resulting in Authentication failure
is caused by concurrent accessing of the MessageDigest object.
.
It appears that the failing user is not getting an LTPA Token
because
he is not even being looked up in LDAP. The failing user's uid
is
T0011JM, and his DN is
daimlerchryslerDGID=0000000011,ou=people,o=dcx.com. Notice, his
DN does
not show up in our LDAP Access Log -->
Here is the unsuccessful LoadRunner trace for this user --> Here
is one
for a good user for comparison -->
From the WAS side of things, we see the following in our
standard out:
.
smtia: sm_user header: T0011JM
smtai: sm_user header: sm_user, value: T0011JM
smtia: User Name: T0011JM
addOtherHeaders - entry.
addOtherHeaders - looking for entry header.1
addOtherHeaders - placing sm_serversessionspec,
dKmSi17f1Km/n8pGm5COHRXommM6JNoC
aRmF2wtuB+kEl+dZxBzWgvZIDvrQb4EPKSvuhu4fwubV6pId/NStNtBUrp/pj9qi
Rm+b8egM
pSWyp9JT
m36PH/HLRVzgxM6EUlyV+8s2G2v5nqfmjQgHKgn3CcekVSdIVncUZwmw8F0hrLWt
gYv2l5Yz
GgEoYnQo
VP0b8nis4SR5/uRmv2ijDWYuGjRkwGSIf7thjk9S1tteRVCxirgiigLIgZ89JBIw
HagRxPLP
Wh6tqtlF
alGDyRQ2qkiJ++UcDZ6Bqzt9x/0Qsq17ooeaF+LTj1xF6R1+x2JuizWPFFa3Kcx3
9cagVblb
bk6pal+u
KS6uuPoSfYUeQhLsBhnZkQ== into Hashtable.
addOtherHeaders - looking for entry header.2
addOtherHeaders - placing sm_serversessionid,
jpUTZik/S5cZkXCD8uHexjp/MC8= into
Hashtable.
addOtherHeaders - looking for entry header.3
addOtherHeaders - MRE for entry header.3; exit.
addOtherHeaders: cookie number: name : value
0 : SMCHALLENGE : YES
addOtherHeaders - looking for entry cookie.1
addOtherHeaders - looking for entry cookie.2
addOtherHeaders - MRE for entry cookie.2; exit.
%%%% login session null
[01.12.06 11:14:45:406 EST] 505cb325 IExtendedSecu A 2001.12.06
11:14:45.406 Pri
ncipalAuthenticatorImpl validate IBM WebSphere Security 0, 0,
org.omg.CORBA.IN
TERNAL: ELTPA F C A
*effe236c5c8b11c7:46d2b306:ea9ce3bc36:-7ff7 Q10.10.10.4:standard out:.smtia: sm_user header: T0011JMsmtai: sm_user header: sm_user, value: T0011JMsmtia: User Name: T0011JMaddOtherHeaders - entry.addOtherHeaders - looking for entry header.1addOtherHeaders - placing sm_serversessionspec,dKmSi17f1Km/n8pGm5COHRXommM6JNoCaRmF2wtuB+kEl+dZxBzWgvZIDvrQb4EPKSvuhu4fwubV6pId/NStNtBUrp/pj9qiRm+b8egMpSWyp9JTm36PH/HLRVzgxM6EUlyV+8s2G2v5nqfmjQgHKgn3CcekVSdIVncUZwmw8F0hrLWtgYv2l5YzGgEoYnQoVP0b8nis4SR5/uRmv2ijDWYuGjRkwGSIf7thjk9S1tteRVCxirgiigLIgZ89JBIwHagRxPLPWh6tqtlFalGDyRQ2qkiJ++UcDZ6Bqzt9x/0Qsq17ooeaF+LTj1xF6R1+x2JuizWPFFa3Kcx39cagVblbbk6pal+uKS6uuPoSfYUeQhLsBhnZkQ== into Hashtable.addOtherHeaders - looking for entry header.2addOtherHeaders - placing sm_serversessionid,jpUTZik/S5cZkXCD8uHexjp/MC8= intoHashtable.addOtherHeaders - looking for entry header.3addOtherHeaders - MRE for entry header.3; exit.addOtherHeaders: cookie number: name : value0 : SMCHALLENGE : YESaddOtherHeaders - looking for entry cookie.1addOtherHeaders - looking for entry cookie.2addOtherHeaders - MRE for entry cookie.2; exit.%%%% login session null[01.12.06 11:14:45:406 EST], 505cb325 IExtendedSecu A 2001.12.0611:14:45.406 PrincipalAuthenticatorImpl validate IBM WebSphere Security 0, 0,org.omg.CORBA.INTERNAL: ELTPA F C A
32898 minor code: 0 completed: No
[01.12.06 11:14:45:542 EST] 505cb325 WebCollaborat A
Authentication
failed
.
And in the tracing of the app server, we find:*effe236c5c8b11c7:46d2b306:ea9ce3bc36:-7ff7 Q10.10.10.4:32898 minor code: 0 completed: No[01.12.06 11:14:45:542 EST], 505cb325 WebCollaborat AAuthenticationfailed.
.
[01.12.06 11:14:45:406 EST] 505cb325 IExtendedSecu A 2001.12.06
11:14:45.406 PrincipalAuthenticatorImpl validate IBM WebSphere
Security
0, 0, org.omg.CORBA.INTERNAL:And in the tracing of the app server, we find:.[01.12.06 11:14:45:406 EST], 505cb325 IExtendedSecu A 2001.12.0611:14:45.406 PrincipalAuthenticatorImpl validate IBM WebSphereSecurity
LTPA *effe236c5c8b11c7:46d2b306:ea9ce3bc36:-7ff7
minor code: 0 completed: No
[01.12.06 11:14:45:408 EST] 505cb325 LTPAValidatio D Validation
failed
for the LTPA token
[01.12.06 11:14:45:408 EST] 505cb325 SecurityColla D resumed the
suspended transaction
[01.12.06 11:14:45:408 EST] 505cb325 WebAuthentica < validate:0, 0, org.omg.CORBA.INTERNAL:LTPA *effe236c5c8b11c7:46d2b306:ea9ce3bc36:-7ff7minor code: 0 completed: No[01.12.06 11:14:45:408 EST], 505cb325 LTPAValidatio D Validationfailedfor the LTPA token[01.12.06 11:14:45:408 EST], 505cb325 SecurityColla D resumed thesuspended transaction[01.12.06 11:14:45:408 EST], 505cb325 WebAuthentica<
LTPA
token validation failed
.
And from the admin server, we see:validate:LTPAtoken validation failed.
.
[01.12.06 11:14:44:926 EST] 81c88a56 SecurityServe >
getLTPATokenCache
[01.12.06 11:14:44:926 EST] 81c88a56 SecurityServe <
getLTPATokenCache
[01.12.06 11:14:44:926 EST] 81c88a56 LTPATokenCach >
getCredential
[01.12.06 11:14:44:926 EST] 81c88a56 SecurityColla D suspended
current transaction
[01.12.06 11:14:44:926 EST] 81c88a56 LTPATokenCach > update
[01.12.06 11:14:44:926 EST] 81c88a56 LTPAServerObj > validate
[01.12.06 11:14:44:931 EST] 81c88a56 LTPAServerObj < validate:And from the admin server, we see:.[01.12.06 11:14:44:926 EST], 81c88a56 SecurityServe>getLTPATokenCache[01.12.06 11:14:44:926 EST], 81c88a56 SecurityServegetCredential[01.12.06 11:14:44:926 EST], 81c88a56 SecurityColla D suspendedcurrent transaction[01.12.06 11:14:44:926 EST], 81c88a56 LTPATokenCach>, update[01.12.06 11:14:44:926 EST], 81c88a56 LTPAServerObj>, validate[01.12.06 11:14:44:931 EST], 81c88a56 LTPAServerObj<
token not valid
[01.12.06 11:14:44:931 EST] 81c88a56 LTPATokenCach D
InvalidTokenException from LTPAServerObject.validate method
[01.12.06 11:14:44:931 EST] 81c88a56 LTPATokenCach D update
.
com.ibm.WebSphereSecurity.InvalidTokenException at
com.ibm.ejs.security.ltpa.LTPAServerObject.validate(LTPAServerOb
ject.jav a:175) at
com.ibm.ejs.security.util.LTPATokenCache.update(LTPATokenCache.j
ava:71)
at com.ibm.ejs.security.util.Cache.get(Cache.java:114) at
com.ibm.ejs.security.util.LTPATokenCache.getCredential(LTPAToken
Cache.ja va:41) at
com.ibm.ejs.security.SecurityServerBean.validate(SecurityServerB
ean.java :204) at
com.ibm.ejs.security.EJSRemoteSecurityServer.validate(EJSRemoteS
ecurityS erver.java:158) at
com.ibm.ejs.security._SecurityServer_BaseStub.validate(_Security
Server_B aseStub.java:874) at
com.ibm.ejs.security._SecurityServer_Stub.validate(_SecurityServ
er_Stub. java:218) at
com.ibm.WebSphereSecurityImpl.SecurityServerImpl.validateCredent
ialToken (SecurityServerImpl.java:75) at
com.ibm.WebSphereSecurity._SecurityServerImplBase._invoke(_Secur
ityServe rImplBase.java:85) at
com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:3
68) at com.ibm.CORBA.iiop.ORB.process(ORB.java(Compiled Code))
at
com.ibm.CORBA.iiop.WorkerThread.run(WorkerThread.java(Compiled
Code)) at
com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run(ThreadPool.java:validate:token not valid[01.12.06 11:14:44:931 EST], 81c88a56 LTPATokenCach DInvalidTokenException from LTPAServerObject.validate method[01.12.06 11:14:44:931 EST], 81c88a56 LTPATokenCach D update.com.ibm.WebSphereSecurity.InvalidTokenException atcom.ibm.ejs.security.ltpa.LTPAServerObject.validate(LTPAServerObject.jav a:175) atcom.ibm.ejs.security.util.LTPATokenCache.update(LTPATokenCache.java:71)at com.ibm.ejs.security.util.Cache.get(Cache.java:114) atcom.ibm.ejs.security.util.LTPATokenCache.getCredential(LTPATokenCache.ja va:41) atcom.ibm.ejs.security.SecurityServerBean.validate(SecurityServerBean.java :204) atcom.ibm.ejs.security.EJSRemoteSecurityServer.validate(EJSRemoteSecurityS erver.java:158) atcom.ibm.ejs.security._SecurityServer_BaseStub.validate(_SecurityServer_B aseStub.java:874) atcom.ibm.ejs.security._SecurityServer_Stub.validate(_SecurityServer_Stub. java:218) atcom.ibm.WebSphereSecurityImpl.SecurityServerImpl.validateCredentialToken (SecurityServerImpl.java:75) atcom.ibm.WebSphereSecurity._SecurityServerImplBase._invoke(_SecurityServe rImplBase.java:85) atcom.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:368) at com.ibm.CORBA.iiop.ORB.process(ORB.java(Compiled Code))atcom.ibm.CORBA.iiop.WorkerThread.run(WorkerThread.java(CompiledCode)) at
535) .
[01.12.06 11:14:44:979 EST] 81c88a56 SecurityServe D Exception
from LTPATokenCache
.
com.ibm.ejs.security.util.CacheException at
com.ibm.ejs.security.util.LTPATokenCache.update(LTPATokenCache.j
ava:79)
at com.ibm.ejs.security.util.Cache.get(Cache.java:114) at
com.ibm.ejs.security.util.LTPATokenCache.getCredential(LTPAToken
Cache.ja va:41) at
com.ibm.ejs.security.SecurityServerBean.validate(SecurityServerB
ean.java :204) at
com.ibm.ejs.security.EJSRemoteSecurityServer.validate(EJSRemoteS
ecurityS erver.java:158) at
com.ibm.ejs.security._SecurityServer_BaseStub.validate(_Security
Server_B aseStub.java:874) at
com.ibm.ejs.security._SecurityServer_Stub.validate(_SecurityServ
er_Stub. java:218) at
com.ibm.WebSphereSecurityImpl.SecurityServerImpl.validateCredent
ialToken (SecurityServerImpl.java:75) at
com.ibm.WebSphereSecurity._SecurityServerImplBase._invoke(_Secur
ityServe rImplBase.java:85) at
com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:3
68)
at com.ibm.CORBA.iiop.ORB.process(ORB.java(Compiled Code)) at
com.ibm.CORBA.iiop.WorkerThread.run(WorkerThread.java(Compiled
Code)) at
com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run(ThreadPool.java:com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run(ThreadPool.java:535) .[01.12.06 11:14:44:979 EST], 81c88a56 SecurityServe D Exceptionfrom LTPATokenCache.com.ibm.ejs.security.util.CacheException atcom.ibm.ejs.security.util.LTPATokenCache.update(LTPATokenCache.java:79)at com.ibm.ejs.security.util.Cache.get(Cache.java:114) atcom.ibm.ejs.security.util.LTPATokenCache.getCredential(LTPATokenCache.ja va:41) atcom.ibm.ejs.security.SecurityServerBean.validate(SecurityServerBean.java :204) atcom.ibm.ejs.security.EJSRemoteSecurityServer.validate(EJSRemoteSecurityS erver.java:158) atcom.ibm.ejs.security._SecurityServer_BaseStub.validate(_SecurityServer_B aseStub.java:874) atcom.ibm.ejs.security._SecurityServer_Stub.validate(_SecurityServer_Stub. java:218) atcom.ibm.WebSphereSecurityImpl.SecurityServerImpl.validateCredentialToken (SecurityServerImpl.java:75) atcom.ibm.WebSphereSecurity._SecurityServerImplBase._invoke(_SecurityServe rImplBase.java:85) atcom.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:368)at com.ibm.CORBA.iiop.ORB.process(ORB.java(Compiled Code)) atcom.ibm.CORBA.iiop.WorkerThread.run(WorkerThread.java(CompiledCode)) atcom.ibm.ejs
535)
.
[01.12.06 11:14:45:004 EST] 81c88a56 SecurityServe D Credential
returned from LTPATokenCache is NULL .oa.pool.ThreadPool$PooledThread.run(ThreadPool.java:535).[01.12.06 11:14:45:004 EST], 81c88a56 SecurityServe D Credentialreturned from LTPATokenCache is NULL Local fix
We have found out that the cause is due to concurrent access to
a static MessageDigest object
in our codes. If multiple threads access the MessageDigest
object concurrently, one thread
may incorrectly compute digest, and either create an invalid
LtpaToken, or unable to verify a
LtpaToken.
.
Note that digest objects can compute only one digest. So that in
order to compute intermediate
digests, a caller should retain a handle onto the digest object,
and clone it for each digest
to be computed, leaving the orginal digest untouched.
.
After modify our codes to using clone technique(instead of
static object, or instance object),
stress testing by loadrunner performed by portal service
indicated the problem has been resolved,
and stress testing aslo indicated no significant performance. Problem summary
****************************************************************
* USERS AFFECTED: All WebSphere Application Server users who *
* use LTPA as authentication mechanism *
****************************************************************
* PROBLEM DESCRIPTION: Under heavy load, some user may not *
* be authenticated. The error message *
* in admin server security trace show *
* token invalidation failed. *
****************************************************************
* RECOMMENDATION: Apply this efix. *
****************************************************************
Under heavy load, some users may not be authenticated due to
token invalidation failure. The problem occurs randomly. Problem conclusion
The token invalidation failure is caused by concurrently
accessing MessageDigest object. Temporary fix
PQ55804-354.jar Comments
APAR information | APAR number | PQ55804 | Reported component name | WAS ADVANCED AI | Reported component ID | 5648C8400 | Reported release | 350 | Status | CLOSED PER | PE | NoPE | HIPER | NoHIPER | Submitted date | 2001-12-12 | Closed date | 2001-12-14 | Last modified date | 2001-12-14 |
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:APAR is sysrouted FROM one or more of the following:
Modules/Macros APAR is sysrouted TO one or more of the following:Modules/Macros
|
Fix information |
Fixed component name | WAS ADVANCED AI | Fixed component ID | 5648C8400 |
Applicable component levels | R350 PSY | UP |
|