JVM’s couldn’t start through console

I had a strange issue on my prod and dev env’s … we were not able to start the JVM’s through console. No changes are made to the JVM configurations but suddenly JVM’s are not starting through console/wsadmin script after the weekly restart.

We have seen these errors in nodeagent log and dmgr systemout.log

Node Agent Log
[8/31/09 9:38:05:828 CDT] 00000079 AdminHelper   A   ADMN1000I: An attempt is made to launch CU01-Ar-0230 on node NU01-0230. (User ID = 636/)
[8/31/09 9:38:08:881 CDT] 00000079 NodeAgent     W   ADML0011E: The server launched, but failed initialization. See the server log files for failure information.

DMGR SystemOut.log

8/30/09 23:25:32:777 CDT] 000002cd MBeanHelper   E   Could not invoke an operation on object: WebSphere:platform=common,cell=00230,version=6.0.2.19,name=NodeAgent,mbeanIdentifier=NodeAgent,type=NodeAgent,node=NU01-00230,process=nodeagent because of an mbean exception: com.ibm.websphere.management.exception.AdminException: Process failed to launch: A11-0230
[8/30/09 23:25:59:399 CDT] 000002cd MBeanHelper   E   Could not invoke an operation on object: WebSphere:platform=common,cell=00230,version=6.0.2.19,name=NodeAgent,mbeanIdentifier=NodeAgent,type=NodeAgent,node=NU01-0230,process=nodeagent because of an mbean exception: com.ibm.websphere.management.exception.AdminException: Process failed to launch: A11-0230

We did all the below:

• cleared /server/logs
• cleared /wstemp
• cleared /Tranlog
• cleared /config/temp/
• cleared ffdc logs under dmgr/logs
• verified if there are any port conflicts
• verified hostname in dmgr/serverindex.xml and plugin-cfg.xml
• After deleting everything manually sync’d the nodes

But nothing worked.We raised PMR with IBM and still they couldnt help.

So we took a big step to remove all the configurations made for JVM’s and started added one by one ..

We came to know that some variables defined under LIBPATH  were giving us issues. We removed the LIBPATH and started appending each variable and then found out which one was causing the issue …

Well that solved our prob 🙂

~ by moturu on September 1, 2009.