CPU Starvation

This is the most common for many application espcially the ones which uses many resourses and have dedicated servers for them.

We have cognos application deployed on 6.0.2.19 and 6.1.0.23. Websphere is installed on Aix 5.2 and 5.3 for the same. We have a very few JVM’s and have a very good amount of RAM on the servers.Still we see CPU starvation warning under every JVM logs. We get the application from vendoe (IBM) so the point of bugs, leaks etc is ruled out.

We tuned the AIX servers for better performance .We had the min heap size as 128 and max heap size as 1580 still we the GC happening very frequently. We have generic variables defined for JVM’s as: -Xloratio0.2 -Xk20000

When we opened a PMR with IBM,they suggested us to run the dbxtrace on the AIX servers and send them the must gather info.

They found out that arrays with 10s of thousands of this object (below), in total occupying over 50% of our heap.  This seems like a leak from IBM standpoint.

org/dom4j/tree/DefaultText   –    3.7million references

101,665,480 (8%) [40] 5 array of java/lang/Object 0xa09982c8
101,631,720 (8%) [32] 4 org/dom4j/tree/DefaultElement 0xa099eaa0  <— leak suspect
101,631,408 (8%) [24] 1 java/util/ArrayList 0xa099ea48
101,631,384 (8%) [56,224] 12,008 array of java/lang/Object 0x93330858
24,888 (0%) [32] 4 org/dom4j/tree/DefaultElement 0xbd8cc9b8
24,872 (0%) [32] 4 org/dom4j/tree/DefaultElement 0xb28dd438
24,856 (0%) [32] 4 org/dom4j/tree/DefaultElement 0xbd726d80
24,608 (0%) [24] 1 java/util/ArrayList 0xbd726d28
216 (0%) [24] 1 java/util/ArrayList 0xbd726d68

In some arrays, this org/dom4j/tree… Object is about 20K, sometimes it is double that.  But in each array, there are thousands of these, which add up to quite a large amount of memory usage.

There was still 150MB free on our heap, but this was unusable due to being marked for large objects.

But when we approached our application team .. they inturn approached their vendor (IBM again) said that this is the standard behaviour of the app and its not a legitimate leak .. so we are still battling to solve the issue.

About these ads

~ by moturu on September 1, 2009.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: