This is the most common for many application espcially the ones which uses many resourses and have dedicated servers for them.
We have cognos application deployed on 22.214.171.124 and 126.96.36.199. Websphere is installed on Aix 5.2 and 5.3 for the same. We have a very few JVM’s and have a very good amount of RAM on the servers.Still we see CPU starvation warning under every JVM logs. We get the application from vendoe (IBM) so the point of bugs, leaks etc is ruled out.
We tuned the AIX servers for better performance .We had the min heap size as 128 and max heap size as 1580 still we the GC happening very frequently. We have generic variables defined for JVM’s as: -Xloratio0.2 -Xk20000
When we opened a PMR with IBM,they suggested us to run the dbxtrace on the AIX servers and send them the must gather info.
They found out that arrays with 10s of thousands of this object (below), in total occupying over 50% of our heap. This seems like a leak from IBM standpoint.
org/dom4j/tree/DefaultText – 3.7million references
101,665,480 (8%)  5 array of java/lang/Object 0xa09982c8
101,631,720 (8%)  4 org/dom4j/tree/DefaultElement 0xa099eaa0 <— leak suspect
101,631,408 (8%)  1 java/util/ArrayList 0xa099ea48
101,631,384 (8%) [56,224] 12,008 array of java/lang/Object 0x93330858
24,888 (0%)  4 org/dom4j/tree/DefaultElement 0xbd8cc9b8
24,872 (0%)  4 org/dom4j/tree/DefaultElement 0xb28dd438
24,856 (0%)  4 org/dom4j/tree/DefaultElement 0xbd726d80
24,608 (0%)  1 java/util/ArrayList 0xbd726d28
216 (0%)  1 java/util/ArrayList 0xbd726d68
In some arrays, this org/dom4j/tree… Object is about 20K, sometimes it is double that. But in each array, there are thousands of these, which add up to quite a large amount of memory usage.
There was still 150MB free on our heap, but this was unusable due to being marked for large objects.
But when we approached our application team .. they inturn approached their vendor (IBM again) said that this is the standard behaviour of the app and its not a legitimate leak .. so we are still battling to solve the issue.