Java process - RSS/VSS are growing unstopped

oznevo

Hi,

We have a weblogic 9.2 server with Java1.5.0.16 on RHEL5.3 that we deploy on it a web service and an Alfresco content management system.

We were running it fine for ~3 years on HP-UX i11.23 and a month ago we moved to Linux RH5.3 and from time to time (it happened 3 times) we noticed that the process is starting to use more and more memory until all the memory and swap on the machine ends.

The process still works fine and all the log files looks normal (as if nothing happened) including GC log.

Glance for process ID 25450:

B0000A Glance C.04.70.000 06:54:05 supra2 x86_64 Current Avg High

CPU Util SU | 2% 2% 2%

Disk Util D D | 97% 97% 97%

Mem Util U U | 98% 98% 98%

Swap Util U U | 60% 60% 60%

Resources PID: 25450, java PPID: 25394 euid: 664 User:afspr04

CPU Usage (util): 5.40 Total RSS : 40.6gb

User CPU : 3.60 Text VSS : 56kb

System CPU : 1.80 Data VSS : 66.1gb

Priority : 15 Stack VSS : 2.0mb

Nice Value : 0 Total VSS : 66.5gb

Blocked On : SLEEP

Major Faults : 235

Minor Faults : 164

Processor : 1

Argv1: weblogic.Server

Cmd : /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -Doracle.net.tns_admin=/etc -server -javaagent:/opt/MercuryDiagn

ostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -Dprobe.id=supra2_afspr04_dmcms_ear_p4 -Dprobe.group=CMS_SERVER -D

points.file.name=/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/etc/supra2_afspr04_dmcms_ear_p4 -Dcom.wily.introsco

pe.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -X

X:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Xnoclassg

c -Xloggc:logs/gc.log -Doracle.net.tns_admin=/etc -Dweblogic.Stderr=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.l

og -Dweblogic.Stdout=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.log -Damdocs.system.home=/app/afspr04/dmcms_ear_

p4/properties/jesi -Damdocs.messageHandling.home=/app/afspr04/dmcms_ear_p4/properties/jesi -Djesi.config.loader=amdocs.

ecommerce.esi.utils.config.InterfaceConfigXPathLoader -Damdocs.uams.config.resource=config/mvc/ldap ...

pmap shows the big allocation as anonymous

pmap (sorted by the big once):

25450: /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -Doracle.net.tns_admin=/etc -server -javaagent:/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -Dprobe.id=supra2_afspr04_dmcms_ear_p4 -Dprobe.group=CMS_SERVER -Dpoints.file.name=/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/etc/supra2_afspr04_dmcms_ear_p4 -Dcom.wily.introscope.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -XX:SurvivorRatio=4 -XX:TargetSurvivo

00002ab0f8000000 10518548 rwx-- [anon]

00002ab798009000 8388612 rwx-- [anon]

000000005fcce000 8038976 rwx-- [anon]

00002aac7aab0000 7602176 rwx-- [anon]

00002aaf74000000 5259284 rwx-- [anon]

00002ab688000000 4194308 rwx-- [anon]

00002aae4b930000 1684124 rwx-- [anon]

00002aab80000000 1314836 rwx-- [anon]

00002aab20000000 655376 rwx-- [anon]

00002aac28000000 532488 rwx-- [anon]

00002aac50000000 524292 rwx-- [anon]

00002aaaec000000 327696 rwx-- [anon]

00002aaad8000000 131088 rwx-- [anon]

00002ab658000000 131060 rwx-- [anon]

00002ab0dc000000 131044 rwx-- [anon]

00002aaacc2f5000 114708 rwx-- [anon]

...

total 69733292K

Have anyone encountered something similar?

Thanks,

Oz