Memory Dump Diagnostic for Java
If you think Java application never leaks memory, just run the following java code for 5 minutes and see:
public class MyClass {
static HashSet myContainer = new HashSet();
public void leak(int numObjects) {
for (int i = 0; i < numObjects; ++i) {
String leakingUnit = new String("this is leaking object: " + i);
myContainer.add(leakingUnit);
}
}
public static void main(String[] args) throws Exception {
System.out.print("May I start leaking? (hit enter)");
System.in.read(new byte[4]);
{
MyClass myObj = new MyClass();
myObj.leak(1000000000);
}
System.out.println("oops!!! I have leaked " +
myContainer.size() + " objects!!!!!");
System.out.print("hit enter to exit"); System.in.read();
}
}
|
The above example shows how it is possible to unintentionally hold references to
objects (Strings) causing these objects to live beyond the lifecycle of the creating
object (MyClass), and preventing them from getting garbage collected.
The only effective way to debug a memory leak problem in java is to diagnose the heap dumps
taken over the time interval when the Java application is experience high garbage collecting
activities (usually reflected in 100% CPU usage on a cpu), or the application appears to hang.
Steps for getting heap jumps for WebSphere Application Server or non-WebSphere JVMs.
WebSphere Application Server V4.0, on an AIX system
1. In the administrative console, navigate to and select the default application server.
In the file tree, click WebSphere Administrative Domain > Nodes > localhost >
Application Servers > Default Server.
2. On the JVM Settings tab, click Advanced JVM Settings and enable verbose garbage collection.
3. On the General tab, click Environment, and set both
IBM_HEAPDUMP = true and IBM_HEAPDUMP_OUTOFMEMORY = true.
4. To direct the destination of the heap dump to an alternate directory,
click Environment, and set IBM_HEAPDUMPDIR to the appropriate directory
(that is, \heapdumps).By default, the memory dumps are created in
the \Websphere4\AppServer\bin\ directory.
5. Go to the /usr/WebSphere40/AppServer/bin/ directory and issue the command:
ulimit -f unlimited
so file size problems do not occur when writing the memory dumps.
6. Take a heap dump by issuing this command: kill -3 XXXXX where XXXXX is the process ID.
The heap dump and an accompanying Java core file, which contains information about the
heap dump are created in the ~/WebSphere40/AppServer/bin/ directory, unless specified
otherwise. The heap dump file name looks similar to heapdump28176.1089822616.txt.
The first group of numbers is the process ID, and the second group of numbers is the
system timestamp.
|
WebSphere Application Server V4.0, on a Windows system
1. In the administrative console, navigate to and select the default application server.
In the file tree click WebSphere Administrative Domain > Nodes > localhost >
Application Servers > Default Server.
2. On the General tab, click Environment and input both
IBM_HEAPDUMP = true and IBM_HEAPDUMP_OUTOFMEMORY = true.
3. By default, the memory dumps is created in the ~\Websphere4\AppServer\bin\. directory.
To direct the destination of the heap dump to an alternate directory, click Environment,
and set IBM_HEAPDUMPDIR to the appropriate directory, that is\heapdumps).
4. On the JVM Settings tab, click Advanced JVM Settings and enable garbage collection
verbose mode.
5. Start the application server (Upon starting the application server,
tail the >Default_Server_stdout.log file and note the port number given to the
application server in the DrAdmin line).
6. Invoke a heap dump by opening a command prompt and going to
the ~\Websphere4\AppServer\bin\ directory. Issue the command:
"DrAdmin -serverport XXXX -dumpThreads"
where XXXX is the port number.
The heap dump and an accompanying Java core file that contains information about
the heap dump are created in the \Websphere4\AppServer\bin\ directory,
unless specified as otherwise. The heap dump file looks similar to
"heapdump.20040709.105232.1140.txt". The first group of numbers is the date,
the second group is the time, and the third group is the process ID.
|
WebSphere Application Server V5.x, V6.x on an AIX system
1. In an Internet browser, access the administrative console at
http://hostname:9091/admin.
2. Navigate to: Servers > Application Servers > Server1 (or the name of
the server to get a heap dump) > Process Definition > Environment Entries.
3. Click New.
4. In the name field, input IBM_HEAPDUMP. In the value field, input true".
5. Click OK.
6. Repeat Steps 3 through 5, except set IBM_HEAPDUMP_OUTOFMEMORY to true.
7. Memory dumps are created in the ~/WebSphere/AppServer/ directory by default
(for WebSphere Application Server V6.x, the default directory is:
~/WebSphere/AppServer/profiles/default). To direct the destination of the
heap dump to an alternate directory, go to Environment Entries, click New,
set IBM_HEAPDUMPDIR to the appropriate directory, that is, /heapdumps,
and click OK.
8. Click Save, and in the next screen click Save again.
9. Navigate to: Servers > Application Servers > server1 (or the name of
the server to get a heap dump) > Process Definition > Java Virtual Machine.
10. Select Verbose garbage collection.
11. Click Save, and on the next screen click Save again.
12. Restart the server.
13. Open a command prompt and go to the /WebSphere/AppServer/bin directory.
14. Invoke a heap dump by issuing a kill -3 XXXXX command, where XXXXX is the process ID.
The heap dump and an accompanying Java core file that contains information about
the memory dump are created in the ~/WebSphere40/AppServer/directory, unless an
alternate directory is specified. The heap dump file name looks similar to:
heapdump57128.1090349140.txt. The first group of numbers is the process ID,
and the second group of numbers is the system timestamp.
|
WebSphere Application Server V5.x, V 6.x, run on a Windows system
1. In an Internet browser, access the administrative console at
http://hostname:9091/admin.
2. Navigate to: Servers > Application Servers > Server1 (or the name of
the server to get a heap dump) > Process Definition > Environment Entries
3. Click New.
4. In the name field, input IBM_HEAPDUMP. In the value field, input true.
5. Click OK.
6. Repeat Steps 3 through 5, except set IBM_HEAPDUMP_OUTOFMEMORY to true.
7. To direct the destination of the heap dump to an alternate directory, go to
Environment Entries, click New, set IBM_HEAPDUMPDIR to the appropriate directory,
that is, /heapdumps, and click OK. Memory dumps are created in the
~/WebSphere/AppServer/ directory by default (for WebSphere Application
Server V6.x, the default directory is: ~/WebSphere/AppServer/profiles/default.
8. Click Save, and in the next screen click Save again.
9. Navigate to Servers > Application Servers > Server1 (or the name of
the server to get a heap dump) > Process Definition > Java Virtual Machine
10. Select Verbose garbage collection.
11. Click Save, and on the next screen click Save again.
12. Restart the server.
13. Open a command prompt and go to the ~\WebSphere\AppServer\bin directory.
14. Issue the command: wsadmin. The command creates an interactive session to server1.
15. When you see "wsadmin>, issue the command:
set jvm [$AdminControl completeObjectName type=JVM,*],
which sets the variable jvm with the application server server1.
16. Issue another command: $AdminControl invoke $jvm dumpThreads, which
tells the JVM to start a heap dump.
The heap dump and an accompanying Java core file that contains information
about the memory dump are created in the ~/WebSphere40/AppServer/ directory,
unless an alternate directory is specified. The heap dump file name looks
similar to heapdump.20040630.140939.3944.txt. The first group of numbers is
the date, the second group is the time, and the third group is the process ID.
|
WebSphere Application Server V5.x,V6.x on a Solaris system
1. In an Internet browser, access the administrative console at
http://hostname:9091/admin.
2. Navigate to: Servers > Application Servers > Server1 (or the name of
the server to get a heap dump) > Process Definition > Environment Entries.
3. In General arguments type: -Xrunhprof:depth=0,heap=dump,format=a,thread=n,doe=n
By default, the memory dumps are created in the ~/Websphere/AppServer/directory.
To direct the destination of the heap dump to an alternate directory, add
the HProf argument: file=/heapdumpdir/hprof.txt, where heapdumpdir is the
appropiate directory and hprof.txt is the appropriate file name. If multiple
memory dumps are taken, each is appended to the one hprof.txt file.
4. Select Enable garbage collection verbose mode.
5. Restart the server.
6. Take a heap dump by issuing this command: kill -3 XXXXX, where XXXXX is
the process ID.
The hprof dump is created in the ~/WebSphere/AppServer/ directory and the
file name looks similar to java.hprof.txt, unless specified otherwise.
7. Shut down the application server and then move the hprof dump file.
The hprof dump file is not complete until the application server is
shut down properly.
Attention: Check each hprof dump for two sets of HEAP DUMP BEGIN and
HEAP DUMP END markers. If both markers do not exist for each dump,
then the hprof dump is not complete and cannot be used for analyzing.
During the run of a leaking application, it is necessary to take multiple
memory dumps at separate intervals.
|
WebSphere Application Server V5.1, V6.x, on a z/OS system
Before WebSphere Application Server for z/OS Version 510.200, no easy means
of gathering a memory dump from the WebSphere Application Server address
spaces exists. If you have a version of the JVM 1.4.1 with a build date
later than July, 2004, you might be able to code a servlet that invokes
the heapdump method.
As a result, the procedure to analyze the heap on z/OS systems involves
gathering a SVCDUMP of the address space with the heap that you want to analyze.
Prepare to get a SVCDUMP on a z/OS system.
Before you take the dump:
* Try to run the application with just one SR/Servant.
* It is best to take the memory dump when the bad behavior is occuring,
for example, stop, slow response, and so on.
When done, the basic instructions for taking the dump are as follows:
1. Bring up the application server with GC tracing running only.
2. Type the console command: CD SET,SDUMP,MAXSPACE=2048M.
3. Start your workload.
4. When the workload is running steadily and assuming that the
performance issue is occurring, type the console command: ,999K,BR=ON.
5. After 30 seconds, type the console command: DUMP COMM=([some comment])
6. Answer the WTOR. If you use SDSF, type a slash (/) on the command
line to get to a four-line entry screen:
[xx],ASID=([yy]),SDATA=(RGN,TRT,CSA,NUC,PSA,GRSQ,LPA,SQA,SUM)
where xx is the WTOR reply number and zz is the ASIDX of the address
spaces to dump. In V4.x with the plug-in, either dump the plug-in
address space or the server and control region address spaces,
depending on where the current investigation is focused.
7. Wait 30 seconds after replying to the dump WTOR, and type the
console command: TRACE ST,OFF
Move the dump to your workstation for processing:
1. Use File Transfer Protocol to the host from your workstation
2. Change directory to the name of the dump in quotes, for example,
cd 'my.dump.dataset'
3. Change to binary mode ex. bin .
4. Type the command: get 'my.dump.dataset'. This action can take
30 minutes or so; the minimum dump size noticed is 800M.
|
Windows systems without WebSphere Application Server
1. In the shell where the JVM is going to run, type:
set IBM_HEAPDUMP=true
2. In the shell where the JVM is going to run, type:
set IBM_HEAPDUMP_OUTOFMEMORY=true
3. To direct the destination of the heap dump to an alternate directory, type:
set IBM_HEAPDUMPDIR=/heapdumps,
or the appropriate directory.
Memory dumps are created in the current working directory by default.
4. Enable Verbose GC:
* For IBM Software Developer Kits, add this line to the
JVM command line: -verbose:gc -Xverbosegclog:verbosegc.log
* For the Sun JDK, add this line to the JVM
command line: -Xloggc:verbosegc.log
5. Start the JVM process and do not run the JVM as a background process.
6. In the same shell where the JVM process is started, press Cntrl-Break.
This action produces a memory dump and can also stop the current process.
7. If the current process is stopped, restart the JVM and take the
second memory dump later.
|
AIX systems without WebSphere Application Server
1. In the shell where the JVM is going to run, for the Bourne shell type:
export IBM_HEAPDUMP=true or for CSH:setenv IBM_HEAPDUMP true
2. In the shell where the JVM is going to run, type:
export IBM_HEAPDUMP_OUTOFMEMORY=true
3. To direct the destination of the heap dump to an alternate directory,
type: set IBM_HEAPDUMPDIR=/heapdumps, or the appropriate directory.
Memory dumps are created in the current working directory by default.
4. Enable verbose GC by adding this line to the JVM command line:
-verbose:gc -Xverbosegclog:verbosegc.log
5. Start the JVM process.
6. Issue the command: kill -3 XXXXX, where XXXX is the process ID.
|
Solaris systems without WebSphere Application Server
1. On a Java command line, add the following parameters to enable HProf:
-Xrunhprof:depth=0,heap=dump,format=a,thread=n,doe=n
2. Enable verbose GC to the verbosegc.log file by adding this line to
the JVM command line: -Xloggc:verbosegc.log
3. Start the JVM process.
4. Issue the command: kill -3 XXXXX, where XXXX is the process ID.
|
ISeries IBM SDK (Version V5 R3) without WebSphere Application Server
1. In an x5250 session, type strsst and sign in.
2. Select option 1 - start a service tool.
3. Select option 4 - display/alter/dump.
4. Select option 2 - dump to printer.
5. Select option 2 - licensed internal code (lic) data.
6. Select option 14 - advanced analysis.
7. Scroll down and find JAVAGCINFO, type a 1 next to it, and press Enter.
8. Press Enter to run the javagcinfo macro with no parameters.
9. Press Enter again.
10. Type wrksplf, for which it is useful for a second x5250 session.
11. Use F18 to get to the bottom of the list.
12. Type 5 by your spool file and press Enter to display the file. The spool
file contains a short summary of each of the JVMs on the system.
13. Page down until you find your JVM (The job number, user, and name for each
JVM is located at the beginning of the data for each JVM). Select the hex
string following VM*= on the line with the job number, and so on.
14. Return to the screen to run the javagcinfo macro through sst. If you use separate
x5250 sessions, the first session is already at this screen after step 9.
15. Type -heap on the options line, where is the string copied in
step 13, and press Enter.
16. Type 100000 for your Through page value to ensure you have enough space, and
press Enter. This action creates a spool file with the heapdump.
17. To retrieve the heap dump, follow these steps to download the spool file
with iSeries Navigator.
1. Double click "Add a connection" under the Environment Tasks work area.
2. Type in the name of the server and enter.
3. Type in your userid on the server and enter.
4. Mouse click on Verify Connection.
5. Once the connection is verified, click on OK and then click on Finish.
6. In the My Connections work area, double click on the server you added
7. Type in your password on the server and enter.
8. Select Basic Operations.
9. Select Printer Output.
10. Drag and drop the spool file from the list to a folder on your client system.
If you retrieve more than one spool file, the resulting text file names will
include spaces; you may find it convenient to change the file names before
proceeding to the next step.
11. If you prefer to use Convert and the other tools on the iSeries, FTP the
resulting file to an ifs directory on the iSeries sever.
18. After getting the iSeries dump in a unicode text file, use the Convert command
from the updated svcdump.jar file to convert the text file to PHD format.
The ordinary syntax for this action, which works from QSHELL is:
java -classpath _path_to_svcdump.jar file_ com.ibm.jvm.ras.findroots.Convert
_file_name_. Note that the svcdump.jar file is available with the Memory Dump
Diagnostic for Java in the IBM Support Assistant installation folder in the
directory: plugins/com.ibm.websphere.mdd4j_1.0.0/WEB-INF/lib/svcdump.jar.
|
After you get the heap dump, use the Memory Dump Diagnostic for Java from IBM to analyze them.
|