HP (Hewlett-Packard) 5992-1918 User Manual

Java™ Troubleshooting Guide for HP-UX Systems

HP Part Number: 5992-1918 Published: July 2007 Edition: 3

Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial

Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under

vendor's standard commercial license. The information contained herein is subject to change without notice. The only warranties for HP products

and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as

constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. UNIX is a registered

trademark of The Open Group.

About This Document.......................................................................................................11

1 Diagnostic and Monitoring Tools and Options........................................................13

1.1 HP-UX Java Tools and Options Tables............................................................................................13

1.1.1 Crash Analysis Tools...............................................................................................................13

1.1.2 Hung and Deadlocked Processes............................................................................................13

1.1.3 Fatal Error Handling...............................................................................................................14

1.1.4 Monitoring Memory Use.........................................................................................................14

1.1.5 Performance Tools...................................................................................................................15

1.1.6 Miscellaneous Tools and Options...........................................................................................15

1.1.7 JDK Tools Not Available on HP-UX........................................................................................16

1.2 Ctrl-Break Handler..........................................................................................................................16

1.3 Fatal Error Log (hs_err_pid<pid>.log).............................................................................................17

1.4 gcore................................................................................................................................................18

1.5 gdb...................................................................................................................................................18

1.5.1 Java Stack Unwind Features....................................................................................................19

1.5.2 gdb Subcommands for Java VM Debugging..........................................................................19

1.6 HPjconfig ........................................................................................................................................21

1.7 HPjmeter .........................................................................................................................................24

1.7.1 Static Data Analysis.................................................................................................................26

1.7.1.1 Using HPjmeter to Analyze Profiling Data.....................................................................26

1.7.1.2 Using HPjmeter to Analyze Garbage Collection Data....................................................28

1.7.2 Dynamic Data Analysis...........................................................................................................28

1.7.2.1 Using HPjmeter to Monitor Applications.......................................................................28

1.7.2.2 Connect to the Node Agent From the HPjmeter Console...............................................29

1.7.2.3 Set Session Preferences....................................................................................................30

1.7.2.4 Viewing Monitoring Metrics During Your Open Session...............................................32

1.7.2.5 Running the HPjmeter Sample Programs.......................................................................32

1.7.2.5.1 Sample Memory Leak Application.........................................................................33

1.7.2.5.2 Sample Thread Deadlock Application....................................................................34

1.8 HPjtune............................................................................................................................................35

1.9 hat....................................................................................................................................................36

1.10 hprof..............................................................................................................................................36

1.11 java.security.debug System Property............................................................................................37

1.12 JAVA_TOOL_OPTIONS Environment Variable............................................................................37

1.13 jconsole (1.5+ only)........................................................................................................................38

1.14 jdb..................................................................................................................................................39

1.15 jhat.................................................................................................................................................39

1.16 jps (1.5+ only).................................................................................................................................40

1.17 jstat (1.5+ only)...............................................................................................................................40

1.18 jstatd (1.5+ only).............................................................................................................................41

1.19 jvmstat Tools..................................................................................................................................41

1.20 -verbose:class.................................................................................................................................42

1.21 -verbose:gc.....................................................................................................................................42

1.22 -verbose:jni.....................................................................................................................................42

1.23 visualgc..........................................................................................................................................42

1.24 -Xcheck:jni .....................................................................................................................................45

1.25 -Xverbosegc....................................................................................................................................46

1.26 -XX:ErrorFile..................................................................................................................................47

1.27 -XX:+HeapDump and _JAVA_HEAPDUMP Environment Variable............................................48

Table of Contents 3

1.27.1 Other HeapDump Options....................................................................................................48

1.27.2 -XX:+HeapDumpOnCtrlBreak ..............................................................................................48

1.27.3 -XX:+HeapDumpOnOutOfMemoryError.............................................................................49

1.27.4 -XX:+HeapDumpOnly...........................................................................................................49

1.27.5 Using Heap Dumps to Monitor Memory Usage...................................................................49

1.28 -XX:OnError...................................................................................................................................49

1.29 -XX:+ShowMessageBoxOnError....................................................................................................50

2 Useful System Tools for Java Troubleshooting...........................................................51

2.1 GlancePlus.......................................................................................................................................51

2.2 tusc...................................................................................................................................................51

2.3 Prospect...........................................................................................................................................51

2.4 HP Caliper.......................................................................................................................................51

2.5 sar....................................................................................................................................................51

2.6 vmstat..............................................................................................................................................51

2.7 iostat................................................................................................................................................51

2.8 swapinfo..........................................................................................................................................52

2.9 top....................................................................................................................................................52

2.10 netstat.............................................................................................................................................52

2.11 Other Tools....................................................................................................................................52

3 Getting Help from Hewlett-Packard............................................................................53

3.1 Problem Report Checklist................................................................................................................53

3.2 Collecting Problem Data.................................................................................................................54

3.2.1 Collecting Core File Information.............................................................................................54

3.2.1.1 Core File Checklist..........................................................................................................54

3.2.1.1.1 Estimate Core File Size............................................................................................54

3.2.1.1.2 Ensure Process Can Write Large Core Files............................................................54

3.2.1.1.3 Verify Amount of Disk Space..................................................................................55

3.2.1.1.4 Check If Directory Supports Large File Systems....................................................55

3.2.1.1.5 Ensure Permissions Allow Core Files.....................................................................56

3.2.1.2 Generating a Core File.....................................................................................................56

3.2.1.3 Verifying a Core File........................................................................................................56

3.2.2 Collecting Fatal Error Log Information...................................................................................56

3.2.3 Collecting Stack Trace Information.........................................................................................57

3.3 Collecting System Information........................................................................................................58

3.4 Collecting Java Environment Information......................................................................................58

3.4.1 Environment Variables............................................................................................................58

3.4.2 Libraries...................................................................................................................................59

3.5 Packaging Files................................................................................................................................60

4 Core File Analysis........................................................................................................61

4.1 Sample Java Application.................................................................................................................61

4.1.1 StackTraceJob...........................................................................................................................61

4.1.2 StackTrace.java........................................................................................................................62

4.1.3 stacktrace.c...............................................................................................................................63

4.2 Building the Application.................................................................................................................64

4.3 Verify Core File................................................................................................................................65

4.4 Debugging On Same System...........................................................................................................65

4.5 Packaging Files For Debugging On Different System.....................................................................65

4.6 Unpacking Files On Debugging System.........................................................................................66

4.7 Example gdb Session.......................................................................................................................68

4.8 Summary.........................................................................................................................................73

4 Table of Contents

Glossary............................................................................................................................75

Index.................................................................................................................................77

Table of Contents 5

List of Figures

1-1 HPjconfig - System Tab..................................................................................................................22

1-2 HPjconfig - Application Tab..........................................................................................................22

1-3 HPjconfig - Patches Tab.................................................................................................................23

1-4 HPjconfig - Tunables Tab...............................................................................................................23

1-5 HPjmeter - Profile Data.................................................................................................................27

1-6 HPjmeter - Threads/Locks Metrics................................................................................................27

1-7 HPjmeter - Garbage Collection Analysis.......................................................................................28

1-8 HPjmeter - Connecting to Server...................................................................................................30

1-9 HPjmeter - Setting Session Preferences.........................................................................................31

1-10 HPjmeter - Collecting Metrics.......................................................................................................31

1-11 HPjmeter - Choosing Metrics to Monitor......................................................................................32

1-12 HPjmeter - Memory Leak Alert.....................................................................................................33

1-13 HPjmeter - Heap Monitor Display................................................................................................34

1-14 HPjmeter - Thread Histogram.......................................................................................................35

1-15 HPjtune Screen..............................................................................................................................36

1-16 jconsole Screen...............................................................................................................................39

1-17 visualgc Application Information Window..................................................................................43

1-18 visualgc Graph Window...............................................................................................................44

1-19 visualgc Survivor Age Histogram Window..................................................................................45

List of Tables

1-1 Tools and Options for Crash Analysis...........................................................................................13

1-2 Tools and Options for Debugging Hung and Deadlocked Processes...........................................14

1-3 Options for Fatal Error Handling..................................................................................................14

1-4 Tools and Options for Monitoring Memory Use...........................................................................14

1-5 Performance Tools.........................................................................................................................15

1-6 Miscellaneous Tools and Options..................................................................................................15

1-7 JDK Tools Not Available on HP-UX..............................................................................................16

1-8 Java Version Information for gdb Java VM Debugging Features.................................................18

1-9 Java VM Debugging Commands...................................................................................................20

1-10 Java Subcommands.......................................................................................................................20

1-11 HPjmeter 3.0 Features....................................................................................................................25

1-12 Java SDKs and JDKs Supported by HPjmeter 3.0.........................................................................25

1-13 Options to the jstat Command......................................................................................................40

1-14 jstat — New Generation Statistics.................................................................................................41

1-15 Garbage Collection Field Information ..........................................................................................46

1-16 Overview of HeapDump Options.................................................................................................48

3-1 Libjunwind Library Location for PA-RISC Systems.....................................................................60

3-2 Libjunwind Library Location for Integrity Systems.....................................................................60

About This Document

The information in this document will help application developers and support engineers debug their Java applications on HP-UX systems.

Intended Audience

This document is intended for application developers and support engineers who are debugging Java applications on HP-UX systems. Note that some features described in this document are only available on HP-UX systems.

New and Changed Information in This Edition

This is the third version of this document. It contains fixes to the second version as well as a new chapter, which is a tutorial about analyzing core files.

Document Organization

This document contains four chapters:

Chapter 1: Diagnostic and Monitoring Tools and Options—This chapter provides information on tools and options useful for Java troubleshooting on HP-UX.

Chapter 2: Useful System Tools for Java Troubleshooting—This chapter provides information about HP-UX system tools to aide in Java troubleshooting.

Chapter 3: Getting Help from Hewlett-Packard—This chapter contains information about collecting necessary data before opening a Java-related support call.

Chapter 4: Core File Analysis—This chapter contains a step by step tutorial for performing core file analysis.

Typographic Conventions

This document uses the following typographical conventions:

%, $, or #

audit(5) A manpage. The manpage name is audit, and it is located in

Command Computer output

Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that you

ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH. [ERROR NAME] Key The name of a keyboard key. Return and Enter both refer to the

Term The defined use of an important word or phrase.

User input

Variable

[] The contents are optional in syntax. If the contents are a list

A percent sign represents the C shell system prompt. A dollar sign represents the system prompt for the Bourne, Korn, and POSIX shells. A number sign represents the superuser prompt.

Section 5. A command name or qualified command phrase. Text displayed by the computer.

must hold down the key labeled Ctrl while you press another key or mouse button.

The name of an error, usually returned in the errno variable.

same key.

Commands and other text that you type. The name of a placeholder in a command, function, or other

syntax display that you replace with an actual value.

separated by |, you must choose one of the items.

{} The contents are required in syntax. If the contents are a list

separated by |, you must choose one of the items.

... The previous element can be repeated an arbitrary number of

times.  Indicates the continuation of a code example. | Separates items in a list of choices. WARNING A warning calls attention to important information that if not

understood or followed will result in personal injury or

nonrecoverable system problems. CAUTION A caution calls attention to important information that if not

understood or followed will result in data loss, data corruption,

or damage to hardware or software. IMPORTANT This alert provides essential information to explain a concept or

to complete a task. NOTE A note contains additional information to emphasize or

supplement important points of the main text.

Related Information

This document contains information specific to troubleshooting Java problems on HP-UX systems. More information can also be found in the HP-UX Programmer's Guide for Java™ 2. In addition,

the Trouble-Shooting and Diagnostic Guide for Java 2 Platform, Standard Edition 5.0 and the

Troubleshooting Guide for Java SE 6 with HotSpot VMfrom Sun Microsystems also contain some

information that may be useful.

Publishing History

The document printing date and part number indicate the document’s current edition. The printing date will change when a new edition is printed. Minor changes may be made at reprint without changing the printing date. The document part number will change when extensive changes are made. Document updates may be issued between editions to correct errors or document product changes. To ensure that you receive the updated or new editions, you should subscribe to the appropriate product support service. See your HP sales representative for details. The latest version of this document is available online at:

http://www.docs.hp.com

Manufacturing Part Number

Systems

Publication DateEdition NumberSupported VersionsSupported Operating

December 2006Edition 1Versions 1 and 2HP-UX 11i5991-7463

February 2007Edition 2Versions 1 and 2HP-UX 11i5992-0551

July 2007Edition 3Versions 1, 2, and 3HP-UX 11i5992-1918

HP Encourages Your Comments

HP encourages your comments concerning this document. We are committed to providing documentation that meets your needs. Send any errors found, suggestions for improvement, or compliments to:

feedback@fc.hp.com

Include the document title, manufacturing part number, and any comment, error found, or suggestion for improvement you have concerning this document.

12 About This Document

1 Diagnostic and Monitoring Tools and Options

This chapter describes the tools and options available for postmortem diagnostics, analysis of hung/deadlocked processes, monitoring memory usage, and performance monitoring.

The tools and options are listed in tables by their respective functions in the first section of this chapter. Many of them are listed in multiple tables since they can be used for multiple functions.

The tools and options are described in detail with examples, where applicable, in the remaining sections of this chapter. All the tools and options described in this chapter are either included in the Java 2 Platform Standard Edition Development Kit (JDK 1.5+), are included with Hewlett-Packard's Java product, or are available for download at the Go Java! website:

http://www.hp.com/products1/unix/java

1.1 HP-UX Java Tools and Options Tables

The tools and options are categorized into the following table groupings:

• Crash Analysis Tools

• Hung and Deadlocked Processes

• Fatal Error Handling

• Monitoring Memory Use

• Performance Tools

• Miscellaneous Tools and Options

• JDK Tools Not Available on HP-UX

1.1.1 Crash Analysis Tools

Several of the options and tools described in this chapter are designed for postmortem diagnostics. These are the options and tools that can be used to obtain additional information if an application crashes. This analysis may either be done at the time of the crash or at a later time using information from the core file. In addition to these tools, many other tools have features useful for crash analysis.

Table 1-1 Tools and Options for Crash Analysis

wdb/gdb

Fatal Error Log (hs_err_pid<pid>.log)

-XX:OnError

-XX:+ShowMessageBoxOnError

Description and UsageTool or Option

An HP-supported implementation of the gdb debugger that has Java support. For simplicity, this document will refer to wdb/gdb as gdb from this point forward. gdb can be used to attach to a running process.

Contains information obtained at the time of the crash. Often one of the first pieces of data to examine when a crash occurs.

Specify filename to use for the fatal error log.-XX:ErrorFile

Specify a sequence of user-supplied scripts or commands to be executed when a crash occurs.

Suspend the process when a crash occurs. Depending on the user response, it can launch the gdbgdb debugger to attach to the Java VM.

1.1.2 Hung and Deadlocked Processes

The following options and tools can help you debug a hung or deadlocked process:

Java language debugger.jdb

1.1 HP-UX Java Tools and Options Tables 13

Table 1-2 Tools and Options for Debugging Hung and Deadlocked Processes

Description and UsageTool or Option

wdb/gdb

HPjmeter

Ctrl-Break Handler

-XX:+HeapDump and _JAVA_HEAPDUMPEnvironment Variable, starting with JDK 1.5.0.03 and SDK 1.4.2.10

1.1.3 Fatal Error Handling

The following options are useful for retrieving more information when fatal errors occur:

An HP-supported implementation of the gdb debugger that has Java support. For simplicity, this document refers to wdb/gdb as gdb from this point forward. gdb can be used to attach to a running process.

Used to identify and diagnose performance problems in Java applications running on HP-UX. It can also be used to debug thread and heap issues.

Used to retrieve thread dump information. It also executes a deadlock detection algorithm and reports any deadlocks detected involving synchronized code. Heap dumps are also generated beginning with JDK 1.5.0.05 and SDK

1.4.2.11 when the -XX:+HeapDumpOnCtrlBreak option is specified.

Used to observe memory allocation in a running Java application by taking snapshots of the heap over time. It can be set by providing the -XX:+HeapDump option or setting the _JAVA_HEAPDUMP environment variable.

Creates a core image of a running process.gcore (11.31 only)

Java language debugger.jdb

Table 1-3 Options for Fatal Error Handling

-XX:OnError

-XX:+ShowMessageBoxOnError

-XX:+HeapDumpOnOutOfMemoryError, starting with

SDK 1.4.2.11 and JDK 1.5.0.04

1.1.4 Monitoring Memory Use

The following options and tools are useful for monitoring memory usage of running applications:

Table 1-4 Tools and Options for Monitoring Memory Use

HPjmeter

HPjtune

Description and UsageOption

Used to specify a sequence of user-supplied scripts or commands to be executed when a crash occurs.

Used to suspend the process when a crash occurs. After the process is suspended, the user can use gdb to attach to the Java VM.

Enables dumping of the heap when an out of memory error condition occurs in the Java VM.

Description and UsageTool

Used to identify and diagnose performance problems in Java applications by examining and monitoring the heap and threads.

HP's garbage collection (GC) visualization tool for analyzing garbage collection activity in a Java program.

-XX:+HeapDump and _JAVA_HEAPDUMPEnvironment Variable, starting with JDK 1.5.0.03 and SDK 1.4.2.10

14 Diagnostic and Monitoring Tools and Options

Table 1-4 Tools and Options for Monitoring Memory Use (continued)

Description and UsageTool

-Xverbosegc (HP only) and -verbose:gc

hat

jconsole (1.5+ only)

1.1.5 Performance Tools

The following tools are useful for identifying where the application spends its time. Some tools allow you to monitor performance in real time (dynamic analysis) and other tools allow you to analyze captured profiling data (static analysis):

Table 1-5 Performance Tools

HPjmeter

HPjtune

Used to enable logging of garbage collection information. The HP-only -Xverbosegc option generates additional GC information that is used by HPjtune. It is preferable to use -Xverbosegc instead of -verbose:gc.

This third-party tool may be used to perform Java heap analysis.

Used to monitor and manage an application launched with a management agent on a local or remote machine.

Description and UsageTool

Use statically collected eprof data to understand where the application is spending time. Use dynamic real-time monitoring to identify performance issues.

HP's GC visualization tool for analyzing garbage collection activity statically collected in a Java program.

jstat (1.5+ only)

jconsole (1.5+ only)

hprof

1.1.6 Miscellaneous Tools and Options

The following tools and options do not fall into any of the previous categories:

Table 1-6 Miscellaneous Tools and Options

JAVA_TOOL_OPTIONS Environment Variable

jvmstat Tools

visualgc

Attaches to the Java VM and collects and logs performance statistics dynamically.

Launches a simple console tool enabling you to dynamically monitor and manage an application launched with a management agent on a local or remote machine.

Simple static profiler agent used for heap and CPU profiling.

Description and UsageTool or Option

Used to augment the options specified in the Java command line.

Tools include jps, jstat, and jstatd. These tools are included with JDK 1.5+.

Uses jvmstat technology to provide visualization of garbage collection activity in the Java VM.

Enables logging of class loading and unloading.-verbose:class

-Xcheck:jni

Enables logging of JNI (Java Native Interface).-verbose:jni

Performs additional validation on the arguments passed to JNI functions.

1.1 HP-UX Java Tools and Options Tables 15

1.1.7 JDK Tools Not Available on HP-UX

Some JDK tools are not available on HP-UX, so they are not described in this document. They are provided in JavaSoft JDK as unsupported tools. Equivalent functionality is available via gdb Java support, HPjmeter, and the HeapDump options.

Table 1-7 JDK Tools Not Available on HP-UX

Description and HP-UX AlternativeTool

jinfo

jmap

jstack

1.2 Ctrl-Break Handler

A thread dump is printed if the Java process receives a SIGQUIT signal. Therefore, issuing the command kill -3 <pid> causes the process with id <pid> to print a thread dump to its standard output. The application continues processing after the thread information is printed.

In addition to the thread stacks, the ctrl-break handler also executes a deadlock detection algorithm. If any deadlocks are detected, the ctrl-break handler also prints out additional information on each deadlocked thread. The SIGQUIT signal can also be used to print heap dump information when using the -XX:+HeapDump or -XX:+HeapDumpOnCtrlBreak options described further on in this chapter.

Following is an example of output generated when SIGQUIT is sent to a running Java process:

Full thread dump [Thu Oct 12 14:00:56 PDT 2006] (Java HotSpot(TM) Server VM 1.5.0.03 jinteg:02.13.06-21:25 IA64 mixed mode):

"Thread-3" prio=10 tid=00a78480 nid=24 lwp_id=2669798 runnable [0bfc0000..0bfc0ae0] at java.lang.Math.log(Native Method) at spec.jbb.JBButil.negativeExpDistribution(JBButil.java:795) at spec.jbb.TransactionManager.go(TransactionManager.java:234) at spec.jbb.JBBmain.run(JBBmain.java:258) at java.lang.Thread.run(Thread.java:595)

"Thread-2" prio=2 tid=009fb7a0 nid=23 lwp_id=2669797 runnable [0c1c0000..0c1c0b60] at spec.jbb.Order.dateOrderlines(Order.java:341)

- waiting to lock <444ba618> (a spec.jbb.Order) at spec.jbb.DeliveryTransaction.process(DeliveryTransaction.java:213) at spec.jbb.DeliveryHandler.handleDelivery(DeliveryHandler.java:103) at spec.jbb.DeliveryTransaction.queue(DeliveryTransaction.java:363)

- locked <154927e8> (a spec.jbb.DeliveryTransaction) at spec.jbb.TransactionManager.go(TransactionManager.java:431) at spec.jbb.JBBmain.run(JBBmain.java:258) at java.lang.Thread.run(Thread.java:595)

"Thread-1" prio=10 tid=008ffa80 nid=22 lwp_id=2669796 runnable [0c3c0000..0c3c0de0] at spec.jbb.infra.Collections.longStaticBTree.get(longStaticBTree.java:1346) at spec.jbb.Warehouse.retrieveStock(Warehouse.java:307) at spec.jbb.Orderline.validateAndProcess(Orderline.java:341)

- locked <48563610> (a spec.jbb.Orderline) at spec.jbb.Order.processLines(Order.java:289)

- locked <48563128> (a spec.jbb.Order) at spec.jbb.NewOrderTransaction.process(NewOrderTransaction.java:282) at spec.jbb.TransactionManager.go(TransactionManager.java:278) at spec.jbb.JBBmain.run(JBBmain.java:258) at java.lang.Thread.run(Thread.java:595)

"Thread-0" prio=2 tid=00781240 nid=21 lwp_id=2669795 runnable [0c5c0000..0c5c0e60] at spec.jbb.infra.Util.DisplayScreen.privIntLeadingZeros(DisplayScreen.java:448) at spec.jbb.infra.Util.DisplayScreen.putDollars(DisplayScreen.java:1214) at spec.jbb.NewOrderTransaction.secondDisplay(NewOrderTransaction.java:416)

- locked <154d4828> (a spec.jbb.NewOrderTransaction) at spec.jbb.TransactionManager.go(TransactionManager.java:279) at spec.jbb.JBBmain.run(JBBmain.java:258)

Prints Java configuration information for a given Java process, core file, or remote debug server.

Prints shared object memory maps or Java heap memory details of a given process, core file, or remote debug server. Use the HeapDump options or gdb heap dump functionality instead.

Prints a Java stack trace of Java threads for a given Java process, core file, or remote debug server. Use gdb stack trace back functionality instead.

Not yet ported to HP-UX.Serviceability Agent (SA)

16 Diagnostic and Monitoring Tools and Options

at java.lang.Thread.run(Thread.java:595)

"Low Memory Detector" daemon prio=10 tid=00778b80 nid=19 lwp_id=2669774 runnable [00000000..00000000]

"CompilerThread1" daemon prio=10 tid=00772c30 nid=17 lwp_id=2669772 waiting on condition [00000000..0a7ff728]

"CompilerThread0" daemon prio=10 tid=007703f0 nid=16 lwp_id=2669771 waiting on condition [00000000..0afff5b8]

"AdapterThread" daemon prio=10 tid=0076c8d0 nid=15 lwp_id=2669770 waiting on condition [00000000..00000000]

"Signal Dispatcher" daemon prio=10 tid=0076a2e0 nid=14 lwp_id=2669769 waiting on condition [00000000..00000000]

"Finalizer" daemon prio=10 tid=00530a60 nid=13 lwp_id=2669768 in Object.wait() [750c0000..750c0e60] at java.lang.Object.wait(Native Method)

- waiting on <11000100> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)

- locked <11000100> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:197)

"Reference Handler" daemon prio=10 tid=0052de80 nid=12 lwp_id=2669767 in Object.wait() [752c0000..752c0ce0] at java.lang.Object.wait(Native Method)

- waiting on <11003dc8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:474) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:123)

- locked <11003dc8> (a java.lang.ref.Reference$Lock)

"main" prio=8 tid=0047dc90 nid=1 lwp_id=-1 waiting on condition [7fffd000..7fffe398] at java.lang.Thread.sleep(Native Method) at spec.jbb.JBButil.SecondsToSleep(JBButil.java:740) at spec.jbb.Company.displayResultTotals(Company.java:942) at spec.jbb.JBBmain.DoARun(JBBmain.java:387) at spec.jbb.JBBmain.DOIT(JBBmain.java:1137) at spec.jbb.JBBmain.main(JBBmain.java:1490)

"VM Thread" prio=10 tid=004ff510 nid=11 lwp_id=2669766 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=004d0520 nid=3 lwp_id=2669758 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=004d0600 nid=4 lwp_id=2669759 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=004d06e0 nid=5 lwp_id=2669760 runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=004d07c0 nid=6 lwp_id=2669761 runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=004d08a0 nid=7 lwp_id=2669762 runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=004d0980 nid=8 lwp_id=2669763 runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=004d0a60 nid=9 lwp_id=2669764 runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=004d0b40 nid=10 lwp_id=2669765 runnable

"VM Periodic Task Thread" prio=8 tid=00500ad0 nid=18 lwp_id=2669773 waiting on condition

1.3 Fatal Error Log (hs_err_pid<pid>.log)

When a fatal error occurs, an error log is created in the file hs_err_pid<pid>.log, where <pid> is the process id of the process. The file is created in the working directory of the process, if possible. In the event that the file cannot be created in the working directory (for example, if there is insufficient space, a permission problem, or another issue), then the file is created in the temporary directory,/tmp. The error log contains information obtained at the time of the fatal error. This includes :

• Operating exception or signal that provoked the fatal error

• Version and configuration information

• Details on the thread that provoked the fatal error and its stack trace

• List of running threads and their states

• Summary information about the heap

• List of native libraries loaded

• Command-line arguments

• Environment variables

• Details about the operating system and CPU In some cases, only a subset of this information is output to the error log. This happens when a

fatal error is so severe that the error handler is unable to recover and report all details.

1.3 Fatal Error Log (hs_err_pid<pid>.log) 17

1.4 gcore

The gcore command creates a core image of a running process. By default, the name of the core file for a process-id will be core.process-id. The process information in the core image can be obtained by using gdb or other debuggers.

When gcore creates a core image of each specified process, the process is temporarily stopped. When the creation of the core image is complete, the process continues to execute.

This command is only available on HP-UX 11.31.

1.5 gdb

Java stack unwind enhancements have been added to gdb to enable it to support unwinding across Java frames and provide an effective way to examine stack traces containing mixed language frames (Java and C/C++) of both live Java processes and core files. This has been implemented by adding subcommands for Java VM debugging to gdb.

The following table shows which Java versions on PA-RISC and Integrity systems have the stack unwind and the gdb Java subcommands features. These features are available in gdb version

4.5 and later versions.

Table 1-8 Java Version Information for gdb Java VM Debugging Features

Platform

Enhancements

-pa11 (PA_RISC)

(PA_RISC2.0)

(PA_RISC2.0W)

(IA64N)

(IA64W)

(IA64N), 64-bit (IA64W)

GDB VersionJava SubcommandsStack Unwind

4.5+SDK 1.4.1.05+SDK 1.3.1.02+PA-RISC 32-bit

4.5+SDK 1.4.1.05+SDK 1.4.1.01+PA-RISC 64-bit

4.5–5.2SDK 1.4.1.05+SDK 1.3.1.06+Integrity 32-bit

4.5–5.2SDK 1.4.1.05+SDK 1.4.0.01+Integrity 64-bit

*5.3+SDK 1.4.2.10+SDK 1.4.2.10+Integrity 32

*5.3+JDK 1.5.0.03+JDK 1.5.0.03+Integrity 32

*gdb version 5.3 requires SDK 1.4.2.10 and later versions or JDK 1.5.0.03 and later versions in order to use the Java VM debugging features.

In order to use this functionality, the GDB_JAVA_UNWINDLIB environment variable must be set to the path name of the Java unwind library. The default location of the Java unwind library on various systems is shown following. The examples are for SDK 1.4; if you are using JDK 1.5, substitute /opt/java1.5 for /opt/java1.4.

/opt/java1.4/jre/lib/PA_RISC/server/libjunwind.sl /opt/java1.4/jre/lib/PA_RISC2.0/server/libjunwind.sl /opt/java1.4/jre/lib/PA_RISC2.0W/server/libjunwind.sl /opt/java1.4/jre/lib/IA64N/server/libjunwind.so /opt/java1.4/jre/lib/IA64W/server/libjunwind.so

Following are a few examples. If you are using ksh on a PA-RISC machine, this is how you set the environment variable for a 32–bit Java application:

export GDB_JAVA_UNWINDLIB=/opt/java1.4/jre/lib/PA_RISC2.0/server/libjunwind.sl

18 Diagnostic and Monitoring Tools and Options

Additionally, this is how you set the environment variable on an Integrity machine for a 32–bit Java application:

export GDB_JAVA_UNWINDLIB=/opt/java1.4/jre/lib/IA64N/server/libjunwind.so

If the SDK is installed in a location other than the default, substitute the non-default location for /opt/java1.4 in the previous commands.

1.5.1 Java Stack Unwind Features

The Java stack unwind features are useful for troubleshooting problems in the Java VM. Following is a list of the Java stack unwind features:

• View mixed language frames information, including Java frames and C/C++ native frames,

in a gdb backtrace.

• Distinguish various Java frame types including interpreted, compiled, and adapter frames.

• View Java method name, signature, and class package name for Java method frames.

Additional stack unwind features are available starting with SDK 1.4.2. These features fall into three categories: Java stack unwind enhancements, Java heap support, and Java threads support.

These additional features are available as part of the Java stack unwind enhancements:

• View Java compiled frame inlined methods.

• View Java interpreted or compiled frame specific information.

• View Java interpreted or compiled frame arguments and local variables.

• Disassemble Java method bytecodes.

• Print out the Java unwind table.

These additional features are available as part of the Java heap support:

• View Java heap parameters.

• Dump Java object.

• Print Java heap histogram.

• Find all the instances of a given Java class.

• Find all the references to a given object in the Java heap.

• Find out the object OOP (object-oriented pointer) of the given field address.

These additional features are available as part of Java threads support:

• View Java threads state information.

• View current Java thread information.

• View Java interpreted frame monitors information.

1.5.2 gdb Subcommands for Java VM Debugging

To view the gdb commands that support Java VM debugging, type help java at the gdb prompt.

(gdb) help java Java and JVM debugging commands.

List of java subcommands:

java args -- Show the current or specified Java frame arguments info java bytecodes -- Disassemble the given Java method's bytecodes java heap-histogram -- Show the Java heap object histogram java instances -- Find all the instances of the given klassOop in the Java heap java jvm-state -- Show Java virtual machine's current internal states java locals -- Show the current or specified Java frame locals info java mutex-info -- Print out details of the static mutexes java object -- Print out the given Java object's fields info java oop -- Find the Java object oop of the given Java heap address java references -- Find all the references to the given Java object in the Java heap java unwind-info -- Show the unwind info of the code where the given pc is located java unwind-table -- Print out the dynamically generated Java Unwind Table

1.5 gdb 19

Type "help java" followed by java subcommand name for full documentation. Command name abbreviations are allowed if unambiguous.

The following two tables list Java VM debugging commands and Java subcommands:

Table 1-9 Java VM Debugging Commands

backtrace

info frame

info threads

thread

Print backtrace of mixed Java and native frames

Print Java frame specific information if this is a Java frame

Print state information for all threads

Print detailed state information for the current thread

Table 1-10 Java Subcommands

java args

java bytecodes

java heap-histogram

java instances

java jvm-state

java locals

java object

java oop

java references

java unwind-info

Show the current or specified Java frame arguments information

Disassemble the given Java method's bytecodes

Show the Java heap object histogram

Find all the instances of the given klassOop in the Java heap

Show the current internal state of the Java VM

Show the current or specified Java frame locals information

Print the given Java object's fields information

Find the Java object OOP of the given Java heap address

Find all the references to the given Java object in the Java heap

Show the unwind information of the code where the given pc is located

java unwind-table

Print the dynamically generated Java unwind table

Type help java followed by the subcommand name for full documentation. Command name abbreviations are allowed if they are unambiguous.

Following are examples that illustrate the gdb command-line options for invoking gdb on a core file and on a hung process.

The first set of examples illustrate how to invoke gdb on a core file:

• Invoke gdb on a core file generated when running a 32-bit Java application on an Integrity

system with /opt/java1.4/bin/java:

$ gdb /opt/java1.4/bin/IA64N/java core.java

• Invoke gdb on a core file generated when running a 64-bit Java application on an Integrity

system with /opt/java1.4/bin/java -d64:

$ gdb /opt/java1.4/bin/IA64W/java core.java

• Invoke gdb on a core file generated when running a 32-bit Java application on PA-RISC

using /opt/java1.4/bin/java:

$ gdb /opt/java1.4/bin/PA_RISC2.0/java core.java

• Invoke gdb on a core file generated when running a 64-bit Java application on PA-RISC

using /opt/java1.4/bin/java:

$ gdb /opt/java1.4/bin/PA_RISC2.0W/java core.java

When debugging a core file, it is good practice to rename the file from core to another name to avoid accidentally overwriting it.

20 Diagnostic and Monitoring Tools and Options

If the Java and system libraries used by the failed application reside in non-standard locations, then the GDB_SHLIB_PATH environment variable must be set to specify the location of the libraries.

The following example illustrate how to invoke gdb on a hung process:

• Determine the process id:

$ ps -u user1 | grep java 23989 pts/9 8:52 java

• Attach gdb to the running process:

$ gdb -p 23989

HP gdb 5.0 for HP Itanium (32 or 64 bit) and target HP-UX 11.2x. Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest 5.0 (based on GDB) is covered by the GNU General Public License.Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support.

Reading symbols from /opt/java1.4/bin/IA64N/java... (no debugging symbols found)...done. Attaching to program: /opt/java1.4/bin/IA64N/java, process 23989 (no debugging symbols found)... Reading symbols from /usr/lib/hpux32/libpthread.so.1... (no debugging symbols found)...done. Reading symbols from /usr/lib/hpux32/libdl.so.1... ...

NOTE: If the version of gdb on the system is older than version 4.5, it will be necessary to specify the full path of the Java executable in order to use the gdb subcommands. For example:

gdb /opt/java1.4/bin/PA_RISC2.0/java –p 23989

A tutorial on gdb may be found at the following website:

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1677,00.html

1.6 HPjconfig

HPjconfig is a configuration tool for tuning your HP-UX 11i system to match the characteristics

of your application. It provides kernel parameter recommendations tailored to your HP-UX hardware platform and application characteristics. HPjconfig has features for saving and restoring configurations so you can distribute customized recommendations across your customer base.

HPjconfig can also be used to verify that your systems has all the necessary patches required for Java. The patches required for Java can be found at the following website:

http://www.hp.com/products1/unix/java/patches

HPjconfig runs on SDK 1.3.1 and later versions, SDK 1.4.x, and JDK 1.5.0.x. HP-UX 11.00 or later versions is required. All HP-UX 11i HP Integrity and HP 9000 PA-RISC systems are supported.

For more information about HPjconfig including the download, go to:

http://www.hp.com/products1/unix/java/java2/hpjconfig/index.html

HPjconfig can be run in either graphical user interface (GUI) mode or non-GUI (command-line) mode. In either mode, it generates a summary of the configuration information in the log file named HPjconfig_<hostname>_<date>_<timestamp>.log. This log file name can be specified using the -logfile option.

Following is usage information for the HPjconfig command:

usage: HPjconfig [ options ] -gui

1.6 HPjconfig 21

HPjconfig [ options ] <object> <action>

objects: -patches &| -tunables actions: -listreq | -listmis | -listpres | -apply

options:

-patches operate on java-specific patches

-tunables operate on java-specific tunables

-listreq list all java required patches or tunables that are applicable to this system

-listmis list missing java-specific patches or tunables on the system

-listpres list applied (installed) java-specific patches or tunables on the system

-apply apply (install) missing java-specific patches or tunables on the system

-javavers s java versions for selecting patches e.g 1.2, 1.3, 1.4, 5.0

-[no]gui run in GUI mode

-logfile s name of log file

-proxyhost s HTTP proxy host name for accessing live data

-proxyport s HTTP proxy port for accessing live data

-help show help string and exit

-version show version string

Following are examples of invoking HPjconfig in GUI mode from the csh and the ksh:

(csh) $ setenv DISPLAY <Display's IP Address>:0.0 $ setenv PATH $PATH:/usr/sbin $ java -jar HPjconfig.jar

(ksh) $ export DISPLAY=<Display's IP Address>:0.0 $ export PATH=$PATH:/usr/sbin $ java -jar HPjconfig.jar

The following four figures show the System, Application, Patches, and Tunables tabs for the HPjconfig tool:

Figure 1-1 HPjconfig - System Tab

Figure 1-2 HPjconfig - Application Tab

22 Diagnostic and Monitoring Tools and Options

Figure 1-3 HPjconfig - Patches Tab

Figure 1-4 HPjconfig - Tunables Tab

Following are the commands for invoking HPjconfig in non-GUI mode. The -help option lists options you can use in this mode.

$ cd <hpjconfig_installation_dir> $ java -jar ./HPjconfig.jar -nogui -help

Following is an example using HPjconfig in non-GUI mode to list missing patches for Java SDK 1.4:

$ java -jar HPjconfig.jar -nogui -patches -listmis -javavers 1.4 Log written to HPjconfig_mutant_20060915_040458.log List of missing patches: PHSS_34201 solves problem emulating floating point conversion when running PA2.0 Java on an IPF system.. solves problem with Aries signal handling that overlaps Java signal handling. solves problem emulating floating point conversion when running PA2.0 Java on an IPF system.. solves problem with Aries signal handling that overlaps Java signal handling.

Following is an example using HPjconfig to show the values for HP-UX tunables required by Java:

$ java -jar HPjconfig.jar -nogui -tunables -listreq Log written to HPjconfig_mutant_20060915_040934.log List of required tunables: Name Recommended value nproc 2048+20 max_thread_proc 3000 nkthread 6000 nfile 30000

1.6 HPjconfig 23

maxfiles 2*1024 maxfiles_lim 2*1024 maxdsiz 2000*1024*1024

Following is an example of using HPjconfig to display tunables that are set to values less than those recommended:

$ java -jar HPjconfig.jar -nogui -tunables -listmis Log written to HPjconfig_mutant_20060915_040955.log List of tunables whose values are less than the recommended values: Name Recommended value max_thread_proc 3000 maxdsiz 2000*1024*1024

Following is an example log file produced by HPjconfig:

$ more HPjconfig_server1_20060915_042600.log Fri Sep 15 16:26:00 PDT 2006

HPjconfig 3.0.01 (Thu Jul 21 14:52:47 2005)

Machine name: server1 IP address: 15.244.94.25 System type: ia64 hp server rx5670 Architecture: IA64N OS name: HP-UX OS version: B.11.23 Processors: 4

Java version: 1.4

Reading required patches/tunables information from /tmp/HPjconfig.xml Read required patches/tunables information

Reading patch list from system Read patch list from system

List of required patches: PHCO_30476 supports HPjmeter profiling of unbound (MxN) threads. PHKL_30192 solves kernel panic with thousands of MxN threads. PHSS_30015 solves problem with Aries signal handling that overlaps Java sig nal handling. PHSS_34201 solves problem emulating floating point conversion when running PA2.0 Java on an IPF system.. solves problem with Aries signal handling that ove rlaps Java signal handling. solves problem emulating floating point conversion w hen running PA2.0 Java on an IPF system.. solves problem with Aries signal handl ing that overlaps Java signal handling.

1.7 HPjmeter

With the release of HPjmeter 3.0, all previous versions of HPjmeter (1.x, 2.x) are no longer available for download and are no longer supported by HP.

If you have an old version of HPjmeter, please download HPjmeter 3.0 from:

http://www.hp.com/products1/unix/java/hpjmeter/index.html

HPjmeter can be used to identify and diagnose performance problems in Java applications running on HP-UX. It can be used for both static and dynamic data analysis. For example, for static data analysis it can be used to analyze profiling data generated by the following command-line options: -Xrunhprof:heap=dump, -Xeprof, -Xverbosegc, -Xloggc, and –XX:+HeapDump. Additionally, when using JDK 1.5.04 or later releases, HPjmeter can capture profiling data with zero preparation (that is, without pre-planning). HPjmeter can also be used for dynamic data analysis by monitoring live Java applications.

The following table lists the features of HPjmeter 3.0. The first two rows are static features and the remaining four rows are dynamic features.

24 Diagnostic and Monitoring Tools and Options

+ 54 hidden pages