Hp COMPAQ PROLIANT 4000, COMPAQ PROLIANT 6000 Java Performance

WHITE PAPER

.

August, 1997 Prepared By

Internet Business Unit Compaq Computer

CONTENTS

Introduction .................... 3

Executive Summary......... 3

Test Objectives...................... 5

Summary of the Test Results..6 Summary Charts of

CaffeineMark and Linpack Test

Suite.....................................7

Analytical Summary of the

Test Results .................... 8

Java Security and

Performance.................... 9

Java Security Architecture...... 9

Appendix A .................... 14

Details of CaffeineMark Test

Results................................14

Appendix B .................... 27

Details of JMark Test Suite

Results................................27

References ..................... 35

ECG039.0897

.

Java Performance on Compaq’s ProLiant

.

Servers

.

Java Performance has been a major debate in the Internet/intranet industry ever since

.

Java was introduced to the computing world. The initial JVM (Java Virtual Machine) and

.

browser product offerings interpreted Java applets, and the performance was found to be

.

significantly slower than C and C++ programs. Then JIT (Just In Time) Compilers,

.

which compile the byte code on a function by function basis on-the-fly, were introduced.

.

This significantly improved the applet performance in many areas. JIT Compilers vastly

.

improved the performance of most non-graphical benchmarks over the interpreters but

.

did not significantly improve the performance for graphical benchmarks.

.

This White Paper describes the results of Java benchmark tests run on Compaq's

.

ProLiant 800 servers and Sun Ultra 2 servers. The ProLiant 800 outperformed Sun

.

Ultra2 by an impressive 250% in price and 235% in performance. Detailed test results

.

are provided in Appendix A, “Details of Caffeine Mark Test Results” and Appendix B,

.

“Details of JMark Test Suite Results”.

.

1

WHITE PAPER (cont.)

.

NOTICE

.

The information in this publication is subject to change without notice.

.

ECG039.0897

.

COMPAQ COMPUTER CORPORATION SHALL NOT BE LIABLE FOR TECHNICAL

.

OR EDITORIAL ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR

.

INCIDENTAL OR CONSEQUENTIAL DAMAGES RESULTING FROM THE

.

FURNISHING, PERFORMANCE, OR USE OF THIS MATERIAL.

.

This publication does not constitute an endorsement of the product or products that were tested.

.

The configuration or configurations tested or described may or may not be the only available

.

solution. This test is not a determination of product quality or correctness, nor does it ensure

.

compliance with any federal, state or local requirements. Compaq does not warrant products other

.

than its own strictly as stated in Compaq product warranties.

.

Product names mentioned herein may be trademarks and/or registered trademarks of their

.

respective companies.

.

Compaq, Compaq Insight Manager, ProLiant, registered United States Patent and Trademark

.

Office.

.

Other product names mentioned herein may be trademarks and/or registered trademarks of their

.

respective companies.

.

Microsoft, Windows, Windows NT, Windows NT Advanced Server, SQL Server for Windows NT

.

are trademarks and/or registered trademarks of Microsoft Corporation.

.

Java Performance on Compaq’s ProLiant Servers

.

First Edition (August 1997)

.

Document Number ECG039.0897

.

2

WHITE PAPER (cont.)

.

INTRODUCTION

.

ECG039.0897

Java applet or application performance depends on many aspects of the Java Virtual Machine and

.

the associated packages and class libraries. Even before Java came on to the computing horizon,

.

interpreters were almost always slower than compilers. Compilers, (excluding non-JIT compilers)

.

have the luxury of performing extensive optimizations on the source code. They perform extensive

.

dataflow and control flow analysis on an intermediate representation of the original source code

.

and produce a highly optimized hardware specific code. All of this processing is being done "ahead

.

of time" or at compile time, not at runtime in the traditional languages like C and C++. JIT

.

Compilers are a compromise between the low performing interpreters and high performing non-

.

dynamic, non-real-time, "ahead of time" traditional compilers. JIT Compilers compile the Java

.

byte code on-the-fly when an applet is running in the browser and they compile a function at a

.

time. This has advantages and disadvantages. Because of their real-time compilation of the code

.

function at a time, they do not have the luxury of doing extensive optimizations like C and C++

.

compilers do, especially inter-procedural (across the functions or methods) optimizations; however

.

they compile only those functions or methods that are called. JITs are quickly catching up with the

.

traditional compilers, but at the time of writing this white paper they still lag the C and C++

.

compilers in performance.

.

Among all the VM components, the interpreter, JIT compiler and the security components affect

.

the applet performance more than any other components. An applet performance is directly

.

dependent on how fast the interpreter interprets the code on the native system and to the extent it is

.

security checked by the security components of the VM. With JIT compilers, it also depends on the

.

optimizations performed by the JITs on the byte code. Extensive optimizations by the JIT

.

compilers could result in efficient machine code, which reduces the runtime of the applet but adds

.

to the compile time of the applet.

.

EXECUTIVE SUMMARY

.

Java is a language and a runtime system designed for Internet/intranet applications. The single

.

most important component of the Java system is the Java Virtual Machine (JVM). It is the JVM

.

that enables Java applets to be platform independent and architecturally neutral. The JVM is a

.

stack based "software machine" or "soft CPU" that has a set of instructions, data types, a set of

.

registers, a stack, a garbage collection heap and a method area. All these are logical abstract

.

components of the JVM. It supports 248 byte codes, each performing a basic CPU operation such

.

as adding an integer to a register, jumping to subroutines, storing a result in memory, and

.

incrementing or de-incrementing a register. The VM is, in effect, a stacked ALU with local and

.

global variables.

.

Java is considered an interpreted language. This is not entirely true. The Java source language is

.

compiled into an intermediate form known as byte codes by the javac compiler. This intermediate

.

language (IL) or the byte codes are JVM instructions and data. In non-java compilers the IL is an

.

abstraction of hardware for which the compiler is written. These compilers optimize the IL and

.

generate code for the specific hardware in the backend of the compiler which is then linked with

.

libraries, loaded and executed. Java JIT (Just In Time) Compilers on the other hand optimize the

.

byte code and produce code for the native hardware and execute it at the same time. The compile,

.

link, load and execute operations are separate and discrete on non-java compilers whereas it is a

.

single continuous dynamic on-the-fly operation in Java interpreters or JIT compilers. See Figure 1

.

for an overview of Java applet processing (i.e. compile, load/download, interpret or JIT compile

.

and run cycle).

.

3

.

Java Compiler

JIT

Operating System

Hardware

ECG039.0897

WHITE PAPER (cont.)

.

Figure 1 – Overview of Java applet processing

.

4

Interpreter

Java Source (.java)

Java Byte Code (.class)

Java Runtime

(javac)

WHITE PAPER (cont.)

.

Java's performance over the last 2 years has been disappointing. Java applets still do not run as fast

.

as C, or C++ applications. Java security has a profound effect on JVM performance. For instance,

NOTE:

SUN has recently released JDK

1.1, which is the latest release, however at the time of these benchmark trials, only JDK 1.0.2 had a JIT compiler.

ECG039.0897

.

on some JMark 1.0 tests run by PC magazine, Borland's 5.0 JIT Compiler was 50 to 60% faster

.

than Netscape's JIT compiler, despite the fact that Netscape licensed the same JIT Compiler from

.

Boreland. The reason for this degradation in performance is Netscape's very aggressive "Sandbox"

.

security and byte code verifier.

.

Asymetrix SuperCede, one of the top 3 VMs in the industry, bypasses the security checks made

.

during the runtime of an applet. It scored very high on many benchmark tests not only because of

.

its "flash" compiler technology but also because of bypassing the VM security. SuperCede's results

.

are included in this paper’s charts, mainly to illustrate the performance enhancement a VM can

.

gain by bypassing the extreme security requirements of Java security.

.

The fundamental problem with Java Security is that it is a run-time/real-time security system. The

.

sandboxing, byte code verification, and security manager checks are all done in real-time when the

.

applet is running within the browser in front of a user.

.

Test Objectives

.

The Intranet and Groupware department at Compaq ran Java benchmark tests on Compaq's

.

ProLiant 800 Servers and Sun Ultra 2 Servers to investigate three major questions:

.

• How do Compaq's servers perform against Sun Microsystems in the Java space?

.

• Which VMs perform the best on Compaq servers?

.

• How do the JITs perform in comparison to the interpreters on Compaq servers?

.

Three major benchmark test suites for the Java industry (CaffeineMark, JMark and Linpack) were

.

selected to run on the Compaq ProLiant 800 servers and Sun Ultra 2 machines. The following

.

JVMs were benchmarked:

.

• Microsoft's VM (MSVM) using Microsoft Internet Explorer (MSIE) 3.0

.

• Symantec's Visual Café (VCafe) 1.0 applet viewer

.

• Asymetrix's SuperCede (SC) 1.0 applet viewer

.

• Netscape's VM (Nav) in its Navigator Browser 3.0

.

• SUN Java Developer Kit (JDK) 1.0.2

.

CaffeineMark tests suite and Linpack ran successfully on both ProLiant 800 systems and Sun Ultra

.

2 systems. JMark 1.0 test suite could not be run on the Sun server.

.

The ProLiant 800 had 32M memory with a 200Mhz CPU and 256k cache. The Sun Ultra 2 had

.

256M memory, two 167Mhz CPUs, 256k cache. Each test was run three times to get an average

.

score.

.

5

WHITE PAPER (cont.)

.

Summary of the Test Results

.

NOTE:

The ProLiant 800 outperformed Sun Ultra2 by an impressive 250% in price (the ProLiant 800 costs $8000 vs. Sun Ultra 2 systems costing $20,000) and 235% in performance.

ECG039.0897

• Compaq's ProLiant 800 server outperformed Sun Ultra2 system in almost all the benchmarks

.

6

by an impressive margin, especially in price and performance (Microsoft's VM Just In Time Compiler on ProLiant 800 is 235% faster than Sun Ultra 2's JDK 1.0.2 VM JIT compiler). See the overall CaffeineMark chart and Linpack chart for a quick overview of the results and individual test result charts for details.

• MSVM (Microsoft's Virtual Machine) running in MSIE 3.0 (Microsoft's Internet Explorer)

browser or the VJ++ IDE is the fastest VM in the Java Space currently.

• The top 3 VMs in the Java Space are MSVM, Symantec's Visual Café and Asymetrix's

SuperCede.

• Slowest JIT on ProLiant 800, Netscape Navigator 3.0, is 156% faster than Sun's JDK 1.0.2 JIT

using the CaffeineMark test suite.

• Fastest JIT on ProLiant 800, Microsoft's MSIE 3.0 VM, is 235% faster than Sun's JDK 1.0.2

running the CaffeineMark test suite.

• Compaq's ProLiant 800 running MSIE 3.0 VM (JIT) is 218% faster than Sun's JDK 1.0.2 JIT

on the compute intensive Linpack suite.

• MSIE JIT is 18 to 100 times faster than MSIE interpreter on the ProLiant systems on the non-

graphics tests. On the graphics tests the JITs do not show any significant improvements in speed.

• Sun Ultra 2's JDK 1.0.2 JIT is 3 to 50 times faster than its interpreter on non-graphics tests. On

the graphics tests, again the JIT does not show any significant improvement over the interpreter.

• JDK 1.1 interpreter on Sun Ultra 2 platform is on an average twice as fast as JDK 1.0.2

interpreter on the same platform.

• JDK 1.1 interpreter on the ProLiant platform is twice as fast as JDK 1.1 interpreter on the Sun

Ultra 2 platform on all non-graphics tests. On the graphics tests, Sun's Ultra 2 JDK 1.1 interpreter is 2 to 3 times faster than the interpreter on ProLiant.

WHITE PAPER (cont.)

.

CaffeineMarks

Linpack Score

.

Summary Charts of CaffeineMark and Linpack Test Suite

.

ECG039.0897

.

7

5000 4000 3000 2000 1000

25 20 15 10

5 0

CaffeineMark Overall Performance Results

(higher scores are better)

ProLiant 800

Sun Ultra 2

0

VCafe Nav JDK 1.0.2 MSIE

JIT Compilers

Linpack

(highers scores are better)

ProLiant 800

VCafe SC Nav JDK 1.0.2 MSIE

Sun Ultra 2

ProLiant 800

WHITE PAPER (cont.)

.

ANALYTICAL SUMMARY OF THE TEST RESULTS

.

ECG039.0897

What was learned from the CaffeineMark test suite is that the MSIE 3.0 (Microsoft Internet

.

Explorer) or VJ++ 1.0 (Visual J++ IDE) JIT compiler is currently the fastest JIT in the Java space.

.

Asymetrix SuperCede scored very high on many tests due to it bypassing the virtual machines’

.

security code. All other VMs do not in a real life environment, therefore Asymetrix's results were

.

discounted as a reflection of true performance of a VM in a real life Java environment.

.

The JIT compilers show significant performance improvements over interpreters on all the non-

.

graphical or math heavy/compute intensive tests. On an average, MSIE 3.0 VM (JIT) on the

.

ProLiant 800 is 2 to 10 times faster than Sun's JDK 1.0.2 VM (JIT) on the Ultra 2 system. On the

.

graphics tests, there were some interesting results. The JIT Compilers do not increase the

.

performance of the VMs as well as on the compute intensive tests. This is mainly because of the

.

poor implementation of the AWT package of the Java VM. Most graphics operations are

.

implemented in the native code (X Windows on Solaris platform and Win32 subsystem on NT),

.

consequently the JITs could not optimize the graphics byte code to the extent they could with math

.

tests. In fact the interpreters performed much better than the JITs on these tests, indicating that the

.

JITs may sometimes degrade performance.

.

Please note that most server applications and servlets are non-graphical in nature. It is the

.

optimization of compute intensive and I/O intensive (for database and network applications)

.

operations that are more important for server performance.

.

Linpack is a compute intensive test suite originally designed for super computers. MSVM,

.

Symantec Visual Café and SuperCede have all performed equally well and are twice as fast as

.

Sun's JDK 1.0.2 VM. JIT Compilers speedup performance by a factor of 20 over the interpreters.

.

One interesting result to note here is the performance of Sun's JDK 1.1 interpreter on the ProLiant

.

800 and JDK 1.1 interpreter on the Sun Ultra 2 system. On almost all CaffeineMark and Linpack

.

tests, Sun's interpreter performed better on the ProLiant platform in comparison to Sun's own Ultra

.

platform. This indicates that the underlying OS and HW (NT and INTEL) are responsible for the

.

increased performance over Sun's OS and HW (Solaris and SPARC).

.

JMark test suite confirmed our earlier findings from CaffeineMark and Linpack. JIT Compilers

.

have performed significantly better than interpreters on the compute intensive or non-graphics

.

tests. Interpreters have done as well as the JITs and better on some of the graphics tests.

.

Java security has a profound affect on Java performance. This is confirmed by the test results.

.

Asymetrix's SuperCede does not do security checks while running the applets like all other VMs

.

that were tested. Its performance numbers were excellent. SuperCede derives its better

.

performance due to two reasons:

.

• It does extensive optimizations of the byte code much like the JITs from Microsoft or

.

8

Symantec's Visual Café.

• It bypasses the runtime security checks of the byte code and other VMs that were tested do

not.

So in all the test case results in this white paper, Asymetrix SuperCede's results should be adjusted by a certain percentage before it can be compared against other VMs or JITs fairly. Sun's JIT has performed the poorest on almost all of the tests.

WHITE PAPER (cont.)

.

JAVA SECURITY AND PERFORMANCE

.

ECG039.0897

As mentioned earlier, an applet performance is dependent not only on the interpreter and the JIT

.

compiler performance, but also on the efficiency of the VM security components. For instance, on

.

JMark l.0 test suite, Borland’s 5.0 JIT had a score of 5711 JMarks. Netscape, which licensed

.

Borland's JIT scored 2138, about 50 to 60% less than Borland's JIT. The reason for this

.

degradation of performance, despite the fact that the JIT is the same in both VMs, is Netscape's

.

very aggressive "sandbox" security and byte code verifier. Additional details follow regarding the

.

Java security components: sandbox, byte code verifier, class loader and security manager. A

.

simple mathematical relationship between JVM performance and its components is as follows:

.

1

.

A + B + C + D

.

Asymetrix SuperCede, one of the JVMs we tested, is one of the top three VMs in the Java Space

.

currently. It performs exceptionally well on many tests not only because of its "Flash" compiler

.

technology (which performs extensive optimizations of the byte code), but also because it bypasses

.

the security checks in its VM implementation. A brief description of Java Security Architecture is

.

given below to explain the Java Performance and Java Security relationship.

.

Java Security Architecture

.

From the beginning, Java architects have placed emphasis on Java security at the expense of

.

performance. There are two major aspects of Java technology that affect Java's performance:

.

• It is an interpreted system and interpreted systems are almost always slower than compiled

.

9

systems.

• Java's runtime security that does extensive checks of not only the byte code but also runtime

resource access validation of applets.

Java Security is architected, designed and implemented into 3 major components of the Java System:

1 Java Language 2 Compiler 3 Runtime Mechanisms

Language and Compiler Security Features

First and the foremost is the removal of pointer based operations from Java Language. Unlike in C or C++, in Java, all accesses to memory areas must be done using object instance variables. The absence of pointers eliminates memory-browsing, modification of memory resident code, illegal access to security related objects, and over writing the security manager or the class loader. Java is a strongly typed language. Strong typing also contributes to security. Methods cannot be used with classes to which they do not apply. Objects are associated with a well defined types and cannot be freely converted.

JVM Performance µ ------------------------

A = Execution time of JIT produced code B = Execution time of JVM packages C = JIT compile time D = Execution time of the security components

WHITE PAPER (cont.)

.

The compiler checks all array operations to make sure that they are valid for the array object being

.

accessed and memory overruns do not occur. The compiler checks all class, interface, variable and

.

method accesses to ensure that the accesses are consistent with the access modifiers used in their

.

ECG039.0897

declarations. The compiler also prevents uninitialized variables from being read and constants from

.

being modified.

.

Run Time Security Mechanisms

.

Of all the security mechanisms of Java security architecture, runtime security mechanisms are the

.

most elaborate and time consuming in terms of affecting the performance. The fundamental

.

assumption here is that all code that is "foreign" to the local system and is 100% untrustworthy, so

.

it should be subjected to extensive security checks even though it affects performance.

.

Performance is secondary to security. As mentioned earlier, this run-time/real-time security

.

functionality coupled with an interpreted execution of the applet degrades the performance

.

considerably. This run-time real-time security mechanism is implemented by 3 VM components:

.

1 Byte Code Verifier

.

2 Class Loader

.

3 Security Manager

.

The byte code verifier does extensive security checks of the Java byte code to make sure that the

.

byte code downloaded from a foreign server conforms to all VM specifications. In addition to

.

downloading the classes of the applet from the network, the class loader creates and enforces a

.

runtime entity called the "name spaces". The security manager is responsible for authenticating and

.

validating all applet accesses to local resources like the disk, memory, processes etc. The byte code

.

verifier, class loader and the security manager together create a virtual entity called the "sandbox".

.

The sandbox is a virtual "prison" for the applet. It allows an applet to function freely as long as it

.

does not affect any other sandbox. In summary, these three components basically ensure:

.

• Only the correct classes are loaded

.

• The classes are in correct format

.

• Untrusted classes will not execute dangerous instructions

.

• Untrusted classes are not allowed to access protected systems and resources

.

The following section describes each of these components in detail. Please see the Java security

.

architecture diagram in Figure 2 for an understanding of the relationship between various security

.

components.

.

10

.

Java Compiler

Java Byte Code

Byte Code Verifier

Security

Interpret

JIT

Hardware

ECG039.0897

WHITE PAPER (cont.)

.

Figure 2. Java Security Architecture

.

11

Java Source

Class Loader

Manager

Java Runtime