User Manual
Version 3.5
CONTENTS
ii
Document Version as of 20 Dez. 02
CONTENTS
iii
Contents
Software License Agreement and Limitations ....................................... 1
General .......................................................................................................................... 1
Complaints Concerning Defects.................................................................................................. 1
Software License ........................................................................................................................ 1
Limitations, Warranty, Liability ....................................................................................................2
Applicable law and Place of Jurisdiction...................................................................................... 4
Preface ..................................................................................................... 5
How this Manual is Organized ....................................................................................................5
Conventions Used in This Manual............................................................................................... 6
Further Information and Support .................................................................................................7
1 Introduction ................................................................................ 9
1.1 What is OPERA™?............................................................................................ 9
1.2 Getting to Know the OPERA™ Product Family ............................................... 11
1.2.1 OPERA™ Measurement System ......................................................................... 11
1.2.2 OPERA™ Software Suite ..................................................................................... 12
2 Test Methodology..................................................................... 13
2.1 Assessing Quality ............................................................................................ 13
2.2 Advanced Audio Measurements Employing "Perceptual Modeling" ................. 15
2.3 International Standardization ........................................................................... 16
2.4 Which "Measurement" for Which "Application"?............................................... 18
2.5 Selection of the Reference File........................................................................ 20
2.6 How to Assess "Signal Enhancers".................................................................. 20
3 Installation and Setup .............................................................. 23
3.1 OPERA™ Software Suite ................................................................................ 23
3.1.1 Unpacking the Software ....................................................................................... 23
3.1.2 System Requirements ..........................................................................................23
3.1.3 Installation and Setup ........................................................................................... 24
3.1.4 Verification............................................................................................................ 24
3.2 OPERA™ Measurement System ..................................................................... 24
3.2.1 Workstation version .............................................................................................. 24
3.2.2 Portable PC Version ............................................................................................. 26
CONTENTS
iv
4 Getting to Know the OPERA™ Framework............................. 29
4.1 General Concept ............................................................................................. 29
4.2 Data Acquisition Using OptiCall™.................................................................... 30
4.2.1 POTS Telephony Interfaces..................................................................................30
4.2.2 Audio Interfaces ....................................................................................................33
4.2.3 The OptiCall™ Program........................................................................................40
4.3 The OPERA™ Framework .............................................................................. 54
4.3.1 The Underlying Generic Algorithm Model .............................................................54
4.3.2 The Structure of the OPERA™ Framework..........................................................55
4.4 Basic Operation............................................................................................... 57
4.4.1 The Main Window .................................................................................................57
4.4.2 How to Select a Measurement Algorithm .............................................................58
4.4.3 How to Start a Measurement ................................................................................60
4.4.4 How to Display the Results ...................................................................................68
4.4.5 Setting Markers in Diagrams.................................................................................73
4.4.6 Logging Results ....................................................................................................74
4.4.7 Performing Online Measurements in Realtime .....................................................74
4.4.8 Measuring only parts of the Input Files .................................................................75
4.4.9 Printing ..................................................................................................................76
4.4.10 Exporting Graphs ..................................................................................................76
4.4.11 Summary of the Menu Options .............................................................................77
4.5 Performing Measurements From Batch Files................................................... 81
4.5.1 Syntax of the Command Line Parameters in a Batch File ....................................81
4.5.2 How to Use a Configuration File ...........................................................................82
4.5.3 Example RunPsqm.bat .........................................................................................83
5 Wide Band Audio Quality Testing............................................85
5.1 What To Know About Testing Wide Band Audio Quality.................................. 85
5.2 Reference Files for Wideband Audio Measurements ....................................... 86
5.3 Signal Acquisition ............................................................................................ 86
5.4 Fundamentals of the PEAQ Measurement Algorithm....................................... 87
5.4.1 Background of the PEAQ (ITU-R BS.1387) Development ...................................87
5.4.2 Common Elements of PEAQ Basic and PEAQ Advanced ...................................87
5.4.3 Basic version.........................................................................................................88
5.4.4 Advanced Version .................................................................................................89
5.5 Using PEAQ .................................................................................................... 90
5.5.1 OPERA Software Suite - PEAQ ............................................................................91
5.5.2 OPERA Portable Tester with Audio Interface Option ...........................................91
5.5.3 Algorithm Parameters ...........................................................................................91
5.5.4 Diagram Types, PEAQ Basic................................................................................92
5.5.5 Diagram Types, PEAQ Advanced.......................................................................105
5.5.6 Command Line Arguments .................................................................................107
5.6 Example Measurement Setups...................................................................... 107
5.6.1 Example 1: Online Monitoring.............................................................................108
5.6.2 Example 2: Stand Alone Testing.........................................................................111
5.6.3 Example 3: Measurements From a Batch File....................................................113
5.6.4 More Examples ... ...............................................................................................116
CONTENTS
v
6 Telephony Band Voice Quality Testing ................................ 117
6.1 What To Know About Testing Telephony Band Voice Quality....................... 117
6.2 Reference Files for Voice Quality Testing and Echo Measurements.............. 118
6.3 PSQM as an Example for Perception Based Measurement Algorithms ......... 119
6.4 PSQM or PESQ, which one shall I use? ........................................................ 120
6.5 PSQM Measurement ..................................................................................... 121
6.5.1 Fundamentals of the PSQM Measurement Algorithm........................................ 121
6.5.2 Signal Acquisition ............................................................................................... 123
6.5.3 PSQM Algorithm Properties ...............................................................................124
6.5.4 Diagram Types ................................................................................................... 124
6.5.5 Command Line Arguments................................................................................. 134
6.5.6 Common Mistakes.............................................................................................. 135
6.6 PESQ Measurement and VAD Measurement ................................................ 137
6.6.1 Advantage of using PESQ instead of PSQM ..................................................... 138
6.6.2 Explanation of the Measured Parameters .......................................................... 138
6.6.3 Using PESQ ....................................................................................................... 143
6.6.4 Diagram Types ................................................................................................... 144
6.6.5 Command Line Arguments................................................................................. 148
6.7 Echo Measurement........................................................................................ 148
6.7.1 Fundamentals of the Echo Measurement Algorithm .......................................... 148
6.7.2 Interpretation of Echo Parameters ..................................................................... 149
6.7.3 Signal Acquisition ............................................................................................... 149
6.7.4 Echo Algorithm Properties .................................................................................. 150
6.7.5 Specific Settings for the Echo Measurement ..................................................... 150
6.7.6 Diagram Types ................................................................................................... 151
6.7.7 Command Line Arguments................................................................................. 156
6.8 Measurement Examples ................................................................................ 157
6.8.1 Example 1: Stand Alone Loop Measurement..................................................... 157
6.8.2 Example 2: Measurements From a Batch File ................................................... 165
6.8.3 More Examples .................................................................................................. 167
7 Automation and Programming.............................................. 169
7.1 General.......................................................................................................... 169
7.2 Performing Measurements From Batch Files ................................................. 169
7.2.1 Syntax of the Command Line Parameters ......................................................... 169
7.2.2 Parameters common to all Algorithms ............................................................... 170
7.2.3 How to Use a Configuration File......................................................................... 171
7.2.4 Parameters Specific to the Measurement Algorithms ........................................ 172
7.2.5 Parameters Specific to OptiCall ......................................................................... 173
7.2.6 Example RunPsqm.bat....................................................................................... 175
7.2.7 Example, Bulk Call Testing ................................................................................ 175
CONTENTS
vi
8 Technical Specifications ........................................................177
8.1 Software ........................................................................................................ 177
Framework .................................................................................................................................................177
PEAQ Algorithm .........................................................................................................................................178
PSQM Algorithm ........................................................................................................................................179
PESQ Algorithm .........................................................................................................................................180
Echo Algorithm ...........................................................................................................................................181
8.2 Hardware....................................................................................................... 181
POTS Telephony Board .............................................................................................................................181
Audio Interface Option (LynxONE) ............................................................................................................182
Audio Interface Option (Digigram) .............................................................................................................183
OPERA Workstation...................................................................................................................................183
OPERA Portable ........................................................................................................................................183
References.............................................................................................185
Glossary of Terms.................................................................................189
Index .................................................................................................195
Appendix................................................................................................197
SOFTWARE LICENSE AGREEMENT AND LIMITATIONS
1
SOFTWARE LICENSE AGREEMENT AND
LIMITATIONS
General
Conditions deviating from these General Contract Conditions shall not be deemed valid unless we have
confirmed them expressly in writing. Verbal agreements are not valid unless the obligation to confirm
such agreements in writing has been renounced by mutual agreement and in writing.
We shall carry out delivery and installation of the goods and machines as well as instruction of the
operating personnel at the expense of the buyer. Consulting on application and usage shall be given to
the best of our knowledge, based on our experience. The goods and machines delivered are subject to
change. Changes in design and/or shape shall be accepted by the buyer, unless these changes are not
deemed fundamental modifications substantially limiting the purpose of the purchased goods as agreed.
The buyer shall take responsibility for the lawful usage of our machines as stipulated in the laws, rules
and stipulations applicable.
Complaints Concerning Defects
Any complaints concerning deficiencies in quality and performance or the delivery of the correct number
and types of goods agreed, that can be determined by reasonable efforts shall be filed promptly with the
buyer in writing within fourteen days after delivery. Hidden or latent faults shall be notified to the buyer
promptly after discovery. In case of any complaint the buyer shall on our request undertake to promptly
send back the goods concerned in their original packages.
Software License
The accompanying software to this OPTICOM product is licensed, not sold. OPTICOM hereby grants
the user of the OPERA Software (herewith referred to as 'Licensee') with respect to the Licensed Patents,
Licensed Trade Marks and the OPERA Software a non-exclusive, non-transferable, non-assignable, nonsublicensable limited right to Use the licensed number of copies of the OPERA Software, solely to
facilitate the objective quality evaluation of audio signals in accordance with the respective Standard;
provided that:
SOFTWARE LICENSE AGREEMENT AND LIMITATIONS
2
Licensee shall not disable any copy protection mechanism of the OPERA Software provided by
OPTICOM; and Licensee shall prohibit any additional copying of the OPERA Software in whole or in
part, other than the number of licensed copies and other than it is essential for the proper operation of the
OPERA software or for normal security back-up purposes;
Licensee shall not modify, translate, reverse-engineer or de-compile the OPERA Software except to the
extent permitted by law;
Licensee shall not resell, sublicense or otherwise redistribute the OPERA Software;
Licensee shall maintain the OPERA Software in confidence and ensure that it is protected from
unauthorized copying or disclosure by measures that are no less stringent than those it uses to protect its
own valuable information and that are, in any case, no less than reasonable in the circumstances;
Except as expressly granted, Licensee shall have no other rights in the OPERA Software. For the
avoidance of doubt the rights granted shall not include a license to modify, have modified, create and/or
have created derivative works of the OPERA Software or any Algorithm in Source Code form, and to
make, or have made copies of an Executable version other than the licensed number of copies of the
OPERA Software. Under no circumstances will anything in this Agreement be constructed as granting, by
implication or otherwise, a license to any other technology owned and/or licensed by OPTICOM other
than the licensed OPERA Software.
In such a case as either party vests any patent rights in any enhancements and new features in the OPERA
Software, the relevant party, upon request of the other party shall offer to the other party a license with a
scope similar to the license obtained by Licensee against fair, reasonable and non-discriminatory terms
and conditions.
Limitations, Warranty1, Liability
1. OPTICOM shall retain all right, title and interest in and to the OPERA Software, subject to the license
granted. Licensee shall be entitled to establish all proprietary rights for itself in the intellectual property
represented by enhancements and new features, created by Licensee, whether in the nature of trade
secrets, copyrights or patent rights or other rights. OPTICOM shall be entitled to establish all proprietary
rights for itself in the intellectual property represented by enhancements and new features, created by
OPTICOM, whether in the nature of patent rights or other rights.
Nothing herein shall constitute or be construed as:
a requirement that OPTICOM shall file any patent application, secure any patent or maintain any patent
in force, or
an obligation on the part of OPTICOM to furnish any technical information, technical support, software
of any kind or any information concerning pending patent applications of OPTICOM.
2. OPTICOM warrants that at the purchase date it has full power and authority to grant Licensee the rights
granted herein and that it has no knowledge of any pending legal procedures regarding the Licensed
Patents. OPTICOM does not warrant and shall not be liable for the existence of such disposal subsequent
to the coming into force of this Agreement.
1
shall mean "Maengelhaftung" according to the German Civil Code of 01. January 2002
SOFTWARE LICENSE AGREEMENT AND LIMITATIONS
3
3. OPTICOM warrants that the OPERA Software along with the Accompanying Hardware works
according to this documentation ("Users Manual") which is part of the delivery and that the OPERA
Software properly implements the relevant measurement Algorithm in accordance with respective
recommendations.
4. Although all software has been designed and controlled with due care, it has to be assumed that it
steadily undergoes a development process. Should any programming errors be discovered, and reported
to OPTICOM in writing, then OPTICOM, within twelve months from the delivery date, shall be obliged
to correct the deficiencies as far as prescribed by the warranty. OPTICOM shall remedy defects of the
Licensed OPERA Software along with the Accompanying Hardware that may occur. The obligation to
correct errors in compliance with the warranty granted, is limited to the correction of errors. Any such
defects of the OPERA Software along with the Accompanying Hardware shall be repaired by replacing
the software by a new version, or by replacing the system by a new hardware component, which shall be
delivered by OPTICOM without undue delay.
5. If within a reasonable period of time the defective OPERA Software along with the Accompanying
Hardware will have undergone a replacement twice without success, Licensee has the option of
demanding a reduction of the price to be paid or the return/termination of the delivered items. Further
warranties are expressly excluded, e.g. the Licensee is not entitled to claims based on warranty,
a) if the deficiency has been caused by improper usage of the OPERA Software along with the
Accompanying Hardware, faulty installation, usage of unsuitable accessories or improper operation,
or faulty or incomplete programming by licensee, or in case of any modification to the OPERA
Software along with the Accompanying Hardware carried out by Licensee or a third party;
b) if the OPERA Software along with the Accompanying Hardware will not have been maintained or
serviced in accordance with our recommendations and this has caused the deficiency;
c) if the deficiency has been caused by improper modification of the OPERA Software along with the
Accompanying Hardware;
d) if the damage has been caused by an Act of God, e.g. damage by lightning;
e) if the deficiency results from normal wear and tear, especially as far as working parts are concerned.
6. Such claims according to 6.2 – 6.5 shall become statute-barred one year from the date on which such
claims arose, or upon expiry of the legal prescription period, whichever period is shorter.
7. OPTICOM's liability and the liability of it's legal representatives and those employed in fulfillment of
the Agreement, arising from breaches of contract or tort is limited to cases of intent, gross negligence or
recklessness, lack of warranted quality (characteristics) and violation of a material responsibility, which
would jeopardize the contractual objectives.
8. For each individual case of damage, such liability shall be restricted to the foreseeable typical damage
OPTICOM had to expect when contracting the Delivery in consideration of the circumstances known.
The liability for slight negligence shall not exceed an equivalent of US$ 300.000,-.
9. Licensee shall not be entitled to claim any damages against OPTICOM, including damages for indirect
loss, e.g. missed profit, impossibility of performance, positive violation of a contractual duty or failure to
perform. In case of a loss of data, OPTICOM shall only be liable for the expenses required to reconstruct
the lost data using backup files duly created in regular intervals.
10. Any claims arising from Product Liability Law shall not be affected by the aforegoing terms of this
Article.
SOFTWARE LICENSE AGREEMENT AND LIMITATIONS
4
In the event that any afore-mentioned terms or conditions are found to be invalid, unlawful or
unenforceable to any extent, this shall not effect any other terms and conditions agreed herein. The
parties shall endeavor to agree to such amendments which shall in as far as possible effect the economic
intentions expressed therein. In the case of a gap of these terms this shall apply accordingly.
Applicable law and Place of Jurisdiction
If the customer is either a merchant entered into the commercial register, or a legal person under public
law, or a Special Fund under public law, Erlangen shall be agreed as the place of jurisdiction. In all other
cases, the legal place of jurisdiction shall apply. This agreement shall be construed under and governed
by the laws of the Federal Republic of Germany.
PREFACE
5
PREFACE
A Brief How-to Guide to this Manual and How to
Get More Information and Support.
e are delighted to welcome you as our new customer. As you
might know from our company vision -
quality is our business.
Especially when it comes to signal quality of speech and wide
band audio signals. Our new OPERA™ system, which is an
abbreviation for Objective Perceptual Analyzer, reflects a new generation
approach, integrating both the latest experience and standards in the research
and development of perceptual based, objective methods for the determination
of signal quality.
We will introduce the basic operation of OPERA™ and give some guidelines on
the most common applications in this manual. This manual, however, is not
meant to substitute research reports, papers and standards documentation.
Where appropriate, we will refer to the corresponding literature indicated by a
number in squared brackets.
How this Manual is Organized
This manual is organized in seven chapters.
Chapter 1, "Introduction", will briefly explain the OPERA™ measurement. You
will learn about the ideas behind this measurement tool and get an overview of
the OPERA™ product family.
Chapter 2, titled "Test Methodology", provides you with the necessary basic
knowledge about perceptual measurements. This chapter is meant to guide you
to the correct measurement for all the applications you might want to use your
OPERA™ system with.
Chapter 3, "Installation and Setup", will guide you through the installation
procedure and inform you of the hardware requirements if you are using either
the OPERA™ Software Suite or the OPERA™ Toolkit. It is important to read this
chapter carefully when you unpack and install your OPERA™
system for the first
time.
Chapter 4, "Getting to Know The OPERA™ Framework", will outline the basic
concept and explain the operation of the framework program. The framework is
W
PREFACE
6
a summary of the functionalities, which you will always need, regardless of
specific measurement algorithms.
The next two Chapters,
Chapter 5, "Wide Band Audio Quality Testing", and
Chapter 6, "Telephony Band Voice Quality Testing", deal with the specific
measurement setups and methods for both principle applications. Depending
on the options installed in your OPERA™
system, you may need to refer to one
or the other, or both chapters.
Chapter 7, Automation and Programming explains how to use OPERA from
the commandline and outlines other possibilities of automating tasks in OPERA.
These chapters are followed by the
"Technical Specifications", the
"References", a "Glossary" and finally the "Index".
In the Appendix you will find some background information, such as a
collection of papers and articles along with the relevant standards
documentation.
Conventions Used in This Manual
In this manual we will use some conventions in order to ease the understanding
of the operation. For instance,
• all menu options that can be selected will be in bold style,
• all command line parameters that you might need to type will be
printed in the courier type style,
• basic command line keywords will also be in courier bold
type style.
You will also find a lot of figures displaying screen shots. Please note that due to
the ongoing development and software update process, the screen shots of your
system might differ slightly from the examples in this manual. In the case of
extreme differences, please do not hesitate to inform us.
Quite often you will experience gray shaded symbols next to the text
paragraphs. The legend on the left explains the meaning of the most commonly
used symbols. They will point out passages in the manual containing
information, important hints, basic menu options and command line
parameters. The little "book symbol" indicates that more detailed information is
available in the papers, books or articles cited in the references.
SYMBOL-
LEGEND
Information
Important Hint
Menu Options
Enter parameters
see References
PREFACE
7
Further Information and Support
For all questions arising from the use of OPERA™ or that might be related to the
interpretation of measurement data, please make sure to refer to this user
manual and the relevant standards documents. In case the information given
will not be sufficient to answer your question, you can visit our on-line support
section, available from our website:
http://www.opticom.de
We specifically recommend the support section of our website for a report of
known bugs and problems. This section will be available soon, and should help
you to easily check if you encounter an unknown problem.
In case you would encounter a bug or a problem, which is not yet listed on the
support section of our web page, please make sure to contact the OPTICOM
support with a detailed bug report.
Note
:
In case of hardware problems with your PC-workstation please
refer to the hardware documentation of the OEM manufacturer
first. All OPERA™ products will be based on well supported
standard hardware PCs that will be supported world-wide through
the original manufacturer. The OEM manufacturer will be able to
help you in the case of hardware problems related to your PC, for
instance if the system would not boot anymore, in the case of a
hard disk crash, or when encountering problems with the power
supply unit. The same applies to the monitor. OPERA portable
systems are directly supported by OPTICOM.
To find out your nearest support contact for the OEM PC hardware, please see
the support offices section of the accompanying hardware documentation.
For all other problems, please contact OPTICOM, and ask for your local
support representative.
PREFACE
8
CHAPTER 1: INTRODUCTION
9
1 INTRODUCTION
An Introduction to the OPERA™ Measurement
System and the OPERA™ Product Family.
fter reading this chapter you will be familiar with the basic ideas and
the concept behind this tool. The OPERA™ product family will be
outlined at the end of this chapter.
1.1 What is OPERA™?
Compression has become state-of–the-art technology in modern
communications – thus allowing for a great number, diverse and inexpensive
new components of the information age, such as: mobile phones, VoIP, MP3
internet audio, radio and TV satellite networks, DAB, DVD, and many more.
On the other hand the economic benefit of lowering data rates to a minimum is
contradictory to clear sound. In spite of "all digital technology“, sound quality
and the intelligibility of speech have become issues again, and are of much
more impact than in those "good old analog days".
Our new generation of quality testers, called OPERA™ – short for "Objective
Perceptual Analyzer“ – represent the latest development in objectively
evaluation and assure the quality of compressed speech and wide-band audio
signals by modelling the human ear: OPERA™ is your digital ear. OPERA™ is not
only suitable to assess a single processing device, with OPERA™ you can
achieve a comprehensive analysis of the end-to-end quality, from the studio
source to the receiver, or from the caller to the callee. And because OPERA™
works quite similar to the human ear, it is able to distinguish between
imperceptible, and more or less annoying transmission errors.
Other than traditional measurement methods (like S/N, THD+N), the new
OPERA™ system is able to simulate the subjective evaluation of human subjects.
The analysis is based on the most recent perceptual techniques, such as PEAQ,
PESQ and PSQM. As a major advantage, OPERA™ employs the same kind of
natural stimulus for a measurement as in practical operation: human speech or
music program material. Moreover, this makes it possible to monitor the
quality during network operation. Cultural and language differences may be
taken into account by the evaluation as well. As a consequence of the novel
approach to measure the perceived audio quality instead of signal
characteristics, it is possible for the first time to gain an objective quality metrics
which truly characterizes the quality of service („QoS“) of a network.
Chapter
À
CHAPTER 1: INTRODUCTION
10
OPERA’s flexible scalability may range from a single stand alone tester up to
powerful network-wide setups with distributed systems sharing information over
TCP/IP. OPERA™ may be used interactively as an analyzer, or runs fully
automated according to a predefined schedule.
The open framework concept of OPERA™ allows the addition of advanced
measurement algorithms as plug-ins in the future as soon as they will become
available. In addition, user defined measurement algorithms may be integrated
upon request.
Basically, there are two different versions of OPERA™, a Telecom Version and a
Broadcast Version. Some of the features of the
Telecom Version are at the time:
• ITU-T P.862/PESQ
• ITU-T P.861/PSQM
• PSQM+ (PSQM improved for GSM)
• Echo measurement with real speech
• Delay measurement
• Interfaces to file (*.wav), loop start (a/b), E1/T1, [VoIP, and wireless to
follow]
• ...
Some of the features of the
Broadcast Version are at the time:
• ITU-R BS.1387/PEAQ
• Delay measurement
• Real time measurement
• Interfaces to file (*.wav), analog XLR balanced (20 bit) and digital AES/EBU
• ...
All standard measurement algorithms are based on the reference code, which
was used for the standardization, and all algorithms are tested and verified to be
fully conforming to the standards.
OPERA™ is available as a software version, a completely pre-installed portable
system and a completely pre-installed rackmount system. The Workstation
version is not available as a standard product anymore. In addition, we offer
custom tailored and OEM solutions.
CHAPTER 1: INTRODUCTION
11
1.2 Getting to Know the OPERA™ Product
Family
1.2.1 OPERA™ Measurement System
The OPERA™ Measurement System comprises both hardware requirements and
software. It comes completely pre-configured. It is available as a portable and a
rackmounted version. The rackmounted version is fully compatible with the
portable version and not described separately. In addition there exists a
worstation type version which is not available anymore, but still supported and
described in this manual.
Workstation Version
The basic configuration of the Workstation Version of OPERA™ includes a
completely equipped high performance PC system with selected components
and the OPERA™ framework software. If assessment of speech quality is
required, the Telecom configuration is provided. Audio quality can be
measured using the broadcast configuration. Combinations of telecom and
broadcast configurations of the OPERA™ Measurement System are also
available.
For a detailed description of what is included in the delivery, please refer to
Section 3.2.1.
Figure 1.1 shows a photograph of the OPERA™ Workstation.
Figure 1.1: The Workstation version of OPERA™
Portable PC Version
The basic configuration of the Portable PC Version of OPERA™ includes a
completely equipped portable high performance Dual Processor PC system
with selected components and the OPERA™ framework software.
As with the Workstation Version, there are different configurations available, the
Telecom version for speech quality assessment, and the Broadcast configuration
for measuring audio quality. Both configurations come pre-configured for live
CHAPTER 1: INTRODUCTION
12
measurements, the corresponding required interfaces and cables will be
provided. Of course assessing prerecorded files is possible too. For a detailed
description of the delivery , please refer to Section 3.2.2. Again, combinations
of Telecom and Broadcast Configurations of the OPERA™ Measurement System
are available.
Figure 1.2 shows a photograph of the OPERA™ Portable.
Figure 1.2: The Portable PC version of OPERA™
1.2.2 OPERA™ Software Suite
OPERA™ is also available as a Software Suite, which is delivered without any
hardware. A description of the corresponding hardware requirements can be
found in the "Installation and Setup" chapter in Section 3.1.2 and in the
"Technical Specifications" chapter.
In contrast to the Workstation or the Portable PC Version of OPERA™, the
Software Suite supports file based measurements only. Live measurements are
not supported. Measurement functionality and performance will be the same as
for the hardware based OPERA™ system in the file based mode. However,
processing time may vary with the available processing power of the PC.
Note:
Even if there is an audio board installed in your system, no live
measurements are possible with the Software Suite.
CHAPTER 2: TEST METHODOLOGY?
13
2 TEST METHODOLOGY
Essential Knowledge about Perceptual
Measurements and a Guidepost to the Correct
Measurement Method.
his chapter describes listening test methods which are modeled by the
OPERA™ system, and a description of the underlying concept of the
proposed algorithms for perceptual measurement and the international
standardization of perceptual audio measurement techniques. In
addition, this chapter will provide a guide to the correct measurement method
for your applications.
2.1 Assessing Quality
Until recently, the only widely accepted assessment procedures for audio or
speech codecs were listening tests, due to the lack of international standards for
measuring the perceived audio quality.
Historically related to the assessment of telephone connections, useful methods
for testing telephone band speech signals were first standardized within the ITUT
1
. Recommendation P.800 [ITUT800] defines the absolute category rating test
method (ACR) which has been used for the assessment of speech codecs since
1993. Within the ACR test method, the ITU five grade impairment scale is
applied (see Table 2.1). In the telecommunication environment, testing is
done without a comparison to an undistorted reference. This copes with a
typical situation of a phone call, where the listener has no access to a
comparison with a reference, e.g. the original voice of the other party.
However, it should be noted that the listening test according P.800 could be
regarded as a comparison between a test signal and a reference "in the mind" of
the listener. The reason for this is that the listener is very familiar with the
natural sound of a human voice.
For comparison reasons, and in order to be able to merge the results of different
individuals, it is necessary to adjust the listeners' opinions to an absolute scale.
For this purpose, predefined examples with well defined noise insertions of
fixed modulated noise reference units (MNRU, [ITUT810]) are presented at the
beginning of a test. Each sample represents an example distortion
corresponding to the ITU-T version of the five grade impairment scale.
1
International Telecommunication Union, Geneva, (former CCITT), see also
http://www.itu.org
Chapter
T
ITU-T P.800
CHAPTER 2: WHICH MEASUREMENT DO I NEED?
14
Impairment Grade
Excellent 5
Good 4
Fair 3
Poor 2
Bad 1
Table 2.1: The ITU-T five-grade impairment scale
Based on these test conditions a population of typically 20 to 50 test subjects
will be presented with an identical series of speech fragments. Every test subject
will be asked to score each sample by applying the impairment scale. After
statistical processing of the individual results, a Mean Opinion Score (MOS) can
be calculated. With thorough setups, such test results can be reproduced quite
well, even at different locations. It goes without saying that the effort needed in
terms of subjects and time is tremendous. It is clear that such test methods can
not be applied within a practical or field environment in the daily life.
The ITU has also recommended a test procedure to assess wide band audio
codecs on the basis of subjective tests. Subjective assessments of low bit rate
audio codecs in the past always targeted an almost transparent quality. For this
reason, the test method focuses on the comparison of the coded/decoded signal
to the unprocessed original reference. The relevant recommendation is known
as BS.1116, titled "Methods for the Subjective Assessment of small Impairments
in Audio Systems including Multichannel Sound Systems" [ITUR1116] which
was issued by the ITU-R
2
in 1994 and was updated in 1997.
The test method, which is recommended by BS.1116, is referred to as "doubleblind triple-stimulus with hidden reference". It is extremely sensitive and allows
for the accurate detection of small impairments. The grading scale used should
be treated as continuous with "anchors" derived from the ITU-R five-grade
impairment scale according to ITU-R BS.562 [ITUR562]. It is depicted in Table
2.2.
Impairment Grade SDG
Imperceptible 5.0 0.0
Perceptible, but not annoying 4.0 -1.0
Slightly annoying 3.0 -2.0
Annoying 2.0 -3.0
Very annoying 1.0 -4.0
Table 2.2: The ITU-R five-grade impairment scale
The analysis of the results from a subjective listening test is generally based on
the Subjective Difference Grade (SDG) and is defined as:
Signal ReferenceTest Under Signal
GradeGradeSDG −=
Provided that the listener correctly assigns the hidden reference signal, the SDG
values will range from 0 to –4, where 0 corresponds to an imperceptible
impairment and –4 to an impairment judged as very annoying. The assignment
of the SDG scale is shown in the last column in Table 2.2.
2
Radiocommunication Sector of the ITU (former CCIR)
ITU-R BS.1116
CHAPTER 2: TEST METHODOLOGY?
15
In contrast to the listening test according to ITU-T P.800, an explicit comparison
between the test signal and a reference signal is needed in the case of BS.1116,
since the listener never knows how the original signal sounds.
This method was applied in a variety of international verification tests in the
past. However, keep in mind that because of the scope of BS.1116 it can be
applied to small impairments only, which means a practical limitation to almost
"transparent" studio quality. Another issue which has been discussed among
experts, is the recommendation to use the scale at a resolution of one decimal
place, resulting in 41 (!) discrete steps. There are indications that for some
subjects this is too great a choice, and furthermore the meaning of the
impairment anchors is interpreted differently [KARJ85].
Because of the restrictions to small impairments, there is consensus among
experts that other methods are needed for very low bit rate tests (i.e. of large
impairments). Various approaches have been introduced and work is currently
in progress in several task groups, e.g. the MPEG standardisation work
[GILC96]. The methods according to ITU-T P.800 were adopted for some
assessments to overcome the problem of a "gap" for a useful recommendation
on testing significantly impaired wide band audio signals. While in principle
they seem to be better suited for impaired music signals when compared to the
BS.1116 method, it’s exploitation for very low bit rate audio coding applications
still remains questionable, as there are no clearly defined example distortions in
such a case. The scale was derived from telephone speech quality, and is not
well defined when translated to music coding. The achieved results may
therefore significantly depend on the subjective interpretation of the impairment
levels.
At the time of drafting this manual, an advanced listening test procedure has
been advised by an EBU expert group, known as "MUSHRA". MUSHRA stands
for "Multiple Stimulus With Hidden Reference Anchors". The new method
targets testing significantly impaired audio signals, such as those derived at very
low bit rates. MUSHRA is expected to become adopted as an international
recommendation by the ITU working party 10-11Q. As soon as more
experience has been gained, OPTICOM will provide its customers with more
information at
www.peaq.org.
2.2 Advanced Audio Measurements
Employing "Perceptual Modeling"
Assessing the quality was a pending issue during the years of the development
of compression schemes. Consequently, the idea of substituting the subjective
tests by objective, computer based methods has been an ongoing focus of
research and development. Early work motivated through the development in
speech coding was reported in [KARJ85]. Since then several methods were
introduced.
The underlying concepts of the proposed algorithms for perceptual
measurement techniques are all quite similar. The common structure of these
algorithms is depicted in Figure 2.1. The process of human perception is
modelled by employing a difference –measurement-technique which compares
both, a reference signal (i.e. the "input" signal to a codec) and a test signal (i.e.
MUSHRA
CHAPTER 2: WHICH MEASUREMENT DO I NEED?
16
the "output" signal of the codec). First, the algorithms process an ear model for
the reference and the test signal, in order to calculate an estimate for the audible
signal components. The result can be imagined as the "internal representation"
inside the human auditory system. Comparing the internal representations of
the reference, and the test signal leads to an estimate of the audible difference.
To derive an overall quality figure, this information, which is a function of time,
must be processed accordingly, like the human brain of a subject would do in a
listening test. The respective part of processing within an algorithm is referred to
as cognitive modelling. A total quality figure will be derived as the final result,
which can be compared to a MOS ("Mean Opinion Score") resulting from a
listening test.
Perceptual
Model
Test
(=Output
)
Reference
(=Input)
Perceptual
Model
Internal
Representation
Internal
Representation
Comparison
Cognitive
Model
Total Qualit
y
Figure
Audible
Difference
Figure 2.1: The underlying concept for perceptual measurement
The evaluation of the internal representation is often related to an estimate of
the masked threshold. This estimate is based on data found in a number of
psychoacoustic experiments, such as those conducted by Zwicker [ZWIC67,
ZWIC82]. Most of these experiments model certain isolated effects of the
human auditory system. One way to design a perceptual measurement
algorithm is to generalize these model data and apply them to complex audio
signals. This was for example the approach outlined in the NMR Algorithm in
1987 [BRAN87, BRAN89, BRAN92, GILC96, HERR92a, HERR92b, KEYH93,
KEYH96, KEYH98, SEIT89]. Similar approaches were used for PAQM and
PSQM [BEER95, BEER92, BEER94].
2.3 International Standardization
International standardization of perceptual audio measurement techniques was
mainly driven by two expert groups within the International
Telecommunications Union (ITU).
Within the telecommunication sector of the ITU, in 1996 study group 12
finalized recommendation P.861 [ITUT861] for the objective analysis of speech
codecs. After a wide-ranging comparison of proposed methods, the group opted
for the PSQM algorithm. PSQM correlated up to 98 percent with the scores of
subjective listening tests.
ITU-T P.861
CHAPTER 2: TEST METHODOLOGY?
17
Since P.861 was mainly developed for application to isolated speech codecs in
mobile networks, a new measure was required to cope with real networks as
well as packet based transmission. Driven by this demand for a verified test
procedure for VoIP, an expert group within ITU-T SG12 has been working on
an improved speech quality model. After a competitive phase, the new model
"PESQ" has been devised. PESQ stands for "Perceptual Evaluation of Speech
Quality". It combines a further refinement of PSQM and PAMS. Extensive tests
showed PESQ's superior performance especially for VoIP applications. In
February 2001, PESQ was accepted as the ITU-T Rec. P.862. More information
on PESQ can be found at
www.pesq.org.
Within the study period 1994 – 1998, the ITU-R had established task group
10/4 with the scope to recommend an objective, perception based model to
evaluate the quality of wide band audio codecs. After collecting a set of
proposals, including the most popular ideas such as NMR, PAQM, PERCEVAL,
POM and others, the group of model proponents opted for a joint collaboration
to derive an improved model. In 1998, two versions of this new model were
presented: A "Basic" version, featuring a low complexity approach, and an
"Advanced" version for higher accuracy at the trade-off of higher complexity.
After thorough verification, the model was recommended as a measure for the
perceived audio quality ("PEAQ") under recommendation BS.1387 in late
1998.
Both standards, ITU-T P.861, and ITU-R BS.1387, currently represent the stateof-the-art technique for the objective evaluation of the perceived audio quality
of audio codecs. Both techniques were derived from modelling the
corresponding subjective experiment by an algorithm based approach. Thus it
is essential to understand the scope of the modelled subjective experiment
when trying to interpret the calculated results.
Data rate
64 kbit/s
16 kbit/s
128 kbit/s
typ. Music - Codec*
typ. Speech - Codec*
subjective objective
ITU-R BS.1116 ITU-R BS.1387
(PEAQ)
ITU-T P.800 ITU-T P.861/862
(PSQM / PESQ)
*Only examples,
no fixed
threshold due to
the continous
trend to furthe
r
reduce bit rates
5.0
4.0
3.0
2.0
1.0
Impairment Scale
triple stimulus,
double-blind with
hidden reference
5
4
3
2
1
Impairment Scale
Absolute
Category
Rating
Figure 2.2: Overview on subjective and objective recommendations
Figure 2.2 summarizes the subjective test procedures and their
corresponding objective counterpart in the context of typical data rate limits. As
ITU-T P.862
ITU-R BS.1387
CHAPTER 2: WHICH MEASUREMENT DO I NEED?
18
mentioned earlier, the threshold between both worlds - broadcasting and
telecommunication - is floating due to the steadily attempt to further reduce the
bit rates by more efficient coding schemes. Consequently, the overall data rate
scale depicted in the figure should be taken as a course indicator only.
2.4 Which "Measurement" for Which
"Application"?
A summary of the principle assessment scenarios and the corresponding
measurement algorithms applications with OPERA™ follows. This section can
be used as a reference when uncertain which measurement algorithm to apply.
First, remember the recommended perceptual measurement techniques always
try to model the underlying subjective experiment of the corresponding
listening test.
To decide which one is the proper experiment, remember to ask yourself the
following questions:
• Is this an assessment of
wide band audio signals (music, or
bandwidth > 16kHz)?
• Would the subject be able to compare the test signal with the
original
reference signal?
If the answer to both questions is YES, you should apply the PEAQ algorithm.
If the answer to both questions is NO, you should apply PESQ, (or PSQM if
required).
If none of the above seems to apply, an experimental situation outside the
scope of both measurements should probably be considered. In this case,
always consider how subjects would behave in a listening test. In some cases,
however, you may also find that even subjects would not be able to properly
score the sound quality.
In Figure 2.3 the principle setup for a BS.1116 listening test is shown.
Figure 2.4 shows the corresponding situation for a P.800 compliant test
setup.
CHAPTER 2: TEST METHODOLOGY?
19
DUT
(device under test)
y
`
?
ORIGINAL
;
TEST
5.0
4.0
3.0
2.0
1.0
BS.1116 Setup
5.0
4.0
3.0
2.0
1.0
wide band audio: YES
comparison with reference: YES
Ö PEAQ
Figure 2.3: Illustration of the principle of BS.1116
DUT
(device under test)
y
`
?
ORIGINAL
§
TEST
5
4
3
2
1
P.800 Setup
wide band audio:
NO
comparison with reference:
NO
Ö
PESQ or PSQM
Figure 2.4: Illustration of the principle of P.800
CHAPTER 2: WHICH MEASUREMENT DO I NEED?
20
2.5 Selection of the Reference File
As a rule of thumb, the reference file should be a signal that comes as close as
possible to the kind of signal which shall be applied to the device under test in
real life. E.g., if you design a special headset for female call center agents, you
should use a test stimulus that contains mostly female speech. If the device
should be used by male and female users as well as children, you should
perform separate tests with typical stimuli for each of these cases. For the
assessment of MPEG audio codecs that are used for the transmission of high
quality music between broadcast studios, real music should be used. Especially
with wide band music codecs a variety of at least six to ten different test
samples should be selected, since the performance of audio codecs differs
widely depending on the test material.
The duration of the test sequence should be within the range of approximately
four to eight seconds. Longer tests will lead to averaging effects (short distortions
may be averaged down by a long but almost perfect transmission) and shorter
sequences may not be long enough to contain representative parts of the signal.
If for any reason very long reference files are desired, OPERAs feature of
measuring just a short sequence out of the entire input signals could be
selected. Details regarding this feature under the Trigger option menu are
explained in chapter 4.
The sample rate of the reference file is frequently already defined by the
algorithm that shall be used for the evaluation of the recorded data. PEAQ
according to ITU-R BS.1387 for example requires 48kHz sample rate, although
the implementation in OPERA will deliver reliable results at 44.1kHz, too. Most
speech quality measures are defined for 8 and 16kHz sample rate only. For
more details, refer to the description of the individual algorithms or the standard
documents that apply.
The selection of the sample format should mainly be driven by considering the
capabilities of the underlying hardware. While using the audio interfaces
provided by OPERA, it makes sense to select the 16bit linear. Since currently all
measures use 16bit linear internally, any higher resolution, although supported
by the hardware, will not result in more accurate measurements. When
performing test calls with the voice board, the sample format should be 8bit
mu-law or 8bit A-law (G.711). Otherwise the measurement will include at least
one more step of encoding, since the DSP on the voice board will convert all
input data back to G.711.
A set of typical wide band audio examples is mentioned in the ITU-R rec.
BS.1387. Speech samples are also provided by the ITU-T, in the Series P
Supplement 23.
2.6 How to Assess "Signal Enhancers"
Signal enhancers are pieces of equipment that try to make the processed signal
sound better than the original signal, like e.g. noise reduction systems etc.
When the input signal of the enhancer is taken as the reference and the output
signal as the test signal of any perceptual measure, the result will usually be the
opposite of what is expected. In general, the enhanced signal will be graded
down the more, the better your enhancer works. This is because perception
CHAPTER 2: TEST METHODOLOGY?
21
based measurement algorithms assume that any audible difference between the
two input signals is a distortion, and by definition the "enhanced" signal will
sound different than the unprocessed signal.
To get around this, a clean signal as the reference file (R) is recommended. This
shall be distorted artificially, which results in signal D, the distorted reference
signal. Signal D may now be used as the input to the enhancer. The output of
the enhancer will be E, the enhanced signal.
When assessing speech quality at this time, the clean reference R and the
enhanced signal E as the input signals of OPERA should be chosen. The grade
calculated by the measurement algorithm now indicates how similar the
enhanced signal sounds to the clean reference. This also implies that based on
the measurement no statement can be made weather the enhanced signal
sounds better than the original signal or not.. If the measurement result shall be
compared to the result obtained from a listening test, it is important to
remember that the question to the subjects must be "How much does the
enhanced signal differ from the reference signal?" and not "Does the enhanced
signal sound better or worse than the reference signal?". There is no
standardized measure available today which answers the second question,
which is frequently a matter of personal taste.
Going one step further, calculate the gain achieved by the enhancer when
relating the final MOS derived this way, to the MOS achieved by comparing the
clean reference (R) to the distorted reference (D). Figure 2.5 shows a sketch of
such a setup.
Figure 2.5: Setup for measuring signal enhancers.
CHAPTER 2: WHICH MEASUREMENT DO I NEED?
22
CHAPTER 3: INSTALLATION AND SETUP
23
3 INSTALLATION AND
SETUP
Using OPERA™ for the first time.
lease review this chapter before you continue. Checking the complete
contents of your OPERA™ system may help in the future to solve
potential problems.
This chapter includes the installation and setup information for the
whole OPERA™ product family. Please refer to the section of the product you
have purchased.
3.1 OPERA™ Software Suite
3.1.1 Unpacking the Software
After unpacking the Software Suite, please check the delivery for completeness
first. The delivery should include the following parts:
• One OPERA™ Software Suite CD.
• One CD with reference wave files and the conformance test set
(PEAQ only).
• One dongle (hardware key, a small grey box which is to be
attached to the printer port).
• This manual.
3.1.2 System Requirements
Please verify that your computer meets or exceeds either of the following
system requirements:
• >=128MB of RAM, 256MB are recommended
• Screen resolution 1024*768, >=64k colors.
Chapter
P
CHAPTER 3: INSTALLATION AND SETUP
24
• NVIDIA TNT2 compatible graphics adapter (others may work, but
are not yet tested).
• Windows NT 4.0 SP4 or SP5, Windows 2000 or Windows XP.
3.1.3 Installation and Setup
To install the OPERA™ Software Suite, follow the steps below, where "X:\"
represents the CD-ROM drive containing the OPERA™ setup CD. If you want to
update from a previous version, there is no need to uninstall the previous
version. The setup program will automatically do this for you
1. Attach the dongle to the parallel interface port of your computer.
2. Start the OPERA™ setup program ( X:\ OperaSetupVxxx.exe).
3. Install the required options if any are offered.
4. Reboot the computer.
3.1.4 Verification
If your installation is running too slow, please check if the system is running out
of memory. If this is the case, memory will be swapped out to the harddisk and
accessing this part of the memory is a million times slower than accessing real
memory. If the OPERA Software Suite requires too much memory, adjust the
size of OPERA's history buffer through a parameter in the registry. Please ask a
specialist for assistance, if unfamiliar with the registry and use of regedit.
Modifying the registry may seriously harm your Windows installation and even
prevent it from booting.
In order to adjust this parameter, open regedit and look for the key:
HKEY_LOCAL_MACHINE\Software\Opticom\Opera\Memory\MemoryReserved
A good value for this key is 130 000 000 on a machine with 256MB RAM. The
parameter defines how much of the physical memory is left free by OPERA.
3.2 OPERA™ Measurement System
3.2.1 Workstation version
Unpacking the System
After unpacking the system check the delivery for completeness first. The
delivery should include the following parts:
• PC workstation.
• Monitor (not included in the case of international shipments).
Loading...