How it Works
BUCHI
Loading...
NIRCal
User Manual
283 pgs10.7 Mb9

BUCHI NIRCal User Manual

Download
NIRCal 5.5
Operation Manual
11593588 en
Imprint
Product Identification: Operation Manual, NIRCal 5.5 115 93 58 8 e n
Publication date:
BÜCHI Labortechnik AG Meierseggstrasse 40 Postfach CH-9230 Flawil 1
E-Mail: quality@buchi.com
BUCHI reserves the right to make changes to the manual as deemed necessary in the interest of experience; especially in respect to structure, illustrations and technical depth.
This manual is copyright. Information from it may not be reproduced, distributed, or used for competitive purpos­es, nor made available to third parties. The manufacture of any component with the aid of this manual without prior written agreement is also prohibited.
Table of Contents
1 Introduction .................................................................................................................................. 10
1.1 ....... Introduction to NIR.......................................................................................................... 10
1.1.1 Spectroscopy .................................................................................................................. 10
1.1.2 Chemometrics ................................................................................................................ 11
1.1.3 Cluster calibration ........................................................................................................... 12
1.1.4 Quantitative calibration ................................................................................................... 14
1.2 ....... Introduction to NIRCal .................................................................................................... 18
1.2.1 General ........................................................................................................................... 18
1.2.2 Starting NIRCal ............................................................................................................... 19
1.2.3 NIRCal Project ................................................................................................................ 20
1.2.4 Spectra Overview ........................................................................................................... 20
1.2.5 Properties Overview ....................................................................................................... 21
1.2.6 Calibration Validation Methods ....................................................................................... 22
1.2.7 Calibration Methods........................................................................................................ 22
1.2.8 Calibrations Overview .................................................................................................... 23
1.2.9 Matrices Overview .......................................................................................................... 24
1.2.10 Data ................................................................................................................................ 24
1.2.11 Selections ....................................................................................................................... 25
1.2.12 Graphics ......................................................................................................................... 27
1.2.13 Protocols ......................................................................................................................... 29
2 Tutorials 31
2.1 ....... Qualitative ....................................................................................................................... 31
2.1.1 Qualitative Tutorial.......................................................................................................... 31
2.1.2 Flow Chart Qualitative .................................................................................................... 31
2.1.3 Selecting the Calibration Method ................................................................................... 32
2.1.4 Selecting the Calibration and Validation Spectra ........................................................... 32
2.1.5 Selecting the Calibration Wavelengths........................................................................... 32
2.1.6 Selecting the Calibration Properties ............................................................................... 33
2.1.7 Applying Data Pretreatments ......................................................................................... 33
2.1.8 Performing a calibration (calculation) ............................................................................. 33
2.1.9 Primary Principal Components ....................................................................................... 33
2.1.10 Secondary Principal Components .................................................................................. 34
2.1.11 Judging the Calibration ................................................................................................... 34
2.1.12 Save Calibration - Lifecycle ............................................................................................ 34
2.1.13 SIMCA ............................................................................................................................ 35
2.2 ....... Quantitative .................................................................................................................... 35
2.2.1 Quantitative Tutorial ....................................................................................................... 35
2.2.2 Flow Chart ...................................................................................................................... 35
2.2.3 Selecting the Calibration Method ................................................................................... 36
2.2.4 Selecting the Calibration and Validation Spectra ........................................................... 36
2.2.5 Selecting the Calibration Wavelengths........................................................................... 36
2.2.6 Selecting the Calibration Properties ............................................................................... 37
2.2.7 Applying Data Pretreatments ......................................................................................... 37
2.2.8 Performing a calibration (calculation) ............................................................................. 37
2.2.9 Primary Principal Components ....................................................................................... 38
2.2.10 Secondary Principal Components .................................................................................. 38
2.2.11 Judging the Calibration ................................................................................................... 38
2.2.12 Save Calibration - Lifecycle ............................................................................................ 39
2.2.13 Create a Quantitative Calibration with Cross Validation ................................................ 39
3 Chemometrics .............................................................................................................................. 41
3.1 ....... Calibration Methods........................................................................................................ 41
3.1.1 Principal Component Analysis: PCA .............................................................................. 41
3.1.2 Cluster: CLU ................................................................................................................... 46
3.1.3 SIMCA ............................................................................................................................ 49
3.1.4 Transform Cluster to SIMCA .......................................................................................... 50
3.1.5 Multiple Linear Regression: MLR ................................................................................... 53
NIRCal 5.5 Manual, Version A 3
NIRCal 5.5 Software Manual
3.1.6 Principal Component Regression: PCR ......................................................................... 54
3.1.7 Partial Least Squares Regression: PLS ......................................................................... 54
3.2 ....... Calibration Validation Methods ....................................................................................... 55
3.2.1 Validation Set (VS) ......................................................................................................... 55
3.2.2 Cross Validation (CV) ..................................................................................................... 55
3.3 ....... Selections ....................................................................................................................... 69
3.3.1 Calibration Method Selection ......................................................................................... 69
3.3.2 Validation Method Selection ........................................................................................... 70
3.3.3 Data Sets ........................................................................................................................ 70
3.3.4 Edit Data Sets Dialog ..................................................................................................... 71
3.3.5 Spectra Data Set ............................................................................................................ 72
3.3.6 Wavelength Data Set ..................................................................................................... 78
3.3.7 Properties Data Set ........................................................................................................ 82
3.3.8 PC Data Set .................................................................................................................... 85
3.4 ....... Calibration Wizard .......................................................................................................... 86
3.4.1 Calibration Wizard .......................................................................................................... 86
3.5 ....... Pretreatments ................................................................................................................. 89
3.5.1 Pretreatments ................................................................................................................. 89
3.5.2 Available Pretreatments ................................................................................................. 91
3.5.3 Normalization .................................................................................................................. 92
3.5.4 Offset ............................................................................................................................ 100
3.5.5 Smoothing .................................................................................................................... 105
3.5.6 Derivatives .................................................................................................................... 107
3.5.7 Transformation ............................................................................................................. 113
3.5.8 Filter .............................................................................................................................. 118
3.6 ....... Primary PCs ................................................................................................................. 120
3.7 ....... Adjusting Primary PCs (Calculation Parameters) ........................................................ 123
3.8 ....... Secondary PCs ............................................................................................................. 124
3.8.1 Secondary PCs for qualitative Calibration .................................................................... 124
3.8.2 Secondary PCs for quantitative Calibration ................................................................. 127
3.9 ....... Secondary PC Selection .............................................................................................. 129
3.9.1 Secondary PC Selection using NIR-Explorer ............................................................... 129
3.9.2 Secondary PC Selection using Edit Data Sets Dialog ................................................. 130
3.10 ..... PCs (Factor Selection Wizard) ..................................................................................... 131
3.11 ..... Blow Up Limits .............................................................................................................. 131
3.11.1 Blow Up limits quantitative calibration .......................................................................... 131
3.11.2 Blow Up limits qualitative calibration ............................................................................ 132
3.12 ..... Outlier Detection ........................................................................................................... 133
3.13 ..... Q-Value ........................................................................................................................ 136
3.14 ..... Change X-Unit Wavelength / Wavenumber ................................................................. 136
3.15 ..... Convert and Import spectra from other instruments to DB .......................................... 136
3.15.1 Procedure of Spectra Import ........................................................................................ 137
3.15.2 Procedure of Spectra Conversion: ............................................................................... 137
3.15.3 Explanation of the Spectra Converter .......................................................................... 139
3.16 ..... Calibration Handling ..................................................................................................... 140
3.16.1 Calibrations without Lifecycle ....................................................................................... 141
3.16.2 Calibrations with Lifecycle ............................................................................................ 141
3.16.3 Project Handling ........................................................................................................... 145
3.17 ..... Protocols ....................................................................................................................... 145
3.17.1 Calibration Protocol Qualitative .................................................................................... 145
3.17.2 Calibration Protocol Quantitative .................................................................................. 152
3.17.3 Calibration Protocol Quantitative CV ............................................................................ 157
3.17.4 Prediction Protocol ....................................................................................................... 162
3.18 ..... Matrices ........................................................................................................................ 168
3.18.1 How to display matrices ............................................................................................... 168
3.18.2 Calibration Residual Limits ........................................................................................... 169
3.18.3 Calibration Residual Limits with Blow Up ..................................................................... 169
3.18.4 Calibration Score Limits ............................................................................................... 170
3.18.5 Cluster per Property ..................................................................................................... 170
3.18.6 Consistency .................................................................................................................. 170
3.18.7 Correlation Spectra....................................................................................................... 171
3.18.8 Correlation Wavelength ................................................................................................ 171
4 NIRCal 5.5 Manual, Version A
Table Of Contents
3.18.9 Creator .......................................................................................................................... 171
3.18.10 CV C-Set PRESS ......................................................................................................... 172
3.18.11 CV Group Index ............................................................................................................ 172
3.18.12 CV Leverages ............................................................................................................... 172
3.18.13 CV Predicted Property .................................................................................................. 172
3.18.14 CV Property Residuum ................................................................................................. 173
3.18.15 CV Regression Coefficients[1] ..................................................................................... 173
3.18.16 CV Regression Coefficients[1] Mean ........................................................................... 173
3.18.17 CV Regression Coefficients[1] Sdev ............................................................................ 173
3.18.18 CV Regression Coefficients[1] t-test ............................................................................ 174
3.18.19 CV Regression Coefficients[1] Variance to final Calib ................................................. 174
3.18.20 CV SECV ...................................................................................................................... 175
3.18.21 CV Spectra Residuals .................................................................................................. 175
3.18.22 CV V-Set PRESS ......................................................................................................... 175
3.18.23 C-Set BCAP-PRES....................................................................................................... 175
3.18.24 C-Set BIAS ................................................................................................................... 176
3.18.25 C-Set Mean Property .................................................................................................... 176
3.18.26 C-Set Mean Spectrum .................................................................................................. 176
3.18.27 C-Set PRESS ............................................................................................................... 176
3.18.28 C-Set Property Dependencies ...................................................................................... 177
3.18.29 C-Set Regression Coefficient ....................................................................................... 177
3.18.30 C-Set Regression Intercept .......................................................................................... 178
3.18.31 C-Set Regression Slope ............................................................................................... 178
3.18.32 C-Set SEE (SEC) ......................................................................................................... 178
3.18.33 C-Set Spectra Residuum Limits ................................................................................... 179
3.18.34 C-Set Spectra Residuum Max ...................................................................................... 179
3.18.35 C-Set Spectra Residuum Min ....................................................................................... 180
3.18.36 C-Set Spectra Residuum MinMax ................................................................................ 180
3.18.37 C-Set Std Orig Property ............................................................................................... 180
3.18.38 C-Set Std Pred Property ............................................................................................... 180
3.18.39 C-Set X-PRESS ............................................................................................................ 180
3.18.40 Durbin-Watson Factor Statistics ................................................................................... 181
3.18.41 Eigenvalue .................................................................................................................... 182
3.18.42 Instrument ..................................................................................................................... 183
3.18.43 Leverages ..................................................................................................................... 183
3.18.44 Loadings or Principal Components .............................................................................. 183
3.18.45 Original Property ........................................................................................................... 184
3.18.46 Original Spectra ............................................................................................................ 185
3.18.47 PCs or Factor ............................................................................................................... 185
3.18.48 PCR B-Matrix ................................................................................................................ 186
3.18.49 Predicted Property ........................................................................................................ 186
3.18.50 Pretreated Property ...................................................................................................... 187
3.18.51 Pretreated Spectra ....................................................................................................... 187
3.18.52 Property ........................................................................................................................ 188
3.18.53 Property Adjacency ...................................................................................................... 188
3.18.54 Property Box Center ..................................................................................................... 189
3.18.55 Property Box Center Uniformity .................................................................................... 189
3.18.56 Property Box Radii........................................................................................................ 189
3.18.57 Property F-Test backward ............................................................................................ 189
3.18.58 Property F-Test forward ............................................................................................... 190
3.18.59 Property Interference .................................................................................................... 190
3.18.60 Property missing Values ............................................................................................... 190
3.18.61 Property Nearest Neighbor ........................................................................................... 190
3.18.62 Property Residuum ....................................................................................................... 191
3.18.63 Property Score Diffusion .............................................................................................. 191
3.18.64 Property Score Disorder ............................................................................................... 192
3.18.65 Property Score Extension ............................................................................................. 192
3.18.66 Property Wavelength Regression ................................................................................. 192
3.18.67 Quant Calib. Statistics .................................................................................................. 192
3.18.68 Q-Values ....................................................................................................................... 193
3.18.69 Real time ...................................................................................................................... 194
3.18.70 Reconstructed Spectra ................................................................................................. 195
NIRCal 5.5 Manual, Version A 5
NIRCal 5.5 Software Manual
3.18.71 Residuals ...................................................................................................................... 195
3.18.72 Regression Coefficients [1] .......................................................................................... 195
3.18.73 Regression Coefficients / Property Spectra (CLU) ....................................................... 195
3.18.74 Sample ......................................................................................................................... 196
3.18.75 Scores .......................................................................................................................... 196
3.18.76 Score Adjacency ........................................................................................................... 198
3.18.77 SEP Generalized Cross Validation ............................................................................... 198
3.18.78 Spectra ......................................................................................................................... 198
3.18.79 Spectra Property Index ................................................................................................. 199
3.18.80 Spectra Radii ................................................................................................................ 199
3.18.81 Spectra Residuals ........................................................................................................ 200
3.18.82 Spectra Residuals too big ............................................................................................ 200
3.18.83 Spectra Residuum ........................................................................................................ 200
3.18.84 Spectrum Nearest Neighbor ......................................................................................... 201
3.18.85 Spectrum Nearest Neighbor Index ............................................................................... 202
3.18.86 Time .............................................................................................................................. 202
3.18.87 V-Set BCAP-PRES ....................................................................................................... 202
3.18.88 V-Set BIAS ................................................................................................................... 203
3.18.89 V-Set PRESS ............................................................................................................... 203
3.18.90 V-Set Regression Coefficient ....................................................................................... 204
3.18.91 V-Set Regression Intercept .......................................................................................... 205
3.18.92 V-Set Regression Slope ............................................................................................... 205
3.18.93 V-Set SEE (SEP) .......................................................................................................... 205
3.18.94 V-Set Spectra Residuum Max ...................................................................................... 206
3.18.95 V-Set Spectra Residuum Min ....................................................................................... 206
3.18.96 V-Set Std Orig Property ................................................................................................ 206
3.18.97 V-Set Std Pred Property ............................................................................................... 207
3.18.98 V-Set X-PRESS ............................................................................................................ 207
3.18.99 Wavenumber ................................................................................................................ 207
3.18.100 X-PRESS ...................................................................................................................... 207
3.18.101 Y-Scores ....................................................................................................................... 208
3.18.102 y-loadings ..................................................................................................................... 208
3.19 ..... NIRCal Configuration Options ...................................................................................... 209
3.19.1 Configuration Dialog ..................................................................................................... 209
3.19.2 General ......................................................................................................................... 209
3.19.3 Journal .......................................................................................................................... 210
3.19.4 Calibration Defaults ...................................................................................................... 212
3.19.5 Tables ........................................................................................................................... 213
3.19.6 2D-Plots ........................................................................................................................ 213
3.19.7 3D-Plots ........................................................................................................................ 215
3.19.8 System Modules ........................................................................................................... 217
3.19.9 Calibration Protocols .................................................................................................... 217
3.19.10 Directories .................................................................................................................... 218
3.19.11 Cluster Prediction Protocol ........................................................................................... 219
3.20 ..... Menu commands .......................................................................................................... 220
3.20.1 File menu ...................................................................................................................... 220
3.20.2 Edit menu ..................................................................................................................... 229
3.20.3 View menu .................................................................................................................... 230
3.20.4 Workspace .................................................................................................................... 241
3.20.5 Project .......................................................................................................................... 242
3.20.6 Calibration .................................................................................................................... 245
3.20.7 Wizard .......................................................................................................................... 247
3.20.8 Modules ........................................................................................................................ 248
3.20.9 Windows ....................................................................................................................... 253
3.20.10 Help .............................................................................................................................. 254
3.21 ..... Context menu (right mouse) ......................................................................................... 255
3.21.1 Add Selection to Set ..................................................................................................... 255
3.21.2 Alpha-Blending ............................................................................................................. 255
3.21.3 Anti-Aliasing .................................................................................................................. 255
3.21.4 Autorotate ..................................................................................................................... 255
3.21.5 Choose [x][y] Independent ........................................................................................... 256
3.21.6 Choose [xy] Independent ............................................................................................. 256
6 NIRCal 5.5 Manual, Version A
Table Of Contents
3.21.7 Clear Selection ............................................................................................................. 256
3.21.8 Colors ........................................................................................................................... 256
3.21.9 Copy Plot to Clipboard ................................................................................................. 256
3.21.10 Copy Selection To ........................................................................................................ 256
3.21.11 Edit Selection ................................................................................................................ 257
3.21.12 Export Table ................................................................................................................. 257
3.21.13 Fix Table Titles ............................................................................................................. 257
3.21.14 Flip X-Axis .................................................................................................................... 257
3.21.15 Flip Y-Axis .................................................................................................................... 257
3.21.16 Invert Selection ............................................................................................................. 257
3.21.17 Load Selection from Set ............................................................................................... 257
3.21.18 Large Plots ................................................................................................................... 257
3.21.19 Mouse Select ................................................................................................................ 257
3.21.20 Mouse Select Lasso ..................................................................................................... 257
3.21.21 Mouse X-Axis Select .................................................................................................... 258
3.21.22 Mouse Zoom ................................................................................................................. 258
3.21.23 Number format .............................................................................................................. 258
3.21.24 Open data as ................................................................................................................ 258
3.21.25 Plot Defaults ................................................................................................................. 259
3.21.26 Plot Settings ................................................................................................................. 259
3.21.27 Regression Draw Line .................................................................................................. 259
3.21.28 Regression Copy Equation ........................................................................................... 259
3.21.29 Remove Selection from ................................................................................................ 259
3.21.30 Snap to Y-Grid .............................................................................................................. 259
3.21.31 Snap to X-Grid .............................................................................................................. 259
3.21.32 Show Plane .................................................................................................................. 260
3.21.33 Show Labels ................................................................................................................. 260
3.21.34 Show Line ..................................................................................................................... 260
3.21.35 Show Selection Colors ................................................................................................. 260
3.21.36 Show Cycle Colors ....................................................................................................... 260
3.21.37 Show Property Colors .................................................................................................. 260
3.21.38 Transpose Data ............................................................................................................ 260
3.21.39 Turbo Mode .................................................................................................................. 260
3.21.40 Undo Last Pretreatment ............................................................................................... 260
3.21.41 Undo Sequence of Pretreatments ................................................................................ 260
3.21.42 View Fit ......................................................................................................................... 261
3.21.43 Visibility ......................................................................................................................... 261
3.22 ..... Toolbar ......................................................................................................................... 261
3.22.1 Calibration Toolbar ....................................................................................................... 261
3.22.2 Database Toolbar ......................................................................................................... 262
3.22.3 File & Edit Toolbar ........................................................................................................ 262
3.22.4 Lifecycle ........................................................................................................................ 263
3.22.5 Modules ........................................................................................................................ 263
3.22.6 Options Toolbar ............................................................................................................ 263
3.22.7 Pretreatments Toolbar .................................................................................................. 263
3.22.8 Pretreatments Advanced Toolbar ................................................................................. 264
3.22.9 Pretreatments Minibar Toolbar ..................................................................................... 264
3.22.10 Pretreatments Gap2 ..................................................................................................... 265
3.22.11 Project Toolbar ............................................................................................................. 265
3.22.12 Search Import Export Toolbar ...................................................................................... 266
3.22.13 Window Toolbar ............................................................................................................ 266
3.22.14 Wizard Toolbar ............................................................................................................. 266
3.22.15 Wizard Workspace ....................................................................................................... 266
3.22.16 Zoom & Select Toolbar ................................................................................................. 267
4 Appendix ..................................................................................................................................... 269
4.1 ....... Keyboard Shortcuts ...................................................................................................... 269
4.2 ....... Tips and Tricks ............................................................................................................. 270
4.3 ....... Conventions / Terminology ........................................................................................... 273
4.3.1 Temporary Settings ...................................................................................................... 274
4.3.2 Permanent Settings ...................................................................................................... 274
NIRCal 5.5 Manual, Version A 7
NIRCal 5.5 Software Manual
4.4 ....... Characteristic absorbance table ................................................................................... 274
5 Software installation .................................................................................................................. 277
5.1 ....... Software installation ..................................................................................................... 277
5.1.1 Important Notes ............................................................................................................ 277
5.1.2 System requirements.................................................................................................... 277
5.1.3 Other requirements....................................................................................................... 277
5.1.4 Installation procedure ................................................................................................... 277
5.2 ....... Software update ........................................................................................................... 281
5.3 ....... Software licenses ......................................................................................................... 281
5.3.1 Software Registration ................................................................................................... 281
5.3.2 Activating the License .................................................................................................. 283
8 NIRCal 5.5 Manual, Version A
Welcome
Dear NIRCal user,
NIRCal is recognized as a reliable, comprehensive but also easy to use chemometric software package. It offers a wide variety of tools for method development and optimization.
The NIR spectroscopic technology and the range of applications develops continuously, just as new requirements from users and regulatory authorities. NIRCal is designed to fulfill those requirements today and in the future.
NIR spectroscopy is a powerful technology, which gives insight in product development projects and sound process understanding enabling optimization of processes and quality. NIRCal is designed to support all those tasks. Through high quality plots and informative workspaces, the analyst gets comprehensive overview of the system under study.
Much of the power in multivariate methods like PCA and PLS lies in the information one can get from interpretation of the models. This manual not only describes the functionality of NIRCal but teaches the user how to interpret the models to ensure the maximum return on your investment in NIR technology.
Should you come across any feature which needs improvement or extension, please do not hesitate to contact us. Your feedback helps us to continuously improve our software and is highly appreciated. Please send an e-mail to: info@buchi.com.
Every effort has been made at BUCHI to ensure that the information in this documentation is correct. BÜCHI Labortechnik AG accepts no responsibility for errors or omissions.
Information in this document is subject to change without notice and does not represent a commitment on the part of BÜCHI Labortechnik AG.
The software and/or files may be used or copied only in accordance with the terms of the license agreement.
BÜCHI Labortechnik AG, CH-9230 Flawil, Switzerland, March 2013.
Explanation of the used safety notes:
NOTE
Information on technical requirements. Non-compliance can lead to faults, inefficiency and production losses.
This manual is stored under: C:\Program Files\Buchi\NIRSolutions\NIRCal Manual.pdf"
NIRCal 5.5 Manual, Version A 9
NIRCal 5.5 Software Manual
frequency:
wavelength:
wavenumber:
velocity of propagation:
energy:
Hz Infrared Light
1 Introduction
1.1 Introduction to NIR
1.1.1 Spectroscopy
Light is a fast time dependent sequence of electric and magnetic fields propagating in space. Light can be characterised with physical properties, like:
A summary of the electromagnetic spectrum can be seen in the Picture below. NIR lies between the visible and middle infrared range. With this light, the molecular vibrations are activated, similar to the infrared range.
Most of the molecule vibrations take place at a characteristic frequency which lies between 1012 to 1014 Hz. The osculating molecule interacts with the electrical field or light when the frequencies are the same.
Not only are the basic vibrations absorbed with very high degree of excitation in the infrared range, but also frequencies 2 or 3 times higher. The harmonic frequencies are absorbed with a lower degree of excitation and lie in the near infrared range. Using an infrared spectrum, the characterisations of molecules is possible. In the NIR spectra region mainly the overtones and combination tones of -CH, -OH and -NH groups are absorbed, so NIR is suitable mainly for organic substances.
The absorption bands are very broad and they often overlap, which can cause problems by direct interpretation of the spectra. The most often used evaluation method in spectroscopy, the Lambert-
10 NIRCal 5.5 Manual, Version A
Beer Law:
Introduction
has just limited validity in the NIR spectroscopy. The application of chemometrics for the evaluation of NIR spectra is a must.
Advantages of NIR spectroscopy
The relative low degree of absorption coefficient of overtones and combination vibrations causes a low degree of absorbance in the NIR region. Solids have a high degree of reflected light and liquids can be measured for path lengths of several mm.
no sample preparation is necessary.
The materials for the optics can be quartz, glass or sapphire, which are
cheap and robust.
1.1.2 Chemometrics
NIR spectra are generally characterised by very broad peaks and a multitude of oscillation superpositions. Visual evaluation is therefore all but impossible. Differences in the spectra of similar substances often consist merely of a slight shift or small change in shape of the wide absorption bands. For this reason, NIR spectra are basically evaluated with the aid of mathematical methods, which is why such significance is attached to the chemometric software. Chemometric is the application of mathematical, statistical procedures for processing, evaluating and interpreting large amounts of chemical data (e.g. NIR spectra). The function of the chemometric software in NIR spectroscopy is to find a statistical correlation between the spectral data and the known (e.g. by laboratory analysis) property values of the samples used for the calibration.
If this correlation is systematic, it is possible to predict desired parameters of unknown samples (e.g. identity, quality, quantity) by recording the spectrum and subsequent evaluation by calculation.
The Büchi NIR spectrometer systems allow several hundred intensity values (reflectance / transmittance) of the measured NIR spectral region to be included in the calibration. In order to be able to draw maximum benefit from the measured region, the Principal Component Analysis (PCA) is applied.
The Principal Component Analysis for qualitative calibration allows the identification of different substances or similar product qualities. There are 2 ways to use PCA:
Cluster calibrations and the  SIMCA method
For quantitative analyses, three different calculation procedures have been implemented:
Multiple Linear Regression (MLR),  Principal Component Regression (PCR) and  Partial Least Squares Regression (PLS).
These methods can be tested with a user made selection of independent validation samples or by using each sample for the calibration in cross validation.
Chemometrics is the science of relating measurements made on a chemical system or process to the state of the system via application of mathematical or statistical methods. [International Chemometrics Society (ICS)]
NIRCal 5.5 Manual, Version A 11
NIRCal 5.5 Software Manual
1.1.3 Cluster calibration
Qualitative calibrations are used for identification of different chemical substances and for separation of different qualities of the same substances. The possible applications include identification of:
substances with very different chemical characteristics;  chemically similar substances;  acceptable and rejectable qualities of a given substance.
The possible methods are Cluster or SIMCA. Both methods use PCA with the difference that Cluster method is always used for a group of similar substances, while for each substance a calculation is performed with SIMCA.
Choosing the Calibration Samples
Verifying substances in the laboratory often means, ascertaining if a sample can be assigned to a specific category (property); e.g., when checking incoming raw materials in a pharmaceutical company to see if the incoming raw material is the product that was expected or not. The question can be answered efficiently by recording an NIR-spectrum of the raw material and analyzing the spectrum with a qualitative calibration. The raw material can be correctly identified that it belongs to the expected category or falsely identified that it does not belong to the expected category.
To obtain a useful, qualitative calibration, first, calibration samples should be measured that cover all expected allowable variations of the quality of the product. For each property, several samples must be collected to cover variations such as different particle size, temperature, moisture or supplier. To obtain a representative set of spectra, we recommend measuring samples from at least five to fifteen different batches of each product that have been collected over a period of at least 6 months. This will ensure that all variations within a product will be represented in a calibration. Collecting samples can speed up the building of a proper calibration. But only stable and unchanged substances can be used. Only samples that have been tested with reference analysis should be used for the calibration. The combinations will be chosen randomly or with an adequate experimental design. Two thirds of these samples are composed to be the basic calculation data for the calibration. The remaining third is used for testing the calibration.
NOTE
When selecting the spectra in the calibration- and validation-set, it is important to assign all spectra of one sample either to the calibration or to the validation set.
Calculating the Qualitative Calibration
The spectra of different substances show the physical and chemical characteristics of each substance.
Not all spectral differences are associated with the searched differences. “False“ differences arise, for
example through a varying presentation of the sample because of different particle size or other non-
interesting but allowable variations of the substance. Such “false“ differences can be reduced or partly
eliminated with the help of appropriate data pretreatments during the calculation.
12 NIRCal 5.5 Manual, Version A
Introduction
Figure.1: Solvent spectra (acetone, ethanol, toluene and dichlor-methane), measured through glass cuvettes with measurement option liquid. The slight shifts (light scattering) are not of interest and should be reduced with data pretreatments.
Figure.2: Spectra out of figure 1 after data pretreatment (here: 1st derivation) Systematic differences of the four solvents are clear and reproducible
For the actual calibration, the Cluster calibration using PCA has been used. This means a calculation of a new illustration of the spectral information with the target to show the main differences in the data set. Spectra appear after the calculation in the new illustration as points in a 2D or 3D Plot.*
The calculation can be performed automatically by the Calibration Wizard. According to the required input by the Wizard concerning measurement method and dedicated calibration type, different calibrations are calculated and sorted by a specific quality attribute (Q-Value). For an optimal result, the wavelength selection, the data pretreatments and the optimal number of PCs must be chosen adequately. This is automatically done by the software. Good qualitative calibrations can be recognised as the single spectra are found in well separated tolerance regions -Clusters- where each represent only one of the possible categories (properties).
NIRCal 5.5 Manual, Version A 13
NIRCal 5.5 Software Manual
Figure 3: After transformation with the PCA, the spectra appear as points assorted in clusters, which are well separated from each other.
* In case of many different categories (properties) more than three delineation axes are required to show all important differences. It is not necessarily the case, that the first three axes are the most important ones.
Inspection of the Qualitative Calibration
The attributes of a good calibration are:
all tolerance regions of the single categories (Clusters) are cohesive;  all tolerance regions are convex and engaged consistently;  all calibration and validation spectra are within a valid range and assigned correctly;  all other substance spectra can not be predicted false with a certain calibration;  Q-Value is closest possible to 1.
Application of a qualitative calibration and interpretation of the results
During the application of a qualitative calibration, it is determined whether a new measured spectrum can be associated with a calibrated category. A measured spectrum is identified as OK when these three criteria are fulfilled:
the residual is smaller than the allowed limit;  the distance is within the allowed tolerance (the spectrum is in a known cluster),  the identity of the measured substance (substance-ID entry) matches the identity of a
substance used to build up the calibration model.
1.1.4 Quantitative calibration
NIR Spectra can be seen as fingerprints, which are characteristic for a certain substance. When investigating a substance mixture, different fingerprints are superposed within a complex spectrum. The concentration rate of the substances is present in the spectra, but cannot be seen. Target of a quantitative calibration is, to calculate applicable filters that establish the coherence of the measured intensities and the concentrations of the single components.
The quantitative calibration serves as the determination of parameters such as concentration (e.g. water content, blending ratio, hydroxyl number, etc.) or physical properties (e.g. density, viscosity).
Choosing the calibration samples
As a basis for the calibration work a set of reference data is used for which the interesting concentrations have been determined with the referring lab method. For each component (property) at least 15 samples must be used.
14 NIRCal 5.5 Manual, Version A
Important:
For the choice of the reference samples, all components of the mixture (and eventually varying parameters like temperature) must be considered, even components not being calibrated. For complex compositions, like foodstuffs, the requested amount of samples might be quite high. The choice of calibration samples must be carried out very carefully. Spot test samples taken from the production process might not meet the demands, because the reference values typically show small variations around a nominal value. Suitable calibration samples should whenever possible be chosen with the aid of an experimental design or specially produced. The consistent and independent distribution has to be checked in any case.
Suggestions:
Introduction
Figure 1: The distribution of the reference values is checked with the help of 2D plots. The NIR calibration range should cover at least 20 times the error range of the reference method.
For example for H2O determination, the range of error of ± 0.2% leads to a minimum NIR-calibration range of 4% around an expected mean value. Otherwise the error of the reference method has a high influence on a calibration. In order to avoid extrapolation for the NIR calibration, select the calibration range slightly wider than the working range.
Working range: concentration range of a parameter that usually is measured in a product;  Calibration range: for the calibration, the concentration range must be set broader than the
working range.
As a rule of thumb; 60 samples should ideally cover the calibration range homogeneously. 2/3 of all samples are used for calibration, 1/3 of all samples are used for validation. Spectra of the same sample must remain together and be put either in the calibration set or the validation set. For the calibration set it is recommended to use at least 10 samples per parameter to be determined.
Spectra with the minimum and maximum concentration values (extreme values) must always be selected as calibration spectra.
NIRCal 5.5 Manual, Version A 15
NIRCal 5.5 Software Manual
An incorrect distribution of the samples with two separated concentration ranges should be avoided. Either an independent calibration should be generated for each range or additional samples, covering the missing range, should be measured.
For reference measurement, the probe and reference material need to be absolutely clean. This is valid for the measurement of calibration spectra as well as for the measurement of application spectra. The measuring option and measurement conditions should be identical for generation of both, the calibration and routine application use. For quantitative measurements, the fibre optics probe should be in a fixed position.
The samples should not be measured with increasing or decreasing concentrations, but randomly distributed concentration.
The accuracy of the laboratory method has a huge influence on the quality of the NIR calibration. It is important to use the most accurate laboratory method (not the weight loss with IR quick drying for moisture determination, but the drying oven method or better the water determination by titration according to Karl Fisher). The time between the laboratory determination and collecting the NIR spectra should be as short as possible.
Calculating the Quantitative Calibration
The suggested calculation methods are called PCR (principal component regression) and PLS (partial least squares regression). The algorithms mostly lead to similar results. The choice of the relevant wavelength ranges and data pretreatments can improve the result of the calibration. With the applied data pretreatments unimportant spectral variations can be suppressed or the coherence between intensities and concentrations can be simplified. For example it is recommended to convert reflectance spectral data into absorbance data that depend directly on the concentration values (according the Beer-Lambert law, which is limited valid for liquids in NIR). The difficulties in choosing the data pretreatments are that they always have combined influence and that the effects depend on the applied order. Theoretical considerations take a back seat in the practice. Often the model can be optimised by trying different variations, that are calculated.
Concentration = Bo + B1*Amplitude1 + B2*Amplitude2 + ..... + Bn*Amplituden [1],
These establish the coherence between the measured amplitudes and the searched concentrations. This equation is strongly simplified, a detailed explanation is shown in chapter "Calibration Methods" Link : Principal component regression (PCR) & Partial least squares regression (PLS)
16 NIRCal 5.5 Manual, Version A
Introduction
Figure 2: For the review of a quantitative calibration the prediction values according to equation [1] are compared with the reference values. In an ideal case, the corresponding points are lying on the 45° calibration curve through the zero point both for the calibration and validation samples.
Precision
the SEC and SEP provide the magnitude of the standard deviation for the calibration set and the independent validation set. The two values should be as small as possible, but they are likely to be comparable with the standard deviation of the conventional laboratory
method. With an acceptable calibration, the two values are also roughly equal (Consistency: »100)
Accuracy
the V-Set Bias provides information on the average deviation of the predicted values from the true values. This value gives information on a systematic deviation of the calibration and therefore should be as close to zero as possible. The C-Set Bias is zero by definition.
Regression
coefficients, r
show how well the predicted values (NIR values) match the reference values (original property values) on average.
Q-Value
for optimisation, different parameters are integrated in this quality factor. The Q-Value lies between 0 ( = inoperative calibration) and 1 ( = ideal calibration)
The choice of parameters and calculation methods can be done automatically by using the Calibration Wizard.
Inspection of the quantitative calibration
Possible reasons for bad calibration results
Outliers can be recognised for the differently calculated calibrations as big differences between predicted values and reference values appear.
Remedy: Outliers must be erased from the calibration as well as the validation set. This always must be combined with a careful clarification of the reason for the appearance of the Outlier.
To find out if a reference value or the measured spectra must be regarded as an Outlier, the score plots should be reviewed (Graphics / Scores / 2D-Scatter).
NIRCal 5.5 Manual, Version A 17
NIRCal 5.5 Software Manual
In case spectra breaking ranks, show clearly deviating Scores and Residuals (Graphics /
Spectra / Residuals), these spectra are real outlier and should be eliminated from the calibration / validation.
In case there are big differences between the reference values and the predicted values, but
the Scores of the referring spectra do not have particular deviations with high probability, the Outliers appear because of false reference value. The lab determination should be repeated.
Group of samples with systematically deviations: this effect can be seen from time to time
when samples are evaluated their reference values have been determined in laboratories not using exactly the same reference methods. Here only an alignment of the reference methods can help.
Significantly different results depending on the chosen classification of the samples into the
calibration and validation set: the number of used samples is too small, for instance because of not considered, hidden properties.
Remedy: Selective completion of the master data set that all possible variations flow into the calculation.
Application of a quantitative calibration and interpretation of the results
During the application of a quantitative calibration, the referring concentration values of the measured spectrum are calculated according to equation [1] and indicated. These calculations always lead to a result, even when performing faulty measurements. (e.g. sample of completely different, not calibrated substance class). The calculation of the concentration is therefore enlarged with two further checks:
the concentration values found must be within the original calibration range, a warning- and
action-limit can be adjusted as well.
make sure that the new measured spectrum matches the calibration spectra and that the
Residual is within the allowed limit. A warning appears, when the residual of the measured spectra is bigger than the maximum allowed residual. Results of the spectra with Residual outlier are not taken for the average calculation of multiple measurements.
1.2 Introduction to NIRCal
1.2.1 General
The work flow for typical NIR Application with Büchi NIRWare is:
1. Create an Application in NIRWare Application Designer for reference measurements
2. Data Acquisition Spectra collection with NIRWare Operator
NIR- Spectroscopy
3. Assign Property value to spectra with NIRWare Sample Manager
Spectra + Reference Analysis
4. Calibration build mathematical models with NIRCal 5 Correlation known Spectra <- -> Concentrations / Identity
5. Copy and modify the Application for routine measurements
6. Assign the created calibration to the routine application
7. Use the mathematical models/calibrations within an Application in the NIRWare Operator for
Predictions
unknown Spectra -> Concentrations / Identity
In case a calibration after the first trial is not delivering the correct results, the steps 3-7 should be repeated, the calibration should be expanded with new samples and tested again.
All data (like spectra, property value, projects, calibrations, etc) are saved in the NIRWare Database.
18 NIRCal 5.5 Manual, Version A
Introduction
Icon:
NOTE
To be able to take full profit from all the possibilities of the NIRCal chemometric software, a Buchi training is highly recommended.
1.2.2 Starting NIRCal
Windows 7: Start / All Programs / BUCHI / NIRSolutions /NIRCal Windows 8: Write “NIRCAl” in the main screen and navigate to programs
To use NIRCal Log On with User name and Password.
User rights and setup is managed with the NIRWare Suite "Security Designer". Please refer to the corresponding documentation within the SW-Manual for NIRWare.
NIRCal opens an empty project.
NIRCal 5.5 Manual, Version A 19
NIRCal 5.5 Software Manual
1.2.3 NIRCal Project
NIRCal stores the imported spectra and their properties , which belong together in a project. A project contains:
spectra: the imported spectra as they were measured;  properties, the property names with their values, which belongs to the loaded spectra and
can not be modified in NIRCal;
the calibrations, which have the information about the used data selections and the
calculated results;
matrices, which have all chemometric results of the active calibration.
Note: Instruments is not used in NIRCal 5 anymore. Related Topics for Spectra import:
Search and Import Spectra from NIRWare Database Import Spectra from File Convert and Import spectra from other instruments to DB Use NIRCal for any type of Data
NOTE
It is possible to import and export spectra between NIRWare databases within the Software "NIRWare Management Console - Administrative Tools"
1.2.4 Spectra Overview
All spectra measured or imported into this project can be seen here. Items marked with a yellow pen can be edited by pressing F2 or a double click.
20 NIRCal 5.5 Manual, Version A
Introduction
Explanation of the symbols (right window):
Spectra belonging to the C-Set
Spectra belonging to the V-Set
Spectra in unused Set = U-Set (these spectra are not used for calibration nor for the
validation of a calibration
Spectra in C- and V-Set at the same time. Overlapping is not allowed.
Red color indicates selection. These spectra for example can be assigned to the user set and be plotted separately.
1.2.5 Properties Overview
All properties in this project can be seen here. Items marked with a yellow pen can be edited by pressing F2 or a double click.
NIRCal 5.5 Manual, Version A 21
NIRCal 5.5 Software Manual
Min/Max represents the calibration range for quantitative properties. For qualitative properties Min/Max is shown as 0/1.
1.2.6 Calibration Validation Methods
To be able to judge the performance of a calibration a set of independent validation samples is necessary.
Validation Set (VS)
Normally all spectra within a project are divided into 2 sets with a suggested ratio of 2/3 to 1/3. The two sets should be completely independent from each other.
C-Set (Calibration Set)  V-Set (Validation Set)
Spectra in the V-Set are not used for the calibration, the V-Set spectra are used like unknown samples to judge the quality of the calibration (internal validation set). Only the C-Set spectra are involved in the loading calculation. Enough spectra of the sample should be available.
VS can be used for all calibration methods.
Cross Validation (CV)
Cross validation (CV) uses all samples as the calibration set for quantitative calibrations except one sample (or a small group of samples) which is left out. Validation is accomplished by predicting the left out samples and by systematically varying the selection of left out samples. The procedure is time consuming because for each selection a calibration has to be calculated. The method is especially useful when the total number of samples is small (< 50 samples). Full cross validation (FCV) means that n-calibrations are calculated so that one spectrum has been left out and all other are in a calibration.
Limitations
only available for PCR and PLS;  needs at least 2 CV groups or at least 4 C-Set spectra for one-leave-out (full cross validation;
FCV);
will delete the V-Set spectra selection, in case it is not empty.
1.2.7 Calibration Methods
Qualitative Calibrations / Identification
Target is to identify the membership of a sample to a property group. The property groups can be chemically completely different or similar to the same substance. Both implemented method are using PCA:
Cluster Analysis CLU SIMCA
Quantitative Calibrations Target is to determine the concentration value such as content in %, OH-value or physical
parameters like density, viscosity.
In NIRCal implemented algorithms are:
Principal component regression PCR Partial least squares regression PLS Multiple linear regression MLR
PCR, PLS, PCA (CLU) and SIMCA are principal components based methods, MLR and also library search with spectra comparison are spectra based methods.
22 NIRCal 5.5 Manual, Version A
Calibration
Inputs are the measured spectra with their properties and with the reference values together with the calibration data selection. Outputs are the calibration for the properties together with the validation for the validation set (V-Set) spectra or the result of Cross Validation (CV).
Application
Inputs are the calibration and a measured spectrum. Outputs are the predicted property value(s), calibration limits or hitlist; spectral residuals.
1.2.8 Calibrations Overview
All calibrations calculated in this project can be seen here. Items marked with a yellow pen can be edited by pressing F2 or a double click.
Introduction
New (active) calibration, not yet stored in the database
Stored calibration not in “Edit” mode (availability for “Edit” mode depends on Lifecycle status)
Stored calibration in “Edit” mode
NIRCal 5.5 Manual, Version A 23
NIRCal 5.5 Software Manual
1.2.9 Matrices Overview
The matrices shown here depend on the selected calibration and validation method. Items marked with a yellow pen can be edited by pressing F2 or a double click.
1.2.10 Data
Raw data like spectra and property values are loaded from the NIRWare Database. Spectra and property values, loaded from the database, are not modifiable in NIRCal anymore. Pretreatments never change the original spectra.
These data are stored in a NIRCal Project. Each project has a first calibration, which has an organiser function and this can not be deleted. Calibration specific data like selections (Spectra, Wavelength, Pretreatments,Principal Components) are stored for each calibration separately together with the selected chemometric methods calculated data. Approved calibrations, which are released for the application according the Lifecycle procedure, can be used in the NIRWare Operator. Link: Calibration Handling according to the Lifecycle.
The main and intermediate data are stored in matrices (e.g. Original Spectra - and Original Property matrices), which are available in table or graphic form. Intermediate data e.g. Scores and Loadings are calculated on demand by the active calibration settings. Matrices can be exported to MATLAB as .dat files or to EXCEL as xlm files.
MATLAB : NIR-Explorer / Matrices / “Loadings” / PopUp Menu / Export. EXCEL: NIR-Explorer / Matrices / “Loadings” / PopUp Menu / Table. In the Table
PopUp Menu / Export Table…
Link: Matrices NIRCal 5 can work as File- and as Database - oriented software.
With NIRCal it is possible to import data from files and to store data to files. NIRCal 5.4 downward compatibility of spectra and properties is supported to NIRCal 5.2 as .nir, .ncf and JCAMP-DX file. In NIRCal 5.2 the NIRCal 5.4 files can be imported with: File/ Import/ Spectra. The loaded spectra can be also stored in the database. A spectra conversion from the older NIRCal 4 data or from other software is possible in this way. The following data types can be imported:
Projects in format ".nir";  Spectra with properties in format ".nsf, .bmp, .dat, .csv, .spc, .dx, .jdx, .jcm, .nir "  BCAP-Series in format ".S??" (Spectra file)
24 NIRCal 5.5 Manual, Version A
Introduction
All Spectra
not modifiable, determined through the number of imported spectra.
User Spectra
mainly used for visual comparison, any selection is acceptable, this selection has no correlation with the C-, V- or U-Set spectra.
Calibration Spectra
spectra selected for the calibration: C-Set, about 2/3 of all spectra for VS mode.
Validation Spectra
spectra selected for the validation: V-Set (1/3 for VS).
Outlier Spectra
result of the "Outlier Detection" wizard for the statistical residual, score, property value and leverage outliers in C- and / or V -Set
It is not possible to import data from NIRCal 4 ".ncf" format, because ncf-files do not include spectral data. It is important to take care of the compatibility of the spectra from different instruments!
Link: Convert and Import..in NIRWare DB.
1.2.11 Selections
Data selections are needed to group the data before the calculation of a calibration.
For each calibration the selections can be different.
Calibration method selection
Choose the method according to the target of the application.
For quantitative applications there are MLR, PCR or PLS and for qualitative application Cluster or SIMCA are available.
Validation method selection
To test the validity of a calibration internal test samples are used in the validation mode: VS. For quantitative calibration the Cross Validation: CV is available.
The default method is VS: Validation Set, where the V-Set spectra should be selected by the user. You can also use the spectra selection wizard.
Data Sets Selection
Spectra Selection
The selection can be made in NIR-Explorer:
Each of these sets, except of "All Spectra", can be edited by the user.
NIRCal 5.5 Manual, Version A 25
NIRCal 5.5 Software Manual
Select spectra for the calibration (Calibration Spectra = C-Set) and for the validation of the calibration (Validation Spectra = V-Set). The C-Set will be used for the calculation of the calibration, the V-Set is used as an internal test of the calibration.
The same spectra should not be defined as C-Set and V-Set Spectra at the same time. If so then the spectra will be marked with a question symbol in the spectra overview and following message appears, that easily allows to remove the spectra set overlap automatically:
Link: Chemometrics / Selections / Spectra Data Set
Wavelengths selection:
All Wavelengths: the measured wavelength/ wavenumber range and the number of
wavelength / wavenumber data points, which is determined by the used instrument and by the resolution;
Calibration Wavelengths: only the selected range will be taken into account for the calculation
and later for the application.
Link: Chemometrics / Selections / Wavelength Data Set
Properties selection:
All Properties: number of properties belonging to the imported spectra;  Calibration Properties: only the selected properties will be taken into account for the
calculation and later for the application. Normally all or several will be selected for Cluster calibration, only one should be selected for quantitative and for SIMCA calibration method.
Link: Chemometrics / Selections / Properties Data Set
Principal Components Selection:
All PCs: the number of primary principal components used for the calibration, they are used
for the spectra reconstruction and for the Residual calculation;
Calibration PCs: only the selected PCs will be taken into account for the calculation and later
for the application, they are used in quantitative calibration for the property value calculation or in qualitative calibration for the property separation and for the allowed tolerance area calculation.
Link: Chemometrics / Primary Principal Components & Secondary Principal Components
Pretreatments selection:
The measured spectra can be modified before the calculation with several pretreatments or different combinations of them.
NOTE
The original spectra are still the same and will not be modified. The list of in NIRCal 5 implemented pretreatments indicates the huge flexibility of the software. The
pretreatments used in the calibration will be applied in the application also.
26 NIRCal 5.5 Manual, Version A
Introduction
Blow Up Parameter:
The allowable limits are calculated with a default blow up limit. These limits can be adjusted by the user:
for quantitative calibrations the scores and residual blow up limits;  for qualitative calibrations the scores, residual and radii blow up limits and the radii calculation
formula.
Link: Chemometrics / Blow Up Limits
Outlier detection limit
To calculate the statistical outliers the limit can be edited by the user or can be calculated according the T-distribution for the actual calculation. Link: Chemometrics / Outlier Detection
1.2.12 Graphics
All data that is available in table format can be plotted graphically as well. In NIRCal 5 there are 1, 2 and 3 dimensional plots available. The 1D scatter plot is new in NIRCal 5: it shows only one selected line or column of a matrix. This
scatter plot can be used for selections. The pop-up menu "Options/Show line" or the keyboard shortcut "W " can change the point to a line.
Each 2D plot can be shown as a front view (this is normally the default adjustment), top view and
transposed.
NIRCal 5.5 Manual, Version A 27
NIRCal 5.5 Software Manual
The plots can show different selections, while the selection plotted is always visible on the subtitle. The use of modern rendering techniques, such as Anti-Aliasing and Alpha-Blending (new hardware
accelerated features of newer graphic cards), allows to eliminate jagged edges and stair-stepped lines. NIRCal applies these features to improve the graphical performance.
Anti-Aliasing (hotkey: a) eliminates jagged edges and stair-stepped lines. This can be switched on permanently under: Edit / Options / 2 D Plot. Opening the 2 D plots is quicker with Anti-Aliasing using new graphic cards, but can be slower with an older driver.
Alpha-Blending (hotkey: b) allows a partial transparency with steps of 1, 1/2, 1/4, 1/8. 1/32, 1/64 (by pressing b several times) and helps to look through huge amount of data. The Alpha-Blending level can only be seen from the data color intensity: the more often the point is drawn, the brighter the color. Each NIRCal rendering can be stopped immediately by pressing the ESC key. Anti-Aliasing and Alpha-Blending are very useful for huge amount of spectra especially for 2-D spectra plots (original, pretreated and residuum); Score plots (vs. PC or vs. Scores) and calibration curve (original vs. Predicted).
On the left side the common drawing style is used, where no details can be seen. Anti-aliasing and alpha-blending (right side) allow to look through a huge amount of data and to detect hidden structures.
Note: In case a one dimensional matrix - which is a vector- is plotted, the subtitel is "All One". "One" indicates the first selected "dimension" and not the value of the vector!
The colors are also selection dependent. The plots can be opened as Table with the keyboard shortcut "G".
On each plot the X- and Y-axis can be flipped using the pop-up menu: Option/ Flip X-Axis (x) and Option/ Flip Y-Axis (y).
2D combined scatter plots: beside the standard 2D plots it is possible to combine matrices, even if they have only with one dimension matching. This can be created manually in the NIR-Explorer selecting the two matrices and opening in pop-up menu the "2D-Combined Scatter". There are several plot combinations available in NIRCal 5: under: View.
These plot combinations, which are fixed, can help to make the data selections easier (spectra, wavelengths, properties, PCs) with useful graphics. The dependency plots can help to find hidden correlation between spectra and different users, or time, or instruments.
The users can create own plot combinations and can save them under: Workspace. 28 NIRCal 5.5 Manual, Version A
Introduction
1.2.13 Protocols
The term "protocol" in NIRCal means report.
Calibration Protocol
The calibration protocol is an important validation document, which contains all information about a calibration, like the user specific data selection and the results of the chemometrical calculations applied for the C- and V-Set spectra in the project. The calibration protocol is stored within the calibration. Link: Calibration protocol
Validation Protocol
The prediction protocol is an in important validation document, which contains the results of a prediction of a calibration applied for other spectra, mainly not existing in the project. The prediction protocols help to find out possible interfering substances for the qualitative calibrations. Link: Prediction protocol
NIRCal 5.5 Manual, Version A 29
2 Tutorials
2.1 Qualitative
2.1.1 Qualitative Tutorial
Qualitative calibrations are used for identification of different chemical substances and for separation of different qualities of the same substances. Either Cluster or SIMCA method can be used for identification, both using Principal Component Analysis PCA.
The Cluster method is explained first. For detailed explanation to the method see Link : Cluster (CLU) and Link : SIMCA
2.1.2 Flow Chart Qualitative
The flow chart shows the way how to build a qualitative calibration using the Cluster method.
NIRCal 5.5 Manual, Version A 31
NIRCal 5.5 Software Manual
NOTE
Loop 1 to Loop 3 represent the sequence for optimising a calibration. After each change in the selection, the calculation and principal component selection should be repeated.
2.1.3 Selecting the Calibration Method
The calculation method can be selected in the Menubar: [Calibration / Method / CLU Cluster] or by clicking the icon
2.1.4 Selecting the Calibration and Validation Spectra
Samples of known characteristics, both chemical and physical are used to generate qualitative calibrations. For each substance classes 5-15 different batches should be used for the calibration. The measured spectra are divided into two sets:
Calibration Spectra: spectra selected for the calibration: C-Set, about 2/3 of all measured
spectra, at least 3 spectra.
Generally, the calibration spectra should contain all “acceptable extreme information” to
define the limits of acceptance.
Validation Spectra: about 1/3 of all measured spectra selected for the validation V-Set at least
2 spectra. Only if a calibration treats the validation spectra equal to the calibration spectra, the settings can be considered as OK.
NOTE
These two groups of spectra should be:
independent from each other;  no overlapping allowed.
Suggested selection: Blockwise: 6-3, if always 3 spectra per batch where measured. In case some extreme samples are in the V-Set, see Loop 3.
It is possible to leave out some spectra from the C- and from the V-Set, these spectra are in the Unused Set = U-Set: C-Set + V-Set + U-Set = All Spectra
The U-Set is visible in the calibration results (see Calibration Protocol), but will not influence the calibration and validation results e.g. will not be considered for the Q-Value calculation.
Link: Spectra selection
2.1.5 Selecting the Calibration Wavelengths
The exact wavelength / wavenumber range measured is dictated by the instrument type used for the spectrum measurement. The selected wavelength / wavenumber range depends on the application and the measuring option used.
NOTE
In general the calibration wavelength / wavenumber range should be as wide as possible. Suggested wavenumber range for NIRFlex N-500 with measuring options
Solids and Liquids: 4'000-10'000 cm-1;  Fiber Optics: 4'500-10'000 cm-1.  Solids with Tablet Accessory: 6'000-11'520 cm-1.
The calibration wavelengths define the spectrum range, where the mathematical algorithms PCA is applied.
32 NIRCal 5.5 Manual, Version A
Tutorials
Suggested selection: use all measured wavenumber in the first calculation, otherwise see Loop 2. Link Wavelength selection
2.1.6 Selecting the Calibration Properties
The calibration properties are the substances to be identified in the application. The mathematical algorithm of PCA will be applied for the selected properties. Normally all qualitative properties that must appear in the calibration model must be selected.
Suggested selection: select all properties. Link Calibration property selection
2.1.7 Applying Data Pretreatments
Data pretreatments are used to eliminate non important effects or to enlarge minor effects of the measurements.
Suggested selection: perform the first calculation without pretreatment, using the spectra as they have been measured. The first step to optimize the calibration is to add and change the pretreatments.
Link: Pretreatment selection
2.1.8 Performing a calibration (calculation)
After selecting the spectra (C-Set and V-Set spectra), the wavelength range (calibration wavelength) and the properties (calibration properties), the chemometric parameters (principal components, scores, residuals,etc) will be calculated.
For the calculation of the PCA the software use only the spectra selected into the C-Set referring to the calibration wavelength and the choice of the data pretreatments. The spectra of the V-Set will only be used to prove and judge the calibration
An overview of the calculation can be obtained in the Menubar: View / Overview. The result of the calibration will depend on the selected number of primary and secondary principal
components, the first step is to decide on the number of PCs. The calibration can be optimised manually or using the Calibration Wizard. Here the manual selection
criteria are described.
2.1.9 Primary Principal Components
Primary principal components are the PCs, which are used for reconstruction of the spectra. They determine the residual value. The more primary PCs used, the smaller the allowed residual of the calibration.
Recommended graphics for primary PCs selection:
X-PRESS function,  Loadings graphic;  Spectra Residuum plots.
Suggested selection: avoid overfitting, do not use too many primary PCs. Link: Primary PCs
Adjust the desired number of primary PCs. Link: Adjusting Primary PCs
NIRCal 5.5 Manual, Version A 33
NIRCal 5.5 Software Manual
2.1.10 Secondary Principal Components
The secondary or calibration PCs are used for the separation of the different substances and are responsible for the tolerance radii calculation. The number of secondary PCs is limited by the number of primary PCs.
Recommended graphics for the selection:
2- and 3-D Scores plots,  Scores against Spectra and  Property Box Radii.
These plots are part of the Overview-Plot. Suggested selection: use just as many secondary PCs as necessary to get a selective calibration.
Link: Secondary PCs Adjust the desired number of secondary PCs.
Link: Adjusting Secondary PCs
Setting the Radii Formula and Radii Blow up Limit
Calculation of the tolerances can be made with two different formulas and with different blow up limits.
NOTE
It may be necessary to reduce the blow up limit for different chemical substances, when Formula 1 is used in order to increase the sensitivity of the calibration.
Edit the value for Radii Blow Up Limits or Formula.
NOTE
After each new selection a recalculation should be performed again.
2.1.11 Judging the Calibration
In a good calibration, the different properties should appear as a separated and ideally compact cluster. The proper clusters are visible in the Overview.
For judging the quality of calibration the following criteria are used:
Cluster per Property: should be one, so only one cluster for each property:  Spectra Residuals too big: zero (no residual outlier);  Property Residuum: should be zero which means, that all spectra are in the right cluster;
The 2nd column of the Overview contains important control windows.
Q-Value: should be as close to 1 as possible;  Calibration Protocol: show the important adjustments and result of the calibration  Prediction Protocol show possible wrong identification for substance spectra not involved in
the calibration
After several calibration optimizations and running the automatic calibration wizard, there will be several calibrations in the project. Sort the calibrations with a click on the Name or Q-Value.
NOTE
Keep only the best calibration and delete all others.
2.1.12 Save Calibration - Lifecycle
Save the project and the calibrations to the database. Only approved calibrations according to Lifecycle will be available in the NIRWare Application Designer. See Calibration handling for further details.
34 NIRCal 5.5 Manual, Version A
Tutorials
2.1.13 SIMCA
It is possible to transfer a cluster calibration into several SIMCA calibrations. See Transform Cluster to
SIMCA.
This tool creates for all C-Set properties one SIMCA calibration, the name of the calibration is "SIMCA + calibrated property name".
2.2 Quantitative
2.2.1 Quantitative Tutorial
Quantitative calibrations are used for the determination of different concentrations or physical parameters. For quantitative calibration either the the PCR or PLS method can be used, less useful is the MLR method.
For detailed explanation see: PCR and PLS
2.2.2 Flow Chart
The flow chart shows how to build a quantitative calibration
NIRCal 5.5 Manual, Version A 35
NIRCal 5.5 Software Manual
NOTE
Loop 1 to Loop 3 represent the sequence for optimising a calibration. After each change in the selection, the calculation and principal component selection should be repeated.
2.2.3 Selecting the Calibration Method
For the quantitative calibration, basically two calculation methods are available:
Principal Component Regression (PCR) consists of a Principal Component Analysis (PCA)
with subsequent MLR.
PLS Partial Least Squares Regression calculates the PCs with iteration in several steps,
whereas spectral information and property values are considered.
The calculation method can be selected in the Menubar: Calibration / Method / PCR Principal Component Regression or Calibration / Method / PLS Partial Least Square regression or by clicking the icon PCR or PLS
2.2.4 Selecting the Calibration and Validation Spectra
Samples of known characteristics, both chemical and physical are used to generate quantitative calibrations. Ideally 60, but minimum 10 samples for each parameter with different concentration should cover the calibration range homogeneously. The measured spectra are divided into two sets:
Calibration Spectra: spectra selected for the calibration: C-Set, about 2/3 of all measured
spectra.
Generally, the calibration spectra should contain all “extreme information” to define the limits
of acceptance. The spectra with the highest and lowest property values should always belong to the C-Set.
Validation Spectra: about 1/3 of all measured spectra selected for the validation V-Set. The V-
Set should be spreaded over the whole calibration range, possible equally. Only if a calibration treats the validation spectra equal to the calibration spectra, the settings are considered as OK.
NOTE
These two groups of spectra should be:
independent from each other;  no overlapping allowed.
Suggested selection: use the selection in the calibration curve graphic. In case some extreme samples are in the V-Set, see Loop 3.
It is possible to leave out spectra from the C- and from the V-Set (e.g. unknown property values), these spectra are in the Unused Set = U-Set:
C-Set + V-Set + U-Set = All Spectra
The U-Set is visible in the calibration results (see Calibration Protocol), but will not influence the calibration and validation results e.g. will not be considered for the Q-Value calculation.
Link: Spectra selection
2.2.5 Selecting the Calibration Wavelengths
The exact wavelength / wavenumber range measured is dictated by the instrument type used for the spectrum measurement. The selected wavelength / wavenumber range depends on the application and the measuring option used.
NOTE
In general the calibration wavelength / wavenumber range should be as wide as possible.
36 NIRCal 5.5 Manual, Version A
Tutorials
Suggested wavenumber range for NIRFlex N-500 with measuring options
Solids and Liquids: 4'000-10'000 cm-1;  Fiber Optics: 4'500-10'000 cm-1.  Solids with Tablet Accessory: 6'000-11'520 cm-1.
The calibration wavelengths define the spectrum range where the mathematical algorithms is applied. Suggested selection: use all measured wavenumber in the first calculation, otherwise see Loop 2. Link Wavelength selection
2.2.6 Selecting the Calibration Properties
The calibration property is the parameter, that is required to determinate in the application. The mathematical algorithm will be applied for the selected property values
Suggested selection: create separate calibration for each parameter - single property calibrations: only one quantitative property is selected in each calibration.
NOTE
Quantitative calibration with several properties can not be stored to the Database, because it can not be used in the application.
Link Calibration property selection
2.2.7 Applying Data Pretreatments
Data pretreatments are used to eliminate non important effects or to enlarge minor effects of the measurements.
Suggested selection: perform the first calculation without pretreatment, using the spectra as they have been measured. The first step to optimize the calibration is to add and change the pretreatments.
Link: Pretreatment selection
2.2.8 Performing a calibration (calculation)
After selecting the spectra (C-Set and V-Set spectra), the wavelength range (calibration wavelength) and the properties (calibration properties), the chemometric parameters (principal components, scores, residuals, etc) will be calculated.
For the calculation of PCR or PLS, the software uses only the spectra selected into the C-Set referring to the calibration wavelength and the choice of the data pretreatment’s. The spectra of the V-Set will only be used to prove and judge the calibration
An overview of the calculation can be obtained in the Menubar: View / Overview. The result of the calibration will depend on the selected number of primary and secondary principal
components. The first step is to decide on the numbers of PCs. The calibration can be optimised manually or by using the Calibration Wizard. Here the manual
selection criteria are described.
NIRCal 5.5 Manual, Version A 37
NIRCal 5.5 Software Manual
Precision
SEC/SEP
as small as possible (around the standard deviation of the reference method)
Accuracy
V-Set Bias
around 0
regression coefficients
r
close to 1
Q-Value
close to 1
Consistency
around 100 (80-110)
2.2.9 Primary Principal Components
Primary principal components are the PCs, which are used for reconstruction of the spectra. They determine the residual value. The more primary PCs used, the smaller the allowed residual of the calibration.
Recommended graphics for primary PCs selection:
X-PRESS function,  Loadings graphic;  Spectra Residuum plots.
Suggested selection: avoid overfitting, do not use too many primary PCs. Link: Primary PCs
Adjust the desired number of primary PCs. Link: Adjusting Primary PCs
2.2.10 Secondary Principal Components
The secondary or calibration PCs are used for the parameter calculation. The number of secondary PCs is limited to the number of primary PCs.
The target is the best matching between the original reference values and the predicted NIR values. This can be seen in the "Predicted Property vs. Original Property" plot and can be judged with statistical values like:
standard errors: SEC / SEP;  V-Set Bias;  regression coefficients;  PRESS-values;
These plots are part of the Overview-Plot. Suggested selection: use the optimal number of secondary PCs, which gives the best result for all
spectra in the C-Set and V-Set. Link: Secondary PCs
Adjust the desired number of secondary PCs. Link: Adjusting Secondary PCs
NOTE
After each new selection a recalculation should be performed again.
2.2.11 Judging the Calibration
The target is the best matching between the original reference values and the predicted NIR values. This can be seen in the calibration curve and can be judged with statistical values like:
These values are documented in the Calibration Protocol.
38 NIRCal 5.5 Manual, Version A
Tutorials
After several calibration optimizations and running the automatic calibration wizard, there will be several calibrations in the project. Sort the calibrations with a click on the Name, Q-Value or e.g. SEP.
NOTE
Keep only the best calibration and delete all others.
2.2.12 Save Calibration - Lifecycle
Save the project and the calibrations to the database. Only approved calibrations according to Lifecycle will be available in the NIRWare Application Designer. See Calibration handling for further details.
2.2.13 Create a Quantitative Calibration with Cross Validation
The above tutorial is valid for a user selected validation set (VS method). It is possible to create a quantitative calibration using the Cross Validation (CV method), in that case
all spectra should be in the C-Set, the V-Set should be empty.
NIRCal 5.5 Manual, Version A 39
3 Chemometrics
3.1 Calibration Methods
3.1.1 Principal Component Analysis: PCA
Principal Component Analysis is a mathematical, statistical evaluation of a large amount of chemical data. In this case the chemical data are the measured NIR spectra.
PCA is made for two reasons:
- to reduces the data amount without loosing necessary information. Noise is truncated by the number of primary PCs;
- to evaluate the measured spectrum automatically after creating a calibration. With today's powerful computers, the prime object is no longer the reduction of the data volume. Today, the main goal of PCA is to find and automatically evaluate characteristics of identity, quality and quantity in the spectra.
Each spectrum measured with NIRFlex N-500 consists of 1.501 data, which correspond to the intensity values of the 1.501 support points on the wavenumber scale. In order to obtain a good calibration, a large number of spectra is needed. For 100 substance spectra, this already gives us 150.100 data points, which places an enormous computing workload on computers. To achieve acceptable computing times, the spectral data are therefore efficiently compressed with the aid of PCA without loosing any important information. For this purpose, PCA utilises the redundancy occurring in the spectra. With PCA, so-called principal components are extracted which are statistically independent from each other and which are therefore orthogonal relative to one another, yet are still capable of adequately reconstructing the original spectra. The PCA will always be performed with the calibration spectra set in the selected wavenumber with the selected pretreatment. A geometric explanation will serve to visualize the PCA: it is not possible to imagine a space of 1.501 dimensions (selected wavelengths), with each wavelength or wavenumber corresponding to a dimension. But in this space, a spectrum can be represented as a point. For three dimensions, this can be shown graphically:
In mathematical terms this point is equivalent to a vector with 1.501 components (I1....I1501). Several
calibration spectra produce a "cloud" of points – a cluster- in space. For a set of spectra or points in the 1.501-dimensional space, a coordinate's transformation is now performed in a way that the new origin comes to lie in the mean centre of all the spectra - mean centering - and the new space directions - principal components - lie along the greatest variance in the spectra. The new space directions are calculated in such a way that the features with the widest variances – differences- of all spectra are included in the first space directions and the higher space directions gradually evolve into noise. Space directions, which contain no any other information than noise, are no longer taken into account.
NIRCal 5.5 Manual, Version A 41
NIRCal 5.5 Software Manual
A reduction of the dimensions is performed when the number of PC's (i) is not higher than 1.501. Through this type of data reduction it is impossible to lose information. With today's powerful computers, the prime object is no longer the reduction of the data volume. Today, the main goal of PCA is to find and automatically evaluate characteristics of identity, quality and quantity in the spectra. As a result of the PCA, the following is obtained:
Mean spectrum: <I (k)>
Calculated by averaging the intensity values at each wavelength. The centre of the new coordinate system is shifted to that point: mean centering.
Formula:
Principal Components: U i (k)
New space directions in the point cluster, which are also called principal components. PCs are artificial differential spectra. Each calibration can have i PCs (default is: 15).
The mean spectrum and PCs are always fixed for the calibration.
42 NIRCal 5.5 Manual, Version A
Chemometrics
Scores: v in
Weightings of each PC after the pretreated spectrum has been transformed into the cluster. A score is the portion of a PC used for the reconstruction of the original spectrum. Each spectrum has different and up to max. i scores.
Residuum: R n (k)
The difference between the pretreated spectrum and the reconstructed spectrum is the residuum spectrum. When the residuum is summed across the wavelength, a number is obtained, the Residual.
The scores and the residual are variables for each spectrum.
Leverage
The direct distance of the spectrum from the centre of the coordinate system in the score place.
Reconstruction of a spectrum
Now, each spectrum can be reconstructed on the basis of a sum through multiplying of scores and PCs. Any desired spectrum I n (k) of the calibration set is developed:
Formula: For the reconstruction it is now only necessary to save the scores v in and the residual R n of each
individual spectrum, since the mean spectrum and PCs are constant for the entire set of spectra in a calibration. The spectrum can also be described as the linear combination of the PCs -U i (k)- and their scores (mean-centred data matrix).
The following figure shows the reconstruction of a spectrum:
Mathematically speaking, the Principal Component Analysis is then a breakdown of the spectrum matrix into 2 smaller matrices. This matrix operation can be represented graphically for, e.g., 15 PC's in the following way:
NIRCal 5.5 Manual, Version A 43
NIRCal 5.5 Software Manual
For 100 spectra with1501 data points the following data reduction is obtained with 15 PCs: 100 x 1.501 ==> 1.501 + [15x1.501] + [100x 15] + 100
150.100 ==> 1.501 + [22.515] + [1.500] + 100 Example for a Principal Component Analysis: 4 different acetone qualities: without and with 0.3 %; 0.7 % and 1.0 % added water. There are only 3
PCs necessary to reconstruct the spectra. The scores according to PC 1 and PC 2 are repeatable. The residuum spectra have only noise character.
As a result of the PCA, we obtain those PCs which themselves represent spectra and scores for each spectrum.
44 NIRCal 5.5 Manual, Version A
Chemometrics
The scores can be represented in two- or three-dimensional PC plots. Each number represents a spectrum, v in its score.
Here, the scores of the spectra 1, 2 and 3 of the PCs 1 and 2 are graphically represented. Each spectrum with "i" PCs will also have "i" scores. The closer together the points in the plot, the "more similar" the spectra. It is now possible to break down an unknown spectrum with regard to these two PCs, i.e. to determine the scores. If this spectrum is located, e.g. in the region of 1, it will be identified as 1.
User-allocated properties of the spectra (e.g. quantity, good/poor quality, identity) do not have any effect on the Principal Component Analysis.
The Mahalanobis distance
The introduction of Mahalanobis distances means an artificial scaling of the scores with the square scores sum being normalised. At the same time this leads to a stretching or compression of the PCs, since the product obtained from the score and PC is not changed. A new normalisation is performed so that the scores v in of the spectra n will retain roughly the same magnitude as the PC index increases. Scores are variables without unit that must only be considered relative to one another. The purpose behind all this is to make physical or chemical properties which have only slight effects on the spectral data and which therefore only manifest themselves in higher PCs as visible as those clearly shown in the spectra.
The scores are normalised as follows:
Formula: For this reason, the points in the 2-D plots are evenly distributed, i.e. the scores of all PCs have the
same average magnitude. On the other hand, individual PCs are appropriately reduced or increased. For further evaluation, only the normalised scores are used.
NIRCal 5.5 Manual, Version A 45
NIRCal 5.5 Software Manual
The Büchi NIRCal software package only works with the normalised distances, the user can not see the result without it.
3.1.2 Cluster: CLU
Goal: to identify different chemical substances using the PCA and as a result to get a well separated cluster area in the scores plot for each substance.
The clusters are created according the selected secondary PCs. Secondary PCs are that PCs among the primary PCs, which shows a clear separation of the substances and the scores are good repeatable. These secondary PCs are responsible and used for the tolerance radii calculation.
Tools for the selection of secondary PC's:
the scores of the spectra will be shown dependent to the PCs: Scores against Spectra. The
scores of all spectra of each substance should be close together, but separated from the scores of all other substances;
the Property Box Radii plot shows the repeatability [(max-min) scores / 2] of the scores for
each property. PCs showing small Property Box Radii values (normally below 0.1) are important for the calibration;
2 and 3 dimensional score plots: PCs having repeatable scores should be selected.
46 NIRCal 5.5 Manual, Version A
Chemometrics
Additional tool:
The Score Disorder values show how effectively a particular PC separates different properties
(substances) from each other. By scanning the score values in one direction of a PC and counting the changes between the membership of A or B the disorder value is achieved. If a PC completely separates all calibration properties, the disorder values is the [number of calibration properties - 1]. PCs with small disorder values are possible calibration PCs.
The selected number of secondary PCs should be adjusted and the calibration should be recalculated. Link: Secondary PCs
The tolerance ring radii are determined according to the secondary PCs. For this calculation the smallest possible rectangle (rectangle in 2D, a cuboid in 3D, an n­dimensional cuboid in n-dimensions) constructed around each C-Set. The sides of the rectangles (a and b) are parallel to the axis of the PCs. The distance from the center to the side of the smallest possible rectangle is a measure of the extension of the C-Set, this is the so called “Property Box Radii”. A ‘virtual rectangle’ is created 5 times greater than the rectangle around the C-Set.
The following distances will be calculated:
R1 smallest distance between a spectrum (substance B) to the closest spectrum of a different
property (substance A).
R2 smallest distance between a spectrum (substance B) to the side of its virtual rectangle.  R3 smallest distance between a spectrum (substance B) to the closest spectrum of its own
property (substance B).
R4 mean value out of all R3 distances of the same property.
With these distances it is possible to calculate tolerance circles with a radius r for every calibration spectra by using Formula 1, 2 or 3.
Formula 1:
Min. of r = R1/2 * f or r = (R1+R2)/4 * f
The smaller of the two possible values for the radius “r” is used. The default setting for the Radii Blow Up factor "f" is 1.
Depending on the extension of the cluster (R2) and the distance between the two closest clusters (R1), the circles are closer or further away from each other. With a Radii Blow Up of 1 two circles can just touch each other but do not overlap. For chemically different substances the Radii Blow up (f) may be reduced (0< f <1) in order to increase the sensitivity of the calibration.
NOTE: It is not suggested to have a Radii Blow up (f) higher than 1: it can cause overlapping rings!
NIRCal 5.5 Manual, Version A 47
NIRCal 5.5 Software Manual
Formula 2
If generally small tolerances are required, Formula 2 can be used:
min. of r = R1*0.499 (0.499: to avoid overlapping) or r = R3 * Pre Blow Up * f
or r = R4*0.5*Pre Blow Up * f The smallest of the three possible values for the radius “r” is used. The pre Blow Up factor is 5, this is an empirically evaluated value. The default Radii Blow Up f = 1,
this value can be adapted by the user. In general it should be increased to get connected spheres.
Formula 3
It is used only for SIMCA calibration, where only one substance is calculated (there is no R1).
Min. of r = R2 * 0.5
or r = R3 * Pre Blow Up * f
or r = R4 * 0.5 * Pre Blow Up * f The smallest of the three possible values for the radius “r” is used. The tolerance circles can now be plotted for all calibration spectra:
These tolerance ranges will show if the sum of the radii of one property generates only one cluster: cluster / property should be one. In this case the number of the secondary PC selection is OK. If more than one cluster is generated, the number of the secondary PCs is not optimal or other calculation (e.g. pretreatment) should be tried.
Assessing the calibration
Cluster per Property
The Cluster per Property plot shows if all tolerance circles build one connected cluster for each property. Here only a value of 1 is acceptable.
All spectra in the C- and V-Set should be identified correctly
The identification is made according to the distance in the scores plot and residual:
48 NIRCal 5.5 Manual, Version A
Chemometrics
Distance
The distance to the next calibration spectra should be smaller, than the tolerance ring radius of the neighboring calibration spectrum. In this case the distance criterion is OK, the spectrum is in a
cluster.
Property Residuum zero means, all spectra are in the correct cluster.
Here only a value of 0 is acceptable. Property Residuum +1 means, that a spectrum is outside the cluster: it is not identified. Property Residuum –1 means, that a spectrum is in a wrong cluster: it is false identified.
Residual
The [(maximum residual of a C-Set) * 2] is the max. allowed residual for the calibration and later for the application. The default residual Blow up factor is 2, it may be changed by the user (it is not suggested to use smaller values as 1). Spectra Residual too big should be for all spectra zero.
These three criteria are showed in the Overview plot in the 2nd column.
3.1.3 SIMCA
SIMCA is a calibration method used for identification of substances. Using SIMCA a Principal Component Analysis (PCA) is made for each substance/property in the project, but each calibration is made for only one substance.
Cluster calibrations can be transformed to several SIMCA calibrations. See Transform Cluster to
SIMCA.
This tool creates for all C-Set properties one SIMCA calibration, the name of the calibration is "SIMCA + calibrated property name".
All SIMCA calibrations take over the following default parameters:
the spectra C-Set and V-Set selection of the last active cluster calibration;  the wavelength selection from the selection of the last active cluster calibration;  pretreatments of the last active cluster calibration;  the "Mean centering after Pretreatments" is still switched on; represent the mean value
spectrum;
NIRCal 5.5 Manual, Version A 49
NIRCal 5.5 Software Manual
Yes
Transform the actual Cluster Calibration into multiple SIMCA Calibration using default SIMCA Parameters. The existing Cluster Calibration will not be modified.
No
Change only to SIMCA method without any calculation.
Cancel
Cancel the transformation and keep the current method.
NOTE
Switching the "Mean centering after Pretreatment" OFF, the first principal component represent almost the mean value spectrum. This is suggested for only one property in the calibration, which is always the case for SIMCA. SIMCA can also be created without a transformation from a cluster calibration.
the number of primary principal components is selected with the "Factor Selection Wizard";  the number of secondary principal components is selected also with the "Factor Selection
Wizard" according the Q-Value.
3.1.4 Transform Cluster to SIMCA
An active Cluster Calibration can be transformed to SIMCA by changing the calibration method to SIMCA. Before the wizard starts with the calculation the following pop-up window appears:
For each C-Set Property of the Cluster Calibration a separate SIMCA Calibration is created.
Hereby the SIMCA default parameters are used and calculated. An automatic estimation of primary and secondary PCs is made.
50 NIRCal 5.5 Manual, Version A
Chemometrics
Plot name
Description
1
Pretreated Spectra
using the same pretreatment, as the cluster calibration
2
scores against PCs
scores against the PCs for each spectra in the project
3
scores vs. scores
in this score plot the calibrated substance is around the center (Mean Centering distance is calculated for only the C-set spectra), each other spectra, which are not in the calibration, have normally huge score values
4
spectra residual
the residuals of the calibrated substance spectra are normally smaller, as each other substance residuals
5
residuals vs. leverages
the so-called Coomans plot shows the residuals against the leverages (leverage is the direct distance of a spectrum in the score plot from the centrum)
6
NIR-Explorer
SIMCA Overview Plot
The Overview plot is automatically opened after the SIMCA calculation is finished. Each calibration has the name:
SIMCA + substance name (property name) selected for the calibration.
The Overview plot contains the following plots:
For a SIMCA calibration the limits are also called:
Residual : outer model distance;  Leverage : inner model distance.
NIRCal 5.5 Manual, Version A 51
NIRCal 5.5 Software Manual
PC (Factor) Selection Wizard
Primary PC selection:
Test 1.5 : C-Set X-PRESS Slope Ratio Highest test
w(i+1) = ( y(i) - y(i+1) ) / ( y(i+1) - y(i+2) )
Limit = 2
PC i for highest i where w(i) > Limit Precise
Secondary PC selection:
Test 2.2 : Q-Value maxima (limited) test
PC i for Max ( QValue(i) ) I < NumPrimaryPCs NIRCal calculates the allowed residual using the primary principal components for reconstruction.
Default parameter for residual
Residual Blow Up = 2.5
Allowed residual for calibration is 2.5 x max. C-Set Residual. For each C-Set spectrum NIRCal creates a tolerance sphere using the Formula 3 for radii calculation according to the Mahalanobis distances with the secondary PCs. This calibration sphere "inner" space defines the area for a substance. Default parameters for scores and radii:
Scores Blow Up = 1.05
Radii Formula = 3
Radii Blow Up = 2.5
SIMCA Q-Value
The Q-Value for SIMCA calibrations can not take into account the "Property Interference" value, because there is only one cluster type in each calibration. This value is always Zero. This causes a slightly higher Q-Value against the Cluster calibrations, in case no outliers are in the SIMCA calibration.
SIMCA Method Validation
SIMCA allows that the principal component spaces cover each other or can partially overlap another spaces. To check possible overlapping the "Prediction Protocols" can be used, as it is also suggested for the Cluster calibrations.
The number “Total not identified, Cluster BAD (&)” and „Total not identified, Cluster OK (%)“ can indicate the correct adjustment of the Blow up (Radii and Residual). SIMCA tolerance spheres normally lie close to the PC center, other spectra can lie also here, which causes losts of "Total not identified, Cluster BAD (&)" cases. This can be reduced by reducing the
52 NIRCal 5.5 Manual, Version A
Chemometrics
Result
Residual
Distance
Identified
OK
OK
Not identified
not OK
not OK
Not identified
not OK
OK
Not identified
OK
not OK
Prop
property of the „n“th Spectrum
a
intercept
b1
correlation coefficient of the first wavelength
I1
intensity at the selected (first) wavelength
Radii Blow Up limit. The number of "Total not identified, Cluster OK (%)" can be reduced by reducing the number of primary PCs or by increasing the Residual Blow Up limit.
Using SIMCA in application
For the identification of an unknown substance the residual should be below the allowed limit and the unknown spectrum distance should be smaller as the allowed tolerance sphere to the nearest known calibration spectrum (inside the "inner space"). In the application mode there are 2 answers possible:
3.1.5 Multiple Linear Regression: MLR
Multiple Linear Regression is an extension of the linear regression to several dimensions. The analysis is based on a few selected wavelengths and does not require any PCA calculation. In this procedure, the properties are calculated through intensity values and correlation coefficients, e.g. it is valid for two selected wavelengths (I1 and I2).
Where:
Note: I1 and I2 must describe independent characteristics. Select at least 3 wavelengths. Because with MLR only few wavenumber (min. 3) are used and the rest of the measured 1501 (in
case of NIRFlex N-500) are automatically discarded, this simple method is not suggested to use. The residual cannot be used for outlier detection during the application because of the extreme
wavelength reduction. This method is only suggested for filter instruments. For Interferometers (full wavelength range) it is
suggested to use PCR or PLS.
NIRCal 5.5 Manual, Version A 53
NIRCal 5.5 Software Manual
3.1.6 Principal Component Regression: PCR
Principal Component Analysis (PCA) with subsequent MLR is called Principal Component Regression (PCR). As a first step, the principal components and scores are calculated with PCA. The second step is a multiple linear regression MLR using the scores and property values (concentrations). Since the calculation of the principal components is performed with the spectral data – independently of the subsequent regression calculation for the correlation of the quantitative values – any number of parameters can be simultaneously included in a PCR calibration. This also means that the relevant PCs for the determination of the property are not necessarily the ones describing the biggest spectral variations.
3.1.7 Partial Least Squares Regression: PLS
Partial Least Squares Regression (PLS) calculates the PC's with iteration in several steps, with spectral information and property values being taken into account simultaneously. This calculation method is more up to date than the PCR. Based on the principle of recursion, PC's and scores are also calculated as with PCR, but the quantitative reference values are included in the calculation from the beginning.
Each of the calculated PC's in the PLS procedure contains information about the original property values (true concentration) of the samples, with the first PCs (unlike PCR) always showing the highest correlation.
54 NIRCal 5.5 Manual, Version A
Chemometrics
If two parameters are not systematically correlated, the mathematical approximation of the spectra via PLS can never be performed for both parameters when each parameter is calibrated using its own PLS. For this reason, it is recommended to calculate properties which are not systematically correlated (e.g. ethanol and acetone contents in any given solvent mixture) in separate calibrations. Therefore, whereas PCR reduces spectral data to the most dominate dimensions, the PLS aims at the most relevant dimensions (relevant here means: best match between predicted and original values). With PLS, the PC's are calculated exactly in relation to the highest correlation in the first PC.
3.2 Calibration Validation Methods
3.2.1 Validation Set (VS)
C-Set and V-Set
C-Set (Calibration Set). From all spectra within a project only the spectra which are in the C-Set are used for the calculation of the calibration.
V-Set (Validation Set). From all spectra within a project only the spectra which are in the V-Set are used for an internal validation of the calibration.
Normally for VS mode 2/3 of the spectra are selected as C-Set and 1/3 are in the V-Set. This can be done with Toolbox “Set Creation”.
3.2.2 Cross Validation (CV)
Cross Validation Method
Instead of dividing the samples into two groups, a calibration set and a validation set, all samples are used for calibration in CV mode. Several calibration runs are performed with all samples except a small group with which the actual calibration is tested. This group is changed for all trial runs. The validation results of all the runs are stored and lead to a standard error of the cross validation (SECV) which compares well with the standard error of predictions (SEP) obtained in VS-mode.
In case each CV group consists of only one sample (one leave out) the method is called full cross validation. This is the default setting for CV-grouping in NIRCal.
Cross validation is recommended for calibrations based on a small amount of samples. If this number is larger than 50, NIRCal suggests to use a validation set instead (this comment can be suppressed; see NIRCal Configuration Edit / Options / Calibration Defaults).
NOTE
All spectra of one sample must be assigned to the same CV-group.
NOTE
For CV mode all spectra should be assigned to the C-Set. Limitations:
Cross validation is available for PCR and PLS only  Cross validation requires at least 4 spectra in the C-Set  For cross validation at least 2 CV groups have to be assigned  Cross validation will delete the V-Set assignment
NIRCal 5.5 Manual, Version A 55
NIRCal 5.5 Software Manual
Menu:
Calibration \ Change Data Sets \ Edit CV Groups...
Icon:
One leave out
Each spectrum represents a group. Full cross validation
Alternate
The spectra are grouped to different groups one after the other. Number of Groups and Spectra per Groups can be varied, they have an interdependency.
Sequence
Consecutive spectra are grouped to the same group. Number of Groups and Spectra per Groups can be varied, they have an interdependency.
Random
Spectra are grouped randomly. The groups are filled to the maximal amount of Spectra per Group.
Property Segments
For the selected Property the range (min - max value) grouped into segments. An empty segment will not build a group.
Property Equal
All spectra with the identical property value are grouped. The property values are compared over all properties.
Spectra Name
All spectra with the same spectra name are grouped. The characters to compare can be defined with Start at Character and Number of Characters.
Spectra Name (autom.)
Number of Characters is appraised incrementally until the spectra can be grouped to more than one group.
Start at Character is always 1.
Cross Validation Grouping
In order to define the cross validation groups, the CV Group selector is opened.
The various possibilities for selections are summarized in the following table. Methods
56 NIRCal 5.5 Manual, Version A
Custom assign Group to Spectra
This is the default method when the CV Group Selector is started . In this mode it is possible to display a plot (Group Plot) and/or a table (Group Table) with the CV Group Index
Method
Select a method from the drop-down list.
Group Table
Display table: CV Group Index. Enabled only for the method: Custom assign Group to Spectra
Group Plot
Display plot: CV Group Index. Enabled only for the method: Custom assign Group to Spectra
OK
OK
Cancel
Cancel
Number of Groups
Spectra per Group
Start at Character
Used for Spectra Name Default is 1, the character comparison starts at the first character. (e.g. Start at Character=10 then all Spectra names that are shorter than 10 characters like "Name xy","Batch01","1" are grouped into 1 group.
Property
Number of Segment
Number of Characters
Used for Spectra Name Default is 500 characters, up to 500 characters are compared.
Short form
Enabled : e.g. 1-5 Disabled: e.g. 1,2,3,4,5
List to Clipboard
The group list is copied to the clipboard.
Highlight Group
All spectra of a group are highlighted red in NIRCal-plots.
Plot Group
Select a group and plot
Buttons / Selections
Chemometrics
After creating or changing the groups, the CV calculation should be performed again.
CV Methods
One leave out (FCV)
Each spectrum represents a group: Full Cross Validation.
NIRCal 5.5 Manual, Version A 57
NIRCal 5.5 Software Manual
Alternate
The spectra are grouped successively by increasing number (1st spectrum to the 1st group, 2nd spectrum to the 2nd group, etc.).
The number of Groups or the number of Spectra per Groups can be changed by clicking on the + or ­symbols.
The number of Groups and Spectra per Groups are depending on each other.
58 NIRCal 5.5 Manual, Version A
Chemometrics
Sequence
The spectra are grouped with consecutive number (1st to 3rd spectra to the 1st group, 4th to 6th spectra to the 2nd group, etc.).
The number of Groups or the number of Spectra per Groups can be changed clicking on the + or ­symbols.
The number of Groups and Spectra per Groups are depending on each other.
NIRCal 5.5 Manual, Version A 59
NIRCal 5.5 Software Manual
Random
The spectra are grouped randomly.
The number of Groups or the number of Spectra per Groups can be changed clicking on the + or ­symbols.
The number of Groups and Spectra per Groups are depending on each other. The smallest number of Group is 2.
60 NIRCal 5.5 Manual, Version A
Property Segments
Chemometrics
For the selected property the min.-max. value range will be divided into Number of Segments (default:
20). Each spectrum, which has the value belonging to a segment, builds a group. Segments without property value are empty (here e.g. concentration range from 5 till 10 %). The cross validation calculates the groups for the selected property. Only one property must be
selected.
NIRCal 5.5 Manual, Version A 61
NIRCal 5.5 Software Manual
Property Equal
Spectra with the same property value (concentration) selected for a group. Each property is taken into account.
62 NIRCal 5.5 Manual, Version A
Chemometrics
Spectra Name
Spectra with the same name are selected for a group. The comparison will start from the 1st character (default), but it can be changed by the user. The length of spectra name can be limited by: Number of Characters. Default: 500.
NIRCal 5.5 Manual, Version A 63
NIRCal 5.5 Software Manual
Spectra Name (autom)
Spectra with the same name are selected for a group. The comparison will start always from the 1st character. The length of spectra name will automatically be determined.
64 NIRCal 5.5 Manual, Version A
Custom assign Group to Spectra
Chemometrics
For custom-defined grouping, open the Matrix CV Group Index with the button "Group Table".
NIRCal 5.5 Manual, Version A 65
NIRCal 5.5 Software Manual
Group name
In C-Set
In CV-Group (CV Group Index > 0)
Unused Spectra
No
No
CV-unused Group
No
Yes
CV-permanent C-Set
Yes
No
CV-mutable V-Set
Yes
Yes
Group Index numbers define to which group a spectrum is assigned. There are 4 different types of selection possible:
Spectra with Group Index zero and not selected in the C-Set: unused spectra. Spectra with Group Index higher than 0, but the spectra are not selected in the C-Set build the CV-
unused Group. Advantage of this group: outliers, which belong to a spectra group, can be removed from the C-Set without making a new spectra group. In case the spectra should be removed from this group, the Group Index should be edited to zero.
Spectra with Group Index zero and selected in the C-Set: CV-permanent C-Set. These spectra will never be left out from the calibration, for each calculation they will be used within the C-Set. Advantage of this group: spectra with extreme values (min., max.) can be permanently kept in the C­Set, which is highly recommended. Spectra with Group Index higher than 0 and selected in the C-Set: CV-mutable V-Set. These spectra are removed once out of the C-Set and used as V-Set during the CV cycles. These spectra are listed in the calibration protocol with their Group number/Group Index and the spectra belonging to each group.
66 NIRCal 5.5 Manual, Version A
Chemometrics
CV Plots
It is possible to show the result of each calibration step (e.g. CV Predicted Property) and also the result of the final calibration (e.g. Predicted Property).
The Overview contains the following plots:
[1] The 1st C-Set property is chosen automatically for the plots.
Pretreated Spectra:
it is suggested to start the first calibration without pretreatment, but try later some pretreatments and combinations.
CV Property Residuum vs. Original Property:
property residuum = original property -predicted property. This plot shows for each sample group the property residuum for the calibration, which had this group in the V-Set during the CV. A small CV Property Residuum and a regression coefficient between original property and predicted property residuum close to 0 shows a stable calibration. Spectra with big deviations are possible outliers and removing them from the C-Set can improve the calibration.
Predicted Property vs. Original Property:
This plot shows the result of the final calibration for the spectra which are in the calibration set (unused spectra are not visible per default: it can be changed by the user under Visibility).
CV Regression Coefficients [1] (called the property spectra in NIRCal 4.21)
Shows the coefficients of the linear relationship between the NIR amplitudes (of the pretreated spectra) and the selected C-property. [1] refers to the 1st C-property. In general for each application only one C-Set property is allowed, so for each property a separate calibration is necessary (NIRCal could handle more, but NIRWare is designed for only one property / calibration for quantitative measurements).
CV SECV; see further details under Matrix CV SECV.
In general the first local minimum of CV SECV for the secondary PC selection will be taken, in
NIRCal 5.5 Manual, Version A 67
NIRCal 5.5 Software Manual
case several minimum SECV exist. Use the Factor Selection Wizard. The selected number of secondary PCs will be red.
Beside these plots there are several other plots to choose outlier spectra or to make a better view of the results. The plots under has the following plots:
Property Residuum:
The property residuum of each spectra showed with the final calibration.
Spectra Residuals vs. Leverages:
The final calibration result with the used primary PCs are showed for the calibration spectra.
CV Spectra Residuals vs. Spectra Residuals:
The spectra residual with the final calibration against the residuals with the CV calculation results are compared. Big difference shows, which spectra groups can be outliers.
Spectra Residuum:
= pretreated spectrum-reconstructed spectrum. The residuum of the final calibration is shown.
CV Leverages vs. Leverages:
The leverages for the final calibration against the CV leverages are ploted.
CV Spectra Residuals vs. CV Leverages:
Large residuals together with large leverage (points in the upper right square) are typical for possible outliers.
68 NIRCal 5.5 Manual, Version A
3.3 Selections
Menu:
Calibration / Method / ...
Icon:
for Principal Component Regression / for Partial Least Squares / for
Multiple Linear Regression
for Cluster / for SIMCA
3.3.1 Calibration Method Selection
Chemometrics
Example: For the calibration of Saccharose the method PCR is selected and for the calibration of Lactose PLS is chosen.
NIRCal 5.5 Manual, Version A 69
NIRCal 5.5 Software Manual
Menu:
Calibration / Validation Method / ...
Icon:
for Validation Set for Cross Validation
3.3.2 Validation Method Selection
The default method is VS (Validation Set). For quantitative calibrations CV (Cross Validation) is available.
3.3.3 Data Sets
Data sets are permanent selections that are stored and loaded with the project.
Spectra Data Set Wavelength Data Set Properties Data Set PC Data Set
In case one of the above data set is left empty by the user the software automatically will create a selection with all data within the project (e.g. wavelengths property).
70 NIRCal 5.5 Manual, Version A
3.3.4 Edit Data Sets Dialog
Menu:
Calibration / Change Data Set / Edit Data Sets...
Icon:
Monte Carlo Random
this selection is suggested where there is only one spectrum for each substance was measured but is not recommended (measurement mistake is not clear). 3 measured spectra of the same sample could be separated.
Sequence
e.g. 70% of the spectra are selected automatically. The measured spectra will be separated time dependent. 3 measured spectra of the same sample can be separated.
Blockwise
to distribute 2/3 of all measured spectra into the calibration set, 6 spectra are selected to the C-Set and 3 left out for V-Set in the range from 1 to the last measured spectra. This is the most common selection method.
Custom (Spectra)
user selected by spectrum number. To separate two spectra blocks, a comma and space are used.
Chemometrics
Under the drop-down list Name you can choose to edit the selections of the sets:
Calibration Spectra  Validation spectra  and more
NOTE
Click Apply after a selection, click OK to close the Dialog. Invert: a simple tool to invert the selections previously made. It is very useful when dealing with large
numbers of spectra. Method
NIRCal 5.5 Manual, Version A 71
NIRCal 5.5 Software Manual
Range from ... to ...
The first and the last spectra index.
Block select ... leave ...
Number of selected and left out spectra per block. Amount ... %
Amount of selected spectra in %.
Parameter
3.3.5 Spectra Data Set
Samples of known characteristics, both chemical and physical are used to generate calibrations. Measurement conditions should remain constant for all samples using the full spectrum range. Several sample from different batches should be collected for a robust calibration and each samples should be analysed in the laboratory with the classical method. Only acceptable samples can be used for the calibration. The measured spectra are normally divided into two sets.
Calibration Spectra (spectra selected for the calibration) C-Set, about 2/3 of all measured
spectra (min. 3). Only the C-Set spectra are used for the calculation of the principal components and the calibration limits.
Generally, the calibration spectra should contain all “extreme information” to define the limits
of acceptance. For quantitative calibration the spectra with the highest and lowest property values always belong to the calibration spectra.
Validation Spectra, about 1/3 of all measured spectra selected for the internal validation V-
Set. Only if a calibration treats the validation spectra equally to the calibration spectra, the settings are considered as OK
NOTE
These two groups of spectra should be:
independent from each other: all spectra of one sample should belong to the same set;  no overlapping allowed.
It is possible to leave out some spectra from the C- and from the V-Set, these spectra are in the Unused Set = U-Set:
C-Set + V-Set + U-Set = All Spectra The U-Set is visible in the calibration results, but will not influence the calibration and validation results
e.g. will not be taken for the Q-Value calculation. The calibration protocol is stored within the calibration.
C- and V-Set can be selected by the user in different ways:
in the NIR-Explorer,  in the Property table.  in the graphic
72 NIRCal 5.5 Manual, Version A
Spectra Selection in the NIR-Explorer
Open NIR-Explorer
Chemometrics
Steps sequence:
1. Open the folder “Calibrations” by clicking on the box + in front of the folder or double clicking the
folder
2. Open the active calibration, indicated by the red dot.
3. Open the folder “Data Sets”.
4. Select "Spectra".
5. Select "Calibration Spectra" in the right part of the window. To open the Edit window press the right mouse button and click on Edit Set or double click on the selected line. Choose Blockwise with Block select 6 and leave 3, press OK for applying this selection. For the " Validation Spectra" the Blockwise method is selected, press first Apply, than Invert and press OK to get the rest.
NIRCal 5.5 Manual, Version A 73
NIRCal 5.5 Software Manual
Selection methods: Monte Carlo Random: this selection is suggested, when only one spectrum for each substance was
measured, which is not recommended (measurement mistake is not clear). For 3 spectra / samples this method is not suggested, while the 3 spectra can be randomly in C- or V-Set.
Sequence: e.g. 70 % of the spectra are selected automatically into the C-Set. 3 spectra / sample can be separated!
Blockwise: to distribute 2/3 of all measured spectra into the C-Set, 6 spectra are selected to the C-Set and 3 left out for the V-Set of the range from 1 to 80. This is the most common selection method.
Custom: spectra selected by spectrum number in the project. To separate two spectra blocks, a comma and space are used.
NOTE
When each sample has been measured three times, all three spectra should be designated to either the calibration or validation set. Therefor it is ideal to use Blockwise selection e.g. if each sample was measured with 3 spectra block select 6 and leave 3 can be used. Blockwise selection is only recommended for qualitative calibrations.
Choose Blockwise with Block select 6 and leave 3, press OK for applying this selection for the C-Set. For the " Validation Spectra" the Blockwise method is selected, press first Apply, than Invert and press OK to get the rest.
Spectra Selection in the Property Table
Open the table Original Property in Menubar: Tables / Properties / Original:
74 NIRCal 5.5 Manual, Version A
Chemometrics
The table consists of the spectrum number, the name of spectrum with batch number and the property membership (1 indicates: it belongs to a property, 0 indicates: it does not belong to a property).
NIRCal 5.5 Manual, Version A 75
NIRCal 5.5 Software Manual
Creating the selection: mark the first selected row with the mouse, press the left button only once. Press the “Shift-key” and double click on the last marked row. All selected spectra are highligthed in red colour.
To remove spectra from the selection: mark the selected row with the mouse, press the left button only once. Press the “Ctrl-key” and double click on the row. When the selection is created, the spectra need to be copied to the calibration spectra and/or the validation spectra:
1. press the right mouse button in the graphic and Copy Selection to – Row - Calibration Spectra;
2. for setting the Validation Spectra select in the Popup-Menus the command Invert Selection. The rest of the spectra will be selected;
3. copy this selection in the Popup-Menus with the command Copy Selection to – Row - Validation Spectra.
NOTE
The menu “Copy Selection to” overwrites the existing selection in the data sets.
Add and Remove Selection from Datasets
Mark the spectra rows of the position to be changed in the selection. Press the right mouse button and add to or remove this selection from the desired data set. In this way, the selection made before will be enlarged / reduced with only the new selected data.
76 NIRCal 5.5 Manual, Version A
Chemometrics
Spectra should not be designated to both, Calibration and Validation-Set. Should this happen the system forces the user to make a clear decision.
Clicking "Yes" will remove the overlapping spectra from the Calibration Set and keep them in the Validation Set.
Spectra Selection in Graphic
A very practical way to make the spectra selection is in the calibration plot using the mouse. This is especially useful for the quantitative calibration because in the plot "Predicted Property vs. Original Property" the concentration distribution of the C- and V-Set is visible.
Click on the Overview-Button in the NIRCal-Toolbar. If the C-Set and the V-Set are empty, a message will appear. Confirm by clicking on the Yes button to get the Overview.
Enlarge the window ”Predicted Property vs. Original Property" in the 3rd column. The ”Predicted Property vs. Original Property” can be opened in the Menubar: Graphics / Properties /
Original vs. Predicted as well: Press the “Minus”-button on the keyboard to expand the X-and Y-ranges of the graphical display.
Clear the selection made before by pressing the right mouse button and click on Clear Selection. To create selections in a graph, the function of the mouse must be changed from the Zoom function to the Window Select function.
Press the right mouse button and choose "Mouse Select" under "Options". The symbol of the
mouse will change from to .
Draw a box around the spectra with the mouse keeping the left mouse button pressed.  Draw boxes around all the other spectra you want to select by holding the “Shift-key”. All
selected spectra will be highlighted in red as soon as the left mouse button is released.
NOTE
While adding spectra DO NOT release the “Shift-key”. To remove spectra from the selection: Draw a box around the spectra you wish to deselect with the left
mouse button pressed while holding the “Ctrl-key”. Release the left mouse button still holding the “Ctrl­key”. All remaining selected spectra will stay highlighted in red.
NIRCal 5.5 Manual, Version A 77
NIRCal 5.5 Software Manual
When the selection is created, the spectra can be copied to the calibration spectra, the
validation spectra or the user spectra.
1. press the right mouse button in the graphic and Copy Selection to – Row - Calibration Spectra;
2. for setting the Validation Spectra select in the Popup-Menus the command Invert Selection. The rest of the spectra will be selected;
3. copy this selection in the Popup-Menus with the command Copy Selection to – Row - Validation Spectra.
NOTE
The function “Copy Selection to” overwrites the existing selection in the data sets.
Add and Remove Selection from Dataset
Mark the spectra, which should be removed from a selection. Press the right mouse button and use "Add / Remove Selection to". This will enlarge or reduce the
previously selection with the new spectra.
NOTE
By copying, adding and removing spectra to (from) the sets, a homogeneous distribution between calibration and validation spectra over the whole concentration range can be achieved.
3.3.6 Wavelength Data Set
The exact wavelength / wavenumber range depends on the instrument type used for the spectrum measurement and their settings (e.g. resolution). The selected wavelength / wavenumber range depends on the application and the measuring option used. The calibration wavelengths define the spectrum range used by the mathematical algorithms, PCA, PCR or PLS.
Calibration Wavelengths are the wavelengths used to create the calibration model. Removing certain wavelengths from a calibration can lead to some improvement.
78 NIRCal 5.5 Manual, Version A
Chemometrics
Steps sequence:
1. Select Custom;
2. Type in the selection;
3. Click on OK.
NOTE
It is recommended to use only custom selection.
NOTE
In general, the calibration wavelength / wavenumber range should be as wide as possible. Suggested wavenumber range for NIRFlex N-500 for measuring options with
Solids and Liquids: 4'000-10'000 cm-1;  Fiber Optics: 4'500-10'000 cm-1.  Solids with Tablet Accessory: 6'000-11'520 cm-1.
Selecting the Calibration Wavelengths in the NIR-Explorer
To make the selection use the NIR-Explorer: double click "Calibration Wavelengths" and the Edit selection dialog appears
NIRCal 5.5 Manual, Version A 79
NIRCal 5.5 Software Manual
In the project, the selected wavenumber as data points and range are shown (here: 1376 data points in the range of 4'500-10'000 cm-1 ).
Selecting Calibration Wavelengths using Graphics
To review and select the suitable calibration wavelength range, several graphics can be used. An example is shown with the pretreated spectra, but it can also be done in the original spectra, loading or property wavelength regression graphic in the same way.
Open the pretreated spectra in Menubar: Graphics / Spectra / Pretreated. When the graphic is opened, press the right mouse button and choose:Options / Mouse X-Axis
Select.
80 NIRCal 5.5 Manual, Version A
Chemometrics
Now it is possible to select the wavelengths with the mouse. The cursor position as wavenumber can be read in the status bar. Keep the left button pressed for selection. The selected range is marked with red color. Copy the selected range into the calibration wavelengths with the right mouse button Popup-Menu:
Copy Selection to / Calibration Wavelengths.
NIRCal 5.5 Manual, Version A 81
NIRCal 5.5 Software Manual
Steps sequence:
1. Select Custom;
2. Type in the selection;
3. Click on OK.
3.3.7 Properties Data Set
In qualitative methods the calibration properties are the substances that are required to be identified in the application. In cluster calibration the mathematical algorithm of PCA will be applied for the selected properties. Normally all qualitative properties are used in the calibration.
In quantitative methods the calibration property (single property calibration) is the selected parameter which should be predicted.
Selecting the Calibration Properties in the NIR-Explorer
To make the selection use the NIR-Explorer: double click "Calibration Properties" and the Edit selection dialog appears.
82 NIRCal 5.5 Manual, Version A
Chemometrics
In the project the selection is shown
Selecting the Calibration Properties using the Property Table
NIRCal 5.5 Manual, Version A 83
NIRCal 5.5 Software Manual
Open the property table.
Select the desired column.
Copy the selection into calibration properties.
84 NIRCal 5.5 Manual, Version A
Chemometrics
3.3.8 PC Data Set
Principal Components or Loadings (old name Factors) are artificial difference spectra. All PCs: Number of selected primary PCs. User PCs: use this set as a selection buffer (e.g. for visibility). Calibration PCs: these secondary PCs are used to create the calibration model. Calibration PCs or
secondary PCs form a subset of the primary PCs of the calibration.
NIRCal 5.5 Manual, Version A 85
NIRCal 5.5 Software Manual
Menu:
Wizard / Calibration Wizard
Icon:
1. Select type of substance.
2. Select type of calibration. Important for PCR/PLS or CLU method selection!
3. Select behavior of calibration.
4. Select used measuring option.
5. Start the wizard with OK.
3.4 Calibration Wizard
3.4.1 Calibration Wizard
The quality (robustness, sensitivity, selectivity, portability...) of a calibration for quantification and/or identification mainly depends on the data selections: choice of calibration and validation spectra, wavelength range, data pretreatments and PCs used. The chemometric software NIRCal offers the calibration wizard for an easy and fast generation of calibrations without a profound understanding of chemometrics. The automatic Calibration Wizard guides the user. While the user only has to answer a few questions about the used samples, the installed sampling options or the expected quality of the calibration, the automatic calibration uses its knowledge base for selecting spectra, wavelength ranges or the number of PCs. The pretreatment selector for permutation and calculating several pretreatment combinations is an integral part of the calibration wizard. After all calibration is calculated the summary is documented and the five best calibrations are stored, all other deleted automatically. The calibration wizard can be started without any data selection before, but a precalculation can be useful: the wizard never selects properties! To do the data selection press the Data Sets button.
The Calibration Wizard with the Advanced option:
86 NIRCal 5.5 Manual, Version A
remove detected Outlier from C­Set and V-Set
Based upon outlier detection module with c=2.5, removed spectra are listed in the calibration protocol. Not in all cases it is advantageous to use this option.
Fast mode reduces data points up to 1/5
Every 5. wavelength value will be used to reduce the calculation time. Not suggested for the end calculation! Use "Turbo Mode" (M) in plots e.g. Loading plot.
show calibration rules after initialisation
The used pretreatment, wavelength and method selection is showed; "Use all rules" with up to 2000 combinations are available (the time of calculation will increase dramatically).
keep preselected Pretreatments for all calibrations
The pretreatment or pretreatment sequence, used in the active calibration, is kept and the pretreatments from the wizard are added afterwards.
exclude this method
For quantitative calibration the PLS or PCR method can be eliminated to reduce the calculation time.
exclude Kubelka Munk
Kubelka Munk will not be used if this option is checked.
stop after 10 calibrations
Calculating 10 calibrations reduce the time for the calculation (mainly for demo purposes).
built calibration name with property names
Adequate for single property calibrations; to be switched off for a larger number of properties.
additional setting
The wavelength selection used in the active calibration, will be used additional to the predefined selection.
boundary for internal settings
The wavelength selection is never bigger, as the selection in the active calibration. The boundary selection can also include gaps.
Advanced Settings:
Chemometrics
The wavelength selection can be influenced in one of these ways:
or
For the pretreatments of derivatives and smoothings there are 3 possibilities:
- Gap0: all data points are in the calculation;
- Gap2: each 3rd data point is in the calculations;
- Segment3: there is an additional 3 point smoothing (as3) for the Gap2 pretreatment. With these selections predefined wavelength and pretreatment combinations and the calculation
algorithm will be activated. The wizard can be stopped with the keeping the Esc button pressed.
NIRCal 5.5 Manual, Version A 87
NIRCal 5.5 Software Manual
The Q-Value is a measure of the quality of the calibrations. It ranges from 0 to 1. The higher the Q­Value, the better the calibration. Q-Value 1 is theoretical, in practice not achievable, if the value is higher as 0.75 (green), the calibration is acceptable, between 0.5-0.75 (blue) the calibration is useable, but not very accurate. Calibrations with Q-Value below 0.5 (red) should be inspected very carefully before routine use. The general observation is that qualitative calibrations yield considerably better Q-Value than quantitative calibrations. The Q-Value is a very good tool for the judgement of calibrations especially when comparing different calibrations. After all calibrations have been calculated the results are summarized, the calibrations are sorted by the Q-Value. In addition to the calculated property the used secondary (calibration) / primary factors = PCs, the selected wavelengths regions, the used algorithm and pretreatments are listed as well.
88 NIRCal 5.5 Manual, Version A
Chemometrics
It is possible to save or print this list by clicking on the appropriate buttons. The ten best calibrations are kept in the project. The calibration with the highest Q-Value is set active.
For this calibration the results are summarized using the Overview. It is possible to inspect the active calibration manually.
NOTE
It is the responsibility of the user to judge the calibrations and to release them for routine use. The Q­Value is one among other helpful tools for the judgement of the quality of the calibrations. It is important to test the calibration with independent well characterized samples
3.5 Pretreatments
3.5.1 Pretreatments
NIR spectra are influenced by various parameters. Variations of chemical and physical properties of samples as well as the measurement process and changes at the spectrometer will have an influence on the spectrum. These effects will mainly appear as problems with:
overlapping absorption bands;  non-linearity;  light scattering;  random noise;
One possibility to overcome these problems is to improve the signal by mathematical transformations of the spectra using pretreatments. They are used to improve the quality of the spectra and to minimize unwanted effects.
NOTE
Pretreatments do not change or affect the original spectra. NIRCal provides a variety of data pretreatments, there are 34 pretreatment possibilities available in 6
groups. Each pretreatment can be combined with another and the order of combination is also important. Some pretreatments, which have a star "*" behind the name, are wavelength dependent. In this way there are a hugh number of combinations available. The size of a pretreatment sequence is only limited by the memory.
NOTE
According to our experience it is not suggested to use more than 3 pretreatments. Be aware of trashing your data to nonsense by misusing pretreatments.
Applying Pretreatments
The pretreatment selection is available in the Menubar: Calibration / Pretreatments.
NIRCal 5.5 Manual, Version A 89
NIRCal 5.5 Software Manual
The pretreatments can be selected in the toolbar via Icons as well; or select Pretreatments in the NIR Explorer and use the right mouse button; or select Pretreatments in the NIR Explorer, or in the Pretreated Spectra Plot and use the right mouse button.
Removing Pretreatment
with Undo Last it is possible to cancel the last pretreatment; with Undo Sequence the whole sequence of pretreatments will be canceled.
90 NIRCal 5.5 Manual, Version A
3.5.2 Available Pretreatments
Pretreatment
Type
Short
Normalization
by Closure*
ncl
by Maxima*
nma
by Sdev*
nsd
to Unit Length*
nle
between 0 and 1*
n01
MSC Amplification * **
ma
MSC Full* **
mf
Divide by Spectrum
div
Standard Normal Variate*
SNV
Variance Scaling**
vs
Offset
Subtract DC*
dc
Shift Negative to Zero*
n2z
MSC Offset* **
mo
Add constant
+c
Mean Centering **
mc
Subtract Spectrum
sub
Smoothing
Average 3 points
sa3, sa3g2 ***
Average 9 points
sa9, sa9g2 ***
Savitzky-Golay 9 points
sg9, sg9g2 ***
Derivatives
1st BCAP 5 points
db1, db1g2 ***
1st Taylor 3 points
dt1, dt1g2 ***
1st Savitzky-Golay 9 points
dg1, dg1g2 ***
2nd BCAP 3 points
db2, db2g2 ***
2nd Taylor 3 points
dt2, dt2g2 ***
2nd Savitzky-Golay 9 points
dg2, dg2g2 ***
2nd Taylor 3 points, Segment5, Gap5 (Linear Filter - with fixed coefficients)
ds2, ds2g2 ***
3rd Taylor 5 points
dt3, dt3g2 ***
Transformation
Absorbance Log10(1/x)
log
Absorbance inverse 1/(10-x)
ilg
2nd Derivative/Logarithm
SDL
Kubelka Munk
kmu
Square x2
sqr
Reciprocal 1/x
1/s
Filter
Linear Filter - with editable coefficients
flt
Chemometrics
* These pretreatments are wavelength dependent. The used wavelength is the selected calibration wavelength, or can be edited in the NIR-Explorer under Pretreatments. *** g2 stands for Gap2-filtering. These pretreatments have been adapted for improved performance for N-500 spectra.
NIRCal 5.5 Manual, Version A 91
NIRCal 5.5 Software Manual
Original Spectra
Normalization by Closure*
Normalization by Maxima*
** MSC and Mean Centering are also depending on the C-Set spectra selection. NIRCal handles this dependency automatically. The necessary data is stored in the pretreatments itself so they can also operate automatically in the predictor of the application.
Legend of formula on the following pages: Capital Letters: Vectors Small Letters: Scalars T: Transmittance or Reflectance A: Absorbance S: Spectrum h: delta X, distance of base point on the x-axis
3.5.3 Normalization
Normalization
The aim of normalization is to reduce baseline variations. MSC is used to reduce or to increase baseline effects caused by scattering.
Tip
for solids  reduce particle size effects  reduce pressure difference
Overview of all Normalization pretreatments:
92 NIRCal 5.5 Manual, Version A
Chemometrics
Normalization by Sdev*
Normalization to Unit Length*
Normalization between 0 to 1*
MSC Amplification*
MSC Full*
Divide by spectrum
Standard Normal Variate*
Variance Scaling
NIRCal 5.5 Manual, Version A 93
NIRCal 5.5 Software Manual
* These pretreatments are wavelength dependent.
Normalization by Closure
Formula:
Martens, Naes 1989, p. 337 Use
Reduction of baseline variations. Type
The Calibration wavelengths are used when the pretreatment is added; the wavelength range can be changed afterwards in the pretreatment that can be different from the calibration wavelength.
Normalization by Maxima
Use Reduction of baseline variations.
94 NIRCal 5.5 Manual, Version A
Chemometrics
Type: The Calibration wavelengths are used when the pretreatment is added; the wavelength range can be changed afterwards in the pretreatment that can be different from the calibration wavelength.
Normalization by Sdev
Division of each spectrum through the Standard Deviation of its Intensity Value within the Wavelength selection.
Use Reduction of baseline variations.
Type: The Calibration wavelengths are used when the pretreatment is added; the wavelength range can be changed afterwards in the pretreatment that can be different from the calibration wavelength.
Normalization to Unit Length
Vector Normalization to Unit Length.
Use Reduction of baseline variations.
NIRCal 5.5 Manual, Version A 95
NIRCal 5.5 Software Manual
Type The Calibration wavelengths are used when the pretreatment is added; the wavelength range can be changed afterwards in the pretreatment that can be different from the calibration wavelength.
Normalization between 0 to 1
Use Reduction of baseline variations.
Type The Calibration wavelengths are used when the pretreatment is added; the wavelength range can be changed afterwards in the pretreatment that can be different from the calibration wavelength.
MSC Amplification
96 NIRCal 5.5 Manual, Version A
Chemometrics
Use Can increase baseline effect, can be good for particle size separation.
Type This pretreatment is not depending on the calibration wavelengths.
MSC Full
MSC Multiplicative Scatter Correction (full)
Use Eliminates scattering effects. Reduction of baseline variations.
Type The Calibration wavelengths are used when the pretreatment is added; the wavelength range can be changed afterwards in the pretreatment that can be different from the calibration wavelength.
Divide by Spectrum
When selecting this pretreatment the number of the spectrum, which should be used for the division, has to be entered.
NIRCal 5.5 Manual, Version A 97
NIRCal 5.5 Software Manual
In this way it is possible to enhance the differences in the data set.
NOTE
Having applied Divide by Spectrum the selected spectrum (spectrum No. 10 in this example) must not be put into the C-Set; its ordinate values will contain 1.0 only. The selected spectrum is copied into the pretreatment once. The selected spectra contains all previous pretreatments.
Type This pretreatment is not depending on the calibration wavelengths.
Standard Normal Variate
The SNV transformation centers each spectrum and then scales it by its own standard deviation (mean zero and variance equal to one). It corrects shifts on the ordinate.
98 NIRCal 5.5 Manual, Version A
Chemometrics
Y_SNV = (y – mean(y)) / Sdev (Y) Use
Reduction of baseline variations. Type
The Calibration wavelengths are used when the pretreatment is added; the wavelength range can be changed afterwards in the pretreatment that can be different from the calibration wavelength.
Variance Scaling
The spectra are divided by the standard deviation vector of the C-Set spectra. It is dependent on the C-Set selection.
NOTE
NIRCal handles the changing of the C-Set spectra automatically (by a refresh F5 or recalculation) and changes the the standard deviation and the result of the pretreatment. The standard deviation vector is stored in the pretreatment, so it can be used in the predictor and application.
Use Can increase baseline variations, can be good for particle size separation.
Type This pretreatment is not depending on the calibration wavelengths.
NIRCal 5.5 Manual, Version A 99
NIRCal 5.5 Software Manual
Original Spectra
Subtract DC*
Shift negative to zero*
MSC Offset*
Add Constant
3.5.4 Offset
Offset
The aim of Offset is to make baseline correction which caused by scattering. Overview of all Offset pretreatments:
100 NIRCal 5.5 Manual, Version A
Loading...
Buy Points How it Works FAQ Contact Us Questions and Suggestions Privacy Policy Cookie Policy Terms of Use