HP Smart Update Manager
3 and 2008, Linux,
of the risk of data loss, only individuals experienced with using firmware and
User Guide
Abstract
This guide is intended for individuals familiar with the configuration and operation of Microsoft Windows, Windows Server 200
smart components, and deployment of firmware and software. Because
software should implement the procedures.
Part Number: 613175-005
September 2011
Edition: 5
© Copyright 2011 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall
not be liable for technical or editorial errors or omissions contained herein.
Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government
under vendor’s standard commercial license.
Microsoft®, Windows®, Windows Vista®, and Windows Server® are U.S. registered trademarks of Microsoft Corporation.
Contents
Introduction .................................................................................................................................. 6
HP SUM overview ........................................................................................................................................ 6
Using this guide ........................................................................................................................................... 7
New features of HP SUM .............................................................................................................................. 7
Minimum requirements .................................................................................................................................. 7
Minimum requirements for Windows servers ......................................................................................... 7
Minimum requirements for Linux x86 servers ......................................................................................... 8
Supported deliverables overview ................................................................................................................. 10
Support and limitations of HP SUM .................................................................................................... 10
Deployment from HP SUM ........................................................................................................................... 11
Executing HP SUM ........................................................................................................................... 11
Deploying firmware for ProLiant servers using the HP Smart Update Firmware DVD ................................ 13
Deploying Integrity firmware bundles with HP SUM ............................................................................. 15
HP SUM User Interface ................................................................................................................ 16
Deployment scenarios ................................................................................................................................. 16
Keyboard support ...................................................................................................................................... 16
Source selections ........................................................................................................................................ 17
Set Proxy Server options ................................................................................................................... 18
Using multiple repositories ................................................................................................................ 19
Add Repository ................................................................................................................................ 19
Edit Repository ................................................................................................................................. 20
Remove Repository ........................................................................................................................... 20
Configure Components ..................................................................................................................... 20
Download components for target discovery ......................................................................................... 21
Select targets ............................................................................................................................................. 22
Inter-target dependency .................................................................................................................... 23
Dependency checking ...................................................................................................................... 23
Target types ..................................................................................................................................... 24
Find and discover targets .................................................................................................................. 24
Add Single Target ............................................................................................................................ 24
Enter Target Credentials .................................................................................................................... 24
Edit Target ....................................................................................................................................... 25
Remove Target ................................................................................................................................. 25
Manage Groups .............................................................................................................................. 25
Install updates ............................................................................................................................................ 25
Installing updates ............................................................................................................................. 26
Abort installation .............................................................................................................................. 27
View failed dependencies ................................................................................................................. 27
Select bundles .................................................................................................................................. 27
Select components ............................................................................................................................ 28
View firmware mismatch details ......................................................................................................... 28
Select devices .................................................................................................................................. 28
Installation options ........................................................................................................................... 29
Reboot options ................................................................................................................................. 29
Schedule Updates ...................................................................................................................................... 29
Generate and View Reports ........................................................................................................................ 30
Contents 3
Generate a Report ............................................................................................................................ 30
View a Report .................................................................................................................................. 30
Scripted deployment ................................................................................................................... 31
Command-line interface .............................................................................................................................. 31
Command-line syntax ....................................................................................................................... 31
Command-line arguments .................................................................................................................. 32
Component configuration for Windows components only ..................................................................... 38
Command-line examples ................................................................................................................... 38
Return codes .................................................................................................................................... 40
Windows smart component return codes ............................................................................................ 40
Linux smart component return codes ................................................................................................... 40
Linux RPM return codes ..................................................................................................................... 41
Input files ................................................................................................................................................... 41
Command-line usage of input file ....................................................................................................... 42
Input file format and rules .................................................................................................................. 42
File encoding ................................................................................................................................... 42
Error reporting ................................................................................................................................. 43
Input file parameters ......................................................................................................................... 43
Reports ...................................................................................................................................................... 51
Advanced topics ......................................................................................................................... 56
Configuring IPv6 networks with HP Smart Update Manager ............................................................................ 56
Configuring IPv6 for Windows Server 2003 ....................................................................................... 56
Configuring IPv6 for Windows Server 2008 ....................................................................................... 57
Limitations of IPv6 for Windows Server 2003 and Windows Server 2008 ............................................. 58
Configuring IPv6 for Linux ................................................................................................................. 58
Limitations of IPv6 for Linux ............................................................................................................... 60
Server virtualization detection and support.................................................................................................... 60
Troubleshooting .......................................................................................................................... 61
Recovering from a failed ROM upgrade ....................................................................................................... 61
Recovering from a failed system ROM upgrade ................................................................................... 61
Recovering from a failed option ROM upgrade ................................................................................... 62
Recovering from an installation failure .......................................................................................................... 63
Collecting trace directories ................................................................................................................ 63
Recovering from a discovery failure ............................................................................................................. 64
Troubleshooting connection errors ...................................................................................................... 64
HP SUM hangs during discovery ........................................................................................................ 65
HP SUM hangs during boot ......................................................................................................................... 65
Recovering from a loss of Linux remote functionality ....................................................................................... 66
Configuring firewall settings .............................................................................................................. 66
Recovering from a blocked program on Microsoft Windows ........................................................................... 66
Configuring Windows firewall settings ............................................................................................... 66
Enabling ports in HP Smart Update Manager ...................................................................................... 67
Recovering from operating system limitations when using a Japanese character set ........................................... 69
Displaying the user-specified reboot message using a Japanese character set when running on a Linux
operating system .............................................................................................................................. 69
Rebooting with the user-specified reboot message using a Japanese character set when running on a Windows
operating system .............................................................................................................................. 69
Running in a directory path containing multi-byte characters ........................................................................... 69
Recovering a lost HP Smart Update Manager connection ............................................................................... 69
HP Smart Update Firmware DVD mounted using iLO virtual media ........................................................ 69
Troubleshooting HP Smart Update Manager in IPv6 networks ......................................................................... 70
Contents 4
Troubleshooting HP Smart Update Manager in IPv6 Windows Server 2003 environment ........................ 70
Troubleshooting HP Smart Update Manager in IPv6 Windows Server 2008 environment ........................ 71
Troubleshooting HP Smart Update Manager in IPv6 Red Hat and Novell SUSE-based Linux environments . 71
HP SUM found new hardware message........................................................................................................ 72
Non-matching systems error reported when building source Linux RPMs or installing Linux RPMs built from source 72
Linux component version discrepancy for source RPMs ................................................................................... 72
HP SUM displays No components found in the selected repository(ies) message ............................................... 73
Additional/Optional Actions columns are grayed when HP SUM is maximized ................................................ 73
Installation of components failed with 'Update returned an error' when installing Linux RPMs ............................. 73
HP SUM fails on Windows Vista® due to McAfee firewalls ............................................................................ 73
Downgrade of ProLiant G6 Server ROM from 2010.03.30 or later to 2010.03.01 or earlier using HP SUM is not
supported .................................................................................................................................................. 76
Technical support ........................................................................................................................ 77
Reference documentation ............................................................................................................................ 77
Operating system information ...................................................................................................................... 77
HP contact information ................................................................................................................................ 78
Acronyms and abbreviations ........................................................................................................ 79
Index ......................................................................................................................................... 81
Contents 5
Introduction
Throughout this document, references to the deployment of firmware apply only to
HP SUM overview
HP SUM is a technology included in many HP products for installing and updating firmware and software
components on HP ProLiant and HP Integrity servers, enclosures, and options.
HP SUM provides a GUI and a command-line scriptable interface for:
• Deployment of firmware for single or one-to-many HP servers and network-based targets such as iLOs,
OAs, and VC Ethernet and Fibre Channel modules.
• Deployment of software for single or one-to-many HP ProLiant servers (supported in Windows and Linux
environments) and HP Integrity servers.
IMPORTANT:
HP SUM when used with HP ProLiant servers. For HP Integrity servers, deploying firmware using
HP SUM has an integrated hardware and software discovery engine that finds the installed hardware and
current versions of firmware and software in use on target servers. This capability prevents extraneous
network traffic by sending only the required components to a target host. HP SUM installs updates in the
correct order and ensures that all dependencies are met before deploying an update. HP SUM prevents
version-based dependencies from destroying an installation, and ensures firmware updates are handled in a
manner that reduces any downtime required for the firmware update process.
HP SUM only applies to BL8x0c i2, rx2800 i2 and Superdome 2 Servers.
HP SUM does not require an agent for remote installations, because it copies a small, secure SOAP server to
the target server for the duration of the installation. After the installation is complete, the SOAP server and all
remote files associated with the installation, except installation log files, are removed. HP SUM copies the log
files from the remote targets back to the system where HP SUM is executed.
Key features of HP SUM include:
• Dependency checking, which ensures appropriate installation order and dependency checking
between components
• Intelligent deployment deploys only required updates
• Simultaneous firmware and software deployment for multiple remote targets in both GUI and CLI modes
(ProLiant servers and options only)
• Improved deployment performance
• Local or remote (one-to-many) online deployment
• Local offline firmware deployments with the earlier HP Firmware Maintenance CD or the HP Smart
Firmware Update DVD for ProLiant servers and options
• Remote offline deployment when used with the SmartStart Scripting Toolkit or iLO Virtual Media
• GUI- or CLI-scriptable with extensive logging
• Remote command-line deployment
Introduction 6
• Support for updating firmware on network-based targets such as the OA, iLO 2/iLO 3 through Network
Management Port, and VC Ethernet and Fibre Channel modules on HP ProLiant servers
NOTE: HP SUM does not support third-party controllers. This includes flashing hard drives
behind these controllers.
Using this guide
HP SUM is included with many different deliverables. A deliverable is an HP product which contains HP SUM
and a set of firmware or software components, for example, the HP Smart Update Firmware DVD. Read your
deliverable’s overview section to verify specific HP SUM limitations, and then, review the SUM features
sections. For more information, see "Overview of supported deliverables ("Supported deliverables overview"
on page 10)."
New features of HP SUM
HP SUM 5.0.0 includes support for:
• New Graphical User Interface
• Ability to deploy firmware from Windows servers to Linux targets
• Support for Integrity BL8x0c i2, rx2800 i2, and Superdome 2 servers
• Automatic discovery of blades and interconnects when given the Onboard Administrator as the target
Minimum requirements
• Minimum requirements for Windows operating systems ("Minimum requirements for Windows servers"
on page 7)
• Minimum Requirements for Linux Servers ("Minimum requirements for Linux x86 servers" on page 8)
The minimum display settings for optimum performance are 1024 x 768 pixels resolution.
NOTE: For Windows operating systems, the respective device drivers for the embedded/add-on
Minimum requirements for Windows servers
To successfully use HP SUM to deploy firmware and software to targets running a Windows® operating
system, the following must be available:
• A local administrative system with 1 GB of memory
devices in the target servers must be installed for HP SUM to discover the devices.
• Sufficient hard-drive space of at least twice the file size of the components to be deployed
• WMI enabled
• All remote targets must be visible by the administrative system running HP SUM.
• An account with administrator privileges on each target server
HP recommends that the user name and password for the administrator account on each target server
be the same as those on the local administrative system. If administrator privileges are not set up in this
Introduction 7
manner, you must have the user name and password available for each remote server available.
Alternatively, you can use a domain account on the local administrative system that has administrator
privileges on the target servers.
• The beginning and ending IP addresses entered for the range of targets must both be on the same
subnet.
NOTE: HP Smart Update Manager requires the user to be logged in using Administrator
credentials. A user account elevated to “Run as administrator” in most cases does not have
sufficient rights because access to protected system files and the Windows registry, which is
needed to deploy most firmware and driver updates, is restricted. If the logged in account is
unable to perform the net use *\\server\ADMIN$ for Microsoft Windows targets, you do not
have sufficient privileges to run HP SUM to remote targets. Deployment of some software updates
on a local server may still be possible in some instances, but is not recommended.
NOTE: When attempting to use the remote deployment functionality of HP SUM on any edition of
Windows Server 2008 or Windows Vista, you must ensure that the File and Print Services feature
is enabled and that the File and Print Services exception has been enabled in the Windows
firewall. Failure to do so prevents HP SUM from deploying remote Windows targets.
• Enable the network ports that HP SUM uses. For more information, see "Enabling ports in HP Smart
Update Manager (on page 67)."
Minimum requirements for Linux x86 servers
To successfully use HP SUM to discover perform dependency checks on targets running a Linux operating
system, the following must be available:
• A local administrative system with 1 GB of memory
• glibc 2.2.4-26 or later
• gawk 3.1.0-3 or later
• sed 3.02-10 or later
• pciutils-2.1.8-25.i386.rpm or later
Be sure that the following platform-specific compatibility libraries are also installed.
• For Red Hat Enterprise Linux 4x86 servers:
o libnl (required for QLogic and Emulex drivers)
• For Red Hat Enterprise Linux 4 AMD64/EM64T servers:
o libnl (required for QLogic and Emulex drivers)
• For Red Hat Enterprise Linux 5 x86 servers:
o libnl (required for QLogic and Emulex drivers)
• For Red Hat Enterprise Linux 5 AMD64/EM64T servers:
o libnl (required for QLogic and Emulex drivers)
• For SLES 10 x86 servers:
o libnl (required for QLogic and Emulex drivers)
• For SLES 10 AMD64/EM64T servers:
o libnl (required for QLogic and Emulex drivers)
Introduction 8
o
e2fsprogs-32bit
• For SLES 11 x86 servers:
o libstdc++33-3.3.3-11.9.i586 or later
o libnl (required for QLogic and Emulex drivers)
• For SLES 11 AMD64/EM64T servers:
o libstdc++33-32bit-3.3.3-11.9 or later
Minimum requirements for Red Hat Enterprise Linux 6 servers
o libnl (required for QLogic and Emulex drivers)
NOTE: The following versions listed are needed as a minimum. Later versions of these might be
used as well.
NOTE: Ensure that each NIC has an IP address assigned. To get the IP address assigned to a
NIC, enter the dhclient command in the console, which enables the NIC interfaces and
retrieves an IP address. If a NIC does not have an IP address, HP SUM appears to hang on start
• RHEL6 console mode
up.
NOTE: No X console in either x86 or x86_64 servers. To run HP SUM in silent mode, you must
install the base server with defaults and the following RPMs.
o lm_sensors-libs-3.1.1-10.el6.<arch>.rpm
o net-snmp-libs-5.5-27.el6.<arch>.rpm
o net-snmp-5.5.27.el6.<arch>.rpm
o kernel-headers-2.6.32-71.el6.<arch>.rpm
o redhat-rpm-config-9.0.3-25.el6.noarch.rpm
o kernel-devel-2.6.32-71.el6.<arch>.rpm
o rpm-build-4.8.0-12.el6.<arch>.rpm
o gcc-4.4.4-13.el6.<arch>.rpm
• RHEL6 graphical mode
NOTE: If the user elects to install the XWindows support, then the following steps apply to both
x86 and x86_64 servers.
NOTE: The following items must be the 32-bit version even under x86_64 architecture as HP
SUM and several of the RPMs require 32-bit libraries installed.
o libuuid-2.17.2-6.el6.i686.rpm
o freetype-2.3.11-5.el6.i686.rpm
o libSM-1.1.0-7.1.el6.i686.rpm
o libICE-1.0.6-1.el6.i686.rpm
o libXi-1.3-3.el6.i686.rpm
o libX11-1.3-2.el6.i686.rpm
o libXext-1.1-3.el6.i686.rpm
Introduction 9
o
•
•
•
•
•
•
•
•
•
available from ftp.hp.com through the HP SUM
•
libxcb-1.5-1.el6.i686.rpm
o libXau-1.0.5-1.el6.i686.rpm
o libXrender-0.9.5-1.el6.i686.rpm
o libXrandr-1.3.0-4.el6.i686.rpm
o libXfixes-4.0.4-1.el6.i686.rpm
o libXcursor-1.1.10-2.el6.i686.rpm
o fontconfig-2.8.0-3.el6.i686.rpm
o expat-2.0.1-9.1.el6.i686.rpm
o expect-5.44.1.15-2.el6.<arch>.rpm
o zlib-1.2.3-25.el6.i686.rpm
o libstdc++-4.4.4-13.el6.i686.rpm
o net-snmp-5.5-27.el6.<arch>.rpm
Supported deliverables overview
HP SUM support is specific to the deliverable in which it is delivered. Make sure you use the correct version
of HP SUM that is released along with the deliverable supporting the environment.
NOTE: Not all functionality is available in each version of HP SUM. Before using HP SUM that
comes with a deliverable, review the Support and limitations ("Support and limitations of HP
Support and limitations of HP SUM
Before installing HP SUM, use the following table to verify that you are using the appropriate software
version.
HP SUM
version
4.0.0
4.0.0.64
4.0.1
SUM" on page 10) of that version of HP SUM to determine if it has the features that you need.
Delivered with Support and limitations
For ProLiant servers:
HP ProLiant Support Pack for Windows
HP ProLiant Support Pack for Linux 8.40
HP Smart Update Firmware DVD 9.0
For ProLiant Workstation Blades:
HP Workstation Support Pack for
HP Smart Update Firmware DVD 9.0
HP Integrity Support Pack for Windows
HP ProLiant Support Pack for Windows
8.40
Windows for HP WS460c G6 Blade
(offline only)
on Itanium-based systems
No support for Integrity server firmware or support
packs
Support for Integrity Support Pack for Windows
7.00
No support for Integrity server firmware
bundles
No support to download the latest components
GUI and command line options
No support for Integrity server firmware or support
Introduction 10
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
HP SUM
Delivered with Support and limitations
version
8.50
packs
HP ProLiant Support Pack for Linux 8.50
4.0.2
4.1.0
HP Smart Update Firmware DVD 9.10
HP ProLiant Support Pack for Windows
8.60
No support for Integrity server firmware or support
packs
No support for Integrity server firmware or support
packs
HP ProLiant Support Pack for Linux 8.60
4.2.0
4.2.1
4.9.5
HP ProLiant Support Pack for Linux 8.70
HP ProLiant Support Pack for Windows®
8.70
HP Smart Update Firmware DVD 9.30
HP Integrity Server Bundles
No support for Integrity server firmware or support
packs
No support for Integrity server firmware or support
packs
HP BL8x0c i2 server
HP rx2800 i2 server
HP c-Class Onboard Administrators
No VC support
5.0.0
HP Smart Update Firmware DVD
HP Integrity Server Bundles
HP Superdome 2 server
VC support
Deployment from HP SUM
Executing HP SUM
The following sections discuss deployment from HP SUM.
HP SUM provides three modes of execution for users:
• GUI—For firmware and software deployments
• Express—For software deployments only (local host only)
• Scripted—For silent firmware and software deployments
HP SUM supports local and remote deployments. HP SUM runs in standard Windows and Linux operating
systems in online and offline mode (available on HP Smart Update Firmware DVD) through special boot
environments based on Linux boot kernels when the prerequisites are met.
Use the following table to determine the correct execution mode for the server use case.
IMPORTANT: Before deploying software updates to a target server, be sure that a recent backup
Scenario Description HP SUM command HP deliverable
Interactive, graphical
deployment on a local
host (Run HP SUM on the
server being updated.)
of the target server is available in the event the deployment procedure fails.
Use this scenario when you:
Are not familiar with
command line tools
Are deploying
components on a local,
single host
Windows:
hpsum
Linux:
./hpsum
Online:
HP ProLiant Support Pack for
Windows and Linux
or
Smart Update Firmware DVD
Offline:
Introduction 11
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
target
•
•
•
•
•
•
Scenario Description HP SUM command HP deliverable
Do not require scripting
Smart Update Firmware DVD
Want to select the
components to be
installed
Scripted deployment on a
local host (Run HP SUM
from a network share on a
local server.)
Use this scenario when you:
Are familiar with
command line tools
Are deploying
components on a local,
single host
Must perform a
customized, scripted
deployment
Are unable to unlock
the required HP SUM
network ports for
remote deployment
Interactive, graphical
deployment to a remote
host (Run HP SUM from a
workstation and update
remote hosts over the
network using the GUI)
Use this scenario when you:
Are not familiar with
command line tools
Do not require scripting
Want to select the
components to be
installed
Need to update
multiple targets
including OA, VC, and
server-based firmware
Need to select
individual firmware to
update
Need to select
individual hardware
devices to be updated
Scripted deployment to a
remote host
(Run HP SUM from a
workstation and update a
remote host over the
network using scripts.)
Use this scenario when you:
Are familiar with
command line tools
Must perform a
customized, scripted
deployment
Need to update
multiple targets
including OA, VC, and
server-based firmware
Need to update all
enclosure firmware
Automatic Deployment to
one or more servers in a
HP BladeSystem enclosure
Use this scenario when you:
Do not have an
operating system
Windows:
hpsum -s
Linux:
./hpsum -s
Windows:
hpsum
Linux:
./hpsum
Targets can be a
server, OA, iLO, or VC
IP address or DNS
name.
Windows:
hpsum -s Target1 -target
Target2
-username User1
-password PW1
Linux:
./hpsum -s
-target Target1
-target Target2
-username User1
-password PW1
The HP Smart Update
Firmware DVD takes
care of this
Online:
HP ProLiant Support Pack for
Windows and Linux
or
Smart Update Firmware DVD
or
HP Integrity Support Pack for
Windows
Offline:
None
Online:
HP ProLiant Support Pack for
Windows and Linux
or
Smart Update Firmware DVD
or
HP Integrity Support Pack for
Windows
or
HP Integrity BL8x0c i2,
rx2800 i2 and Superdome 2
Servers Bundles for Windows
and Linux
Offline:
None
Online:
HP ProLiant Support Pack for
Windows and Linux
or
Smart Update Firmware DVD
or
HP Integrity Support Pack for
Windows
or
HP Integrity BL8x0c i2,
rx2800 i2, and Superdome 2
Servers Bundles for Windows
and Linux
Offline:
None
Online:
None
Offline:
Introduction 12
•
are
•
•
•
•
•
•
Scenario Description HP SUM command HP deliverable
(use the HP Smart Update
Firmware DVD with
embedded HP SUM to
deploy firmware updates
to one or more blades)
installed on a server
blade
Want an automatic
way to deploy firmw
updates
automatically HP Smart Update Firmware
DVD
Do not require log files
or user interaction with
the installation process
Want to update
multiple blades in an
enclosure in a single
operation
Update an existing
enclosure manually
For more information, see
the HP website
(http://www.hp.com/go
/matrixcompatibility).
The same scenario as
shown in the “Interactive,
graphical deployment to a
remote host” or “Scripted
deployment to a remote
host” scenarios discussed
previously in this table
Windows:
hpsum
Linux:
./hpsum
Online:
HP BladeSystem Firmware
Update Bundles for Windows
and Linux
or
HP Smart Update Firmware
DVD
Offline:
HP Smart Update Firmware
DVD
Deploying firmware for ProLiant servers using the HP Smart Update Firmware DVD
The HP Smart Update Firmware DVD provides a collection of firmware bundles containing HP SUM and
firmware for ProLiant servers in an ISO image that can be used either in offline (an automatic or interactive
mode), or online (a scripted or interactive mode) modes. The HP Smart Update Firmware DVD provides
firmware for supported HP ProLiant servers and options. The Firmware DVD contains all the firmware for the
supported HP ProLiant servers including BladeSystem enclosures and Virtual Connect firmware.
• In offline mode, the HP Smart Update Firmware DVD boots a small Linux kernel and enables firmware
updates to occur on a single server using the HP SUM software.
• In online mode, users can leverage the autorun utility to launch HP SUM or browse the DVD to the
\hp\swpackages directory and then execute it directly.
All firmware smart components are placed in the \hp\swpackages directory for use by HP SUM. If
additional firmware smart components are required, then the HP Smart Update Firmware DVD can be copied
to a USB key, and these additional components added to the \hp\swpackages directory. If HP SUM supports
the type of firmware added, then it is automatically added the next time HP SUM is executed.
CAUTION: The HP Smart Update Firmware DVD and its contents must be used only by
individuals who are experienced and knowledgeable with HP SUM. Before using HP SUM to
update firmware, back up the target server, and take all other necessary precautions so that
mission-critical systems are not disrupted if a failure occurs.
NOTE: Because firmware might be able to update only in online or offline mode for some
components, you might need to execute the HP Smart Update Firmware DVD both online and
offline to update all the firmware in an enclosure.
Introduction 13
Loading...