HP JetAdvantage Security Manager Troubleshooting Manual

Technical white paper

HP JetAdvantage Security Manager

Troubleshooting Issues

Table of Contents

Overview ............................................................................................................................................. 1

Log Files and Enabling Debugging ..................................................................................................... 1

Security Manager Log Files Locations ......................................................................................................... 1

Enabling HP JetAdvantage Security Manager debugging (service int eractions) ....................................... 4

Enabling HP JetAdvantage Security Manager debugging (device communication issues) ....................... 5

Enabling HP JetAdvantage Security Manager debugging (Certificate Management) ............................... 7

Enabling HP JetAdvantage Security Manager debugging (SCEP issues) ................................................... 8

Enabling HP JetAdvantage Security Manager debugging (EST issues) ...................................................... 8

Enabling HP Print License Service debugging ............................................................................................. 8

Enabling logging for Qualys Policy Compliance .......................................................................................... 9

Audit log files ................................................................................................................................................ 9

Installation Issues ............................................................................................................................ 10

Common SQL issues ................................................................................................................................. 10

Database Upgrade Failure

Using SQL script to repair/upgrade/install the database ........................................................................ 14

Problems Launching Security Manager Web Interface .................................................................. 17

Issues when running HPSM Application Pool: web page not displayed ................................................. 17

Bindings/firewall problem .................................................................................................................................. 17

Browser settings causing issues........................................................................................................................ 19

Issues when running HPSM Application Pool as network service (sql exception) ................................. 19

32-Bit Applications not enabled for hpsm application pool ........................................................................... 20

........................................................................................................................ 13

Issues when running HPSM Application Pool as non-admin user or service account (error 503) ........ 21

Access Denied when trying to logon to HPSM ......................................................................................... 23

Licensing Issues ................................................................................................................................ 23

Licensing Issues in multi-homed environment: Failed to connect to licensing server .......................... 23

Other Licensing Issues .............................................................................................................................. 24

Required Network Ports .................................................................................................................. 26

Ports Diagram................................................................................................................................... 31

Changing Firewall Settings and Testing Open Ports ........................................................................ 32

Proxy for Network Service Blocking Communication ...................................................................... 34

Device Status Errors and Credential Failures .................................................................................. 35

Credential Management ............................................................................................................................ 33

SNMP ..................................................................................................................................................................... 34

Admin (EWS) Password ...................................................................................................................................... 35

Check Credentials Example ............................................................................................................................... 35

Device Communication Log Files .............................................................................................................. 37

Hanging tasks ................................................................................................................................... 38

Hung Tasks vs. Slow Tasks ....................................................................................................................... 38

Credential Failed or Network Connection Error Impacting Performance ..................................................... 38

Performance Impact of Instant On Remediations ........................................................................................... 39

Growing Database and Nightly Maintenance failing ....................................................................... 41

Certificates Installation Failures ...................................................................................................... 43

Using Network Traces for Troubleshooting .................................................................................... 43

Using Event Viewer for Troubleshooting ......................................................................................... 43

Appendix A Following a task in HPSM_service.log .......................................................................... 44

Overview

HP JetAdvantage Security Manager is a security compliance software used to assess and remediate a fleet of devices against a desired security policy. This document provides troubleshooting techniques to use if issues arise when either installing or running Security Manager. As with any software, challenges can be presented by servers, operating systems, networks, devices, etc. Security Manager relies upon Microsoft IIS and SQL Server, both of which can present challenges as well. While this document discusses known issues, new issues are always going to be encountered. Successful troubleshooting always involves removing variables to determine root cause. Google is also an excellent troubleshooting tool as most errors are generic Microsoft errors and are likely encountered by users of other software.

Log Files and Enabling Debugging

Log files are always an excellent place to start looking for clues as to why an issue occurred. There are three areas where logs reside: Security Manager, licensing, installation. Debug logging can also be enabled to produce more verbose logging if needed.

Security Manager Log Files Locations

Log files created by HP JetAdvantage Security Manager service:

The service HP JSM creates log files in two different directories:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log

C:\Program Files (x86)\HP JetAdvantage Security Manager\WebApp\log

Log file created by HP Print License Service:

C:\ProgramData\HP\HP Print License Service Files\HPPLS.log

Log file created when interacting with SCEP server:

C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

Log file created by FlexeraLicensingService:

C:\ProgramData\HP\HP Print License Service Files\Flexera.log

Log file created by the security manager installer:

C:\Users\”username”\AppData\Local\Temp

Note: This file will begin with “MSI” and end with “.log” but will contain random characters in the middle as this is produced by the Microsoft Windows installer.

Enabling debug logging is sometimes crucial for finding clues on issues such as hanging or slow tasks.

There are two places debug can be enabled. One is in the services area where the HPSM_service.log is filled with debug statements. The other one is in what is called the DAT layer responsible for

communications to and from devices. Two log files in particular named EapNetworkLIb.log and EapDevice.log capture all data to and from devices, and debug statements can be added to these files to

see if devices are responsible for hangs.

HPSM 3.4 and older is using the UTC timestamp for the EAP*.log files and localtime for all other log files. In order to get the same timestamps in all log files, do the following:

1. Open the file EapLogConfig.XML in the directory C:\Program Files (x86)\HP JetAdvantage Security

Manager

2. Replace %utcdate entries with %date

3. After making the changes, restart the HP Jetadvantage Service

In Security Manager 3.0 and later versions, both areas of debug can be enabled by configuring settings in two configuration files and restarting the Security Manager service.

For Security Manager 2.1.5 and earlier, only the service debug can be enabled and a special build of Security Manager has to be produced to turn on DAT debugging.

Enabling HP JetAdvantage Security Manager debugging (service interactions)

To enable debugging for non-device specific issues (like problems starting a task, problems starting the service, edit the following lines in the HPSM_service.exe.config file. Make changes highlighted below in yellow and restart the HP JetAdvantage Security Manager service:

<appender-ref ref="ServerAppender" /> </logger> <logger name="NHibernate.Loader.Loader">

<appender-ref ref="ServerAppender" /> </logger> <logger name="Assessment">

<appender-ref ref="ServerAppender" /> </logger> <logger name="Common">

<appender-ref ref="ServerAppender" /> </logger> <logger name="Data">

<appender-ref ref="ServerAppender" /> </logger> <logger name="Task">

<appender-ref ref="ServerAppender" /> </logger> <logger name="Service">

<appender-ref ref="ServerAppender" /> </logger> <logger name="WCF">

<appender-ref ref="ServerAppender" /> </logger>

The Data logger (highlighted in green) refers to the interactions between the HPSM service and the SQL database. Thus INFO level for the Data logger is fine for troubleshooting non-database related issues.

HPSM 3.4 introduced auto group policies. In order to see in the debug logs the order in which tasks are executed the enableTaskSequencingLogging must be set to true.

Note: These additional log settings will only be available if the logger value for the logger name

Service is set to DEBUG.

By default the logs will roll over when they reach 100 MB in size, and each log will only roll over once before they start to become deleted in each subsequent rollover. These values can also be configured in each of the configuration files mentioned above by changing the following entries:

Enabling HP JetAdvantage Security Manager debugging (device communication issues)

To enable debugging for issues related to device communication, edit the following lines in the EapLogConfig.xml file. Make changes highlighted below in yellow and restart the HP JetAdvantage Security Manager service:

<!--

**LOGGER DEFINITIONS FOLLOW**

Logging levels represented in order of increasing priority (e.g.

ALL includes everything below it, but excludes INFO):

To turn all logging off use:

-->

<logger name="EapDeviceLibrary">

</logger>

</logger>

</logger>

</logger>

<!-- ERRORs shown in this log file are expected, therefore it's off

by default -->

</logger>

</logger>

</logger>

</logger>

Brief explanation of the different log files:

• EapDeviceLibrary: This log shows general information like DAT version, the values returned from

device, which methods will be used, etc.

• Pipeline: This log shows the methods (pipelines) to perform different operations. This helps to

identify errors/exceptions in pipeline steps.

• Results: This gives the names of the config items which are processed.

• EapNetworkLib: This log shows the device communications. This helps to identify the retries,

timeouts, etc.

• McoTranslation: This log file shows methods (pipelines) for custom processes. This helps to

identify errors/exceptions in custom processes.

• Bindings: This log shows which interface of the device has been selected. Normally not needed to

enable debug mode for this, only in cases with errors like McoNotFound, McoNotSupported is listed in other log files.

Enabling HP JetAdvantage Security Manager debugging (Certificate Management)

When HPSM is interacting with a Microsoft (Standalone or Enterprise CA), OpenTrust CA or Symantec CA, it’s

using the Certificate Management log files (HPCM.log). These log files are located in three locations:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log C:\Program Files (x86)\HP JetAdvantage Security Manager\WebApp\log

Debug logging for ID Certificate requests can be enabled in two files:

C:\Program Files (x86)\HP JetAdvantage Security Manager\HPSM_Service.exe.config C:\Program Files (x86)\HP JetAdvantage Security Manager\WebApp\Web.config

Make the following changes highlighted:

<appender-ref ref="HPCMAppender"/>

</logger>

The changes in the HPSM_Service.config file require a restart of the service HP JetAdvantage Security Manager and will enable the debug logging for the HPCM.log file in the following directories:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

The last file will contain information about the interaction with the CA server. The changes in the Web.config file require a restart of IIS (or recycle the application pool hpsm) and will enable the debug logging for the HPCM.log file in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\WebApp\log

Enabling HP JetAdvantage Security Manager debugging (SCEP issues)

SCEP debugging can be enabled in the HPSM_Service.exe.config file. Make changes highlighted below in yellow.

</logger>

The changes in the HPSM_Service.config file require a restart of the service HP Jetadvantage Security Manager and will enable the debug logging for the SCEP.log file in the following directories:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

The last file will contain information about the interaction with the SCEP server.

Enabling HP JetAdvantage Security Manager debugging (EST issues)

EST debugging can be enabled in the HPSM_Service.exe.config file. Make changes highlighted below in yellow.

</logger>

The changes in the HPSM_Service.config file require a restart of the service HP Jetadvantage Security Manager and will enable the debug logging for the Est.log file in the following directories:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

Note: The EST plug-in is available from HPSM 3.5 onwards.

Enabling HP Print License Service debugging

To enable debugging for the HP Print License Service, edit the following lines in the HP.Print.License.Host.WindowsService.exe.config file in the following directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\HP Print

License Service

Make changes highlighted below in yellow, then stop the service HP JetAdvantage Security Manager and then stop the HP Print License Service. Restart first the HP Print License Service and after that restart the service HP JetAdvantage Security Manager.

<appender name="FileAppender"

type="log4net.Appender.RollingFileAppender">

<param name="File" value="C:\ProgramData\HP\HP Print License

Service\HPPLS.log" />

<param name="ConversionPattern" value="%d [%t] %-2p %c - %m%n"

</layout> </appender> <root>

<!--Possible value for level

1.ALL

2.DEBUG

3.INFO

4.WARN

5.ERROR

6.FATAL

7.OFF

-->

<appender-ref ref="FileAppender" />

</root>

</log4net>

Enabling logging for Qualys Policy Compliance

HPSM 3.5 offers service integration with Qualys. In Qualys debugging can be enabled in the HPSM_Service.exe.config file. When enabled log entries will be created when printer assessment results are sent to Qualys. Password are not transmitted to Qualys, only the password assessment result itself. To enable debugging for the Qualys policy integration, make changes highlighted below in yellow.

The changes in the HPSM_Service.config file require a restart of the service HP Jetadvantage Security Manager and will enable the debug logging for the Qualys.log file in the following directories:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log

Audit log files

Besides the debugging log files, Security Manager logs events for who did what and when in two different Audit logs:

• C:\Program Files (x86)\HP JetAdvantage Security Manager\

WebApp\log\HPSM_WebAudit.log

This file will have events that are triggered from the user.

• C:\Program Files (x86)\HP JetAdvantage Security Manager\log\

HPSM_ServiceApp.log

This file contains events about stopping/starting service and license limitations.

This file will have events that are triggered from HPSM service. An explanation of entries in those log files, can be found in the whitepaper Reporting, Email Alert Subscriptions & Remediation Summary, Auditing &

Syslog Functionality under the section HPSM User Activity Logging (Auditing)

Installation Issues

Security Manager uses the Microsoft installer for installing and upgrading the software. If anything goes wrong during installation, the logs file for the installation attempt may provide some clues.

For proper Security Manager installation and operation, specific Microsoft software must be present. The requirements are listed below:

•

Microsoft SQL Server Systems CLR Types [x86] - (part of installation script)

•

Microsoft SQL Server Systems CLR Types [x64] - (part of installation script)

•

Microsoft Primary Interop Assembly - (part of installation script)

•

Microsoft Report Viewer 2012 Runtime - (part of installation script)

•

Microsoft .NET Framework 4.6.1 or greater - (install prior to installation script)

•

Microsoft .NET Framework 3.5 or greater - (install prior to installation script)

•

Microsoft SQL Server Database - (see supported databases above)

•

Microsoft Internet Information Services (IIS) - (part of installation script)

If these are not present on the system, the installation process installs some of the required software. If errors occur when launching Security Manager related to the Microsoft Primary Interop Assembly or Report Viewer, there is a chance that the installer saw them installed thus did not attempt to install them. However, they may be bad versions that had been installed from some other application. It may be required to download and install newer versions from Microsoft.

The installer checks for the presence of IIS and attempts to enable it and necessary configuration elements if not present. The installer also provides an option to install SQL Express if desired or to use an existing SQL server location. It launches a series of SQL scripts to ensure the database is at the current schema version to match the software version. Proper permissions need to be present for the user running the upgrade on the SQL database in order for the installer to upgrade the database.

Recommended .NET versions are .NET 3.5 and 4.6.2. Earlier versions of .NET can be used such as 4.6.1, but some issues were seen in testing on Windows 10 and Windows Server 2016 operating systems using these older versions.

Common SQL issues

By far the most common reason an installation goes awry is related to SQL issues. Whether using local or remote SQL Server, Express or Full, the rules are essentially the same. In every case, Security Manager needs access to a SQL server instance. It can either create a new database, upgrade an existing database, or attach to an existing database, depending upon the situation and the user rights. If Security Manager is instructed to install SQL Express on the local machine, a SQL instance and database for Security Manager will be created by the Security Manager installer. If Security Manager is pointed to an existing local or remote SQL server and instance during installation, proper rights must be present for the user running the installation to be able to create or update a SQL database wherever SQL server may reside. Proper rights

must also exist on the database itself for the user which the Security Manager service runs under to be able to read from and write to the database.

KEY POINT: Just to reiterate, for installing and upgrading Security Manager, the user who is logged into

the machine and running the installer executable must have proper rights on the SQL server to either create a database or update an existing database. All the installer does is run SQL scripts to create or alter a database, and naturally any user running those commands needs to have proper SQL rights. In this case it is the Windows user who is running the installer.

Creating a database requires Create database rights on the SQL instance. Upgrading an existing database requires DBO rights on the database. For normal operation of Security Manager after installation, the user running the Security Manager service (default as Network Service) needs to have permissions to at least read and write to the database, DBO rights preferred.

Each Security Manager installation must point to its own unique database, multiple installations cannot share a database. The instance can be named or default, and the instance can have as many databases it desires including a Web Jetadmin database, but there will be only one HPSM database. Several techniques are available to allow Security Manager to install/use a SQL database in any location.

There are three scenarios where Security Manager will interact with Microsoft SQL:

• Creating a database during installation of Security Manager

• Upgrading a database during upgrading of Security Manager from one version to another

• Running Security Manager to manage security features on a fleet of devices

Each scenario requires a different set of SQL rights for potentially different users.

• Create Database – Windows user running the installer executable needs at minimum Create

Database rights on the SQL instance. SA rights would certainly work.

• Upgrade Database – Windows user running the installer executable to upgrade versions needs

DBO rights on the database to perform potential commands on the database such as insert, update, alter, create table.

• Run Security Manager – the Windows account that runs the Security Manager service (default of

Network Service) needs DBO rights on the database to perform operations such as reading and writing. NOTE: The “HP JetAdvantage Security - Using SQL Server” whitepaper explains how to run with less rights or a different account if desired.

The first sign of trouble if something goes wrong might be an error when launching Security Manager that indicates the SQL database cannot be opened.

Clues can be found in the following log file:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log\HPSM_Service.log

A successful login attempt will display the server\instance, database name, database version and Security manager version

2018-01-11 11:31:29,338 INFO Service [4] - TaskSupervisor.Init - HPSM Starting

2018-01-11 11:31:41,427 INFO Service [4] - Successfully started DSSessionVariables() - hibernate session

2018-01-11 11:31:41,430 INFO Service [4] - ScheduleTaskManager.RetryDBConnection Testing DB connection to: Server=(local)\EXP2014;initial catalog=HPIPSC;Integrated Security=SSPI;

2018-01-11 11:31:41,437 INFO Service [4] - - Done TestDBConnection

2018-01-11 11:31:41,437 INFO Service [4] - ScheduleTaskManager.RetryDBConnection Testing DB connection successfull to: Server=(local)\EXP2014;initial catalog=HPIPSC;Integrated Security=SSPI;

2018-01-11 11:31:41,593 INFO Service [4] - Service Starting up - Init()

2018-01-11 11:31:42,571 INFO Service [4] - Service Version: 3.1.0.65238

2018-01-11 11:31:42,594 INFO Service [4] - Shrinking the DB log file

In the error above, the message indicates that Windows Authentication is attempting to login the user the Security Manager runs under (Network Service in this case) into the correct remote server\instance name but is being rejected. This would happen if Network Service did not have DBO rights to use the database. Use SQL Management Studio to confirm the user running the Security Manager service has DBO rights.

Also, make sure the Security Manager service was restarted after making the changes to the database rights.

Ensure the HPSM_Service.exe.config file contains the correct entries:

• The server or instance name is not correct. Double-check the spelling of each.

It is possible network related issues are preventing the connection to the remote instance. Common troubleshooting steps include:

• Fully qualify the remote SQL server name in the configuration file if name resolution issues are

present or use the IP address instead of the hostname.

• TCP/IP must be enabled on the remote SQL server instance. Use SQL Server Configuration

Manager to confirm.

• Check firewall settings to ensure the port that is used for the remote connection is open. The

default port is 1433.

• SQL Server may default to using a dynamic port. Either configure to use a fixed port or start the

SQL Browser service to allow for remote connections.

• Use SQL Management Studio and/or Windows ODBC to connect to the remote SQL server/instance

from the same machine as Security Manager to at least prove a Windows user account can access the server/instance from the Security Manager machine.

• If the HPSM_Service.log indicates table columns are missing, the database may not have been

upgraded due to insufficient rights by the user running the upgrade. This scenario is described earlier in this document with steps on how to uninstall/reinstall to rectify.

• The database tables should always begin with dbo as the schema i.e. dbo.DeviceTable. If some of

the tables begin with a Windows username as the schema, it is very likely that the user who upgraded or created the database was a member of a Windows group when assigned SQL rights. The default schema for a user can be defined by using the DEFAULT_SCHEMA option of CREATE USER or ALTER USER. If no default schema is defined for a user account, SQL Server will assume dbo is the default schema. It is important note that if the user is authenticated by SQL Server as a member of a group in the Windows operating system, no default schema will be associated with the user. If the user creates an object, a new schema will be created and named the same as the user, and the object will be associated with that user schema. The fix to this scenario is to either rename the schema in the affected tables to indicate dbo, or better yet, assign a default schema of dbo to the Windows group to which the user belongs.

Database Upgrade Failure

A very common scenario involves the installer completing the installation, but improper rights were present on the account running the installer to upgrade the SQL database tables. When this happens, you have a new Security Manager version trying to use older SQL tables. The log files are filled with statements indication tables and/or columns are missing the database.

Two options exist in the case:

• Roll back to the previous version

• Attempt to repair the database to match the Security manager version

To roll back to the previous version, uninstall Security Manager under Programs and Features, but when

asked if you would like to delete the database, select No.

Finish the uninstall, keeping the license files intact when asked. Install the previous version, pointing to the remote server\instance where the database still resides. Since the database was never upgraded when Security Manager was upgraded earlier, it should still match the previous Security manager version and work fine with it. Now ensure proper rights are in place before attempting the upgrade again.

Using SQL script to repair/upgrade/install the database

SQL scripts are also available that can repair the database and upgrade it to match the Security manager version to update the database. This is certainly a much faster resolution to the problem. The scripts are installed with HPSM (InstallSqlScripts.zip in the directory C:\Program Files (x86)\HP Jetadvantage Security Manager) and can be run by someone such as the DBA who has proper rights to upgrade the database. The SQL scripts are run manually from a command prompt. They essentially mimic the SQL files the installer runs when “install a new or update an existing database” is selected. The script is launched by executing a batch file named InstallDBRMT.bat from a command prompt.

When you run the SQL scripts from a command line, you must use the proper syntax It won’t tell you if your syntax is bad, but the log files that are created in the folder where you ran the scripts will be full of errors, if the syntax is bad.

Proper syntax for SQL server instance on same machine where you are running scripts:

installdbrmt .\instance

installdbrmt server\instance

The “.\” just means same server, you can just as easily specify the server even though it is the same machine you are on.

Proper syntax for SQL server instance on remote machine where you are running scripts:

installdbrmt server\instance

If a default instance is used, you should be able to just enter the server name, SQL knows if no instance is specified then use the default instance.

BAD SYNTAXES:

installdbrmt \server\instance

installdbrmt instance

If you use bad syntaxes, the command line will appear as if it is creating tables. It isn’t. Log files are created in the same folder containing the scripts. The logs will be filled with errors if permissions are not correct or improper syntax was used.

The script first looks to see if the database is present. If not present, the script will create the database as long as the user running the script has Create Database rights on the SQL instance. If an existing database is present, the script walks through a routine to see what schema is present and updates to the latest schema, as long as the user running the scripts has DBO rights on the database After the script completes, restart the HP Security Manager service. All should be well as the software and database now match schema versions. See the whitepaper titled “HP JetAdvantage Security Manager – Using Microsoft SQL Server” for more information on running the SQL scripts.

If Security Manager still generates errors while attempting to launch or just hangs indefinitely, the database is probably still not upgraded. If you have a new Security Manager trying to use an old database, statements appear over and over in the service log indicating required tables aren’t present:

2018-01-04 14:33:53,731 ERROR NHibernate.Util.ADOExceptionReporter [4] - Invalid object name 'ScheduledReportsTable'.

In this example, ScheduledReportsTable is a new table only available in the 3.1 version of Security Manager, and it is complaining it can’t find the table. That is very typical of new version using an old database.

Here is a screen shot generated by viewing the database using SQL Management Server showing the table that provides the database schema version. It should be version 7 for Security Manager 3.1. The HPSM_Service.log file also indicates the schema version. Notice too how there is a dbo.ScheduledReportsTable. That is another quick method to see if the database is upgraded or not.

Reasons why this can happen include:

• The account running the Security Manager installer doesn’t have DBO rights on the database

during an upgrade.

• The user chooses “connect to and existing database” instead of “install or upgrade an existing

database” during an upgrade attempt but the database was never upgraded manually using the

SQL scripts.

• The SQL scripts are run to manually upgrade the database but the account running the scripts

does not have DBO permissions on the database to upgrade it.

• The SQL scripts may be the broken ones included in the Security Manager folder of version 3.1.

• The SQL scripts are run using the wrong syntax on the command line

Problems Launching Security Manager Web Interface

Issues when running HPSM Application Pool: web page not displayed

Bindings/firewall problem

The Security Manager browser-based interface requires Internet Information Services (IIS) in order to operate. The installer will verify that IIS is enabled with the proper settings enabled and will offer to enable the proper settings if desired. The Installation Guide specifies the proper IIS setting to be enabled if it is desired to perform manually. If the installer fails to set some of the IIS settings, it may be necessary to configure them manually. Since the installer is attempting to enable IIS, it may prompt for a machine restart. You can use IIS Manager under Administrative Tools to determine if the HPSM application pool and HPSM web site are present and configured properly.

An easy test to determine if IIS is functioning properly is to see if you can browse to the default IIS web

page: http://localhost:80. If you can’t browse to the default page, it is likely you won’t be able to

browse to the Security Manager page either.

When you still cannot access the web page, you might get the following error message in your browser when Security Manager is launched in a browser :

If the application pool is still running after attempting to reach the page, then there is a binding problem of firewall problem .

If it is desired to change this port, or if the port is being blocked, it can be changed by configuring the bindings for the HPSM web site under IIS Manager. Expand Sites in the left pane, click on hpsm and click on Bindings.

Make sure that Type is set to https, Port to 7637 and that as an SSL certificate the HP Security Manager Self signed self-certificate or a server certificate is selected. Change it to a different port if it’s blocked by the firewall or open the port in the firewall.

The self-signed certificate allows the data to be encrypted between client and server, while an existing server certificate not only encrypts data but also provides trust that the server is who it says it is. IIS will always search and bind for the server certificate in the personal store of computer account. An identity certificate needs to be of the type “Server Authentication” in order to provide trust.

Issues have been seen in cases where there are multiple certificates to choose from with the same name

during installation, and if an improper certificate is chosen, the bindings will not be created for the web site. Security Manager will not launch, and the HPSM web site will also contain a couple of icons instead of the 20 or so it should contain. The easy fix to this problem is to select the correct port for the bindings as shown above. The icons will appear, and Security Manager will now run properly.

Browser settings causing issues

The new browser-based interface supports either Microsoft Internet Explorer or Google Chrome. The following settings may need to be configured on certain machines or operating systems if Security Manager is having difficulty loading and the application pool is running correctly.

Uncheck “Display intranet sites in Compatibility View” if the login screen for Security Manager is not appearing. This can be done by clicking on the ALT key which will bring up the options in IE. Now under Tools select Compatibility View and uncheck “Display intranet sites in Compatibility View”.

Internet Explorer may require the “Bypass proxy server for local addresses” box to be checked under Internet Options, Connections, LAN Settings if the login screen for Security Manager is not appearing.

Windows 10 may require HTTP2 to be disabled in the browser if Security Manager continually logs out the user.

Issues when running HPSM Application Pool as network service (sql

exception)

The Security Manager service and the HPSM application pool must have the proper permissions to access the Security Manager service database. If the service and database are installed on the same computer, the installation process manages the assignment of database permissions by assigning Network Service to run both the HPSM service and the HPSM application pool. If the service and the database are installed on separate computers, you must configure the correct permissions for the remote database, otherwise you might see errors like this when trying to access HPSM:

See the whitepaper titled “HP JetAdvantage Security Manager - Using Microsoft® SQL Server” for more

information about the required rights for a remote database.

It is also possible that some IIS settings either were not configured correctly by the installer for some reason or they changed since installation. Double check that all of the IIS settings were enabled as described in the “HP Security Manager Installation Guide“. If the browser displays an error that it cannot read the web.config file, there is an error code displayed that Googling may provide some clues. For example, other IIS settings may be misconfigured such as:

32-Bit Applications not enabled for hpsm application pool

The Enable 32-bit Applications setting for the HPSM application pool may need to be toggled. In IIS, click on Application Pools, right-click on HPSM and select Advanced Settings (or click on Advanced Settings in the right -pane). This will bring up the following screen:

Change the setting for Enable 32-bit Applications from False to True.

Issues when running HPSM Application Pool as non-admin user or

service account (error 503)

If the service account is NOT a member of the local administrators group on the Security Manager server, additional steps are required to ensure the service account can access the service control manager just as Network Service could. If these additional steps are not taken, the HPSM application pool will stop as soon as you try to open the hpsm web page and a 503 error will likely be seen when attempting to launch Security Manager. The HPSMWeb.log in the directory C:\Program Files (x86)\HP JetAdvantage Security Manager\WebApp\log and the Windows Event Viewer will contain errors indicating there are not sufficient rights to access service control manager:

<Data>HP JetAdvantage Security Manager: Unexpected Error Initializing Shuting down: System.InvalidOperationException: Cannot open Service Control Manager on computer '.'. This operation might require other privileges. ---> System.ComponentModel.Win32Exception: Access is denied

--- End of inner exception stack trace --- at

System.ServiceProcess.ServiceController.GetDataBaseHandleWithAccess(Stri ng machineName, Int32 serviceControlManaqerAccess) at System.ServiceProcess.ServiceController.GetDataBaseHandleWithEnumerateAc cess(String machineName) at System.ServiceProcess.ServiceController.GetServicesOfType(String machineName, Int32 serviceType) at System.ServiceProcess.ServiceController.GetServices() at LocksmithBusinessLogic.BizLogicMgr.ValidateProductAndDB() at LocksmithBusinessLogic.BizLogicMgr.Init()</Data>

By default, non-administrators cannot access the service control manager and cannot stop/start services. To assign the rights needed for a non-admin account to access the service configuration manager, run the following from command prompt then restart the Security Manager service. This command provided privileges so that authenticated users can run the service control manager as required by Security Manager.

Open a command prompt as administrator, browse to C:\Windows\System32, and enter the following command:

sc sdset SCMANAGER D:(A;;CCLCRPRC;;;AU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD)

If successful, the line ‘[SC] SetServiceObjectSecurity SUCCESS’ is displayed.

The final step is to make sure the service account is a member of the local IIS_USRS group.

Restart the Security Manager service, recycle HPSM application pool, and launching Security Manager UI should now be successful.

When the HPSM application pool is running and the UI is launched, the process w3wp.exe will get started. For each application pool inside IIS another w3wp.exe process will get started. When you add the column Command line in task manager you can see which process is used by which application pool. See screenshot:

In the command line column you will see for the w3wp.exe process something like:

C:\windows\SysWOW64\inetsrv\w3wp.exe -ap “HPSM” -v “v4.0” -l “webengine4.dll” -a \\.a\pipe\iisipm324143214-2432424-sdfasfds- h “C:\inetpub\temp\apppools\HPSM\HPSM.config” -w – m 0 -t 20 -ta 0

For HPSM this also means that full access is required to c:\Inetpub\temp\apppools\HPSM\HPSM.config.

If the HPSM application pools is still crashing after trying to access the UI when running running the HPSM application pool as a different user/service, then the account is most likely too restrictive. In that case try to

run the default application of IIS as the same user/service. If the default application pool is also crashing when you try to access the default IIS homepage (http://localhost:80), then it confirms that the problem is outside of HPSM and caused by a too restrictive user or service account. To resolve this add the user/service account to the local administrators group.

Access Denied when trying to logon to HPSM

Users, even (local) Administrators might get an Access Denied when trying to logon to Security Manager, see screenshot:

Only users which are member of the HPIPSC group will have full access to the HP Security Manager and the specific user will have to be added to this group first.

Licensing Issues

Licensing Issues in multi-homed environment: Failed to connect to licensing server

Issue: In a multi-homed server (several NICs installed). It’s possible that the HPSM service tries to reach the HP Print License service on the wrong IP address/NIC with the result: Failed to connect to licensing server in the Dashboard screen and unable to add new licenses.

Perform the following steps to check if this is happening:

1. Retrieve the table dbo.ServerConfigTable (this contains one entry) and check for the

LicenseServerAddress in this table.

2. Retrieve the table dbo.ServerTable and check which IP addresses are listed

If the second IP address in this table is used for communication with all the devices, then HPSM will not be able to communicate to the licensing server.

Resolution:

1. For every NIC go to Control Panel, Network and Sharing Center and click on Change Adapter Settings

2. Right mouse-click on every Adapter/NIC. Select Internet Protocol Version 4 (TCP/IPv4).

3. Click on Properties.

4. Click on Advanced and disable Automatic metric

5. Add a value for the Interface metric.

The interface which is used for communication with the devices should have the lowest number as lower number indicates higher priority.

6. Now restart the HP Jetadvantage Security Manager service. During the service startup, HPSM will

update the priority order in the dbo.ServerTable and update the dbo.ServerConfigTable accordingly.

Note: changing the adapter order can also done in other ways, see for example:

https://www.windowscentral.com/how-change-priority-order-network-adapters-windows-10

Other Licensing Issues

There are a couple of symptoms that would indicate something went wrong when applying the licenses. The

License Server Status

indicates

Error

instead of

Success

This may be accompanied by an error at the bottom of the page indicating “Error: Failed to connect to

the license server.”

Another sign of a problem occurs when attempting to Add Licenses Now, a failure screen appears

indicating no licenses were added.

Here are some steps that can be followed to attempt to resolve these licensing issues:

1. Security Manager requires the proper startup of these 3 services.

• Flexera Licensing Service

• HP Print License Service

• HP JetAdvantage Security Manager

You may need to stop and restart these services. Stop them in this order:

a. HP JetAdvantage Security Manager b. HP Print License Service c. Flexera Licensing Service

Restart them in the reverse order:

d. Flexera Licensing Service e. HP Print License Service f. HP JetAdvantage Security Manager

Now launch the UI.

2. The licensing process in Security Manager is

handled by two services: Flexera and HP Print License service. The latter relies on the former working.

There may be cases where the Flexera service can’t start because of reduced permissions on the service account it is using. Windows Event Viewer may indicate the service cannot start because of reduced permissions. By default, the Flexera service runs under the Local Service account. If permissions are reduced on this account and cannot be increased, try running the service under another account with more permissions. The services may also be starting up too slowly so that the later services cannot determine the initial services are started. Try manually starting the services with a time break of perhaps 30 seconds in between. If this resolves the issue, a delay can be defined for the services when they automatically start.

3. A. Uninstall Security Manager, make sure the uninstaller removes licenses, then reinstall and add

licenses. The uninstaller asks if you want to delete the database also, and you can choose not to do so if it is desired to maintain all the devices, policies, and remediation data from the previous install. When you re-install Security Manager, just point to the existing database and install the licenses

B. If you don’t want to uninstall Security Manager, another technique for removing evaluation licenses is to manually delete them.

• Stop the services in this order:

HP JetAdvantage Security Manager service HP Print License service Flexera service.

• Delete the evaluation license file (don’t delete the DemoLicense file) under:

C:\ProgramData\HP\HP Print License Service\Licenses

• Delete all recovery files (LSRecovery.xml) under the recovery directories (if present):

C:\ProgramData\HP\HP Print License Service\RecoveryFile1 C:\ProgramData\HP\HP Print License Service\RecoveryFile2

• Reboot the HPSM server (in most cases restarting first Flexera service, then the

Print License service and finally HP Jetadvantage Security Manager service, will not fix the licensing issue, a reboot is required in most situations)

4. If both services are running and issues still arise, there may license type conflicts such as trying to

add purchased licenses alongside custom trial\evaluation licenses. Each license has unique parameters such as machine type, expiration date, and feature type. Some varieties cannot be mixed. Purchased licenses can be stacked, but mixing purchased with custom trial/evaluation licenses will usually generate errors. If this has happened refer to step 3A or 3B for resolutions. Examples include:

• Attempt to add a permanent license for 8000 devices with no expiration tied to mac address on

top of a custom trial license that works on any machine: Failure - Cannot mix AnyHost and Mac Address hosts for IPSC_DEVICES.

• Attempt to add a permanent license for 8000 devices with no expiration tied to mac address

over top of the demo license included in product: Success - IPSC_DEVICES replaces IPSC_CMPS.

• Attempt to add a permanent license for 8000 devices with no expiration tied to mac address

over top of the downloaded 60 day trial license from the kiosk: Success - IPSC_DEVICES

replaces IPSC_DEMO.

• Install downloaded 60 day trial license available on kiosk on top of demo license included in

product: Success - IPSC_DEMO replaces IPSC_CMPS.

• Attempt to add another downloaded 60 day trial license from kiosk: Failure - can’t stack

multiple IPSC_DEMO licenses.

• Attempt to add a custom trial license for 100 devices expiring Apr. 30 on top of default demo

license included n product: Success, IPSC_DEVICES replaces IPSC_CMPS.

• Attempt to add another custom trial license for 550 devices expiring Aug 31 on top of another

trial license: Success, added to existing 100 trial, but days remaining reflects first file read. Expired license is just ignored. IPSC_DEVICES can stack together as long as all AnyHost.

5. If problems still exist, there might be a port conflict. For example, Port 8888 needs to be open for

the HP Print License service so check firewall and/or McAfee type firewall products. It might be easier to allow applications through the firewall instead of specific ports, so allow the following through

Rules

C:\Program Files (x86)\ HP JetAdvantage Security Manager\HP Print License Service\lmgrd.exe C:\Program Files (x86)\ HP JetAdvantage Security Manager\HP Print

License Service\HPQ.exe

C:\Program Files (x86)\ HP JetAdvantage Security Manager\HP Print

License Service\HP.Print.License.Host.WindowsService.exe

Even though Flexera and the HP Print License service are on the same machine, other products that use the HP Print License service may not have Flexera on the same machine. The HP Print License Service requires the dedication of this TCP Port and can’t be modified for alternative port assignment.

A successful use of port 8888 appears as such in the HPSM_Service.log file:

2018-01-11 11:31:49,416 INFO

Service

[License Task 1/11/18 11:31]

- LicenseWrapper.Connect using IPs : 15.25.250.161 : 15.25.250.161 device features ::devices version: :host addr: net.tcp://15.25.250.161:8888/LicensingService ok to LogErrors: True

An unsuccessful use of port 8888 appears as such in the HPSM_Service.log file:

2015-04-07 11:15:27,146 ERROR Service - [License Task 4/7/15 11:15]

- LicenseWrapper.Heartbeat - (skipping) unexpected ERROR: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://192.168.181.1:8888/LicensingService. The connection attempt lasted for a time span of 00:00:21.0285341. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

192.168.181.1:8888.

6. Other applications such as ePrint also use this HP Print License service but can use a different

version that conflicts with the version used by Security Manager. For this reason, ePrint and Security Manager cannot coexist on the same machine. You may have to uninstall the existing HP

Print License Service under Programs and Features or Add/Remove Programs and reinstall Security Manager to obtain the proper HP Print License Service.

7. Log files can provide clues as to what may be causing the problem. Two log files exist for the

license service as such:

C:\ProgramData\HP\HP Print License Service Files\Flexera.log

C:\ProgramData\HP\HP Print License Service Files\HPPLS.log

The Security Manager service log can also provide valuable information:

C:\Program Files (x86)\ HP JetAdvantage Security Manager\log\HPHPSM_Service.log

For example:

2015-04-07 11:15:27,146 ERROR Service - [License Task 4/7/15 11:15] - LicenseWrapper.Heartbeat - (skipping) unexpected ERROR: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://192.168.181.1:8888/LicensingService. The connection attempt lasted for a time span of 00:00:21.0285341. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

192.168.181.1:8888.

A “Flexera call completed” message in the HPPLS log file indicates the issue is not related to the

Flexera service.

A Winsock error usually means the service is having trouble opening a TCP socket. This could be network or firewall related, perhaps caused by a security app such as McAfee, etc.

The table below mentions possible error codes that may be seen in the Print License Service log file and a brief description of each.

Error Code

Description

Success.

Error Code

Description

-100

Something is wrong with the service. Could be a security token mismatch between the client and the server or internal exceptions un- handled by the HPPLS.

-101

Null job handler.

-102

An invalid parameter is passed to any of the APIs.

-104

This corresponds to any exception that is thrown by HPPLS. The

details of the exception are logged in the log file.

-105

A valid session is already present, and client tries to create another session.

-106

Either there are no features checked out and client tries to query for feature details or there is no feature available with the given name and version.

-107

An API is called with an empty session.

-108

Flexera server is not available.

-109

Flexera server is not responding.

-110

Some features that are passed to the API are invalid.

-111

Invalid file content.

-112

Token mismatch.

-113

Client does not exist.

-130

Flexera service is either not up & running or not ready to serve.

8. If Security Manager had been installed and the IP Address changed, that could cause issues as the

HP Print License service would be trying to contact the Flexera service as seen in the HPPLS.log file:

net.tcp//Old IP Address:8888/LicensingService

An uninstall and reinstall of HPSM would solve this issue.

9. A purchased license file must be ordered with the exact match of the Security Manager server MAC

address. The HP Print License manager will fail to operate properly without this exact match. If using VMWare, make sure the appropriate virtual adapter MAC address is used. The physical mac address of the NIC for the machine as seen under IPCONFIG is not the mac address of the VM. Check the documentation of the VM vendor for instructions on how to find the mac address. VMWare also recommends a static MAC assignment to accommodate software licensing scenarios.

If set to dynamic, it will not match the mac address in the license file any longer. Check the documentation of the VM vendor for instructions on how to set to static.

If the mac address in the license is not what Security Manager expects, there should be an error in the Flexera.log file found under C:\ProgramData\HP\HP Print License Service indicating what it read in the license file and what it expects for the correct mac address:

10:15:13 (HPQ) Wrong hostid on SERVER line for license file: 10:15:13 (HPQ) C:\ProgramData\HP\HP Print License Service\LicenseFiles\24092015101513_IPSC_DEVICES.lic 10:15:13 (HPQ) SERVER line says d8d38582501d, hostid is d8d38582501c 10:15:13 (HPQ) Invalid hostid on SERVER line 10:15:13 (HPQ) Disabling 500 licenses from feature IPSC_DEVICES

If a trial/evaluation license installs fine but a purchased license does not install, it is likely that either the mac address is wrong in the license file or the license file itself is bad. A purchased license differs from a trial/evaluation license in that the purchased license only works on the machine where the mac address is provided. That mac address must match the mac address in use, and if running on a VM, the physical mac address for the machine is not the same as the mac address of the VM (see #7 above).

Second, it is possible that the license file itself is bad or corrupt, although rare. The feature name in the license file must be either IPSC_DEVICES or IPSC_DEMO. If it is HPSM_DEVICES, contact the licensing support team and request another license. When doing so, make sure to instruct them to generate a new license instead of re-sending the original license as they will not know the original license is bad unless you tell them.

The license has readable text in the first half and encrypted content (SIGN=xxxx ...) in the latter half that matches the readable content in the first half. Content of license files SHOULD NOT be changed/edited at any time. Any edit to the readable content will no longer match the encrypted portion and the file is now corrupt and unusable. If “SIGN=0” in the latter half of the file, it is corrupt, and a new license file needs to be generated.

10. Make sure license files are present under:

C:\ProgramData\HP\HP Print License Service\LicenseFiles.

By default, C:\ProgramData is a hidden folder. Either un-hide hidden folders to view this folder or type the folder name (C:\ProgramData) directly in the address bar.

11. If in the rare case the Security Manager server has obtained a new hostname, Flexera likely needs

some help to reflect that change. Flexera refers to some keys in the windows registry related to server hostname. One option could be to uninstall Security Manager, keep database and licenses intact, then reinstall Security Manager and point to the existing database. This would likely update the registry entries to the new hostname.

Another option involves editing the registry entries manually and restarting the services rather than a full uninstall/reinstall.

A. Stop the following services in this order:

HP JetAdvantage Security Manager Service

HP Print License Service

Flexera Licensing Service

B. Open Registry Editor. C. Find all of the ‘HPQ_LICENSE_FILE’ keys in the registry and replace the values to reflect the new

hostname (they will all have the old hostname referenced after the @ sign). There will be several keys in the registry for each machine and user account.

Example of one of the HPQ_LICENSE_FILE entries in the registry:

D. Delete license files under the following directory (don’t delete DemoLicense file_ForStartup.lic).

C:\ProgramData\HP\HP Print License Service\LicenseFiles

Delete All files under RecoveryXXX directory (don’t delete the directory itself).

C:\ProgramData\HP\HP Print License Service\RecoveryFile1

C:\ProgramData\HP\HP Print License Service\RecoveryFile2

E. Restart the services in the reverse order that you stopped them:

Flexera Licensing Service

HP Print License Service

HP JetAdvantage Security Manager Service

F. Check whether the HPSM server is working well with the demo license. G. Install the issued license again.

Required Network Ports

If a firewall is installed on the computer on which the Security Manager service runs, and the service will be accessed from the user interface on a remote computer, the firewall must be set to allow access to the service. The older Security Manager service listens on port 8002, which must be opened in the firewall to allow remote access to the service. The new browser-based interface listens on port 7637 be default. If you do not want to allow remote access to the Security Manager web service for either version, then you can block the respective ports with a firewall.

The next pages list the ports used by Security Manager.

Port

Protocol

Service

Notes

Client to Server

7637

(version 3.0+)

TCP

HTTPS

Port set during installation to be used to secure data between client and HPSM server via browser. This port may be changed to something else by editing bindings for the HPSM web site under IIS Manager. HPSM versions 3.0 and beyond.

8002

(version 2.1.5-)

TCP

WCF-NET.TCP

WCF with message encryption - port used from a remote client interface to the Security Center service. HPSM versions 2.1.5 and prior.

Server to Devices and to hp.com (for firmware assessments)

80 and 8080

TCP

HTTP

Port used for HTTP communication to devices only when SSL is not supported on the device. Also used to gather the latest firmware versions from the web if firmware assessments are enabled and configured to dynamically retrieve from web.

443

TCP

HTTPS

Port used for secure HTTP communication to devices, HTTP Web over SSL.

N/A

ICMP

PING

Internet Control Message Protocol - port used to check if node is active.

161 UDP SNMP

Simple Network Management Protocol - port used for many configuration items on devices as well as discovery of devices.

7627

TCP

SOAP-HTTP

Web service port used to manage communications on Futuresmart devices.

Devices to Server

3329 TCP HP Instant-On Security

Secure port (uses SSL) used from the device to the Security Manager service for Instant-On discovered devices.

Server to SQL database

1433 TCP MS SQL

Standard DB Connection - port used from the Security Manager service to a remote SQL database with a default instance. Can be customized in a configuration file.

1434 UDP MS SQL Browser service

Standard connection to SQL browser service to retrieve the TCP port for the named SQL instance

dynamic

TCP

MS SQL

Standard DB connection to a named SQL instance using dynamic ports

Server to Email

SMTP

Simple Mail Transfer Protocol

Typical port used for communication to mail server if Automated Output feature is enabled. Port can be customized under File, Settings, Automated Output.

Server to Certificate Authority

135 TCP DCOM/RPC

Certificate management - port used between Security Manager service and CA server.

Random allocated high TCP ports above 1024

TCP DCOM/RPC

Certificate management - port used between Security Manager service and CA server.

Licensing

7000 TCP HP Print License Service

Licensing heartbeat – Heartbeat port used between the Security Manager service and the HP Print License service. This is the communication between two services on the same machine and needs to be open on the incoming and outgoing port

8888 TCP HP Print License Service

Licensing - port used between the Security Manager service and the HP Print License service. This is the communication between two services on the same machine and needs to be open on the incoming and outgoing port.

27000

TCP

Flexera service

Licensing - port used by the Flexera service (lmgrd.exe). This port is used to communicate between the FlexeraLicensingService and the HP Print License Service. These two services reside on the same machine and therefore this port needs to be open for incoming and outgoing communication.

Ports Diagram

Changing Firewall Settings and Testing Open Ports

When configuring firewalls, an administrator can either open up ports used by the application (above table) or allow certain program executables access through the firewall. For the latter, Security Manager includes three separate services represented by four executables:

• C:\Program Files (x86)\HP JetAdvantage Security Manager\HPSM_Service.exe

• C:\Program Files (x86)\HP JetAdvantage Security Manager\HP Print

License Service\HP.Print.License.Host.WindowsService.exe

• C:\Program Files (x86)\HP JetAdvantage Security Manager\HP Print License Service\HPQ.exe

• C:\Program Files (x86)\HP JetAdvantage Security Manager\HP Print License

Service\lmgrd.exe Security Manager primarily uses the following ports to communicate to

devices:

• 80 – used for non-encrypted HTTP traffic to device

• 161 – SNMP traffic to device

• 443 – encrypted traffic to device

• 7627 – web services traffic to device

Therefore, these ports need to be open in order for Security Manager to effectively manage devices. Security Manager and Web Jetadmin are very similar in how they communicate to devices. If Web Jetadmin is managing devices fine without credential failures and you are certain Security Manager has matching credentials in its Credential Store, as a test you might install Security Manager on the same machine as Web Jetadmin to determine if port issues exist on the Security Manager server.

PortQry, a free Microsoft utility, can also be used to determine if ports are open between the Security Manager server and the device. The Port Query user interface tool makes it easy to quickly enter an IP Address for a printer and the port to be queried. Results will be displayed whether the port is Listening or Not Listening.

Proxy for Network Service Blocking Communication

Security Manager runs under the Network Service account be default. It is possible that Network Service has been configured to use a proxy. If Security Manager reports credential failures on the fleet when no passwords are actually present on the fleet, it is quite possible a proxy is blocking the ability for Security Manager to query the pages it requires from the device to determine if an Admin (EWS) Password is present. A utility named bitsadmin can be used to determine if a proxy is present on local service accounts and can clear these settings. Try the following command to determine if network Service is using a proxy:

Bitsadmin /util /getieproxy

networkservice Possible values include:

• NO_PROXY—Do not use a proxy server.

• AUTODETECT—Automatically detect the proxy settings.

• MANUAL_PROXY—Use an explicit proxy list and bypass list. Specify the proxy list and

bypass list immediately following the usage tag. For example, MANUAL_PROXY proxy1,proxy2 NULL.

If the command returns MANUAL_PROXY or AUTODETECT, try setting Network Service to run without a proxy by typing the following command:

Bitsadmin /util /setieproxy networkservice NO_PROXY

Device Status Errors and Credential Failures

It is important to understand how Security Manager interacts with devices to be able to troubleshoot issues such as credential failures or device status errors.

Security Manager uses a variety of techniques to manage different types of devices depending upon how features are exposed on the device.

• SNMP - Simple Network Management Protocol (SNMP) is used quite extensively in Security

Manager for all types of devices, especially during a Verify task to determine status. SNMP is used to read from a device (SNMP GET REQ) or to write to a device (SNMP SET REQ). The packet contains one or more Object Identifiers (OID) that defines the item being read or configured. Security Manager supports both SNMPv1/SNMPv2, which is unencrypted, and SNMPv3, which is encrypted.

• Web Services - Web Services (WS*) is a SOAP-based protocol used mostly with HP FutureSmart

devices. This communication uses port 7627. Security Manager communicates over HTTPS to ensure that all data is encrypted during the transmission.

• LEDM - For the configuration of some non-HP FutureSmart devices, Security Manager uses Low-

end Data Model (LEDM). LEDM is based on the Representational State Transfer (REST) style architecture, which is a design that describes a simple interface for transmitting XML data over HTTP or HTTPS without an additional messaging layer. This configuration is done over HTTP or HTTPS depending on the device configuration and device firmware.

• HTTP (“screen scraping” or “web scraping) – Some features are not exposed through typical

management protocols thus Security Manager resorts to performing what is referred to as “screen scraping” to more or less use HTTP to read a page and extract settings.

• DSMP - DSMP is a proprietary protocol that Security Manager uses for some configuration options

in the Digital Sending category for legacy HP Enterprise printers. DSMP is sent over HTTP.

• PJL - Security Manager may use Printer Job Language (PJL) to test the PJL Password on some

devices.

• CDM - Security Manager uses a proprietary implementation of CDM only in version 3.1 and beyond

to accommodate the “Secure by Default” initiative released in FutureSmart 4.5 firmware. CDM is sent over HTTPS.

The Assessment Status and is defined as such:

Not assessed

Assessed and all settings in compliance

Assessed with only low risk items out of compliance

Assessed with medium risk items (and possibly low) items out of compliance

Assessed with high risk items (and possibly low/medium) items out of compliance

It has

nothing to do with the state or status of the device, it merely indicates which settings were in or out of compliance during the last assessment task.

The Device Status column includes a visual icon to indicate Good (green check mark) or some sort of error

(red x) indicating a problem. It also includes a textual description of the error as follows:

• Network Connection Error - this error typically indicates an issue trying to communicate with a device

over a specific protocol. Many times Security Manager is trying to securely connect to the device over SSL/TLS and cannot for some reason. Most common reasons for this error include:

o No response from device on basic network communications such as ping. Device may be

powered off or disconnected from network. Pings may be filtered at router or firewall.

o Device responds to pings but does not respond to Web Services (WS*) queries. o Can’t browse to EWS page, perhaps EWS has been disabled.

o SSL/TLS handshake fails so transaction cannot be encrypted. Operating system controls

the TLS versions in the handshake, not HPSM. You can select the device and click Do Not Enforce SSL/TLS.

o Certificate has MD5 hash which Microsoft no longer supports, so SSL/TLS handshake is

rejected. Can right-click device and choose Set SSL\TLS Enforcement, then choose Do Not Enforce to temporarily fix. Now Verify again to see if state clears. If so, for a permanent solution, regenerate the self-signed certificate under EWS (might require newer Jetdirect firmware) to generate a new certificate with a supported hash.

o ACL (Access Control List) blocking. o No certificate support, already set to not enforce SSL/TLS (grayed out). o Ports blocked, perhaps by firewall o Some cases have been reported whereby DAT indicates SNMPv1/v2 passes for Gets and

Sets yet SNMPv3 cannot be enabled using SNMPv1/v2 and this error is generated. Many times a second remediation clears the error.

• Connection Refused / Invalid Identity Certificate - If Security Manager installed an identity certificate

on the device, it tags it in the database to enforce trust for future communications.

o Removed, expired or revoked certificates. o Cannot connect to CRL (certificate revocation list) to check revocation.

• Credentials Failed - a mismatch occurred between what is stored in the database for the specific

credential and what is on the device. Security Manager will always try what is stored in database first, then public/blank in case the credential was erased on the device, and finally any credential stored in the Global Credential Store. If all three attempts fail, Security Manager posts Credentials Failed as it needs proper credentials in order to communicate with the device.

o SNMP behavior is to not respond to SNMP REQ packet when community name is wrong. o Older devices had two locations under EWS for Admin Password.

• Device Not Supported - device does not support enough security related items to be deemed

supported by Security Manager. See “Supported Device List” whitepaper for a complete list of supported devices.

• Error - this is a rarely seen state that seems to occur on the devices that answer so little to queries

that Security Manager deems it not manageable.

• Hostname Resolution Error - the hostname cannot be resolved. The DNS name server does not

recognize the hostname that is being presented by Security Manager that once represented the device.

• License Required for Assessment - not enough licenses are available, a license has not been assigned

to the device yet.

• No Information - a Verify task has not yet been performed where Security Manager gathers a dozen

or so attributes about the device and populates the columns. Just adding a device will only perform a hostname lookup, but the device will remain in No Information status until a Verify is performed or an Assessment which begins with a Verify task.

Remember, browsing to EWS is strictly HTTP traffic. Security Manager and tools such as Web Jetadmin use WS*, LEDM, SNMP, DSMP, etc. to communicate to devices for the various settings depending upon how they are exposed for fleet management per device family. For example, you could disable SNMP on devices and not affect EWS browsing at all, but tools such as Web Jetadmin and Security Manager would be severely hampered as they could not communicate with the device using a critical protocol.

Credential Management

It is common for both Web Jetadmin and Security Manager to be managing a fleet, and when one tool changes device credentials, the other tool indicates a credential failure until it too knows the credentials.

The best way to troubleshooting credential failures is to hone in on a single device, highlight one device and check the Properties by clicking on the device IP Address link. Normally for devices with no credential

failure, the credentials will all be in a Valid state as

seen to the right.

However, when the status indicates Credentials Failed, Security Manager does not know that credential

in order to read and/or write settings that require that credential.

SNMP

SNMP v1/v2 credentials are broken into two types: Read and Read/Write. These equate to the Get Community Name and Set Community Name under EWS. If there are no credentials entered for the Get Community Name under EWS, and if the Disable SNMP v1/v2 default Get Community Name of “public” box is not checked, the device is wide open to read information using SNMP with just public used as the Community Name for an SNMP GET REQ packet. This isn’t uncommon as very little if any sensitive data is passed thru just Read attempts.

Starting with version 3.1, Security Manager checks both a Set

Community Name and a Get Community Name during a Verify task and will post a credential failure if either fails. If either an SNMP GET REQ or SNMP SET REQ packet receives no response, it is assumed a credential failure is present because devices will not respond if a Community Name does not match. To resolve this situation, either clear the credentials on the device under EWS, or

add the SNMP v1/v2 Read/write Community Name or the SNMP Write Community Name to

the database (credential store) for the device by

selecting it and choosing the Set Credentials icon and clicking Configure.

Admin (EWS) Password

If the Admin (EWS) Password is claiming Credentials Failed, this means that the test Security Manger performs to determine if an Admin (EWS) Password is present is failing. For Futuresmart devices, Security Manager will attempt to use web services to retrieve a system configuration page. Proper EWS credentials are required to retrieve such a page. If the page is not returned, it is assumed the Admin (EWS) Password that Security Manager has stored for the device in the database or in the global store is incorrect. For older non-Futuresmart devices, an attempt is made over an HTTP request to retrieve the Security Status page under EWS. Again, if the page is not returned, it is assumed the Admin (EWS) Password that Security Manager has stored for the device in the database or in the global store is incorrect.

One technique to resolve this issue is to add the Admin (EWS) Password to the database (credential store) for the device by selecting it and choosing the Set Credentials icon and clicking Configure. Another technique is to clear out the Admin (EWS) Password under EWS, then right-click the device in HPSM and clear the credentials that are stored in the database under Set Credentials, Reset. Basically you are just trying to make sure Security Manager and EWS match. Now perform a Verify task and see if credentials failures are cleared. If it still claims credentials failed, try deleting the device and rediscovering.

Check Credentials Example

Here is an example of a typical device interrogation of a FutureSmart device to check credentials:

During the Verify task, Security Manager starts by pinging the device. If there is no response to pings, a Network Communication Error status is posted. If pings are successful, there are several SNMP GET REQ packets sent with various OIDs to retrieve basic device information. If the OIDs receive no response, Security Manager makes an educated guess that the SNMP credentials are wrong since that would be the exact behavior if the Community Names don’t match and the device still responds to pings. Therefore, a Credentials Failed status is posted blaming the SNMP Read Community Name. The remaining transactions are secure ones over port 443 and port 7627 using to attempt to retrieve an endpoint called SystemConfiguration over web services. If that fails, there will be several indications in log files that it failed, and the status will indicate Credentials Failed as seen below blaming the Admin (EWS) Password because you must know the correct Admin (EWS) Password to retrieve such information over web services:

The EapNetworkLib.log file contains statements as such indicating the request was forbidden:

15:32:47,213 ERROR EapNetworkLib [28] EapNetworkLib.EapUnauthorizedAccessException: XmlRest.Get(NPI851843, /systemconfiguration, 0) failed. HTTP status code: Forbidden ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.

If you try to access the device EWS, it should only show the Information tab unless you login and provide the correct Admin (EWS) Password if an Admin (EWS) Password is set.

If one is set and you can successfully login to EWS by entering it, that same Admin (EWS) Password can be manually entered into the Security Manager database for the device by right-clicking the device and adding it as such:

Now a verify task should succeed because the password matches the device.

Device Communication Log Files

If failures still cannot be resolved for SNMP Community Names or Admin (EWS) Password, log files can be viewed for clues. Log files are stored under C:\Program Files (x86)\HP JetAdvantage Security Manager\logs. The three log files containing the most valuable troubleshooting data include:

• HPSM_Service.log – contains data regarding HPSM service and actions it performs

• EapNetworkLib.log – contains data regarding network traffic to devices

• EapDeviceLib.log – contains data specific to device settings

The EapNetworkLib.log file contains statements such as below that either the SystemConfiguration endpoint cannot be retrieved or the Security Status page cannot be returned for Admin (EWS) Password issues. It also contains statements that SNMP GET REQ or SNMP SET REQ attempts were not returned for SNMP credential issues.

Example Admin (EWS) password failure for Futuresmart device:

Exception message=XmlRest.Get(15.86.190.69, /systemconfiguration, 0) failed. HTTP status code: Forbidden, inner exception message=The remote server returned an error: (403) Forbidden.

Example Admin (EWS) password failure for non-Futuresmart device:

1. 15:52:04,451 WARN EapNetworkLib [4] - Exception

message=Web.Get(15.86.190.170,https://15.86.190.170:443/hp/jetdirect/s ecurity_status.html,0) failed. HTTP status code: Unauthorized, inner exception message. The remote server returned an error: (401) Unauthorized.

Example SNMP Set Community Name failure for non-Futuresmart device:

Exception message=Snmp.Set(15.86.190.69,1 varbind(s)) timed out., inner exception message=No response was received from the agent. Timeout is set to 30000 milliseconds.

Hanging tasks

If a task is started but never seems to complete or takes an extraordinary time to complete, there may very well be a valid reason or some configuration options to try to improve performance.

Hung Tasks vs. Slow Tasks

It is important to determine if the task is hung forever and will never complete or if it is just taking much longer than expected to complete.

First, try the same policy on just a few devices to see if it completes. Also, try an extremely simple task on the fleet to see if a setting in the policy might be responsible for the delay or hang. You want to narrow down settings and/or devices to see if they are causing a task to be slow or hung. Versions of Security Manager prior to 3.0.1 had known issues where certain items in a policy could cause a task to hang indefinitely. For example, setting that required web scraping to assess/remediate relied upon a Microsoft library to perform the HTTP transaction. Some servers had Microsoft libraries installed that would not permit performing this HTTP transaction. Security Manager 3.0.1 starting using a third-party library to perform these transactions that eliminated the hang.

Security Manager relies on a shared library called Microsoft.MSHTML.dll to perform queries on devices using a technique referred to as “web scraping” to manage the features. Older devices rely on this technique more than newer FutureSmart devices. Typically, this file is already present in the Global Assembly Cache (GAC), and if so Security Manager will use the file loaded in GAC. If the file is not present in GAC, Security Manager will load a copy of this file. Issues may arise where functionality of this file is not working or is blocked by browser settings, for example. One such setting is a browser setting called “Run antimalware software on ActiveX controls” under the Security tab that will block the usage of this dll. If so, there is a possibility that a task relying on this .dll file to perform managing of a feature may hang as there is never a return to the query. Security Manager 3.1 uses a newer third-party library to perform web scraping to alleviate this issue on these unique servers that block the usage of the MS library.

If tasks are still hanging, while it may appear deleting the task under the tasks tab stops all the devices from being attempted to assess/remediate, it does not. The devices are still tagged in the database as not complete and will start again if the service is restarted, for example. You will have to wait for incomplete devices to run their course.

Credential Failed or Network Connection Error Impacting Performance

Devices in a Credentials Failed or Network Connection Error status can cause extreme delays in completing tasks. Try to run the task again without devices in this state. If it completes in a more reasonable amount of time, Security Manager 3.1 offers some timeout and threading settings that can be configured to help lessen to negative performance impact of devices in such a state. These settings can

be controlled using configuration items in the HPSM_Service.exe.config file. If the server has the power,

increasing the threads and decreasing the timeouts can substantially reduce the time it takes to complete tasks.

To change these parameters, open the following file in a test editor:

C:\Program Files (x86)\HP JetAdvantage Security

Manager\HPSM_config.exe.config

Edit the following entries, save changes, and restart the Security Manager service.

<add key="snmpRequestTimeout" value="30000"

Value is in milliseconds (30s) – amount of time to wait for responses to SNMP packets

<add key="verificationSnmpRequestTimeout" value="2000"

Value is in milliseconds (2s) – amount of time to wait for SNMP responses during a Verify task

<add key="timeBetweenEapRetry" value="5000"

Value is in milliseconds (5s) – time to wait between retries on SNMP packets

<add key="eapRetryLimit" value="2"

Number of retries if a device fails to respond for a request

<add key="eapMaxThreadCount" value="100”

Total number of active threads to devices

<add key="maxNumberTasks" value="10"

Number of tasks that can be open simultaneously. This means that Security Manager will either have 10 child tasks of 25 devices each open at a time for Verification and Assessment, or 10 tasks of one device each During Instant On, Remediation and Credential retry.

<add key="numberDevicesInEAPTaskCheckpointInterval" value="25"

This only applies to verification and assessment/remediation tasks, it means 25 devices will be present in each child task.

<add key= “caManagerMaxThreadCount" value="100"

Number of devices at a time when a request is sent to CA manager to provide certificate remediations.

Performance Impact of Instant On Remediations

A high volume of Instant on automatic remediations occurring in the background will absolutely affect performance. It is possible there are a few devices causing such a high volume, or it could just be devices sending announcements for legitimate reasons and the server can only handle so many tasks.

If many of the messages are coming from one device, it could be because a faulty device is going off of network and coming back again. There could also be devices constantly rebooting for some reason. Try eliminating such devices if suspicions arise, they may be responsible for the bulk of Instant On remediations.

You can also try turning off Instant On completely in Security Manager, at least temporarily to see if it is the contributing factor of the hangs/delays.

The number of active Instant On assessments can be viewed under the Tasks tab by selecting Instant On Tasks:

Instant on announcements are processed immediately (i.e adding of the devices), but the action of performing the assessment/remediation task is scheduled. These appear as one task at a time for each device, and Security Manager processes a maximum of 10 Instant On remediation tasks at a time. This number of maximum tasks at a time can be controlled using a configuration item in the HPSM_Service.exe.config file found under

\Program Files (x86)\HP JetAdvantage Security Manager

Change “maxNumberTasks” from a value of “10” to as much higher value to see if it makes a difference.

The threading for scheduled tasks can also be increased here. By default each parent task is broken into child tasks of 25 devices each. This number could be increased also to increase performance by changing ="numberDevicesInEAPTaskCheckpointInterval" to a higher number.

It may boil down to if the fleet is exceptionally large and Instant On seems to be consuming a bulk of the bandwidth, a separate server may be required to process just the Instant On tasks. This will allow for the

“instant” remediation benefit of Instant On in cases where a device is cold reset, for example. However, it won’t compromise the ability of the scheduled tasks to keep the entire fleet in compliance if running on a

separate server.

One other possible cause of hanging tasks includes the Email Summary Reports feature. Assess/remediate tasks run forever if an invalid email address is configured in automated output. In such a case, tasks should be cancelled manually and a correct email address should be configured in settings.

Suggestions for improving performance are provided below but certainly not limited to these values. This may take some trial and error. For example, setting eapRetryLImit=0 can shave approximately 24s off total assess/remediation time per device.

Growing Database and Nightly Maintenance failing

If the database keeps growing (for example over 10GB), then you need to verify if the nightly maintenance is getting executed correctly. The HPSM inbuild maintenance starts at 01.00 AM. The HPSM_service.log file will show errors if the nightly maintenance is failing. For example an SQL timeout might occur. Note that some information is only logged as INFO not as ERROR.

The default timeout for the maintenance task is 30 minutes. From 3.5 onwards this can be configured with the configuration parameter in the HPSM_Service.exe.config file:

After making changes a restart of the HPSM service is required. However, as it’s unknown which timeout is required, it’s better to remove the old data with a script from SQL Management Studio to prevent timeouts. In that case, stop the HPSM service, then execute the following script:

DECLARE @X INT=1;

WAY:

SELECT TOP 10000 * into #NEWTABLE FROM (SELECT rec.ID AS recID, rToret.KEY_ID as rToretID, rt.ID AS rtID, rvt.ID as rvtID, rTorv.ID AS rTorvID, rTorat.KEY_ID AS rToratKEY_ID, rat.ID AS ratID, av.ID AS avID, raTop.ID AS raTopID FROM dbo.RecommendationTable rec LEFT OUTER JOIN dbo.RecToReasonsTable rToret ON rToret.KEY_ID = rec.ID

LEFT OUTER JOIN dbo.ReasonTable rt ON rt.ID = rToret.Reason LEFT OUTER JOIN dbo.ReasonToReasonValuesTable rTorv ON rTorv.ID = rt.ID LEFT OUTER JOIN dbo.ReasonValueTable rvt ON rvt.ID = rTorv.ReasonValue_ID LEFT OUTER JOIN dbo.RecToRecommendationActionsTable rTorat ON rTorat.KEY_ID = rec.ID LEFT OUTER JOIN dbo.RecommendationActionTable rat ON rat.ID = rTorat.RecommendationAction LEFT OUTER JOIN dbo.AssessmentValueTable av ON av.ID = rat.ActionValue_REF LEFT OUTER JOIN dbo.RecActionsToParametersTable raTop ON raTop.ID = rat.ID

where rec.AssessmentAndPolicyUniqueID NOT IN ( select distinct dal.assessmentAndPolicyUniqueID as uniqueID from dbo.DeviceAssessmentLogTable dal where dal.State = 2 )) as Sub1

--select count (*) from #NEWTABLE

DELETE a FROM dbo.RecToRecommendationActionsTable a INNER JOIN #NEWTABLE B ON a.KEY_ID= B.rToratKEY_ID DELETE a FROM dbo.RecToReasonsTable a inner join #NEWTABLE B on a.KEY_ID = B.rToretID DELETE a FROM dbo.RecommendationTable a inner join #NEWTABLE B on a.ID = B.recID

DELETE a FROM dbo.ReasonToReasonValuesTable a inner join #NEWTABLE B on a.ID = B.rTorvID DELETE a FROM dbo.ReasonTable a inner join #NEWTABLE B on a.ID = B.rtID

DELETE a FROM dbo.ReasonValueTable a inner join #NEWTABLE B on a.ID = B.rvtID DELETE a FROM dbo.RecActionsToParametersTable a inner join #NEWTABLE B on a.ID = B.raTopID DELETE a FROM dbo.RecommendationActionTable a inner join #NEWTABLE B on a.ID = B.ratID DELETE a FROM dbo.AssessmentValueTable a inner join #NEWTABLE B on a.ID = B.avID

SET @X = (select count (*) from #NEWTABLE)

drop table #NEWTABLE

IF @X=10000 GOTO WAY;

Once the script has been completed, there could be a lot of empty space in the HPSM database. The space which the database needs will be a few GB smaller. Therefore shrink the database with the following command: EXEC sp_configure DBCC Shrinkdatabase ('HPIPSC')

Now restart the HPSM service. From this point onwards the nightly maintenance should be finished within the default timeout of 30 minutes.

Certificates Installation Failures

Troubleshooting certificate installation failures is no different than troubleshooting most configuration issues. The typical scenarios can cause failures such as name resolution issues, network connectivity issues, traffic blocked by firewall, permissions, device issues, etc.

Security Manager uses DCOM over RPC to submit requests to the CA and retrieve certificates, just like workstations do for auto-enrollment of certificates. Remote Procedure Call (RPC) is a mechanism that allows Windows processes to communicate with one another, either between a client and server across a network or within a single system. Numerous built-in Windows components utilize RPC. RPC uses dynamic ports for communication between systems, but a static port (TCP port 135) must also be used as a starting point for communication. The RPC endpoint mapper listens on this static port.

In a typical RPC session, a client contacts a server's endpoint mapper on TCP port 135 and requests the dynamic port number assigned to a particular service. The server responds with the IP address and port number that the service registered with RPC when it started, and the client then contacts the service on that IP address and port.

If the RPC server is unavailable, errors will occur indicating the certificate was not installed. Many other reasons can cause a certificate to not install. For example, the RPC server's name may be resolving to the wrong IP address, resulting in the client contacting the wrong server or attempting to contact an IP address not currently in use. Alternatively, the server's name may not be resolving at all. A firewall or other security application on the server, or a network firewall appliance between the client and server, may be preventing traffic from reaching the server on TCP port 135. The client may be unable to reach the server at all due to a general network problem.

The following troubleshooting steps should help to resolve these issues.

1. Check the policy settings again for accuracy, especially the Certificate Authority Server and Certificate Authority. Ping the server by name from the client to verify that the name resolves to

the correct IP address. If it doesn’t, verify that the client and server are both using the correct DNS

servers, which must be inside the domain and will typically be domain controllers. Try an IP Address instead of a hostname for the server in case the hostname isn’t resolving. It’s also

possible the Key Length or Signature Algorithm values in the policy aren’t supported by the

device either as a value that can be created in a CSR if Jetdirect is chosen as source or as a value in the certificate itself.

2. Check the Certificate Authority settings again to ensure that the account running the HPSM service

has the rights to submit requests to the CA. By default the Network Service account runs the HPSM service, and Network Service manifest itself remotely as the machine name (machine$).

3. Check the CA Template settings again to ensure that the account running the HPSM service has

the rights for Read and Enroll and that Authenticated Users has Read permissions. Make sure

Submit in Request is selected in the template settings under the Subject Name tab, otherwise

the certificate will be created for the Security Manager server and not the printer.

4. If the CA server is on a different domain as the HPSM server, and no trust relationship exists

between the domains, an error will appear claiming the template does not exist. Even though the template clearly exists, templates must be published into Active Directory in order for clients to use them, and the lack of trust relationship is keeping Security Manager from seeing the template. The easiest resolution is to place the Security Manager server on the same domain as the CA server

5. Check firewall settings for ports being blocked. Security Manager uses DCOM over RPC, just like

workstations do for auto-enrollment of certificates, and DCOM uses port 135 for certificate enrollment. If the firewall is enabled on the Security Manager server, make sure traffic on TCP port 135 is allowed to pass. If workstations are successfully auto-enrolling for certificates, it can be reasonably assumed the CA server firewall is not blocking port 135.

The certutil tool can simulate the behavior Security Manager performs to submit a request and retrieve a certificate by checking for the port being blocked or not:

certutil -ping “CA server”

Examples of an unsuccessful attempt to connect to a non-resolvable FQDN and a successful attempt to the IP Address:

6. The PortQry command-line utility or PortQryui.exe user interface utility (both

downloadable from Microsoft, for example: https://www.microsoft.com/en-

us/download/details.aspx?id=24009 ) can be used to test connectivity from the client to

the server and determine which ports are open on the server. It includes support for RPC and can be used to determine which services have dynamic ports registered with RPC and which specific ports they use.

7. If workstations are also having issues auto-enrolling for certificates, then the standard

troubleshooting steps for resolving RPC Server Unavailable errors may apply such as ensuring the RPC service us running, the Authenticated Users group is in the “Certificate Service DCOM Access” group, Enable Distributed COM on this computer is selected in the

Default Properties tab, etc.

8. The printer does not have a valid certificate revocation list (CRL) from the issuing CA that

Security Manager can use to check if a certificate has been revoked. Verify that all certification authorities in the chain have valid CRLs published.

9. In some cases, HPSM can display Medium risk with the message Unable to communicate with the

Device for a certificate remediation. This could mean that HPSM could retrieve a certificate from a CA but was unable to install the certificate on the device. Enable debug mode and check the EAPDeviceLib.log. It could have an error and debug statement like the following:

2020-04-01 09:04:14,007 DEBUG Pipeline [36] uid=_95c5978c54dd_10.133.227.46, Exception while converting certificate to X509Certificate Access denied. 2020-04-01 09:04:14,007 ERROR Pipeline [36] uid=_95c5978c54dd_10.133.227.46, Pipeline execution halted due to step failure, address=10.133.227.46, failed step=PipelineStepGetCertificateKeyLength, failed transform=InstallIDCert, failed with step

If the above messages are displayed in the log, then HPSM is missing READ access for the following directory: C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

Provide at least READ access to NT Authority\System (if HP JetAdvantage Security Manager service is running as network service).

After making the changes, restart the service and re-run the task.

10. In some situation you might see: Unknown certificate Error when remediating ID certificates, see

screenshot:

The HPCM.log might have errors like:

2020-03-23 09:52:49,932 ERROR HPCM [114] - Error while retreiving Provider: The constructor to deserialize an object of type 'HP.HPCM.Contract.Exception.CertificateAuthorityNameException' was not found. 2020-03-23 09:52:49,948 ERROR HPCM [114] - Error while retreiving Provider: The constructor to deserialize an object of type 'HP.HPCM.Contract.Exception.CertificateAuthorityNameException' was not found. 2020-03-23 09:52:49,948 ERROR HPCM [114] - Error while Enrolling Certificate for Request 8fcf2157-a42a-4875-99a4-aa85473c59d7. Exception : The constructor to deserialize an object of type 'HP.HPCM.Contract.Exception.CertificateAuthorityNameException' was not found.

StackTrace :

at HP.HPCM.Provider.ProviderHandler.ExecuteEnroll(String providerName, CertificateEnrollData certData)at HP.HPCM.HPCMService.Execute(CertificateEnrollData request)

This error happens when the CA server which is specified in the ID certificate is not valid or when the CA is not available.

Incorrect Certificate Authority Server or CA server down/unreachable

Check the HPCM.log file in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log

It might have errors like:

2020-04-10 12:30:14,536 ERROR HPCM [7] - Error while retreiving Provider: The constructor to deserialize an object of type 'HP.HPCM.Contract.Exception.ServerUnavailableException' was not

found.. 2020-04-10 12:30:14,537 ERROR HPCM [7] - Error while Enrolling Certificate for Request 8d680a79-af7b-411e-9ffc-0e4ea9c98da8. Exception : The constructor to deserialize an object of type 'HP.HPCM.Contract.Exception.ServerUnavailableException' was not found. StackTrace : at HP.HPCM.Provider.ProviderHandler.ExecuteEnroll(String providerName, CertificateEnrollData certData) at HP.HPCM.HPCMService.Execute(CertificateEnrollData request)

Check the HPCM.log file in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

It might have errors like:

2020-04-10 12:30:14,508 ERROR HPCM System.Runtime.InteropServices.COMException (0x800706BA): The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)at NETcertcli.CCertRequestClass.Submit(Int32 Flags, String strRequest, String strAttributes, String strConfig)at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.SubmitCertRequestAndGetCe rtificate (String request, String certificateServer, Int32 certFormat, String attribute, String& certificate, Int32& requestId) at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.GetJetdirectIdentityCertU singJetdi rectRequest(String certificateServer, String csr, String templateName, String& signedCertificate, Int32& requestId, String& certRequest) [7] - The certificate server is unavailable: [2008R2-DCTME.wrong\upd-tme2008r2-dc-tme-ca] This could be because that certificate server name is not a valid computer name or it cannot be accessed. 2020-04-14 10:21:32,873 ERROR HPCM [26] - Failure in Connecting to the Certificate Server, Server UnavailableHP.HPCM.Contract.Exception.ServerUnavailableException: The certificate server is unavailable: [2008r2-dc-tem\UPD-TME-2008R2-DCTME.wrong] This could be because that certificate server name is not a valid computer name or it cannot be accessed. ---> System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) at NETcertcli.CCertRequestClass.Submit(Int32 Flags, String strRequest, String strAttributes, String strConfig) at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.SubmitCertRequestAndGetCe rtificate (String request, String certificateServer, Int32 certFormat, String attribute, String& certificate, Int32& requestId)at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.GetJetdirectIdentityCertU singJetdi rectRequest(String certificateServer, String csr, String templateName, String& signedCertificate, Int32& requestId, String& certRequest) --- End of inner exception stack trace --- at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.GetJetdirectIdentityCertU singJetdi rectRequest(String certificateServer, String csr, String templateName, String& signedCertificate, Int32& requestId, String& certRequest) at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.EnrollCertificate(Certifi cateEnrol lData certificateEnrollData, Boolean isMicrosoftSA, CertificateEnrollResult& result)at HPCMMicrosoftPKI.MicrosoftEnterprisePKI.Enroll(CertificateEnrollData certificateEnrollData) cert

Incorrect Certificate Authority Name in policy

Check the HPCM.log in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log

2020-04-10 12:11:20,386 ERROR HPCM [5] - Error while retreiving Provider: The constructor to deserialize an object of type 'HP.HPCM.Contract.Exception.CertificateAuthorityNameException' was not found.. 2020-04-10 12:11:20,386 ERROR HPCM [5] - Error while Enrolling Certificate for Request 3b6aac52-4565-449b-a3d2-5b9d4c318bc2. Exception : The constructor to deserialize an object of type 'HP.HPCM.Contract.Exception.CertificateAuthorityNameException' was not found. StackTrace : at HP.HPCM.Provider.ProviderHandler.ExecuteEnroll(String providerName, CertificateEnrollData certData) at HP.HPCM.HPCMService.Execute(CertificateEnrollData request)

Check the HPCM.log in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

It might have errors like:

2020-04-10 12:46:11,738 ERROR HPCM [5] - Failure in fetching the CertificateHP.HPCM.Contract.Exception.CertificateAuthorityNameExceptio n: The Certificate Authority part of the certificate server is invalid: [2008R2-DCTME\upd-tme-2008r2-dc-tme-ca] ---> System.ArgumentException: Value does not fall within the expected range. at NETcertcli.CCertRequestClass.RetrievePending(Int32 RequestId, String strConfig)at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.CheckPendingCertRequest(I nt32 requestId, String certificateServer, Int32 certFormat, String& certificate)at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.RetrievePendingCertificat eJetdirec tCsr(String certificateServer, Int32 requestId, String& signedCertificateString) --- End of inner exception stack trace --- at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.RetrievePendingCertificat eJetdirec tCsr(String certificateServer, Int32 requestId, String& 45 signedCertificateString)at HPCMMicrosoftPKI.MicrosoftPKICertificateBase.EnrollCertificate(Certifi cateEnrol lData certificateEnrollData, Boolean isMicrosoftSA, CertificateEnrollResult& result)at HPCMMicrosoftPKI.MicrosoftEnterprisePKI.Enroll(CertificateEnrollData certificateEnrollData)

Note there are no errors in the file ../WebApp/log?HPCM.log This error happens when the CA server which is specified in the ID certificate is not valid or when the CA is not available.

10. In some situations, you might see the error message: the certificate request to certificate authority

‘AuthorityName’ is pending, see screenshot:

This can have multiple reasons.

• CA server is configured to require manual approval of the request

Check on the CA server if the request is listed under Pending Requests and validate that the checkbox for CA certificate manager approval has been deselected for the CA certificate which is used in the policy.

• The Template Name in the policy for the certificate template is incorrect

Check the HPCM.log file in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log

It might have entries like:

2020-04-14 10:30:09,076 DEBUG HPCM [4] - DoEnroll : Request b7a69e45-c89f-4880-bafe-5141256a5837 2020-04-14 10:30:10,173 INFO HPCM [4] - ParseCertResponse : PkiProviderName MSEnterprise ; Status - Pending ; Request ID - b7a69e45-c89f4880-bafe-5141256a5837 Check the HPCM.log file in the directory: C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log It might have entries like: 2020-04-14 10:30:09,076 DEBUG HPCM [4] - DoEnroll : Request - b7a69e45c89f-4880-bafe-5141256a5837

Check the HPSM_service.log file in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

It might have entries like:

2020-04-14 10:30:14,075 DEBUG Service [4] - HPCMEventManager calling callback for Request ID: b7a69e45-c89f-4880-bafe-5141256a5837

Check the HPSM_service.log file in the directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\PkiProviders\log

It might have entries like:

2020-04-14 10:30:14,075 DEBUG Service [22] - HPCMMangerResponseManager Dequeue b7a69e45-c89f-4880-bafe-5141256a5837 2020-04-14 10:30:14,075 DEBUG Service [22] – HPCMMangerResponseManager Dequeue - b7a69e45-c89f-4880bafe5141256a5837 Status is : Pending 2020-04-14 10:30:14,075 DEBUG Service [22] - recommendation state: = SuggestedCannotFix

On the CA server, you can find the following Warning in the event viewer event ID 53) or in the Failed Requests list from the CA:

Active Directory Certificate Services denied request 147 because The requested certificate template is not supported by this CA. 0x80094800 (-2146875392). The request was for OU=OU, O=HP Inc, C=NL, L=Amstelveen2, CN=m880-flow.updtme.net. Additional information: Denied by Policy Module 0x80094800, The 47 request was for a certificate template that is not supported by the Active Directory Certificate Services policy: DomainControllerAuthenticatoin.

In the above example, the Template Name had a typo: Authenticatoin instead of Authentication.

Comparing log files with a successful enrollment.

When an enrollment is successful you should see in the HPCM.log file:

2020-04-14 16:28:45,925 DEBUG HPCM [30] - DoEnroll : Request 09a561fb-263a4b6e-ac41-58ad50df812f 2020-04-14 16:28:46,515 INFO HPCM [30] - ParseCertResponse : PkiProviderName - MSEnterprise ; Status Success ; Request ID - -58ad50df812f

Using Network Traces for Troubleshooting

Network traces are sometimes used to prove a certain behavior. Traces never lie. Common free applications such as Wireshark can be downloaded and installed to see exactly what is happening and why on the network. If possible, install it on the Security Manager server and capture all traffic to and from the server. Filters can be used when reading the traces to narrow down traffic between server and device.

Using Event Viewer for Troubleshooting

Event Viewer maintains logs about program, security, and system events on the server. Many times additional clues can be found regarding issues with services or applications. Logs can be exported and sent to support for analysis.

Appendix A Following a task in HPSM_service.log

When debug has been enabled for the HPSM_service.log file, the process details of a task can be seen in the log file. The EAPTaskManager starts the task (using the taskname which is displayed in the UI. This task is running under on thread, in this case thread #4 which is the number in the square brackets.

2020-06-04 16:40:40,147 DEBUG Service [4] - EAPTaskManager - ProcessTask: TelnetAndRetainJobTask

Now the task is created (added and a taskID (id) and started under a new threadnumber (11):

2020-06-04 16:40:40,179 DEBUG Service [4] - - AddingTask - TelnetAndRetainJobTask id: 24c16d9f-227b-4150-be28-abd00112d6cc 2020-06-04 16:40:40,179 DEBUG Service [4] - # Tasks waiting in queue: - 0 2020-06-04 16:40:44,522 DEBUG Service [11] - Starting task: TelnetAndRetainJobTask - with ID - 24c16d9f-227b-4150-be28-abd00112d6cc 2020-06-04 16:40:44,538 DEBUG Service [4] - AssessmentTask - ExecuteTask Started: TelnetAndRetainJobTask job: AssessAndRemediate processing step: notExpanded for task dbID: 24c16d9f-227b-4150-be28-abd00112d6cc

If needed child tasks are created. Child task have the same ID as the parent, but get a .xxx.# behind the task name, where xxx are characters and # is a number:

2020-06-04 16:40:45,835 DEBUG Service [4] - BizLogicMgr Create Task name: TelnetAndRetainJobTask.sbc.0 returning - 24c16d9f-227b-4150-be28-abd00112d6cc 2020-06-04 16:40:45,865 DEBUG Service [4] Task.OptimizeNumberOfTasksAndSetupCheckPointRestart - Done

The number of child tasks depend upon the number of devices which are assessed and remediated and the configured number of devices for one thread – by default 25- for the following two settings in the HPSM_service.config:

Sometimes it’s hard which device is causing a hang within a childtask (as a child task will serve 25 devices by default). You can change the numberOfDeviceInEAPRequest to 1 and restart the HPSM service. After that create a new task. Now each child-task will only serve one printer and it will be easier to detect which child task is hanging.

The dbo.ScheduledTaskTable can also help to understand the relationship between parent task and child task as it will show the ID and Names of parent task and child tasks.

The child task is running under it’s own threadnumber and registering it’s existence at the parent task.

2020-06-04 16:40:54,755 DEBUG Service [16] - ScheduleTaskManager - Registereing task: TelnetAndRetainJobTask.sbc.0 type:Worker 16 b8f39960-b872-40ea-b087-abd00112de33 2020-06-04 16:40:54,771 DEBUG Service [16] - ScheduleTaskManager.UpdateTaskTracking parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 total devices: 2 substractCounters: False 2020-06-04 16:40:54,771 DEBUG Service [16] - ScheduleTaskManager.UpdateTaskTracking parent found 2020-06-04 16:40:54,771 DEBUG Service [16] ScheduleTaskManager.UpdateParentsToRunningTasksRelationshipAndCounters with: parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 total devices: 2 substractCounters: False final task status: None propagateStatusMsgToParent: False 2020-06-04 16:40:54,771 DEBUG Service [16] - Parent's Updated Counters: ParentTask dbID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 2 R#: 2 D#: 0 taskStatus: None 2020-06-04 16:40:54,771 DEBUG Service [16] - ScheduleTaskManager.UpdateDictOfRunningTasks parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 total devices: 2 substractCounters: False 2020-06-04 16:40:54,788 DEBUG Service [16] - ScheduleTaskManager - Successfully Registered task: TelnetAndRetainJobTask.sbc.0 type:Worker 16 b8f39960-b872-40ea-b087-abd00112de33

2020-06-04 16:40:54,788 DEBUG Service [16] - --Number of running tasks after Register: 1

When the client task has been completed it will inform the parent and unregister itself.

2020-06-04 16:41:20,303 DEBUG Service [16] - ScheduledTaskMgr.Unregister - Task: b8f39960b872-40ea-b087-abd00112de33 name:TelnetAndRetainJobTask.sbc.0 type:Worker 2020-06-04 16:41:20,303 DEBUG Service [16] - ScheduleTaskManager.UpdateTaskTracking parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 2 R#: 2 D#: 0 total devices: 2 substractCounters: True 2020-06-04 16:41:20,303 DEBUG Service [16] - ScheduleTaskManager.UpdateTaskTracking parent found 2020-06-04 16:41:20,303 DEBUG Service [16] - ScheduleTaskManager.UpdateTaskTracking worker found 2020-06-04 16:41:20,303 DEBUG Service [16] ScheduleTaskManager.UpdateParentsToRunningTasksRelationshipAndCounters with: parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 2 R#: 2 D#: 0 total devices: 2 substractCounters: True final task status: None propagateStatusMsgToParent: False 2020-06-04 16:41:20,303 DEBUG Service [16] - Parent's Updated Counters: ParentTask dbID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 taskStatus: None 2020-06-04 16:41:20,318 DEBUG Service [16] - ScheduleTaskManager.DetermineStatusMsg: 2020-06-04 16:41:20,334 DEBUG Service [16] - --Number of running remaining tasks after UpdateTaskStatus: ParentTask - dbID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 taskStatus: None 2020-06-04 16:41:20,350 DEBUG Service [16] - AssessmentTask - ExecuteTask Ended: TelnetAndRetainJobTask.sbc.0 with status : Completed job: AssessAndRemediate for task: b8f39960-b872-40ea-b087-abd00112de33 time taken : { 25.6411117 }s 2020-06-04 16:41:20,350 DEBUG Service [16] - Task - UnregisterAndSetTaskMsg TelnetAndRetainJobTask.sbc.0 - with msg: Finished interacting with devices 2020-06-04 16:41:20,365 DEBUG Service [16] - Task.UnRegisterTask: TelnetAndRetainJobTask.sbc.0

In other words for each client task you find a registration task and UnregisterTask:

ScheduleTaskManager - Registereing task: TelnetAndRetainJobTask.sbc.0 type:Worker 16 b8f39960-b872-40ea-b087-abd00112de33 2020-06-04 16:41:20,365 DEBUG Service [16] - Task.UnRegisterTask: TelnetAndRetainJobTask.sbc.0

If you cannot find a UnregisterTask for a client task, then you have identified a hanging task.

After all child tasks have been completed, the parent task will be notified that the child tasks completed:

2020-06-04 16:41:20,365 DEBUG Service [16] - ScheduleTaskManager.UpdateDictOfRunningTasks parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 total devices: 2 substractCounters: True 2020-06-04 16:41:20,365 DEBUG Service [16] - ScheduleTaskManager.UpdateDictOfRunningTasks removing from running tasks: parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 total devices: 2 2020-06-04 16:41:20,365 DEBUG Service [16] - TaskBase - ProcessShutdown: set shut down msg to true: b8f39960-b872-40ea-b087-abd00112de33 2020-06-04 16:41:20,365 DEBUG Service [16] ScheduleTaskManager.UpdateParentsToRunningTasksRelationshipAndCounters with: parentID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 total devices: 2 substractCounters: True final task status: Completed propagateStatusMsgToParent: True 2020-06-04 16:41:20,365 DEBUG Service [16] - Parent's Updated Counters: ParentTask dbID: 24c16d9f-227b-4150-be28-abd00112d6cc A#: 0 R#: 0 D#: 0 taskStatus: Completed 2020-06-04 16:41:20,381 DEBUG Service [16] ScheduleTaskManager.CheckParentStateAndRemoveIfAppropriate: 24c16d9f-227b-4150-be28abd00112d6cc 2020-06-04 16:41:20,397 DEBUG Service [16] - ScheduleTaskMgr.GetListOfChildernTasksNotDone

- # tasks: 0

2020-06-04 16:41:20,412 DEBUG Service [16] - -- Updating parent task status to: Completed 2020-06-04 16:41:20,412 DEBUG Service [16] ScheduleTaskManager.CheckParentStateAndRemoveIfAppropriate removing parent info 2020-06-04 16:41:20,459 DEBUG Service [16] ScheduleTaskManager.CheckParentStateAndRemoveIfAppropriate Updating parent task - all childern are done processing 2020-06-04 16:41:20,459 DEBUG Service [16] - TaskBase - SetNextRunTime name: TelnetAndRetainJobTask type: Schedulable status: Completed next run:12/31/99 11:59 2020-06-04 16:41:20,459 DEBUG Service [16] - ScheduleTaskManager UpdateTaskStatusAndRepeatCycleAfterShutdown: AssessAndRemediate name: TelnetAndRetainJobTask to Completed 2020-06-04 16:41:20,505 DEBUG Service [16] - Parent Task Finished: TelnetAndRetainJobTask 2020-06-04 16:41:20,505 DEBUG Service [16] - --Number of running tasks after Unregister: 0 2020-06-04 16:41:20,505 DEBUG Service [16] - ScheduledTaskMgr.Unregister Success - Task: b8f39960-b872-40ea-b087-abd00112de33 name:TelnetAndRetainJobTask.sbc.0 type:Worker

After that the parent task should stop and unregister as well on it’s own thread number:

Line 312: 2020-06-04 16:41:21,535 DEBUG Service [4] - Parent task stopping, created workers: TelnetAndRetainJobTask Line 313: 2020-06-04 16:41:21,DEBUG Service [4] - AssessmentTask - ExecuteTask Ended: TelnetAndRetainJobTask with status : None job: AssessAndRemediate for task: b8f39960-b87240ea-b087-abd00112de33 time taken : { 1.1715505 }s Line 314: 2020-06-04 16:41:21,DEBUG Service [4] - Task - UnregisterAndSetTaskMsg TelnetAndRetainJobTask - with msg:

hp.com/go/support

Current HP driver, support, and security alerts

delivered directly to your desktop.

© Copyright 2020 HP Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

c05561479ENW, Rev.19, Mar 2021

HP JetAdvantage Security Manager Troubleshooting Manual

Specifications and Main Features

Frequently Asked Questions

User Manual