HP Data Protector IDB Purge best practices
Commands that purge obsolete data from the HP Data Protector
(DP) software Internal Database (IDB), to maintain the IDB in
good condition
Table of contents
Introduction......................................................................................................................................... 2
Scope............................................................................................................................................. 2
Three step guide to use this white paper ............................................................................................. 2
How to identify if a DP IDB needs filenames purging? .............................................................................. 3
Automated discovery........................................................................................................................ 3
Manual discovery ............................................................................................................................ 4
Automated IDB purging done by the Daily Maintenance........................................................................... 5
What is the Daily Maintenance? ........................................................................................................ 5
Setting the time Daily Maintenance runs ............................................................................................. 5
Methods to confirm if, and when, Daily Maintenance is being run ......................................................... 5
If Daily Maintenance is not running .................................................................................................... 6
IDB purging done by the Daily Maintenance........................................................................................... 6
Sessions purge................................................................................................................................. 6
Messages purge .............................................................................................................................. 6
Dcbf purge...................................................................................................................................... 7
Mpos purge .................................................................................................................................... 7
Manual IDB purge operations ............................................................................................................... 8
Filenames purge .............................................................................................................................. 8
The consecutive running of a dcbf Purge followed by a fixmpos............................................................. 9
Appendix A ........................................................................................................................................ 9
What does “Requires exclusive access to the IDB” mean? ..................................................................... 9
Appendix B....................................................................................................................................... 11
Further information about the IDB Daily Maintenance: ........................................................................ 11
Appendix C ...................................................................................................................................... 12
Example update to purge.log after a completed forced filenames purge ............................................... 12
Appendix D ...................................................................................................................................... 13
Failed_copies purge ....................................................................................................................... 13
For more information.......................................................................................................................... 14
Glossary........................................................................................................................................... 14
Introduction
Growing volumes of backup data can significantly load the Data Protector (DP) Internal Database
(IDB). Regular IDB maintenance to purge obsolete data can prevent size related problems.
Purges remove obsolete data from the database, making space available within the database. The
space created by the purges can be used by new DP sessions, delaying the requirement for the IDB to
be extended.
This document presents some Data Protector commands that can help keep your Internal
Database healthy.
This white paper is complementary to the white paper HP Data Protector software cell manager
planning and sizing. Please also follow the recommendations therein to facilitate reliable Data
Protector operation.
This White Paper can be found at:
HP Data Protector V6.0 Software
http://www.hp.com/go/dataprotector --> Support Matrices -->
Scope
This document covers Cell Managers on HP-UX or Windows, but the principals apply to Cell
Managers on other platforms as well. This document covers DP versions 5.5, 6.0, and 6.1.
Three step guide to use this white paper
Note:
Before running any of the commands outlined in this document HP
recommends you make a successful backup of the DP IDB.
Step One:
How to identify if a DP IDB needs a filenames purge?
Step Two:
Automated purging done by the Daily Maintenance and making sure these tasks are taking place.
Step Three:
Manual IDB purge operations.
2
How to identify if a DP IDB needs filenames purging?
There are two ways to identify if a DP IDB needs filename purging:
Automated: Part of the daily health check combined with the DP Reporting feature
Manual: Using DP Command line interface (CLI) or running preconfigured IDB reports
Automated discovery
By default, Data Protector checks the condition of the IDB once a day as a part of the daily health
check mechanism. The start time of the check is defined by the global file parameter DailyCheckTime.
# DailyCheckTime=HH:MM
# default: 12:30
# This option is used for starting daily checks at first
# omnitrig run after the specified time each day. Valid values
# are hour:minute, using the twenty-four hour clock notation
# Specifying “None” disables starting daily checks.
The global file is located on the DP Cell Server under:
Windows (default) C:\Program Files\OmniBack\Config\Server\Options\global
HP-UX/SUN Solaris/Linux (default) /etc/opt/omni/server/options/global
The daily check triggers an “IDB Purge Required” notification if:
• The time elapsed since the last IDB filename purge exceeds Days Last Purge [days] (default, 180
days). At the same time, either the number of filename records likely to be purged exceeds Num.
Estimated Filenames [mio] (default, 6 millions) or the estimated time needed for the purge exceeds
Estimated Time Purge [min] (default, 120 minutes).
• The number of filenames in the IDB exceeds Num. Filenames [mio] (default, 100 millions).s
The filenames purge check carried out as part of Daily Check can be system resource intensive.
Therefore, under some circumstances, it might be useful to deactivate this part of the daily health
check. It can be switched off by setting the (undocumented in DP 5.5/6.0) global parameter
DbPurgeCheck = 0:
# DbPurgeCheck = 0 or 1
# default: 1
# If this option is set (=1), IDB purge check will be included
# when daily check is started. If this option is not set (=0),
# IDB purge check will be skipped. DbPurgeCheck=0
If the purge check is switched off as part of the daily health check it needs to be executed manually
on a regular basis.
3
Manual discovery
A filenames purge preview status report can be manually created either using the DP command line
interface (CLI) or the GUI.
• DP CLI:
omnirpt –report db_purge_preview
• DP GUI:
Reporting context, right click on “Reports” and select “Add Reporting Group.” Choose a name
and click on “Finish.” Right click on the newly created Report group, and select “Add Report.” In
the right pane select the icon to the left of “Reports on the Internal Database.” Choose a name
and in the “Type:” field select “IDB Purge Preview Report.” Click “Next.” For “Send method:”
select “Log”, for “Log to file:” select a temp directory and for “Format:” select “ASCII.” Click on
“Add” and then “Finish.” To run the report right click on it and select “Preview.”
Both the preceding points will give an output similar to:
Client # Filename # Est. Obs Est. Durat
________________________________________________________________
alpha.xxxx.com 110 30 23
beta.xxxx.com 481 87 100
delta.xxxx.com 80 0 16
gamma.xxxx.com 0 0 0
zeta.xxxx.com 122 42 25
eta.xxxx.com 17 0 3
theta.xxxx.com 506 134 105
Note:
You either have to be added to the Admin user group or granted the
Reporting, notifications, and Event Log user right.
4
Automated IDB purging done by the Daily Maintenance
What is the Daily Maintenance?
Daily Maintenance runs a series of commands that purge obsolete data from many sections of the DP
IDB. By default, the Daily Maintenance takes place at midday (12:00 Hrs.) each day. It does not
purge all parts of the IDB, just the parts that can be done without exclusive access to the IDB. (For
more information about this, see the Appendix A.)
Setting the time Daily Maintenance runs
The time the Daily Maintenance runs can be changed by the setting Daily Maintenance Time in the
global file.
The Daily Maintenance can be disabled by setting Daily Maintenance Time = 99:99 or None in the
global file.
Note:
Disabling daily maintenance is not recommended by Hewlett Packard,
unless alternative arrangements to maintain the DP IDB have been made.
Methods to confirm if, and when, Daily Maintenance is being run
Method 1
Look at the global file for the values of Daily Maintenance Time:
On Windows the file is (default) C:\Program Files\OmniBack\Config\Server\Options\global
On HP-UX the file is /etc/opt/omni/server/options/global
For example if you only see
# DailyMaintenanceTime=HH:MM daily maintenance is running at 12:00
If you see
DailyMaintenanceTime=99:99 daily maintenance is disabled
DailyMaintenanceTime=None daily maintenance is disabled
DailyMaintenanceTime=08:30 daily maintenance is running at 08:30
Method 2
Run the command omnicc -debug 20 CHECK.txt, and look at the output debug file which will:
• Be in (default)/tmp on HP-UX
• Be in (default) C:\Program Files\OmniBack\tmp on Windows
• Have file name ending with CHECK.txt.
5
Look through the output file for name=DailyMaintenanceTime and see the value:
For example if you see
value=12:00 daily maintenance is running at 12:00 hrs
value=99:99 daily maintenance is disabled
value=None daily maintenance is disabled
value=20:30 daily maintenance is running at 20:30 hrs
If Daily Maintenance is not running
In case Daily Maintenance is disabled consider the following steps:
1. In the global file set a time for the daily maintenance to run.
2. Manually run the daily maintenance with the command omnidbutil –purge –daily.
3. Manually run individual purges that do similar steps to the daily maintenance.
For example
omnidbutil -purge –sessions 30
omnidbutil -purge –messages 10
omnidbutil -purge –dcbf
omnidbutil -purge –mpos
IDB purging done by the Daily Maintenance
Sessions purge
Expired sessions, older than a certain number of days, are removed from an IDB with the command
omnidbutil -purge –sessions [TimeInDays].
For example omnidbutil -purge –sessions 30 removes all expired protection sessions from
the IDB that are more than 30 days old.
Example of messages seen at the command line
omnidbutil -purge -sessions 30
DONE!
Messages purge
Session messages from expired sessions, older than a certain number of days, are removed with
omnidbutil -purge –messages [TimeInDays]
Example of messages seen at the command line
omnidbutil -purge -messages 10
DONE!
6
Dcbf purge
omnidbutil -purge –dcbf removes obsolete data from the dcbf part of the internal database.
Example of messages seen at the command line
omnidbutil -purge -dcbf
DONE!
Mpos purge
Note:
This purge is available by default on DP 6.1, but for DP 5.5 and 6.0 it
requires one of the following patches or their successor.
Windows HP-UX PA Risc HP-UX IA 64
Data Protector 5.5 DPWIN_00396 PHSS_38726 PHSS_38727
Data Protector 6.0 DPWIN_00385 PHSS_38404 PHSS_38405
During 2008 new global file functionality, QuickMediaFormat=1was introduced.
In the course of backups, DP may need to overwrite a media. For tapes containing a large number of
objects and corresponding media positions the purging of these object versions and media position
records within the IDB can put a high and sustained load on the process that manages the DP IDB
(known as RDS). The sustained high load on RDS may give the false impression that DP has hung. This
behavior today is observed more by DP users past because of the large amount of data modern tapes
can hold.
If QuickMediaFormat is set to 1 in the global file, DP will not purge any records during a backup or
when a media is formatted. Those obsolete records will be explicitly removed when a purge mpos
command is run. If the QuickMediaFormat is not set, the records will be purged from the database
during the course of the backup.
The command omnidbutil -purge -mpos can be run at other times to remove all the object
versions and media position records for the overwritten tapes.
The installation of one of the Cell Server patches in the preceding table changes the Daily
Maintenance to include omnidbutil -purge -mpos. So, this purge will take place every day as long as
Daily Maintenance has not been disabled.
Example of messages seen at the command line
omnidbutil -purge -mpos
Starting Purge for formatted medium (-mpos)
Successfully completed Purge for formatted medium (-mpos)
DONE!
7
Manual IDB purge operations
Note:
These commands require Exclusive Access to the IDB, so that no backups
are running or no GUIs are open.
On some very busy Cells it can be difficult to book time when no backups are running. This problem
may be greater if the Cell Manager is part of Manager of Managers (MoM), as all backups have to
be stopped on all Cells in the MoM, in order to run the commands that require exclusive access to the
IDB.
Filenames purge
A filenames purge removes obsolete data from the filenames part of the IDB, but leaves filenames
records referring to protected data intact.
Running a DP IDB filenames purge requires exclusive access to the DP IDB, so no backups can be in
progress and all GUIs must be closed. In a MoM environment, DP must not be busy with backups or
other activities anywhere on the MoM.
Run a filenames purge on a DP Cell Manager as follows:
• Verify that no backups or restores are in progress by checking that “omnistat” returns:
“No currently running session“
• Start the purge:
omnidbutil -purge -filenames -force
Note:
Occasionally it will be necessary to stop (omnisv -stop) and start again
(omnisv -start) DP before the command will run.
If DP is running within Microsoft Cluster Services (MCSC) or
MC/ServiceGuard then omnisv -stop and omnisv -start may not be the
appropriate way to stop and start DP.
You can monitor the filenames purge in two ways:-
• Start one GUI and look in the Monitor context
• Look at the file (default location):
on windows C:\Program Files\OmniBack\log\server\purge.log
on HP-UX /var/opt/omni/server/log/purge.log
See Appendix C for an example of the update to purge.log of a completed filenames purge.
If the purge has not finished, but you need to run backups simply stop the purge with
omnidbutil -purge_stop and use DP as normal.
Later, when you have time, start the purge again as previously explained.
Example of messages seen at the command line
omnidbutil -purge -filenames -force
Filename purge session started.
8
Frequently Asked Questions about filenames purge
Question 1:
Will the force option cause the filenames purge to remove protected data?
Answer 1:
No, a force purge will not remove any protected data from the DP IDB.
A purge without the force option looks for obsolete records for a hostname and only purges those
records if there are more than 1000000 obsolete entries.
The –force switch causes DP to purge all expired records even if there are less than 1000000 expired
records for a hostname. For this reason the command is much more effective if the –force switch is used.
Question 2:
What happens to scheduled backups that are due to start while the omnidbutil -purge filenames is running.
Answer 2:
Scheduled backups will not start if a filename purge is in progress.
The consecutive running of a dcbf Purge followed by a fixmpos
omnidbutil -purge –dcbf, deletes the dcbf records that are no longer required. However, the
mmdb will still have mpos records referring to these removed dcbf records.
Therefore, from time to time we suggest the consecutive running of a dcbf Purge followed by a fixmpos.
omnidbutil -fixmpos requires exclusive access to the IDB, therefore the consecutive running of
these commands must be done when no backups are in progress.
Example of messages seen at the command line
omnidbutil -purge -dcbf
DONE!
omnidbutil -fixmpos
DONE!
Note:-
See Appendix D for information about another manual purge that may, in
some circumstances, be required.
Appendix A
What does “Requires exclusive access to the IDB” mean?
The following three commands mentioned in this document require exclusive access to the IDB
• omnidbutil -purge -filenames -force
• omnidbutil -fixmpos
• omnidbutil -purge_failed_copies
9
To run any of these commands, DP must not be busy doing backups or other activities.
Close all DP GUIs and confirm that no backups, restores or other DP activities are in progress by
checking that ”omnistat“ returns “No currently running session.”
Note:
Exclusive access does not mean that the DP processes must be stopped. But
occasionally it will be necessary to stop and restart them before the
preceding commands can run.
When doing this, it is good practice after stopping DP (omnisv -stop) to look for any remaining DP
processes that may be running and close them before restarting DP (omnisv -start)
On HP-UX proceed as follows:
• Close all DP GUIs
• Make sure DP is not active, omnistat returns ”No currently running sessions”
• Stop DP processes with omnisv -stop
• ps -ef | grep omni
• kill any omni processes
• Start DP with omnisv -start
On Windows proceed as follows:
• Close all DP GUIs
• Make sure DP is not active, omnistat returns ”No currently running sessions”
• Stop DP services with omnisv -stop
• Open Task Manager and go to the Processes tab
• Highlight Data Protector process and “End Process” them
Common DP processes are bsm.exe, bma.exe, Manager.exe, dbsm.exe.
• Start DP with omnisv -start
Note:
If DP is running within Microsoft Cluster Services (MCSC) or
MC/ServiceGuard then “omnisv –stop” and “omnisv –start” may not be the
appropriate way to stop and start DP.
While a command that requires exclusive access to the IDB is running, scheduled DP backups will not start.
If the Cell Manager where you want to run the command requiring exclusive access to the IDB is part
of a Manager of Managers (MoM) you have to stop backups on all Cell Managers that are part of
the MoM.
10
Appendix B
Further information about the IDB Daily Maintenance:
The daily maintenance runs the following commands:
omnidbutil -purge -sessions <days>
omnidbutil -purge -messages <days>
omnidbutil -purge -dcbf
omnidbutil -purge -mpos
The omnidbutil -purge -mpos purge is available by default on DP 6.1, but for DP 5.5 and 6.0
it requires one of the following patches or their successor.
Windows HP-UX PA Risc HP-UX IA 64
Data Protector 5.5 DPWIN_00396 PHSS_38726 PHSS_38727
Data Protector 6.0 DPWIN_00385 PHSS_38404 PHSS_38405
The value for the number of days used by the daily maintenance omnidbutil -purge -
sessions <days> command is determined by the setting of KeepObsoleteSessions in the global
file.
Windows default location of global file: C:\Program Files\OmniBack\Config\Server\Options
HP-UX location of global file: /etc/opt/omni/server/options
The value for the number of days used by the daily maintenance omnidbutil -purge -
messages <days> command is determined by the setting value of KeepMessages in the global file.
The default value for KeepMessages is “0.”
Note:
“0” is a valid value for use in the global file; however, it is invalid at the
command line for example, omnidbutil -purge -messages 0 in not
valid at the command line.
11
Appendix C
Example update to purge.log after a completed forced filenames purge
1/12/2009 2:37:09 PM ASM.5312.4976 ["sm/asm/asm_purge.c /main/dp56/1":470] A.06.00
bDPWIN_00385
Purge session started.
1/12/2009 2:37:09 PM ASM.5312.4976 ["sm/asm/asm_purge.c /main/dp56/1":480] A.06.00
bDPWIN_00385
Filename purge session started.
[REPORT_DATA] 1231771029 1231771052 0 0 6
[REPORT_DATA] 1231771053 1231771067 0 0 6
[REPORT_DATA] 1231771067 1231773602 0 1161430 32
1/12/2009 3:20:02 PM ASM.5312.0 ["db/dc/dcpurge/dcpurge.c
/main/dp56/dp60_fix/2":1167] A.06.00 bQXCM1000887045_TM1
[[[Purge session finished]]]]
Total time: 2573 secs
Total work time: 2573 secs
Total inactivity time: 0 secs
1/12/2009 3:20:02 PM ASM.5312.0 ["db/dc/dcpurge/dcpurge.c
/main/dp56/dp60_fix/2":1175] A.06.00 bQXCM1000887045_TM1
[REPORT_DATA] 1231771029 1231773602 0 1161430 32
1/12/2009 3:20:02 PM ASM.5312.864 ["sm/asm/asm_purge.c /main/dp56/1":237] A.06.00
bDPWIN_00385
Purge session finished.
1/12/2009 3:20:03 PM ASM.5312.4976 ["sm/asm/asm_msg.c /main/13":92] A.06.00
bDPWIN_00385
Filename purge session ended.
12
Appendix D
Failed_copies purge
Note:-
This purge is available by default on DP 6.1, but for DP 5.5 and 6.0 it
requires the following patches or their successor.
Windows HP-UX PA Risc HP-UX IA 64
Data Protector 5.5 DPWIN_00396 PHSS_38726 PHSS_38727
Data Protector 6.0 DPWIN_00385 PHSS_38404 PHSS_38405
In certain circumstances a DP IDB may hold multiple copies of objects made during a backup. These
unrequired copies may overload an IDB. A special purge omnidbutil -purge_failed_copies
removes any copy of objects that are not required.
The preceding patches include two functions with respect to these multiple unrequired copies of
objects, they allow the running of omnidbutil -purge_failed_copies and the correct the
behavior that caused the multiple copies of objects. For these reasons once this command has been
run it should rarely need repeating.
The command requires exclusive access to the IDB so can only be run when no backups are running
and all GUIs are closed.
When run, this purge updates the RDS.log file.
• On Windows the RDS.log file is, by default,
C:\Program Files\OmniBack\db40\datafiles\catalog
• On HP-UX the debug.log file is: /var/opt/omni/log
Example of messages seen at the command line
omnidbutil -purge_failed_copies
DONE!
Example of the update to RDS.log
[19-Dec-2008 15:32:22] overs read[346766] overs deleted[309] mpos deleted
[179431]
13
For more information
• Get Help in the DP GUI.
Help menu, Help Topics. Search tab for searching. Contents tab is the equivalent of an
Administrator’s Guide.
• Chapter 5 of the HP Data Protector Concepts Guide, which is available from the DP GUI, help
menu, Online documentation
• The omnidbutil main page:
On HP-UX this can be accessed by ”man omnidbutil”
On Windows this can be found in the “Command Line Interface Reference,” which is installed (by
default) in C:\Program Files\OmniBack\Docs\MAN
• For DP 5.5, the HP Data Protector Software Release Notes
• For DP 6.0, the Release Notes were renamed to:
– HP Data Protector Product Announcements
– Software Notes
– References
Glossary
Cell
Cell is a set of systems that are under the control of a Cell Manager. The cell typically represents the
systems on a site or an organizational entity, which are connected to the same LAN. Central control is
available to administer the backup and restore policies and tasks. A cell consists of a Cell Manager,
one or more Installation Servers, client systems, and devices.
Cell Manager
The main system in the cell where the essential Data Protector software is installed and from which all
backup, restore, copy, and administrative activities are managed. The GUI used for management
tasks can be located on a different system. Each cell has one Cell Manager system and it can also be
referred to as Cell Server (CS).
CDB
The Catalog Database (CDB) is a part of the IDB that contains information about backups, object
copies, restores, media management sessions, and backed up data. Depending on the selected
logging level, it also contains file names and file versions. This part of the database is always local to
the cell. This part of the database is always local.
DCBF
The Detail Catalog Binary Files (DCBF) part of the IDB stores information about file versions and
attributes. It occupies approximately 80% of the IDB. One DC binary file is created for each Data
Protector medium used for backup.
DC directory
The Detail Catalog (DC) directory consists of DC binary files, which store information about file
versions. It represents the DCBF part of the IDB, which occupies approximately 80% of the IDB. The
default DC directory is called the dcbf directory and is located in the Data_Protector_home>\db40
directory on a Windows Cell Manager and in the /var/opt/omni/server/db40 directory on a UNIX
Cell Manager. You can create more DC directories and locate them as appropriate to you. Up to 50
DC directories are supported per cell. The default maximum size of a DC directory is 4 GB.
14
fnames.dat
The fnames.dat files of the IDB contain information on the names of the backed up files. Typically,
these files occupy about 20% of the IDB. Filenames, for reasons of space efficiency and quick access,
are stored in pieces across six files (and their extension file, if they exist): dirs.dat, fn1.ext, fn2.ext,
fn3.ext, fn4.ext and fnames.dat.
Graphical User Interface (GUI)
A cross-platform [Windows, HP-UX (PA Risc), Solaris] graphical user interface provided by Data
Protector for easy access to all configuration, administration, and operation tasks. It may also be
known as User Interface (UI) or Cell Console (CC).
IDB
The Data Protector Internal Database is an embedded database located on the Cell Manager that
keeps information regarding which data is backed up, on which media it is backed up, how backup
and restore sessions are run, and which devices and libraries are configured.
MoM
Several cells can be grouped together and managed from a central cell. The management system of
the central cell is the Manager-of-Managers (MoM). The cells are called MoM clients. The MoM
enables you to configure and manage multiple cells from a central point.
MMDB
The Media Management Database (MMDB) is a part of the IDB that contains information about
media, media pools, devices, libraries, library drives, and slots configured in the cell, as well as the
Data Protector media used for backup.
RDS
The Raima Database Server process (service) runs on the Data Protector Cell Manager and manages
the IDB. The process is started when Data Protector is installed on the Cell Manager.
SIBF
The Serverless Integrations Binary Files (SIBF) is a part of the IDB that stores raw NDMP metadata.
This data is necessary to perform restore of NDMP objects. Occasionally the SIBF may be used for
other data too.
SMBF
The Session Messages Binary Files (SMBF) part of the IDB stores session messages generated during
backup, restore, object copy, object consolidation, and media management sessions. One binary file
is created per session. The files are grouped by year and month.
Technology for better business outcomes
© Copyright 2009 Hewlett-Packard Development Company, L.P. The information
contained herein is subject to change without notice. The only warranties for HP
products and services are set forth in the express warranty statements accompanying
such products and services. Nothing herein should be construed as constituting an
additional warranty. HP shall not be liable for technical or editorial errors or omissions
contained herein.
4AA2-4988ENW, March 2009
Loading...