SAP Business objects DATA SERVICES Advanced Development Guide

Data Services Advanced Development Guide
BusinessObjects Data Services XI 3.1 (12.1.1)
Copyright
© 2008 Business Objects, an SAP company. All rights reserved. Business Objects owns the following U.S. patents, which may cover products that are offered and licensed by Business Objects: 5,295,243; 5,339,390; 5,555,403; 5,590,250; 5,619,632; 5,632,009; 5,857,205; 5,880,742; 5,883,635; 6,085,202; 6,108,698; 6,247,008; 6,289,352; 6,300,957; 6,377,259; 6,490,593; 6,578,027; 6,581,068; 6,628,312; 6,654,761; 6,768,986; 6,772,409; 6,831,668; 6,882,998; 6,892,189; 6,901,555; 7,089,238; 7,107,266; 7,139,766; 7,178,099; 7,181,435; 7,181,440; 7,194,465; 7,222,130; 7,299,419; 7,320,122 and 7,356,779. Business Objects and its logos, BusinessObjects, Business Objects Crystal Vision, Business Process On Demand, BusinessQuery, Cartesis, Crystal Analysis, Crystal Applications, Crystal Decisions, Crystal Enterprise, Crystal Insider, Crystal Reports, Crystal Vision, Desktop Intelligence, Inxight and its logos , LinguistX, Star Tree, Table Lens, ThingFinder, Timewall, Let There Be Light, Metify, NSite, Rapid Marts, RapidMarts, the Spectrum Design, Web Intelligence, Workmail and Xcelsius are trademarks or registered trademarks in the United States and/or other countries of Business Objects and/or affiliated companies. SAP is the trademark or registered trademark of SAP AG in Germany and in several other countries. All other names mentioned herein may be trademarks of their respective owners.
Third-party Contributors
Business Objects products in this release may contain redistributions of software licensed from third-party contributors. Some of these individual components may also be available under alternative licenses. A partial listing of third-party contributors that have requested or permitted acknowledgments, as well as required notices, can be found at: http://www.businessobjects.com/thirdparty
2008-11-28

Contents

Introduction 7Chapter 1
Welcome to Data Services..........................................................................8
Overview of this guide...............................................................................14
Migration Basics 17Chapter 2
Development phases.................................................................................18
Migration mechanisms and tools...............................................................20
Welcome................................................................................................8
Documentation set for Data Services.....................................................8
Accessing documentation....................................................................11
Business Objects information resources..............................................12
About this guide....................................................................................14
Who should read this guide..................................................................15
About environment migration...............................................................15
Design phase.......................................................................................19
Test phase............................................................................................19
Production phase.................................................................................20
Which mechanism is best?...................................................................21
Export/import migration........................................................................23
Multi-user migration..............................................................................23
Preparing for Migration 25Chapter 3
Naming conventions for migration.............................................................26
Connections to external datastores......................................................27
Directory locations................................................................................29
Schema structures and owners............................................................29
Data Services Advanced Development Guide 3
Contents
Datastore and system configurations........................................................30
Datastore configurations and migration...............................................31
Multiple configurations in multi-user environments..............................33
Command line login to the Designer.........................................................34
Export/Import 37Chapter 4
Exporting/importing objects in Data Services............................................38
The Export editor..................................................................................39
Exporting objects to another repository................................................42
Exporting objects to a file.....................................................................43
Exporting a repository to a file..............................................................44
Importing from a file..............................................................................45
Command line options to export objects to an XML file.......................46
Removing obsolete repository contents....................................................48
To compact your repository by creating a new repository....................48
Backing up repositories.............................................................................49
Maintaining Job Server performance.........................................................49
Multi-user Development 51Chapter 5
Central versus local repository..................................................................52
Data Services and multiple users..............................................................53
Security and the central repository............................................................56
Multi-user Environment Setup 57Chapter 6
Create a nonsecure central repository......................................................58
To create a nonsecure central repository.............................................58
Define a connection to a nonsecure central repository.............................59
To define a connection to a central repository......................................59
Activating a central repository....................................................................60
To activate a central repository.............................................................60
4 Data Services Advanced Development Guide
Contents
To open the central object library.........................................................61
To change central repository connections............................................62
Implementing Central Repository Security 65Chapter 7
Overview....................................................................................................66
Group-based permissions....................................................................66
Permission levels.................................................................................67
Process summary.................................................................................67
Creating a secure central repository..........................................................68
To create a secure central repository...................................................68
To upgrade a central repository from nonsecure to secure..................69
Adding a multi-user administrator (optional)..............................................69
Setting up groups and users......................................................................70
Defining a connection to a secure central repository.................................70
To define a connection to a secure central repository..........................70
Working with objects in a secure central repository..................................72
Viewing and modifying permissions.....................................................72
Working in a Multi-user Environment 75Chapter 8
Filtering......................................................................................................76
Adding objects to the central repository....................................................78
To add a single object to the central repository....................................78
To add an object and its dependent objects to the central repository...79
Checking out objects.................................................................................79
Check out single objects or objects with dependents..........................80
Check out single objects or objects with dependents without
replacement..........................................................................................82
Check out objects with filtering.............................................................83
Undoing check out.....................................................................................84
To undo single object check out...........................................................84
Data Services Advanced Development Guide 5
Contents
To undo check out of an object and its dependents.............................85
Checking in objects....................................................................................85
Checking in single objects, objects with dependents...........................86
Checking in an object with filtering.......................................................87
Labeling objects.........................................................................................88
To label an object and its dependents..................................................90
Getting objects...........................................................................................90
To get a single object............................................................................90
To get an object and its dependent objects..........................................91
To get an object and its dependent objects with filtering......................91
To get a previous version of an object..................................................92
To get an object with a particular label.................................................92
Comparing objects.....................................................................................92
Viewing object history................................................................................92
To examine the history of an object......................................................92
To get a previous version of an object..................................................94
To get an object with a particular label.................................................95
Deleting objects.........................................................................................95
Migrating Multi-user Jobs 97Chapter 9
Application phase management................................................................98
Copying contents between central repositories.......................................100
To copy the contents of one central repository to another central
repository............................................................................................100
Central repository migration.....................................................................101
Index 103
6 Data Services Advanced Development Guide

Introduction

1
Introduction
1

Welcome to Data Services

Welcome to Data Services

Welcome

Data Services XI Release 3 provides data integration and data quality processes in one runtime environment, delivering enterprise performance and scalability.
The data integration processes of Data Services allow organizations to easily explore, extract, transform, and deliver any type of data anywhere across the enterprise.
The data quality processes of Data Services allow organizations to easily standardize, cleanse, and consolidate data anywhere, ensuring that end-users are always working with information that's readily available, accurate, and trusted.

Documentation set for Data Services

You should become familiar with all the pieces of documentation that relate to your Data Services product.
What this document providesDocument
Documentation Map
Release Summary
Release Notes
Getting Started Guide
Installation Guide for Windows
8 Data Services Advanced Development Guide
Information about available Data Services books, languages, and locations
Highlights of key features in this Data Services re­lease
Important information you need before installing and deploying this version of Data Services
An introduction to Data Services
Information about and procedures for installing Data Services in a Windows environment.
Introduction
Welcome to Data Services
What this document providesDocument
1
Installation Guide for UNIX
Advanced Development Guide
Designer Guide
Integrator's Guide
Management Console: Administrator Guide
Management Console: Metadata Re­ports Guide
Migration Considerations Guide
Information about and procedures for installing Data Services in a UNIX environment.
Guidelines and options for migrating applications in­cluding information on multi-user functionality and the use of the central repository for version control
Information about how to use Data Services Designer
Information for third-party developers to access Data Services functionality. Also provides information about how to install, configure, and use the Data Services Adapter for JMS.
Information about how to use Data Services Adminis­trator
Information about how to use Data Services Metadata Reports
Information about:
Release-specific product behavior changes from
earlier versions of Data Services to the latest re­lease
How to migrate from Data Quality to Data Services
Performance Optimization Guide
Reference Guide
Information about how to improve the performance of Data Services
Detailed reference material for Data Services Design­er
Data Services Advanced Development Guide 9
Introduction
1
Welcome to Data Services
Technical Manuals
What this document providesDocument
A compiled “master” PDF of core Data Services books containing a searchable master table of contents and index:
Getting Started Guide
Installation Guide for Windows
Installation Guide for UNIX
Designer Guide
Reference Guide
Management Console: Metadata Reports Guide
Management Console: Administrator Guide
Performance Optimization Guide
Advanced Development Guide
Supplement for J.D. Edwards
Supplement for Oracle Applications
Supplement for PeopleSoft
Supplement for Siebel
Supplement for SAP
Tutorial
A step-by-step introduction to using Data Services
In addition, you may need to refer to several Adapter Guides and Supplemental Guides.
What this document providesDocument
Salesforce.com Adapter Interface
Supplement for J.D. Ed­wards
Supplement for Oracle Ap­plications
Supplement for PeopleSoft
10 Data Services Advanced Development Guide
Information about how to install, configure, and use the Data Services Salesforce.com Adapter Interface
Information about license-controlled interfaces between Data Services and J.D. Edwards World and J.D. Edwards OneWorld
Information about the license-controlled interface between Data Services and Oracle Applications
Information about license-controlled interfaces between Data Services and PeopleSoft
Introduction
Welcome to Data Services
What this document providesDocument
1
Supplement for SAP
Supplement for Siebel
Information about license-controlled interfaces between Data Services, SAP ERP, and SAP BI/BW
Information about the license-controlled interface between Data Services and Siebel

Accessing documentation

You can access the complete documentation set for Data Services in several places.
Accessing documentation on Windows
After you install Data Services, you can access the documentation from the Start menu.
1. Choose Start > Programs > BusinessObjects XI 3.1 >
BusinessObjects Data Services > Data Services Documentation.
Note:
Only a subset of the documentation is available from the Start menu. The documentation set for this release is available in LINK_DIR\Doc\Books\en.
2. Click the appropriate shortcut for the document that you want to view.
Accessing documentation on UNIX
After you install Data Services, you can access the online documentation by going to the directory where the printable PDF files were installed.
1. Go to LINK_DIR/doc/book/en/.
2. Using Adobe Reader, open the PDF file of the document that you want
to view.
Data Services Advanced Development Guide 11
Introduction
1
Welcome to Data Services
Accessing documentation from the Web
You can access the complete documentation set for Data Services from the Business Objects Customer Support site.
1.
Go to http://help.sap.com.
2. Cick Business Objects at the top of the page.
You can view the PDFs online or save them to your computer.

Business Objects information resources

A global network of Business Objects technology experts provides customer support, education, and consulting to ensure maximum business intelligence benefit to your business.
Useful addresses at a glance:
ContentAddress
12 Data Services Advanced Development Guide
Introduction
Welcome to Data Services
ContentAddress
1
Customer Support, Consulting, and Education services
http://service.sap.com/
Data Services Community
https://www.sdn.sap.com/irj/sdn/businessob jects-ds
Forums on SCN (SAP Community Network)
https://www.sdn.sap.com/irj/sdn/businessob jects-forums
Blueprints
http://www.sdn.sap.com/irj/boc/blueprints
Information about Customer Support programs, as well as links to technical articles, downloads, and online forums. Consulting services can provide you with information about how Busi­ness Objects can help maximize your business intelligence investment. Education services can provide information about training options and modules. From traditional classroom learning to targeted e-learning seminars, Business Ob­jects can offer a training package to suit your learning needs and preferred learning style.
Get online and timely information about Data Services, including tips and tricks, additional downloads, samples, and much more. All con­tent is to and from the community, so feel free to join in and contact us if you have a submis­sion.
Search the Business Objects forums on the SAP Community Network to learn from other Data Services users and start posting questions or share your knowledge with the community.
Blueprints for you to download and modify to fit your needs. Each blueprint contains the neces­sary Data Services project, jobs, data flows, file formats, sample data, template tables, and custom functions to run the data flows in your environment with only a few modifications.
Data Services Advanced Development Guide 13
Introduction
1

Overview of this guide

http://help.sap.com/
ContentAddress
Business Objects product documentation.Product documentation
Documentation mailbox
documentation@businessobjects.com
Supported platforms documentation
https://service.sap.com/bosap-support
Send us feedback or questions about your Business Objects documentation. Do you have a suggestion on how we can improve our docu­mentation? Is there something that you particu­larly like or have found useful? Let us know, and we will do our best to ensure that your suggestion is considered for the next release of our documentation.
Note:
If your issue concerns a Business Objects product and not the documentation, please contact our Customer Support experts.
Get information about supported platforms for Data Services.
In the left panel of the window, navigate to Documentation > Supported Platforms > BusinessObjects XI 3.1. Click the Busines­sObjects Data Services link in the main win­dow.
Overview of this guide

About this guide

The guide contains advanced development information. Topics include:
Migration—How to move your projects to different development
environments
Multi-user development—How to manage a project developed by multiple
users
14 Data Services Advanced Development Guide
You will find this guide most useful:
After you have learned product basics
While planning the design, test, and production phases of your
data-movement projects
As an advanced source of information during any phase of your projects

Who should read this guide

This and other Data Services product documentation assumes the following:
You are an application developer, consultant, or database administrator
working on data extraction, data warehousing, or data integration.
You understand your source data systems, RDBMS, business intelligence,
and messaging concepts.
You understand your organization's data needs.
You are familiar with SQL (Structured Query Language).
Introduction
Overview of this guide
1
If you are interested in using this product to design real-time processing,
you are familiar with:
DTD and XML Schema formats for XML files
Publishing Web Services (WSDL, HTTP, and SOAP protocols, etc.)
You are familiar Data Services installation environments—Microsoft
Windows or UNIX.
Further, Business Objects recommends that you review both the Data Services Getting Started Guide and the Data Services Designer Guide before using advanced concepts in this document.

About environment migration

One of the most powerful aspects of Data Services is its architectural flexibility through development, test, and production environments. Data Services is designed to support various configurations including large enterprises with many developers working on multiple projects. Data Services supports multi-site architectures whether centralized or not.
Data Services Advanced Development Guide 15
Introduction
1
Overview of this guide
This guide discusses architectural options for implementing Data Services in development, test, and production environments.
Related Topics
Migration Basics on page 17
Preparing for Migration on page 25
Export/Import on page 37
Multi-user Development on page 51
Multi-user Environment Setup on page 57
Working in a Multi-user Environment on page 75
Migrating Multi-user Jobs on page 97
16 Data Services Advanced Development Guide

Migration Basics

2
Migration Basics
2

Development phases

About this section
Migration as it relates to Data Services is the process of moving applications through multiple development phases into production. Data Services supports simple and complex application migration through all phases into production.
Related Topics
Development phases on page 18
Migration mechanisms and tools on page 20
Development phases
The ETL application development process typically involves three distinct phases:
Design phase
Test phase
Production phase
You can use Data Services in all three phases. Because each phase might require a different repository to control environment differences, Data Services provides controlled mechanisms for moving objects from phase to phase.
Each phase could involve a different computer in a different environment with different security settings. For example, design and initial test may only require limited sample data and low security, while final testing may require a full emulation of the production environment including strict security.
18 Data Services Advanced Development Guide

Design phase

In this phase, you define objects and build diagrams that instruct Data Services in your data movement requirements. Data Services stores these specifications so you can reuse them or modify them as your system evolves.
Design your project with migration to testing and final production in mind. Consider these basic guidelines as you design your project:
Construct design steps as independent, testable modules.
Use meaningful names for each step you construct.
Make independent modules that can be used repeatedly to handle
common operations.
Use test data that reflects all the variations in your production data.

Test phase

Migration Basics
Development phases
2
In this phase, you use Data Services to test the execution of your application. At this point, you can test for errors and trace the flow of execution without exposing production data to any risk. If you discover errors during this phase, return the application to the design phase for correction, then test the corrected application.
Testing has two parts:
The first part includes designing the data movement using your local
repository.
The second part includes fully emulating your production environment,
including data volume.
Data Services provides feedback through trace, error, and monitor logs during both parts of this phase.
The testing repository should emulate your production environment as closely as possible, including scheduling jobs rather than manually starting them.
Data Services Advanced Development Guide 19
Migration Basics
2

Migration mechanisms and tools

Production phase

In this phase, you set up a schedule in Data Services to run your application as a job. Evaluate results from production runs and when necessary, return to the design phase to optimize performance and refine your target requirements.
After moving a Data Services application into production, monitor it in the Administrator for performance and results. During production:
Monitor your jobs and the time it takes for them to complete.
The trace and monitoring logs provide information about each job as well as the work flows and data flows contained within the job.
You can customize the log details. However, the more information you request in the logs, the longer the job runs. Balance job run-time against the information necessary to analyze job performance.
Check the accuracy of your data.
To enhance or correct your jobs:
Make changes in your design environment.
Repeat the object testing.
Move changed objects back into production.
Migration mechanisms and tools
Data Services provides two migration mechanisms:
Export/import migration works best with small to medium-sized projects
where a small number of developers work on somewhat independent Data Services applications through all phases of development.
Multi-user development works best in larger projects where two or more
developers or multiple teams are working on interdependent parts of Data Services applications through all phases of development.
Regardless of which migration mechanism you choose, Business Objects recommends you prepare for migration using one or more tools that best fit
20 Data Services Advanced Development Guide
your development environment for more information). The mechanism and tools you use will depend on the needs of your development environment.
If your source data will come from multiple, homogeneous systems, Business Objects recommends you use Datastore and system configurations tools.
When migrating applications in a multi-user environment, Business Objects strongly recommends you use Naming conventions for migration.
Related Topics
Export/import migration on page 23
Multi-user Development on page 51
Preparing for Migration on page 25
Datastore and system configurations on page 30
Designer Guide: Datastores, Creating and managing multiple datastore
configurations
Naming conventions for migration on page 26

Which mechanism is best?

Migration Basics
Migration mechanisms and tools
2
Although Data Services supports a multi-user environment, you may not need to implement this architecture on all projects. If your project is small to medium in size and only consists of one or two developers, then a Central Repository may not be a necessary solution to integrating the work of those developers.
For example, only two consultants worked on a certain HR data mart application. The Development system was designed so that while Consultant 1 managed the Master Repository, Consultant 2 worked on a new section within a complete copy of the Master Repository.
Consultant 2 then exported this new section back into the Master Repository using the export utility that allows objects to be 'Created', 'Replaced', or 'Ignored'. After updating the Master Repository, Consultant 2 took a new complete copy of the Master Repository, overwriting the previous copy.
Use the following matrix to help you determine which mechanism and tools would work best in your environment.
Data Services Advanced Development Guide 21
Migration Basics
2
Migration mechanisms and tools
Situation/re quirements
Small to medium-sized project
Export/im port
Multi-user
ToolsMigration Mechanisms
Naming con­ventions
Configura tions
OOX
Multiple-team project
Source data from multiple, homoge­neous sys­tems
Different source or tar­get database among envi­ronments
Need a "fast and easy" mi­gration solu­tion
Optimal solution: X Compatible solution: O
OXX
XX
XX
XO
22 Data Services Advanced Development Guide

Export/import migration

Export/import is the basic mechanism for migrating Data Services applications between phases. First, you export jobs from the local repository to another local repository or to an intermediate file which you can then import into another local repository. For example, when moving from design repository to test repository, you export from the design repository to a file, then import the file to your test repository.
Migration Basics
Migration mechanisms and tools
2
If you find application errors during testing, you can correct them in the development environment, then export the corrected version and import it back into the test repository for retesting.
Related Topics
Export/Import on page 37

Multi-user migration

You can also migrate Data Services applications between phases in more complex development environments. Instead of exporting and importing applications, multi-user development provides a more secure check-in, check-out, and get mechanism, using a central repository to store the master copies of your application elements. Multi-user development includes other advanced features like labeling and filtering to provide you more flexibility and control in managing application objects.
Data Services Advanced Development Guide 23
Migration Basics
2
Migration mechanisms and tools
Related Topics
Migrating Multi-user Jobs on page 97
24 Data Services Advanced Development Guide

Preparing for Migration

3
Preparing for Migration
3

Naming conventions for migration

About this section
Before you develop Data Services applications, Business Objects recommends that you first set up a comprehensive structure to facilitate the migration process between development phases.
This section discusses tools that can help you build your migration structure.
Business Objects recommends that you implement standardized naming conventions for connectivity between computer systems. Add datastore and system configurations to more easily work with multiple homogeneous systems.
Related Topics
Naming conventions for migration on page 26
Datastore and system configurations on page 30
Naming conventions for migration
The best way to ensure fast and seamless migration is to use common naming conventions across all systems and phases of all your development environments.
Just as Business Objects recommends you standardize object prefixes, suffixes, and path name identifiers to simplify your projects internally, we also recommend the use of naming conventions externally for migration purposes.
To ease migration, use common naming conventions for:
Connections to external datastores
Directory locations
Schema structures and owners
You want to make it as quick and easy as possible to migrate applications between users and between phases. This translates to significantly reducing or eliminating time spent reconfiguring your jobs to work in each specific environment.
While the actual data you are extracting, transforming, and loading usually differs by database, the essential structure of the data should be the same
26 Data Services Advanced Development Guide
on every database with which you want the same applications to work. Therefore, it makes the most sense to standardize your database naming and structuring before starting the development process.
Related Topics
Designer Guide: Projects and Jobs, Naming conventions for objects in
jobs
Connections to external datastores on page 27
Directory locations on page 29
Schema structures and owners on page 29

Connections to external datastores

In Data Services, migration is the process of moving objects between local repositories, whether directly using the Export/Import method or indirectly using the Multi-user development method. Regardless of method, you must consider how the migration will impact connection configurations associated with your jobs.
Preparing for Migration
Naming conventions for migration
3
Using generic naming for similar external datastore connections reduces the time you spend on reconfiguring the connections to the same database type. For example, you should choose the same logical name for all your Oracle datastore connections to the same type of database structure regardless of migration phase environment.
You can make connection names meaningful to a certain phase and specific computer system names (Test_DW, Dev_DW, Prod_DW), however if you choose this naming structure, Business Objects recommends that you use datastore configurations for migration purposes.
Test phaseDevelopment phase
User name: Test_DWUser name: Dev_DW
Password: Test_DWPassword: Dev_DW
Host String: Test_DWHost String: Dev_DW
Data Services Advanced Development Guide 27
Preparing for Migration
3
Naming conventions for migration
For a job to run against Test and Development, it would have to use Test_DW and Dev_DW and this would require you to create different datastore configurations for when the job runs against the Test or the Dev instance, respectively.
Alternatively, you could call the connection string DW and regardless of what instance you ran the job against, it would run without users having to create multiple datastore configurations.
Test PhaseDevelopment Phase
Database A
Examples:
There is one Oracle source system in your company that processes order
entry data. Multiple instances of this system exist for development, test, and production purposes. Therefore, you name the connection string to your Oracle source system "ORDER_SYSTEM". Then in all phases, you configure that name to point to the correct (phase-specific) instance of the system.
Name the connection string to your target data warehouse "DW" then
point it to different databases depending on whether you are in the development, test, or production environment.
Datastore Con­nection
Database B
Datastore Con­nection
User name: DWUser name: DWUser name: DWUser name: DW
Password: DWPassword: DWPassword: DWPassword: DW
Owner name: DWHost String: DWOwner name: DWHost string: DW
When you use this generic, cross-phase naming method, you cannot access both dev and test from the same computer (since the connection string maps only to one instance). If you require access to both, use multiple datastore configurations.
28 Data Services Advanced Development Guide
Related Topics
Export/Import on page 37
Multi-user Development on page 51

Directory locations

Business Objects recommends you use logical directory names (for example,
X:\) or point to common local drives to standardize directory location. For
example, since every computer has a C:\ drive, pointing to the directory location, C:\TEMP would be a safe, reproducible standard.

Schema structures and owners

To further facilitate a seamless structure between development phases, give all your database instances the same owner name for the same schema structures from which you are reading and to which you are loading. Regardless of name, the owner of each schema structure can vary and Data Services will reconcile them.
Preparing for Migration
Naming conventions for migration
3
Data Services Advanced Development Guide 29
Preparing for Migration
3

Datastore and system configurations

Datastore and system configurations
Datastore and system configurations are powerful tools for reducing the configurations required to execute the same logic against different datastore environments. With configurations, migration between development phases becomes faster and more simplified.
Related Topics
Datastore configurations and migration on page 31
Multiple configurations in multi-user environments on page 33
30 Data Services Advanced Development Guide
Loading...
+ 76 hidden pages