On-Demand Mapping for Business Users - Embedded in an Application

For a top level look at a real-world solution that includes a basic on-demand mapper for business users that leverages the engine API, click here to review the Steve Woods ppt + audio about Eloqua Software and embedded data integration. For an in-depth technical explanation and sample code for this solution, see below the diagram.

Eloqua has named its on-demand mappers "Dynamic Mapper Components" (see reference in diagram).

Leveraging the Pervasive Integration Engine API: Basic On-Demand Mapper

Contributor: Julie Hunt, Julie Hunt Consulting

Overview of Approaches to Creating Transformation Maps

Developers working with the Pervasive Data Integration Platform design tools are well-acquainted with the broad range of capabilities that they provide for any data integration challenges. Besides saving time by using Pervasive design tools, developers don't have to "reinvent the wheel" every time they do a new project, due to the many repeatable functions and features in the products. The Pervasive Data Integration Platform maintains project components in one place, documented and managed for control and ease of use.

But sometimes developers face data integration challenges that may have to be solved without the design tools interface. While the majority of data transformations are accomplished by executing transformation maps created in the Map Designer, there are situations in which the only method to accomplish a specific transformation is to write code outside of the design platform. The code can then leverage the powerful resources of the Pervasive Integration Engine API to perform transformation tasks.

In this article a real-world example of leveraging the Integration Engine API, in conjunction with Java code, will be detailed. The example covers the creation of a basic on-demand mapper that is set up in conjunction with any UI and is used to build Pervasive maps (map.xml, tf.xml). Since this mapper is intended for business users as the end users, the best UI to develop is a wizard-like interface. The maps created are simple, without complex transformation parameters. The engine API is integral to providing source and target connections and metadata acquisition. This article presents what is needed in Java code to complete these tasks.

The Pervasive Integration Engine

The Pervasive Integration Engine is a dedicated execution engine with no user interface components. This allows it to be invoked to perform automatic runtime data transformations quickly and easily. The Integration Engine API supports seamless integration with other programs and extensive control of transformation processes by running maps and processes created in Map Designer and Process Designer.

The API provides for launching transformation processing directly from any "calling" program (Java, .NET or other environments). In addition to execution of pre-designed transformations, it may be used to automatically create, map and run default transformations. The functionality provided by the API allows for tight control of connection parameters and data manipulation, including error handling, on-the-fly filtering and transformation overrides.

Engine SDK - Java and COM

Documentation is provided for the Engine SDK with the installation of Pervasive Data Integrator™. The SDK outlines development in either Java or COM environments. For most data integration projects, the Engine API is leveraged for runtime overrides of parameters in maps and processes that have been created with the Pervasive design tools.

Prerequisites for Working with the Basic On-Demand Mapper Sample in this Article

Skills:

• Strong knowledge of the Pervasive Data Integration Platform
  • Map Designer
  • Maps and their properties as created by the Map Designer (including map.xml and tf.xml created for each map)
  • Runtime engine and its API
  • Runtime deployment options for maps after they have been created
• Java development experience
  
• UI development and usage experience
• Knowledge of source and target data formats, structures and fields

Products:

• Pervasive Data Integrator installed (access to supporting documentation and transformation objects)
• Pervasive runtime engine with a configuration appropriate for the solution platform

A Basic On-Demand Mapper

The most obvious use for a basic on-demand mapper that creates maps without the Pervasive Data Integration design tools is for applications that enable business users to perform mapping activities through a wizard-like UI. This could be for migrating data from one application to another, or to maintain synchronization between two applications. This sort of mapper is useful for applications that need to create on-the-fly maps on a frequent basis where the business user knows the source and target data well enough to be able to easily perform the mapping.

This basic on-demand mapper is best suited for situations that are based on tightly defined data mapping without extensive transformation needs. This mapper is best used as a component of an overall solution where the component performs very specific mapping tasks. The small footprint of the Integration Engine makes it quite appealing as an embedded component in a solution or application.

The mapper described in this article is based on a combination of Java code and the functionality available with the Integration Engine API that allows developers to build out source and target connections, metadata, map and transformation tasks. There are no constraints on the type of UI that a developer could use, as long as it works well for the solution. If the UI is mainly for business users, then it should be simple and easily used.

Data mapping and connectivity could be bi-directional or uni-directional, depending on project requirements. There are no limits on the number of fields that can be acquired from connections to a source and target, except for the usual constraints (i.e., memory). But if the purpose is to provide a business user a wizard-like UI for mapping, then it makes sense to limit the number of fields handled by the on-demand mapper UI.

Once a map has been created and saved, the map is saved to disk and may then be run immediately or run at another time via any scheduler or automation options available to a developer.

Sample code has been provided as a download at the end of the article to illustrate one approach to a basic on-demand mapper that leverages the Pervasive Integration Engine API.

COMPONENTS OF THE SAMPLE SOLUTION

Connecting to Sources and Targets
When developing a UI for an on-demand mapper, the developer will need to understand all parameters that are required by the Integration Engine to connect to source and target data. Some parameters may be stored in a file that is accessed by the mapper code; some or all may be entered dynamically through the UI. Parameters could be tied to a specific login provided by the business user. Again, the developer has many options for providing information that is required to make source and target connections. The sample code shows examples of how parameters are used with the engine API.

While the only call to the engine API is to instantiate "DataJunction.ec," there is extensive coding to be done in Java to work with source and target data. To connect to sources and targets through the Integration Engine, the developer will write a Java wrapper around each adapter to be used, implementing the engine SDK. The wrapper code is detailed but not difficult to write.

Developers should expect complexity when coding wrappers around the adapters, so a careful read-through of the engine API documentation is highly recommended. The sample code download provides more detail on writing wrappers around Pervasive Integration Engine adapters.

Acquiring Metadata from Sources and Targets
Once connectivity has been established to source and target, the Java code must acquire source and target metadata. The Pervasive Integration Engine does not make the metadata directly available to the Java code. The metadata is held in memory, so code is written to read the metadata from memory and save it to disk. The saved metadata is in the form of a partial map or map.xml object, containing source and target connection information as well as the metadata for source and target data. The Java code then opens the partially populated map.xml to parse out the metadata. The extracted metadata is used to populate metadata for source and target in the UI, for the business user.

The sample Java code walks the developer through the methods used to perform the metadata acquisition steps.

Mapping in the UI
The business user must have a good understanding of the source and target data fields to properly perform the mapping in the UI. Additionally, the mapping in the UI should be simple and intuitive.

Once mapping is completed, the business user will click on a "Save" command to enact Java code to save the now fully-completed map.xml to disk. The sample code shows the new map.xml overwriting the original partially completed map.xml that was saved to disk by the Java code after acquiring it from memory. The Save command also invokes Java code to create the transformation object, "tf.xml," in order to run the transformation map with the integration engine.

The newly created map may be viewed in the Pervasive Map Designer to assist with the development phase, or may be viewed as an xml document if the developer is well familiar with the valid structure of map.xml and tf.xml.

Running Completed Maps
Completed maps comprised of valid map.xml and tf.xml can be run any number of ways. Code can be written to run the map immediately. Or the saved map objects can be run on a scheduled or automated basis, accomplished by any of the ways a developer would like to run an automated task. No matter what method is selected, running the maps created by the on-demand mapper will require the Integration Engine.

UML diagram
A UML diagram can be downloaded here. The UML diagram describes the classes used in the sample Java code for the basic on-demand mapper. The UML diagram presents the objects and their relationships to help developers understand and adapt the sample Java code to their projects.

Benefits of Creating a Basic On-Demand Mapper with Java and the Engine API
The most likely scenarios that benefit from the on-demand mapper include:

• Embedding in another application where a small footprint is important
• Map creation by business users (typically, high frequency scenarios)
• Migrating data from one application to another
• Synchronizing data between applications
• Need for customized UI, particularly for business users
• Need to reduce developer or IT involvement in some map creation and running activities

The over-arching benefit of combining Java code with the engine API is the opening of opportunities for runtime overrides, which provide greater control and flexibility for transformations and processes.

Limitations of This Approach
Using a mapper to create maps without the Pervasive Data Integration design tools has limitations:

• Very limited functionality compared to the breadth and depth of design possibilities available in the Map Designer and Process Designer
• Should always keep in mind that this is a work-around (but a very useful one) that works through the engine API
• This mapper does not produce the more complex maps that can be created with the sophisticated design tools (Map Designer and Process Designer)
• Limited access to metadata due to inherent function of the engine API
• Can't "see" database schemas or other metadata as can be done in the Map Designer
• Limited by how much a business user understands about mapping data from source to target
• This approach will only satisfy a narrow need and is not a universal approach to creating maps

Download the Java code for the Basic On-Demand Mapper

Click on this link to download the full code sample.

A simple example for deployment of the Mapper is available in the Stub class in the code sample.

With the proliferation of interoperability utilities for mixed development environments, developers should be able to work with the Java code sample, whether developing in Java or .NET. Additionally, the Java code could be re-developed in .NET.

Other Methods of Creating & Running Maps and Transformations: Message Component Framework (MCF)

Overview of Message Component Framework

The Message Component Framework is a design feature of the Process Designer for creating components to be included as "plug-ins" in processes. MCF components can only be run in a process. The MCF acts as a "container"service within the Integration Engine.

The components are developed in Java. Each different component lists variable parameters that provide further definition of what the component performs in a process. Frequently these parameters contain source and target connection information.

Components are built using the MCF creation tool, which is based on Eclipse. It provides a set of wizards and editors that automate the development process. Documentation for MCF components is in the Help menu for the Process Designer. Additional documentation is listed at the end of this article.

Adding MCF components to a process is fully supported in the Process Designer interface. Pervasive provides a library of existing components that are available in the Process Designer to be included in processes created there. Many other user-created components may be added as they are developed.

Benefits of MCF for developers

• Reusable components
• Extensive flexibility in component types
• Single step functionality for "multi-step" tasks
• Run in memory with message objects for efficiency
• Can be used to extend RIFL functions
• Eclipse-based development tool to create components

MCF Components for Mapping and Transformation

There two ready-to-use MCF components for mapping and transformation tasks.

• Automated Mapping Transformer

  A Transformer component typically performs a custom transformation from a source data format into a target data format. The Source and Target connection parameters are provided to the Transformer as message objects. The use of message objects opens the Transformer component to dynamic acquisition of parameters at runtime. A subsequent Execute action in the Transformer performs the transformation.

  The Automated Mapping Transformer leverages functionality already built into the Integration Engine for automatic one-to-one mapping and transformation between source and target without a pre-existing map (map.xml, tf.xml). The mapping is based on exactly "matched" metadata, either where the field names are the same for source and target (map by name) or where the fields are in the same relative position for source and target (map by position). The source and target connection parameters may be passed in message objects. Essentially this transformer enables on-the-fly mapping for a very narrow definition of source and target metadata, and without complex data manipulation from source to target.

  Typically transformations are run in processes by executing transformation maps as steps, but sometimes the only method to accomplish a specific sort of transformation is to create custom objects like the Automated Mapping Transformer to create maps and transformations on the fly.

• Dynamic Map -Invoker

  An Invoker typically provides the ability to utilize a service available from a third party API or remote application and returns a result to a process in a single step. The Invoker places any output from the service into a target message object. Typical applications for an Invoker component would include using a Web or SOAP service.

  The Dynamic Map -Invoker component enables dynamic calls to run an existing transformation (map.xml, tf.xml) in a single step. The Invoker component uses parameters to call the map or transformation. The map can then be loaded into a message object, and the message URI may be used as the source. Parameters are set up for source and target connections through the Invoker properties window in the Process Designer (the connection strings support authentication). Message objects may also be used to hold source and target data at runtime. Mapping expression overrides at runtime may also be applied to the transformation when using this Invoker.

Developer Hack

MCF component: FileFinder Invoker

The Message Component Framework is a design feature of the Pervasive Process Designer for creating components to be included as "plug-ins" in processes. The MCF component for this Developer Hack is an Invoker. An Invoker typically provides the ability to utilize a service available from a third party API or remote application and returns a result to a process in a single step.

One MCF component that was developed internally by Pervasive technical staff is the FileFinder Invoker. The FileFinder Invoker component extends the FileList function (in the Pervasive RIFL scripting language) to enable searches through any directories to locate certain files for processing in data integration projects.

The original RIFL-based FileList function requires foreknowledge of the folder or directory name that contains the files that are required. The folder or directory name must be directly referenced in the FileList expression. The FileList function then retrieves a list of files in the specified directory and stores the list in an array variable.

Example: FileList("Samples\Data\*.asc", File_List)
where File_List is the array variable that will hold the list of files retrieved from the named directory

However, there are many instances where file lists need to be retrieved based on file name or file extension parameters only, particularly to better automate data integration processes. The FileFinder Invoker searches through directories by cascading from a specified high level root directory to find and list files that match the search criteria. Searches are performed recursively either to match a provided file extension or to match a provided file name. The FileFinder search returns the full path and file list. The resulting list of files may be used as would normally be done in any data integration process.

Download the FileFinder Invoker component.

FootNotes and Additional Resources

Developing with Pervasive Data Integration Engine SDK

To locate documents for the Engine API: Documents are available in the Pervasive Business Integrator or Data Integrator product install. Go to the installation directory for Pervasive:

Install Directory\Pervasive\Cosmos\Common800\Help\PDF\integration_engine.pdf

Install Directory \Pervasive\Cosmos\Common800\Help\PDF\engine_sdk.pdf

When using the Pervasive Integration Engine Java and COM SDKs, remember that while the documentation for classes and methods may appear to be similar, the behaviors in Java and in COM are quite different. Thoroughly read and understand each SDK.

Start > Install Directory > Pervasive > Integration Architect > SDK Documentation > Integration Engine - Java Interface

Java SDK
Install Directory \Pervasive\Cosmos\Common800\SDKs\EngineSDK\Java

COM SDK
Install Directory \Pervasive\Cosmos\Common800\SDKs\EngineSDK\COM

Support for Working with the Basic On-Demand Mapper and the Sample Java Code

Customers with paid support 7:00am - 7:00pm CST
All other Support Customers 9:00am - 6:00pm CST

Main Telephone: 1.512.231.6000
Toll-Free: 1.800.287.4383

Web support: http://ww1.pervasive.com/support/incident/.

Pervasive Professional Services Group

Further assistance with custom data integration projects and the implementation of MCF components is available through engagements with Pervasive Professional Services Group (PSG).

Download the PSG brochure or contact PSG.

Many thanks to the Senior Project Manager in Professional Services Group for his generous and dynamic contributions to the content for the Basic On-Demand Mapper and for the FileFinder Invoker Component.

Working with the Message Component Framework (MCF)

To locate documents for MCF: Documents are available in the Pervasive Business Integrator or Data Integrator product install. Go to the installation directory for Pervasive:

Install Directory \Pervasive\Cosmos\Common800\Help\PDF\message_component_framework.pdf

Install Directory \Pervasive\Cosmos\Common800\Help\PDF\mcf_component_creation_tool.pdf

Install Directory \Pervasive\Cosmos\Common800\SDKs\MCFSDK

Dynamic Map Invoker Documentation
Install Directory \Pervasive\Cosmos\Common800\Help\PDF\cmp_dynamic_map_invoker

Auto Mapping Transformer Documentation
Install Directory \Pervasive\Cosmos\Common800\Help\PDF\cmp_automatic_mapping_transformer

Location of useable MCF Components
Install Directory \Pervasive\Cosmos\Common800\Plug-Ins\

Install Directory \Pervasive\Cosmos\Common800\Plug-Ins\Automatic Mapping Transformer

Install Directory \Pervasive\Cosmos\Common800\Plug-Ins\Dynamic Map Invoker

Other MCF Component Documentation
Install Directory \Pervasive\Cosmos\Common800\Help\PDF\ then browse all files that begin with cmp_

Support for Working with the MCF Components

Customers with paid support 7:00am - 7:00pm CST
All other Support Customers 9:00am - 6:00pm CST

Main Telephone: 1.512.231.6000
Toll-Free: 1.800.287.4383

Web support: http://ww1.pervasive.com/support/incident/

October 2007

A newsletter sponsored by Pervasive Software, providing trends, events, tech tips, and industry news on data integration.