Introduction

Last year I wrote about configuring the BICS Data Sync Tool to extract data from ‘On-Premise’ data sources and to load those into BI Cloud Services (BICS).  That article can be found here.

In March 2016 the Data Sync Tool added the ability to connect and extract data from Oracle Transactional Business Intelligence (OTBI).  OTBI is available on many Oracle Cloud Services, and also On-Premise products.

This approach opens up many more sources that can now be easily loaded into BICS.  It also allows the Data Sync tool to leverage the Metadata Repository in OTBI (the RPD) to make the creation of an extract a simple process.

 

Main Article

This article will walk through the steps to download and configure the Data Sync tool to extract data from OTBI.  These steps are the same for an on-premise or cloud source, with the only difference being the URL used for the connection.

The Data Sync tool provides 3 approaches for extracting data from OTBI.  The ‘pros’ and ‘cons’ of each will be discussed, and then each method explained in detail.  The 3 approaches are:

1. An extract based on a report created in Analyses (also known as BI Answers)

2. An extract based on the SQL from the ‘Advanced’ tab of an Analyses Report

3. An extract based on a Folder from a Subject Area from the /analytics portal

 

Table of Contents

Which Approach Should You Use ?

Downloading Latest Version of Data Sync Tool

Setup Source Connection for OTBI Environment

Create Data Source Based on BI Analyses Report

Create Data Source Based On Logical SQL

Create Data Source Based on Subject Area Folder

Which Approach Should You Use ?

It is the opinion of this author that the second method, the extract based on SQL, is going to be the most useful approach for regular data updates.  Once created, there is no reliance on a saved analysis, and the true incremental update capability reduces the volume of data needing to be extracted from OTBI, improving performance and load times.

It is also not restricted by the 65,000 maximum row limit that many Cloud and On-Premise OTBI environments impose on reports.

Below is a quick summary of the differences of the 3 approaches.

 

Extract based on a report created in Analyses (also known as BI Answers)

In this approach, a report is created in OTBI and saved.  The Data Sync tool is then configured to run that report and extract the data.

• Fully leverages the OTBI reporting front-end, allowing complex queries to be created without the need to understand the underlying table structures or joins, including filters, calculated fields, aggregates, etc
• The select logic / filters in the Analyses report may be changed later, with no need to make changes in the Data Sync tool.  As long as the data fields being returned by the report remain the same, the changes will be transparent to the Data Sync tool. This would allow, for example, a monthly change to a report – perhaps changing the date range to be extracted.
• For Cloud environments, this approach is limited to 65,000 rows, so should only be used for smaller data sets
• It is not possible to restrict the data extract programmatically from the Data Sync tool, so true incremental updates are not possible.  For this functionality, one of the next two approaches should be used.

Extract based on the SQL from the ‘Advanced’ tab of an Analyses Report

This approach is very similar to the previous one, but instead of using a saved report, the logical SQL generated by OTBI is used directly.

• Fully leverages the OTBI reporting front-end, allowing complex queries to be created without the need to understand the underlying table structures or joins, including filters, calculated fields, aggregates, etc
• Allows for true incremental updates, with an Incremental SQL option that will reduce the amount of data being pulled from OTBI, improving performance and reducing load times
• Once created, there is no reliance on a saved OTBI analyses report
• Is NOT limited to 65,000 rows being returned, so can be used for both small, and larger data sets

Extract based on a Folder from a Subject Area from the /analytics portal

This approach bases the extract on a folder within a Subject Area within OTBI.  It allows the creation of many such mappings to be created at once.

• No need to create an OTBI report or logical SQL, or even to log into OTBI.  The extract is set up purely within the Data Sync tool.
• Allows mappings for multiple Subject Area folders to be created in one step, saving time if many mappings are needed and the Subject Area folders are structured in a meaningful way for BICS tables
• Only allows Subject Area ‘folders’ to be imported, with no additional joins to other folders.  This approach will be most useful when the Subject Area folders closely mimic the desired data structures in BICS
• Allows for true incremental updates, with a Filter option option that will reduce the amount of data being pulled from OTBI, improving performance and reducing load times
• Is NOT limited to 65,000 rows being returned

Downloading Latest Version of Data Sync Tool

Versions of the Data Sync Tool prior to 2.0, released in February 2016, do not include this functionality.   The latest version can be obtained from OTN through this link.

For further instructions on configuring Data Sync, see this article.  If a previous version of Data Sync is being upgraded, use the documentation on OTN.

Setup Source Connection for OTBI Environment

No matter which of the 3 approaches is used, an initial Source Connection to OTBI needs to be configured.  If multiple OTBI environments are to be sourced from, then a Source Connection for each should be set up.

The Data Sync tool connects via web-services that many instances of OTBI expose.  This applies to both Cloud and On-Premise versions of OTBI.

To confirm whether your version of OTBI exposes these, take the regular ‘/analytics’ URL that is used to connect to the reporting portal – as demonstrated in this image:

and in a browser try and open the page with this syntax:

https://yourURL.com/analytics-ws/saw.dll/wsdl/v9

if the web-services are available – a page similar to the following will be displayed:

 

If this does not display, try repeating but with this syntax (using ‘analytics’ instead of ‘analytics-ws’)

https://yourURL.com/analytics/saw.dll/wsdl/v9

If neither of these options display the XML page, then unfortunately web-services are not available in your environment, or the version of web-services available are earlier than the ‘v9’ that the data sync tool requires.

Speak with your BI Administrator to see if the environment can be upgraded, or the web-services exposed.

 

Defining Connection in the Data Sync Tool

a. In the Data Sync tool, select ‘Connections’, ‘Sources/Targets’, and ‘New’

b. Give the connection an appropriate name, and select ‘Oracle BI Connector’ as the connection type.

c. Enter the credentials for a user that has Consumer rights in the OTBI environment required to run a report.

d. For a the URL – first test with the syntax

https://yourURL.com/analytics-ws

This has less layers for the Data Sync tool to traverse, and so may offer slightly improved performance.

If the ‘Test Connection’ option fails – then try the following syntax:

https://yourURL.com/analytics

 

In this case, using the ‘analytics-ws’ version of the syntax, the configuration would look like this:

Save the connection, and then use the ‘Test Connection’ button to confirm details are correct.

e. For an On-Premise connection, the process would be identical.  Use the URL that is used to connect to the BI Analytics portal, and edit the URL to either use the ‘/analytics-ws’ or ‘/analytics’ path as described above.  In this example screenshot an internal URL is used, with the port number.  Once again, test the connection to confirm it connects.

 

Create Data Source Based on BI Analyses Report

a. Log into the Analytics portal for OTBI.

b. Create a new Analysis.  In this example, a simple report using Customer data from the ‘Marketing – CRM Leads’ Subject Area.  While not used here, a more complex query, with filters, calculated fields, aggregations, and fields from multiple subject area folders could easily be created.

c. Save the Analysis.  In this example the report was named ‘DataSyncTest’ and saved in /Shared Folders/Customer.  This path will be used in subsequent steps, although the path format will be slightly different.

d. Within the Data Sync tool, create a new ‘Manual Entry’ within the ‘Project’ / ‘Pluggable Source Data’ menu hierarchy:

e. Give the extract a Logical Name, used as the Source Name within the Data Sync Tool.  The ‘Target Name’ should be the name of the table you want to load into BICS. If the table doesn’t exist, the Data Sync Tool will create it.

f. A message provides some additional guidance of best practice.  Select ‘Report‘ as the source at the bottom of the message box as shown below, and then ‘OK’ to continue.

f. In the next screen – enter the path for the BI Answers analysis from step c.  Notice that the syntax for the Shared Folder is ‘/shared’ which differs from how it’s displayed in the OTBI Portal as ‘Shared Folder’.  In this example the folder path is:

/shared/Custom/DataSyncTest

 

g. the Data Sync tool can identify the field types but it does not know the correct length to assign in the target for VARCHAR fields.  By default these will be set to a length of 200.  These should be manually checked afterwards.  A message will inform you which of these need to be looked at.  Click ‘OK’ to continue.

h. The target table defined in step e. will be created.  Go to ‘Target Tables / Data Sets’, select the Target table that had just been created, and the ‘Table Columns’ option, and adjust the VARCHAR lengths as necessary.

i. As long as the Source Report has a unique ID field, and a date field that shows when the record was last updated, then the Load Strategy can be changed so that only new or updated data is loaded into the target table in BICS.  This can be changed in the ‘Project’ / ‘Pluggable Source Data’ menu hierarchy as shown below:

 

j. In this case the ‘Customer Row ID’ would be selected as the unique key for ‘User Key’ and the ‘Last Update Data’ for the ‘Filter’.

 

It is important to realize that while only changed or new data is being loaded into BICS, the full set of data needs to be extracted from OTBI each time.  The next two methods also provide the ability to filter the data being extracted from OTBI, and thus improving performance.

There is also a restriction within some Cloud OTBI environments where the result set is restricted to 65,000 rows or less.  If the extract is going to be larger than this, the other 2 methods should be considered.

You now have a Source and Target defined in BICS and can run the Job to extract data from an OTBI Analyses and load the data into BICS.

Create Data Source Based On SQL

a. While editing the Analyses Reports within OTBI, select the ‘Advanced’ tab and scroll down. The SQL used by the report is displayed.  This example below is the same report created earlier.

b. Cut and paste the SQL and remove the ‘ORDER by’ and subsequent SQL.  Also remove the first row of the select statement (‘0 s_0,’).  Both off these highlighted in the green boxes in the image above.

In this case, the edited SQL would look like this:

SELECT
“Marketing – CRM Leads”.”Customer”.”City” s_1,
“Marketing – CRM Leads”.”Customer”.”Country” s_2,
“Marketing – CRM Leads”.”Customer”.”Customer Row ID” s_3,
“Marketing – CRM Leads”.”Customer”.”Customer Unique Name” s_4,
“Marketing – CRM Leads”.”Customer”.”Last Update Date” s_5,
DESCRIPTOR_IDOF(“Marketing – CRM Leads”.”Customer”.”Country”) s_6
FROM “Marketing – CRM Leads”

c. In the Data Sync tool, Create a ‘Manual Entry’ for the SQL Data Source under the ‘Project’ / ‘Pluggable Source Data’ menu hierarchy.  As before, the Logical Name is used as the Source, and the Target Name should either be the existing table in BICS that will be loaded, or the new table name that the Data Sync Tool is to create.

d. Select ‘SQL‘ as the source type

e. In the ‘Initial SQL’ value, paste the SQL edited from the ‘Advanced’ tab in the Analyses Report

f. As before, a message is displayed reminding you to check the target table and adjust the size of VARCHAR fields as necessary:

g. Edit the newly created target and adjust the lengths of the VARCHAR fields as necessary.

h. This approach allows for true Incremental Updates of the target data, where only new or updated records from the Source OTBI environment are extracted.  To set up Incremental Updates, go back to ‘Project’, ‘Pluggable Source Data’ and select the source created in the previous step.  In the bottom section under the ‘Edit’ tab, select the ‘Incremental SQL’ box.

The Data Sync tool has 2 variables that can be added to the Logic SQL as an override to reduce the data set extracted from the source.

Those 2 variables are:

‘%LAST_REPLICATION_DATE%’ – which captures the date that the Data Sync job was last run

‘%LAST_REPLICATION_DATETIME%’ – which captures the timestamp that the Data Sync job was last run

As long as there is a suitable DATE or TIMESTAMP field in the source data that can be used to filter records, these variables can be used to reduce the data set pulled from OTBI to just the data that has changed since the last extract was run.

This is an example of the Incremental SQL using the %LAST_REPLICATION_DATE% variable.  The SQL is identical to the ‘Initial SQL’, just with the additional WHERE clause appended to the end.

And this is an example of the Incremental SQL using the %LAST_REPLICATION_DATETIME% variable:

i. To utilize the Incremental Approach, the Load Strategy should also be set to ‘Update Table’ from ‘Project’ / ‘Pluggable Source Data’ and by selecting the Source based on the Logic SQL.  Under the ‘Edit’ tab, change the Load Strategy:

Set the ‘User Key’ to a column, or columns, that make the row unique.

For the Filter, use a field that identifies when the record was last updated.

Data Sync may suggest an index to help load performance.  Click ‘OK’ to accept the recommendation and to create the Index.

You now have a Source and Target defined in BICS and can run the Job to extract data from the SQL created from an OTBI Analyses and load the data into BICS.

Create Data Source Based on Subject Area Folder

a. In the Data Sync Tool under the ‘Project’ / ‘Pluggable Source Data’ menu hierarchy, select ‘Data from Object’

b. To save time, use the Filter and enter the full or partial name of the Subject Area from within OTBI that is to be used.  Be sure to use the ‘*’ wildcard at the end.  If nothing is entered in the search field to reduce the objects returned, then an error will be thrown.

c. Select ‘Search’.  Depending on the Filter used, this could take several minutes to return.

d. Select the Subject Area folder(s) to be used.  If more than one is selected, a separate source and target will be defined for each within the Data Sync tool.

e. Some best practice for this method are displayed – hit ‘OK’

f. As in the other methods, a list of the fields cast as VARCHAR with a default length of 200 are shown.  Click ‘OK’

g. After a few moments a ‘Success’ notification should be received:

h. As before, update the length of the VARCHARs as needed, under ‘Target Tables / Data Sets’, and then by selecting the Target table created in the previous step, and then in the bottom section under ‘Table Columns’ the lengths can be altered:

i. The Load Strategy can be updated as needed in the same way, and the true ‘Incremental’ option is available if there is a suitable date field in the source.

NOTE – for this method, if a date is select as part of the Update Strategy, then that is automatically used to restrict the data extract from the source.  No further action is required to implement true incremental updates.

There is the option to add a filter to further restrict the data.  This example below shows how the “Contact” folder within the “Marketing – CRM Leads” could be restricted to just pull back contacts from California.  The value is in the form of a ‘Where Clause’, using fully qualified names.

You now have a Source and Target defined in BICS and can run the Job to extract data from one or more Subject Areas (separate mapping for each) and load that  data into BICS.

 

There is another method for creating an extract based on a single Subject Area.  This may be preferable if an incremental update approach is required, although only allows a single mapping to be set up at a time.

a. Under ‘Project’, ‘Pluggable Source Data’, select ‘Manual Entry’.  Select the OTBI DB connection and enter a suitable name for the logical source and target:

b. Select ‘Subject.Area.Table’ as the data source, and then click ‘OK’.

c. In the Properties, enter the fully qualified name for the Subject Area and Folder (‘Table’), and also the filter with the same format as step (i) above.  Be sure to follow the steps to update the Load Strategy if incremental updates are required.

Summary
This article walked through the steps to configure the Data Sync tool to be able to connect and extract data from a cloud or on-premise OTBI environment.  The second method – ‘Extracting Data based on SQL’ – is the recommended approach for most use cases.

For further information on the Data Sync Tool, and also for steps on how to upgrade a previous version of the tool, see the documentation on OTN.  That documentation can be found here.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission