WDCi Blogs

Salesforce Integration Tips: Using the Bulk API – Serial or Parallel mode options

zylean — Fri, 25 May 2012 09:30:59 +0000

I talked about the usage of Salesforce BULK API in Talend Data Integration Studio and the Dell Boomi Integration Platform previously. So I’m wondering, how many of us really know that there are actually 2 concurrency modes available in the Salesforce BULK API – Serial mode and Parallel mode.

Parallel mode

This is the default mode in the BULK API. Salesforce will process all the batches in a posted job in parallel and this gives you a better performance in uploading data. However, this might lead to lock contention which will cause the upload to fail.

Serial mode

Salesforce will process all the batches in a posted job one by one and this can help to prevent the lock contention issue that can be encountered in Parallel mode. But, please bear in mind that you should only use this mode if you couldn’t get around the lock contention issue.

How to Enable Serial Mode in Apex Data Loader?

You can enable this in Apex Data Loader settings as shown in the screenshot below.

How to Enable Serial Mode in Java Client?

You can set the concurrency mode in the job as the following:

JobInfo job = new JobInfo();

job.setObject(“ObjectName”);

job.setOperation(OperationEnum.query);

job.setConcurrencyMode(ConcurrencyMode.Serial);

job.setContentType(ContentType.CSV);

job = bulkConnection.createJob(job);

If you would like to know more information about concurrency mode in Salesforce BULK API, please visit the Bulk API Developer’s Guide:

- http://www.salesforce.com/us/developer/docs/api_asynch/api_bulk.pdf

- http://wiki.developerforce.com/page/Loading_Large_Data_Sets_with_the_Force.com_Bulk_API

Dell Boomi Integration Tips: Using the Salesforce Bulk API option

zylean — Fri, 25 May 2012 09:21:53 +0000

In the last blog post, I talked about the usage of Salesforce BULK API in one of the integration tools that we use.

This brought up my curiosity to find out how the BULK API option works in the Dell Boomi Salesforce connector. Out of the box the Salesforce connector only supports the BULK API option in parallel mode. Please the screenshot below:

There are 2 options that you need to look at:

1. Use Bulk API – to tell the connector to load data to Salesforce using Bulk API

2. Batch Count – the maximum number of records in a batch

To have your data upload with Bulk API, you just need to check on the “Use Bulk API” checkbox and specify the Batch Count (the default value is 200) in the operation configuration screen. The Dell Boomi Salesforce connector will handle everything for you automatically at the backend including preparing the data into batch according to the Batch Count.

As there is no option to turn on serial model, some options to address lock contention in Salesforce could be:

Reduce batch sizes
Use flow control

Netsuite Integration Tips: Troubleshooting errors

TIM — Fri, 25 May 2012 08:59:26 +0000

Netsuite has a feature called the Web Services Usage Log, this is useful for troubleshooting issues for an integration between Netsuite and other applications. Using this feature, you can view the incoming web services requests received by Netsuite and also the response returned by Netsuite.

The Web Services Usage Log can be accessed under the top menu panel: Setup > Integration > Web Services Usage Log

Here, we have an example on how to troubleshoot the error when a Customer record has failed to be created into Netsuite from another application:

1. Once clicked into the Web Services Usage Log, we can see a list of jobs (each job represents the incoming request sent to Netsuite). The last request that was sent to Netsuite is the one that attempted to create a Customer record. The Status of this job has failed.

2. We can view the request sent to Netsuite. Here is a screenshot of the request:

3. Let’s view the response returned by Netsuite for this job. Inside this response, we can then see the Error Code and Error Message explaining why Netsuite failed to create this Customer record.

Note: Some error messages returned are usually stated as Unknown Error, this type of error doesn’t give us clue and unfortunately will require further detailed troubleshooting to find out the cause.

Boomi Integration Tips: Start Shape Options

mychan — Tue, 22 May 2012 09:54:01 +0000

Before diving into the work of developing an integration process, it’s important to give some thought to error handling and the later stage of troubleshooting the process you have developed. In this topic, we would like to share some information we know with regards to the Start shape of the Boomi component.

There are 3 options that we are mainly focusing on:

Connector
No Data
Data Passthrough

Connector

If Connector is your selected option, then you should expect to see:

The # of records returned from the output Connector of the start shape in your Process Reporting
The process execution appearing in your Process Reporting

For example, using Salesforce connector will produce the following result in Process Reporting:

Note:
The inbound result also depends on the response document of the selected Connector. For Salesforce (example above), the response document returned represents a record individually (8 records in total for the above example). Other connectors may just return a single response representing all the records within one document regardless of the number of records returned.

No Data

If No Data is your selected option, then you will always see:

1 inbound data
The process execution appearing in your Process Reporting

For example:

Data Passthrough

This option is most often selected when the process should receive data from another process or another external source. We can say that it is used within a sub-process. If Data Passthrough is your selected option within the sub-process, then you should expect:

Not to see the sub-process execution where the Data Passthrough is selected in your Process Reporting
The logic of the process flow is performed at the back end

For example:

Note:
If Data Passthrough is used within a parent process where the process is scheduled without a data source, it will run as a No Data start shape.

I hope that the above explanation will help you to decide which Start shape option best suits your design.

With this information in mind, you can decide the necessity of the process execution display within the Process Reporting to assist with your troubleshooting effort.

Tips: Magento API – filter and complex filter

TIM — Tue, 22 May 2012 02:25:18 +0000

When you are querying records in Magento, you can use filter to narrow down the result.

The basic filter uses the EQUAL operator and AND operand. For example, you can query products with two basic filters: type=simple AND set=39

Example:

c9f1ca71de1da4202c84bf90f6df51cf

type

simple

set

Using the complex filter, you can perform a more sophisticated search. For example, you can query customers with two complex filters: store_id=0 AND created_at>2012-05-13 06:11:00

Example:

c9f1ca71de1da4202c84bf90f6df51cf

store_id

created_at

2012-05-13 06:11:00

The complex filter also uses the AND operand. However, you can specify your own operator. For example: eq (equals), neq (not equals), gt (greater than), lt (less than), etc.

Salesforce BULK API in Talend Data Integration

zylean — Tue, 22 May 2012 01:45:07 +0000

Migrating or integrating your data from an in-house application or cloud application to Salesforce can be difficult, time consuming and consume a lot of API calls if you choose the wrong approach.

There are a lot of tools available in the Internet nowadays, and the most common tool that a developer is like to use is the Salesforce Apex Data Loader. The advantage of using this tool is that it does support the BULK API, which can help to save the number of API calls that you need during the upload. However, if you want to implement additional logic to manipulate the data before uploading the data to Salesforce, you’ll need consider to using an integration tool.

Here, I will go through the steps to implement a simple Talend Data Integration job which will upload data from a CSV file to the Salesforce Account object by utilizing the Salesforce Bulk API.

First, you need to create a new job in Talend. Here, I will start with tPrejob component and connect it to a tFileInputDelimited component. This will force the job to execute the tFileInputDelimited component during runtime to read the data that I want to load into Salesforce.

Figure 1 – Read data using tFileInputDelimited

Next, you have to specify the schema for the tFileInputDelimited component according to the fields that you have in the CSV file. Below is the schema that I use in this example:

Figure 2 – Schema for tFileInputDelimited

Now, you have to drag the tSalesforceOutputBulk component into the design workspace and specify the location to save the Salesforce bulk data load file and schema. Please note that the name of the field for the tSalesforceOutputBulk component must be exactly the same as the API name in that you see in Salesforce Account object (Setup -> App Setup -> Customize -> Accounts -> Fields).

Figure 3 – Salesforce bulk data load file location for tSalesforceOutputBulk component

Figure 4 – Schema for tSalesforceOutputBulk component

Once the schema for tFileInputDelimited and tSalesforceOutputBulk are specified, we will do a simple transformation in between and mapping by using tMap component. You need to drag the tMap component from the palette and:

1. connect the tFileInputDelimited to tMap by right click on the component -> Row -> Main

2. connect the tMap to tSalesforceOutputBulk by right click on the component -> Row -> *New Output* (Main) -> name the output (in this example, I name it as sf_data) then click Yes when it prompts you “Do you want to get the schema of the target component?”.

Figure 5 will be the current flow that I have in design workspace.

Figure 5 – Flows from tFileInputDelimited to tSalesforceOutputBulk

In the tMap component, you can map the fields and apply additional logic according to your business logic. In this example, I want to join the address1 and address2 to become BillingStreet in Salesforce

Figure 6 – Field mapping in tMap component

At this point, you have done first part of the process and we will move on to the second part where the job will read the Salesforce bulk data load file using the tSalesforceBulkExec component and save the success and failure result to CSV files (salesforce_account_bulk_success.csv and salesforce_account_bulk_fail.csv).

Figure 7 – tSalesforceBulkExec

Before you move on to the next step, you need to configure the connection settings and the number of rows to commit in the tSalesforceBulkExec component. The default Rows to commit is 10000. You can reduce the number according to your requirement. I will stick with 10000 in this example as my company has data to load in a day and this can help to save the number of API calls

Figure 8 – tSalesforceBulkExec connection setting

Figure 9 – Rows to commit in tSalesforceBulkExec

After that, you need to connect the Main row from tSalesforceBulkExec component to a tFileOutputDelimited to record the success record and the Reject row to another tFileOutputDelimited to record the failure record. The reason of doing this is you will be able to know which record is uploaded to Salesforce successfully and the record id in Salesforce. This makes your life easier if you would want to use the record id in another job. Below are the schemas that you should see in Main row and Reject row:

Figure 10 – Schema for Main row from tSalesforceBulkExec

Figure 11 – Schema for Reject row from tSalesforceBulkExec

Yes, now you are done and you should something similar to this in the design workspace. You can run the job by clicking on the Run button and you should see the data is uploaded to Salesforce.

Figure 12 – Complete flow

Error Handling in Talend using tLogCatcher

TIM — Mon, 21 May 2012 05:35:35 +0000

The tLogCatcher component listens to Warn, Die or Java Exceptions in the Talend Job. It will retrieve log information from the message and send it to a standard Talend output. By using this component you can display the error message(s) in more detail and an organised format.

Formatting the output

Did you notice that there too many columns displayed on the above image and that you are not allowed to edit the tLogCatcher schema directly to remove the columns. The solution is to use a tMap component to filter out those columns you do not want to display:

LDAP Integration: Performing a User search in Group

TIM — Thu, 17 May 2012 05:18:28 +0000

If you are synchronising users from one system to LDAP, you can check if the User is a member of a Group before adding the User into the Group. This can be achieved by using the LDAP connector. You can do this by providing the User CN and Group DistinguishedName into the filters.

For example, we can put the following details into the filters of connector to perform the search:

User CN: 112233
Group DistinguishedName: CN=My_Group,OU=User_Groups,DC=test,DC=wdci,DC=com,DC=au

This is the result we will get if the user is found in the group:

Found
User 112233 found in group CN=My_Group,OU=Students,OU=User_Groups,DC=test,DC=wdci,DC=com,DC=au

Understanding Salesforce Object Relationships

mychan — Thu, 17 May 2012 04:27:08 +0000

There are many skill variations when it comes to Salesforce system implementation and Salesforce integration. One of it requires an understanding on the object relationship within the application. It is not hard to comprehend the relationship once you can identify the data model that you are about to build. The following are some of the common object relationships that are used to define the data model:

1. One to One
2. One to Many/Many to One
3. Many to Many

One of the ways to get yourself equipped with the knowledge is to read through the Salesforce Fundamentals documentation. Well, that’s how I got to it while undertaking the Salesforce Certification – Developer. However, if you find that going through the materials a hassle, you can always refer to the following explanation on a quick run-down on how to build the data model.

One to One object relationship

Scenario:
One School can only have one Headmaster

Implementation:
This object relationship requires that one of the associated object to be unique.
- School (Master object)
- Headmaster (Detail/Child object)
- Create a unique field in Headmaster object
- Create workflow to copy the School id to the unique field for every record creation

One to Many/Many to One object relationship

Scenario:
One school can have many teachers

Implementation:
- School object
- Teacher object: School field (Lookup to School)

When you view the relationship from School, it’s ONE School to MANY Teachers
When you view the relationship from Teacher, it’s MANY Teachers to ONE School

Many to Many object relationship

Scenario:
Student can enrol in many subjects

Implementation:
- This is where junction object comes into play for this relationship
- Student Subject (junction object)
- Subject (Master object to Student Subject)
- Student (Master object to Student Subject)

This model explains that there are many students taking up many subjects and vice versa.

Hope this clears off any confusion you have with regards to the object relationships.

Furthermore, if you plan to perform a Salesforce migration with other applications and require to figure out the object relationship via API, you can simply generate the Salesforce WSDL of your current instance via:

Setup > App Setup > Develop > API > Generate WSDL

From there, you should be able to determine the definition of the object relationship. The following extensions explain on the references:

__c – in reference to the custom field id values

__r – in reference to the custom object relationship with the current object view

Salesforce Integration: The 15 & 18 character alphanumeric ID

jacky — Thu, 17 May 2012 04:13:34 +0000

A Salesforce ID represents a unique Salesforce record. The 15 character ID comes in an alphanumeric format i.e. 0019000000CHAus.

A simple way to retrieve this is via the URL.

A Salesforce record ID comes in an 18 character format i.e. 0019000000CHAusAAH.

The 18 character ID is similar to the 15 characters mentioned above but appended with 3 additional characters.

The 18 character ID is only retrievable through the Salesforce API using a web service query or an Integration tool, Data Loader or Apex Explorer. Here is an example response when querying for Account records in Salesforce:

0019000000CHAusAAH

Abbott Company

These 18 and 15 character IDs represent the same Salesforce record retrieved via different mechamisms. When integrating Salesforce or using the Salesforce API, utilizing either the 15 or 18 characters ID is supported for querying or updating a specific record.