Informatica Cloud: How To Run More Than 2 Data Synchronization Tasks Concurrently

By default, the secure agent can run 2 data synchronisation tasks at a time. This constraint can become limiting quickly especially when multiple developers are building and testing the data synchronisation tasks at the same time. By adding a custom property on the secure agent, you can run more than 2 jobs concurrently.

Steps

(1) Log in to Informatica cloud

(2) Configure -> Runtime Environment -> Pick the secure agent

(3) Go all the way down and you will find Custom Configuration tab. Select service as Data integration server. Set Type as Tomcat and Name as maxDTMProcesses. Value would be the number of jobs you want to run concurrently. For example, setting 4 means 4 jobs run concurrently.

Note

Informatica recommend to distribute the load among the available secure agents instead of running all the tasks using one secure agents. The number of secure agent you can have depends on the contract.

However, running jobs in multiple secure agent servers makes it harder to manage. If you just want to increase it to 4 or 5 jobs, it’s usually fine with updating the configuration.

The max number of concurrent jobs you can run with one secure agent depends on the volume of data as well as the size of the secure agent server. Running concurrent tasks may slow down the speed of individual tasks. The best practice is to experiment it and find the magic number where performance compromise is minimum. I recommend to monitor the server performance and capacity while you are running concurrent tasks.

Data Synchronization is probably the best tool to do one-to-one copy between relational databases in the market. It is super easy to build jobs. It is so solid and never really fails unless source database changes. Coming from the DataStage world, I found that Data Synchronisation minimises the time worrying about one-to-one copy jobs and makes me really focus on transformation, adding more value to the business.

If you are interested in incremental load with Data Synchronization tasks, check out the post here.

Enjoy!

ETL
Running Jobs with Informatica Cloud REST API

Informatica Cloud offers REST API for us to interact with the platform programmatically. At this point in time, the latest official reference is found here. In this post, we are going to use Python to trigger jobs through API. Triggering a job is a 2-step process. We need to authenticate …

ETL
Tips and Troubleshooting For Uploading CSV to Database In Talend

There will be time when you want to upload a big csv file (with many rows and hundreds of columns) to a relational database table. Talend Open Studio is an open source ETL tool that I use regularly to do odd jobs like that. I like using it because it …

ETL
How To Rename Mapping Job In Informatica Cloud

Mappings are where all the magic happens in Informatica Cloud. When I started using it, it took me a while to work out how to rename a mapping job. Since then, a few people asked me the same question. So, I decided to write about it. This is probably the …