DataStage: How To Resolve ‘Scratch Space Full’ Error

When the data volume is large, DataStage uses a scratch disk to process data. The default scratch disk space is usually the Scratch folder in the Server folder where the application is installed. To use a larger scratch disk space, we can create a custom configuration file.

The default configuration file is called default.apt and  sits in the Configuration folder (for example /opt/IBM/InformationServer/Server/Configuration/). In this folder, we can create a new configuration file (new_2node.apt). The example uses two nodes. Change the resource disk and resource scratch disk parameters as below.

{
    node "node1"
    {
        fastname "<Your DataStage Server URL>"
        pools ""
        resource disk "/scratch" {pools ""}
        resource scratchdisk "/scratch" {pools ""}
    }
    node "node2"
    {
        fastname "<Your DataStage Server URL>"
        pools ""
        resource disk "/scratch" {pools ""}
        resource scratchdisk "/scratch" {pools ""}
    }
}

Go back to the parallel job that gave the error. In the job parameter, add the variable APT_CONFIG_FILE and set the new config file path (/opt/IBM/InformationServer/Server/Configuration/new_2node.apt) in order to override the default config file path.

Check out the further reference here.

ETL
Running Jobs with Informatica Cloud REST API

Informatica Cloud offers REST API for us to interact with the platform programmatically. At this point in time, the latest official reference is found here. In this post, we are going to use Python to trigger jobs through API. Triggering a job is a 2-step process. We need to authenticate …

ETL
Tips and Troubleshooting For Uploading CSV to Database In Talend

There will be time when you want to upload a big csv file (with many rows and hundreds of columns) to a relational database table. Talend Open Studio is an open source ETL tool that I use regularly to do odd jobs like that. I like using it because it …

ETL
How To Rename Mapping Job In Informatica Cloud

Mappings are where all the magic happens in Informatica Cloud. When I started using it, it took me a while to work out how to rename a mapping job. Since then, a few people asked me the same question. So, I decided to write about it. This is probably the …