It’s sometimes convenient to have a script to get data from SharePoint. We can automate the user managed data ingesting from SharePoint. For example, business users can upload or update the user managed file and a scheduled ETL task fetch and bring it to the datalake. Using SharePoint API is …

One of the greatest things in Python is that it has heaps of cool modules to use. They make programming fun and easy. Here is the example of an amazing salesforce API module. What makes it even better is that these python modules are easy to customise for your need …

We may encounter a situation where we need to check if the source system adds or drops columns regularly. For example, when the source system is constantly going through a heavy development, the audit process automation can be helpful. The code uses JDBC for both target and source database connections. …

When you ingest data from Salesforce into a relational database, you first need to create a table for the object you want to ingest. Writing a create statement manually is cumbersome and you often need to debug it a few times. Salesforce data types are quite different from database ones. …

In the previous post, we had a look at Python code examples of basic data engineering with AWS infrastructure. By using Qualtrics API, I would like to present a coding example of API data ingestion into S3 and Redshift. This code can be scheduled hourly, daily or weekly in a …

AWS offers a nice solution to data warehousing with their columnar database, Redshift, and an object storage, S3. Python and AWS SDK make it easy for us to move data in the ecosystem. In this post, I will present code examples for the scenarios below: Uploading data from S3 to …

Python has a plethora of modules that makes programming fun and easy. If you need to use Salesforce API with Python, the simple-salesforce module is your best friend.  The module takes care of boring stuff like authentication and let you use different APIs. You can check the documentation and source …

Informatica does not have a dedicated Postgres database connector. Therefore, we need to use the ODBC connector. In this post, I will discuss how to configure Postgres ODBC in both Linux and Windows servers for the Informatica Cloud ODBC connector. Linux Server (Red Hat) There are a few instructions, but …

In the previous post, we discussed a strategy to ingest Google Analytics data and presented the Python code example (How To Get Data From Google Analytics With Python). Generally speaking, I prefer using Python code to ingest API data, however, every once in a while, we get requested to write …

When you ingest data from Google Analytics, you need to create a series of reports based on GA dimensions and metrics. The granularity is determined by dimensions you add in the report. The most important thing is to understand business requirements before start ingesting data. Good requirement analysis will enable …