Installing all the required Python packages one by one can be time consuming. When you are setting up multiple development environments or creating a virtual environment, you can have a list of Python packages in a text file and install them all together by pip. Steps (1) Create a text …

If you are using Anaconda as your choice of Python distribution, chances are that you have encountered this error message when you try to install a package with conda install command. There are two ways to fix this issue. Use pip install instead of conda install. Search the package and …

In the previous post, we discussed how to ingest data from Liveperson with Python. In this post, I want to use R to make the same API call and create an aggregated table instead of preparing data for ingestion. The code is based on the example here. For further information …

Engagment History API let you grab livechat interaction data from Liveperson. It is based on the REST architecture and uses OAuth1.0. You first need to retrieve API Keys. In this example, I am using the requests and requests_oauthlib modules to make API calls from Python. Liveperson offers a good code …

It’s sometimes convenient to have a script to get data from SharePoint. We can automate the user managed data ingesting from SharePoint. For example, business users can upload or update the user managed file and a scheduled ETL task fetch and bring it to the datalake. Using SharePoint API is …

One of the greatest things in Python is that it has heaps of cool modules to use. They make programming fun and easy. Here is the example of an amazing salesforce API module. What makes it even better is that these python modules are easy to customise for your need …

We may encounter a situation where we need to check if the source system adds or drops columns regularly. For example, when the source system is constantly going through a heavy development, the audit process automation can be helpful. The code uses JDBC for both target and source database connections. …

When you ingest data from Salesforce into a relational database, you first need to create a table for the object you want to ingest. Writing a create statement manually is cumbersome and you often need to debug it a few times. Salesforce data types are quite different from database ones. …

In the previous post, we had a look at Python code examples of basic data engineering with AWS infrastructure. By using Qualtrics API, I would like to present a coding example of API data ingestion into S3 and Redshift. This code can be scheduled hourly, daily or weekly in a …

AWS offers a nice solution to data warehousing with their columnar database, Redshift, and an object storage, S3. Python and AWS SDK make it easy for us to move data in the ecosystem. In this post, I will present code examples for the scenarios below: Uploading data from S3 to …