Event-Driven Data Ingestion with AWS Lambda (S3 to RDS)
In the previous post, we discussed how to move data from the source S3 bucket to the target whenever a new file is created in the source bucket by using AWS Lambda function. In this post, I will show you how to use Lambda to execute data ingestion from S3 …
Event-Driven Data Ingestion with AWS Lambda (S3 to S3)
Let’s say you have data coming into S3 in your AWS environment every 15 minutes and want to ingest it as it comes. The best approach for this near real-time ingestion is to use AWS lambda function. To demonstrate how to develop and deploy lambda function in AWS, we will …
Comprehensive Guide to Download Files From S3 with Python
Using AWS SDK for Python can be confusing. First of all, there seems to be two different ones (Boto and Boto3). Even if you choose one, either one of them seems to have multiple ways to authenticate and connect to AWS services. Googling solutions can quickly become confusing as you …
New JSON Data Ingestion Strategy by Using the Power of Postgres
Postgres always had a JSON support with somehow limited capability before the 9.2 version added the native JSON support. The release of version 9.3 has really taken the JSON feature to the next level with additional constructor and extractor methods. The capability of querying and transforming the JSON data type …
How To Ingest AES Encrypted Data With Python
To ingest encrypted data into DWH, we may ingest the data as it is or decrypt and load it to the database, depending on the business requirements. It is always good to know how to decrypt encrypted data. There are many encryption methods. Encryption usually happens at the application (either …
How To Convert Non-UTC Timestamp Into UNIX Epoch Time In Python
When we ingest API data, the query URI string often takes Unix epoch time (or Unix time) in order to specify the datetime range. The epoch time is the way to represent timestamp as the number of seconds that have elapsed since 1970-01-01 00:00:00 UTC. When you have an input …
How To Get Data From MongoDB With Python
How to get data from MongoDB with Python MongoDB is one of the most popular no SQL databases used as a backend database for web and mobile applications. Data is stored in MongoDB as BSON, which looks like JSON files. Once you understand the way MongoDB stores data, all you …
How To Get Data From Liveperson And Create Aggregated Table With R
In the previous post, we discussed how to ingest data from Liveperson with Python. In this post, I want to use R to make the same API call and create an aggregated table instead of preparing data for ingestion. The code is based on the example here. For further information …
How To Ingest Data From Liveperson With Python
Engagment History API let you grab livechat interaction data from Liveperson. It is based on the REST architecture and uses OAuth1.0. You first need to retrieve API Keys. In this example, I am using the requests and requests_oauthlib modules to make API calls from Python. Liveperson offers a good code …
How To Get Data From SharePoint With Python
It’s sometimes convenient to have a script to get data from SharePoint. We can automate the user managed data ingesting from SharePoint. For example, business users can upload or update the user managed file and a scheduled ETL task fetch and bring it to the datalake. Using SharePoint API is …