Infrastructure

Create Custom Python Module and Distribute with Git

When we create a custom Python module, we can distribute it through a Git repository. By using a private repo, you can keep your module private and use it in your deployment pipeline by calling pip install from the private repo. Pretty cool trick. In this example, I created a …

Data Engineering

Event-Driven S3 Data Ingestion With Node.js Lambda Function and Deploy it with Serverless

Ingesting data upon the file creating on S3 bucket enables near real-time data ingestion. For example, you may need to ingest log files from applications or API monitoring tools as soon as they land on the bucket. Just to get it started, let’s move the file from the source bucket …

Data Engineering

Uploading and Downloading Files in S3 with Ruby

To date, the latest Ruby AWS SDK is version 3. In this version, each resources has its own module while the version 2 had the one with everything, aws-sdk. To interact with S3 with the v3 SDK, let’s use the aws-sdk-s3 module. Ruby has its own way to upload to …

Data Engineering

Uploading and Downloading Files in S3 with Node.js

AWS S3 is probably the most utilised AWS storage services. It is affordable, highly available, convenient and easy to use. To interact with any AWS services, Node.js requires AWS SDK for JavaScript. Let’s first create a project folder called nodeS3 and install SDK. Then, create the main program file and …

Infrastructure

How to Set up Dual-Authentication to GitHub and AWS CodeCommit

Both GitHub and AWS CodeCommit are used to version-control your source code. I use GitHub to share my code with the wider community and CodeCommit for production code for AWS. You may use other source control systems like BitBucket. Whatever the choice you make, it is convenient to set up …

AWS

Creating Publicly Accessible RDS with CloudFormation

AWS CloudFormation let us create AWS resources with JSON or YAML files. With CloudFormation, you can create and update your AWS infrastructure by code. In the previous post, we discuss how we can create publicly available RDS (How to Make RDS in Private Subnet Accessible From the Internet). In this …

AWS

How to Make RDS in Private Subnet Accessible From the Internet

When you have RDS in a private subnet, you cannot access from the Internet. Accessing RDS from the Internet is not necessary and is not a good practice if it is used for the application backend. However, your RDS is an analytics database and needs to be accessed by another …

ETL

Running Jobs with Informatica Cloud REST API

Informatica Cloud offers REST API for us to interact with the platform programmatically. At this point in time, the latest official reference is found here. In this post, we are going to use Python to trigger jobs through API. Triggering a job is a 2-step process. We need to authenticate …

Node.js

Sending Email After Form Submission with Node.js

Nodemailer is for sending email without the hustle of writing many lines of code. It uses the existing email services (Outlook, Hotmail, Gmail and so on) to authenticate and send messages. Authentication is done by simply adding your email credentials as a mail option. It also handles various transport protocols …

Node.js

Website Page Routing with Node.js Express and EJS

Express is pretty much the default framework to create a website with Node. Just like any website creation, the first step to create an Express website is to set up an template and routing. EJS is an easy templating language to generate HTML with Javascript. It works well with Node …