Ingesting data upon the file creating on S3 bucket enables near real-time data ingestion. For example, you may need to ingest log files from applications or API monitoring tools as soon as they land on the bucket. Just to get it started, let’s move the file from the source bucket …

AWS S3 is probably the most utilised AWS storage services. It is affordable, highly available, convenient and easy to use. To interact with any AWS services, Node.js requires AWS SDK for JavaScript. Let’s first create a project folder called nodeS3 and install SDK. Then, create the main program file and …

Nodemailer is for sending email without the hustle of writing many lines of code. It uses the existing email services (Outlook, Hotmail, Gmail and so on) to authenticate and send messages. Authentication is done by simply adding your email credentials as a mail option. It also handles various transport protocols …

Express is pretty much the default framework to create a website with Node. Just like any website creation, the first step to create an Express website is to set up an template and routing. EJS is an easy templating language to generate HTML with Javascript. It works well with Node …

Now that we got data from MongoDB and loaded into Postgres with some transformation, let’s put the data back into MongoDB. This post is the continuation of the previous post about ingesting data from MongoDB. You should totally start from How to Ingest Data From MongoDB with Node.js. The aim …

Data ingestion from a NoSQL database often involves the denormalisation of their schema-less data before loading into a relational database. In this post, we will try to grab the restaurant data from MongoDB, denormalise the collection into a parent and a child table, and load them into Postgres. This exercise …

Once you get your head around MongoDB CRUD operations with Mongo Shell, getting Node.js to do them becomes easy. If you are still not confident with MongoDB, check out MongoDB skills essentials exercise to brush up your MongoDB skills. We will create a collection called, students. Then, we will insert …

In MongoDB, the primary key is reserved for the _id field. This is a system field and gets created by default when inserting new records. When you want to enforce the uniqueness in other fields, you can use unique index. Here is how to create unique index with Mongo Shell. …

The classic REST API data ingestion pattern is (1) to make an API call to the endpoint, (2) get the data, (3) transform it to a structured table and (4) load it to a database. Let’s have a go at it with Node.js. We are using JSONPlaceholder which offers a …

To convert JSON to CSV, I love using json2csv. It really does all the hard work of working the JSON structure out and converting it to a flat file. For nested JSON elements, you can simply specify them by the dot notation (like transaction.itemId). When it contains an array element, …