Data Engineering

Converting CSV to JSON and Loading it to Postgres with Node.js

To convert csv to json, Node has an awesome module, csvtojson. It takes a json file and convert it to csv asynchronously. Once we convert csv to json, let’s load it to a Postgres table with jsonb data type. Postgres supports JSON data and you can query it (see the …

Data Engineering

Bulk Loading Postgres with Node.js

The fastest way to bulk load data into Postgres is to call Copy, which is a SQL command to load data into a table from a flat file. To connect to Postgres with Node.js, we can use the node-postgres module (pg). To use the copy function, we can use the …

Infrastructure

Setting up Docker and Serverless Development Environment in Ubuntu

Docker and Serverless do not work well with Windows. I have tried so many different ways to make them work, to no avail. My solution is to use an Ubuntu virtual machine. I primarily use those frameworks for AWS lambda developments with both Python and Node.js. Here is how I …