Bulk loading with the copy command from a CSV file is the fastest option to load a large table with Postgres. In fact, loading data from a flat file is the fastest option in any relational databases. When you have a large table and need to load it to another …

MySQL has a parameter called max_connect_errors to prevent user from connecting to the database if they make too many connection errors (e.g. from wrong password) for security reason. The default for this value is 10. In case of AWS RDS, I don’t think the value is set. Therefore, when you …

The conventional wisdom is that, if you do not know what you are doing, you’re better off using the default configuration. Databases usually perform well with the default parameter values and premature performance optimisation often results in worse performance. On the other hand, Postgres is a different story. Postgres default …

Managing user access in MySQL is similar to Postgres (see Postgres user creation here). You can either manage the access privileges through roles or give privileges directly to users. This post is a quick reference for granting privileges to users. Let’s have a look at a simple use case. We …

You can deploy lambda functions manually with hard-coded environment variables. But, that is not a good development practice. When we deploy the function to production, the deployment process eliminates the necessity of changing any input parameters or configuration. Serverless is a lambda function deployment tools that enable good development practice …

In the previous post, we discussed how to move data from the source S3 bucket to the target whenever a new file is created in the source bucket by using AWS Lambda function. In this post, I will show you how to use Lambda to execute data ingestion from S3 …

Let’s say you have data coming into S3 in your AWS environment every 15 minutes and want to ingest it as it comes. The best approach for this near real-time ingestion is to use AWS lambda function. To demonstrate how to develop and deploy lambda function in AWS, we will …

To deploy a AWS lambda function written in Python, you need to package all the modules that are used in your code before pushing it to your environment. Serverless is a great lambda deployment framework. It takes care of all the packaging and deployment. With Python, the best approach to …

Docker and Serverless do not work well with Windows. I have tried so many different ways to make them work, to no avail. My solution is to use an Ubuntu virtual machine. I primarily use those frameworks for AWS lambda developments with both Python and Node.js. Here is how I …

If you need to use the latest Python 3 version in Linux as a non-root user, you can download the latest python in your space and install it there. In this way, you will have the separate Python development environment from the one that OS relies on. Once you installed …