Let’s say you have data coming into S3 in your AWS environment every 15 minutes and want to ingest it as it comes. The best approach for this near real-time ingestion is to use AWS lambda function. To demonstrate how to develop and deploy lambda function in AWS, we will have a look at a simple use case of moving file from source S3 to target S3 as the file is created in the source.

AWS lambda supports a few different programming languages. I will use Python for this example. To deploy lambda functions, you need to package the modules used in the function. For some reason, modules downloaded and packaged from Windows sometimes do not work (e.g. pyscopg2 never works even if I download the binary version). From my experience, Python lambda development is the best to be done in Linux or Mac.

In this example, we will use Amazon Linux EC2 instance to develop the package and deploy it from there. You can of course use your local Mac or Linux machine for this.

To deploy lambda functions, I usually use the deployment framework called Serverless, which makes deployment much easier. Here, I would like to use the manual deployment from the AWS documentation here. Doing the deployment manually will teach you how the lambda function works. I think it is good to try it at least once before using more automated solutions.

We will also cover the topic on setting the environment variables in order not to hard-code the parameters that change between environments (e.g. database connection url, access details, or bucket names).

Summary Steps

1. Launch Linux EC2 instance
2. Install Python 3 and create virtual environment
3. Install required packages
4. Code Lambda function
5. Package function & modules into a zip file
6. Create Lambda Function Role
7. Deploy Lambda
8. Test function manually
9. Configure S3 source bucket as event source
10. Execute the function by pushing file to the source bucket

Steps

(1) Launch a Linux EC2 Instance

Micro instance is usually good enough for developing lambda function. After deployment, you can terminate the instance. Make sure to choose the right security group and subnet so that the instance can access the appropriate resources.

(2) Install Python 3 and create virtual environment

This step and the next step can be bootstrapped (How to Launch EC2 With Bootstrap in AWS). For Centos, Redhat or Amazon Linux, follow the steps here: How to Install Python 3 and Create Virtual Environment in Centos, Redhat and Amazon Linux. For Ubuntu, follow the steps here: How to Install Python 3 and Create Virtual Environment In Ubuntu.

(3) Install required packages

Make sure to activate the virtual environment where you want to develop the lambda function. The installed packages will be in the site-packages folder. You need to pack all the site-packages content into a zip file. AWS SDK for Python (like boto3 or botocore) is pre-installed for Lambda function and you do not need to pack them. Although we do not use any other package for this code example, let’s pack psycopg2 and import it in the function to make sure the packaged modules work.

Note that you need to install the binary version for psycopg2.

(4) Code Lambda Function

The PUT API call to the source S3 is the event to trigger the function. In Python, the event data is processed as dictionary.  See how the input file looks in the AWS documentation here. Make sure to change the parameter for your environment. Save this as inputFile.txt.

From the event input, you can grab bucket name and key to specify the newly created file in the source bucket.

The function grab the data from S3 and copy to a local tmp folder. Then, push it back to the target bucket. You can add transformation logic here if you need to.

Note that the target bucket name is taken from the environment variable. When we call the create lambda function API, we can set it. Alternatively, you can do it from the management console. This is recommended for the values that change between environment like database connection or access credentials. Make sure to import os to get environment variables.

PaAWS documentation about setting environment variables in Lambda functions is here. It uses Node.js as an example. It is the same in Python. You can also check out the blog about setting environment variables for Python here.

Once the function is created, use the input file (inputFile.txt) to test the function. You can add this for local testing.

(5) Package function & modules into a zip file

Now that your function is working, you can package the modules and code. You need to go to the site-packages folder for your virtual environment first then pack all the modules and then pack the python file.

(6) Create Lambda Function Role

By using IAM, you need to create a role for Lambda function. Follow the steps in the documentation here.

(7) Deploy Lambda

Once you have the zip file and lambda role, you can deploy the function. Make sure that you have the correct role and region selected. Handler has to be your-python-file-name.main-function-name.

This is where you can define environment variables. In this example, we are getting the target bucket name from the environment variable.

If you want to update the existing function, run the update script.

If you want to update the list of environment variables, run the update config script.

(8) Test the function manually

When your deployment is successful, invoke the function with the input file (inputFile.txt) set as payload.

(9) Configure S3 source bucket as event source

First of all, you need to add the permission to s3 (see here).

Go to the source bucket and configure event as below. In this example, my source bucket is lambda.test.source.

(10) Execute the function by pushing file to the source bucket

Let’s push a file to S3 with AWS console and check if the function moved the data into the target bucket. Here is the s3 copy command reference.

You can check the execution outcome in CloudWatch, too.

Yay!