Uploading and Downloading Files in S3 with Ruby

To date, the latest Ruby AWS SDK is version 3. In this version, each resources has its own module while the version 2 had the one with everything, aws-sdk. To interact with S3 with the v3 SDK, let’s use the aws-sdk-s3 module.

Ruby has its own way to upload to S3. First of all, you need to create the s3 object and then call methods on that object. With Node.js, we first read the file and convert to the binary format and upload the data into the object (see here). With Python, we can simply pass the bucket name, key, and local file path in the upload function on the S3 object (see here).

Downloading is the same. We first need to create the S3 object that you want to retrieve and call the get() method.

The authentication is handled by configuring AWS CLI on your local machine or server. To configure AWS CLI, you can check the link here.

First of all, you need to install the module.

gem install aws-sdk-s3

Uploading File

The function takes 3 arguments, bucket name, prefix and local file path. From the File object, you can extract the file name in the path with basename(). Once the target S3 object is created, you can all the upload_file() function.

Downloading File

Downloading takes the same flow as uploading. You first need to create the object you want to retrieve with bucket name and object key. Then, call the get() method.

Data Engineering
Node Module to Retrieve Multiple Parameters from AWS Parameter Store

I created a node module to retrieve multiple parameters as the parameterName-parameterValue Json object from AWS Parameter Store. It is called aws-ssm-parameters and is available from Npm. The module is namespaced. So, you need to add the namespace @mdhnpm when you install it. npm i @mdhnpm/aws-ssm-parameters The main motivation for creating …

Data Engineering
Downloading All Public GitHub Gist Files

I used to use plug-ins to render code blocks for this blog. Yesterday, I decided to move all the code into GitHub Gist and inject them from there. Using a WordPress plugin to render code blocks can be problematic when update happens. Plugins might not be up to date. It …

Data Engineering
6
Loading Data Frame to Relational Database with R

Once you create a data frame with R, you may need to load it to a relational database for data persistence. You might have a data transformation batch job written in R and want to load database in a certain frequency. Here, I created a function to load data into …