When we create a custom Python module, we can distribute it through a Git repository. By using a private repo, you can keep your module private and use it in your deployment pipeline by calling pip install from the private repo. Pretty cool trick.

In this example, I created a simple Python module to do S3 download and upload and pushed it to a Git repo for pip installation.

First of all, we need to have the right folder structure with setup.py.

The folder structure will look like:

s3uploader
    - README.txt
    - LICENSE.txt
    - setup.py
    - s3uploader
        - s3.py
        - __init__.py

The key is to have the correct setup.py file. This describes the metadata about the module as well as is used for installation.

setup.py

The top level project folder is the name of the module. There is another folder with the same name, including __init__.py and actual python code that contains the class.

The init file references the class that can be imported by the module.

__init__.py

Here is the actual S3 class that does upload and download.

s3.py

Once everything is done, let’s push it to the repo and all done. Installation is easy. You can pass the repo url in pip install as below.

Installation

pip install git+https://github.com/mydatahack/s3uploader

You can now test the module.

It works!

Yay!