Infrastructure

Create Custom Python Module and Distribute with Git

When we create a custom Python module, we can distribute it through a Git repository. By using a private repo, you can keep your module private and use it in your deployment pipeline by calling pip install from the private repo. Pretty cool trick. In this example, I created a …

Infrastructure

How to Set up Daul-Authentication to GitHub and AWS CodeCommit

Both GitHub and AWS CodeCommit are used to version-control your source code. I use GitHub to share my code with the wider community and CodeCommit for production code for AWS. You may use other source control systems like BitBucket. Whatever the choice you make, it is convenient to set up …

Infrastructure

Advanced AWS Lambda Python Function Deployment with Serverless

You can deploy lambda functions manually with hard-coded environment variables. But, that is not a good development practice. When we deploy the function to production, the deployment process eliminates the necessity of changing any input parameters or configuration. Serverless is a lambda function deployment tools that enable good development practice …

Infrastructure

How to Deploy Python Lambda Function with Serverless in AWS (Introduction)

To deploy a AWS lambda function written in Python, you need to package all the modules that are used in your code before pushing it to your environment. Serverless is a great lambda deployment framework. It takes care of all the packaging and deployment. With Python, the best approach to …

Infrastructure

Setting up Docker and Serverless Development Environment in Ubuntu

Docker and Serverless do not work well with Windows. I have tried so many different ways to make them work, to no avail. My solution is to use an Ubuntu virtual machine. I primarily use those frameworks for AWS lambda developments with both Python and Node.js. Here is how I …

Infrastructure

How to Install Python 3 and Create Virtual Environment In Ubuntu

If you need to use the latest Python 3 version in Linux as a non-root user, you can download the latest python in your space and install it there. In this way, you will have the separate Python development environment from the one that OS relies on. Once you installed …

Infrastructure

How To Install Python 3 and Create Virtual Environment in Centos, Redhat and Amazon Linux

The scope of this post is to install Python 3 for a user and create a virtual environment in Centos, Redhat or Amazone Linux. Linux comes with Python 2.7 and the best practice is not to mess with. This is because Linux OS has some dependency and upgrading it to …

Infrastructure

Resolving Docker Deamon Is Not Running Error From Command Prompt

I recently installed Docker Toolbox on Windows machine (Windows 10 Home version). We all know Docker is awesome and serverless microservice architecture is hot. Docker has a better tool, called Docker for Windows. Unfortunately, this does not work with Windows 10 Home version (it works with Windows 10 Professional versions …

Infrastructure

How To Install Apache Spark On Windows

Apache Spark is a powerful framework to utilise cluster-computing for data procession, streaming and machine learning. Its native language is Scala. It also has multi-language support with Python, Java and R. Spark is easy to use and comparably faster than MapReduce. For example, you can write Spark on the Hadoop …

Infrastructure

How To Resolve Refusing to serve hidden directory, via 404 Error with Jupyter notebook.

I hate 404 errors. Whenever I get it, I stare at the screen with disdain because it cannot be fixed most of the time. What cannot be found cannot be found. But, fixing this one is easy! By default, Anaconda serves notebooks from local directory: C:\ in Windows. To resolve …