In modern web development, we try to avoid building a website from scratch if we can. Websites are usually built on top of a platform like Sitecore, Drupal, WordPress and so on. When I first thought about creating this blog, I tried to code everything from scratch, going against the …

Joiner is the stage to join tables in Informatica Cloud (see a quick introduction for Joiner Transformation here). If you have a large volume of data, the joiner transformation becomes very slow without performance optimisation. In this post, we will show you a few tricks that you can use to …

By default, the secure agent can run 2 data synchronisation tasks at a time. This constraint can become limiting quickly especially when multiple developers are building and testing the data synchronisation tasks at the same time. By adding a custom property on the secure agent, you can run more than …

In the previous post, we used grid search to find the best hyper parameter for the neural network model with R’s caret package. Here, let’s use Python and scikit-learn package to optimise a neural network model. Just like the caret package, scikit-learn has a pre-built function for hyper parameter search. …

Once you finish training the model and are happy with it, you may need to consider saving the model. Otherwise, you will loose the model once you close the session. The model you create in R session is not persistent, only existing in the memory temporarily. Most of the time, …

Writing code to do machine learning is easy. What makes it difficult is the optimisation. By large, there are two ways to optimise your model. Feature selection & transformation Model parameter optimisation Both are hard-core topics and neither can be covered in this post. Feature selection and transformation often require …

Once you create an awesome data science application, it is time for you to deploy it. There are many ways to productionise them. The focus here is deploying Spark applications by using the AWS big data infrastructure. From my experience with the AWS stack and Spark development, I will discuss …

I love dissecting and analysing my favourite books by reading them again and again, discussing them with my like-minded friends, getting to know the authors and reading other people’s writings about them. My obsession with books lead me to thinking, how can I visualise them in interesting ways? Making Word …

Making a word cloud is fun and easy. It is a way of looking at text data and gain a different perspective. For example, if you have a bunch of customer feedback about your product, you can quickly create a word cloud to get some ideas. When I work with …