Getting started with AutoML: feature engineering

Predicting the future: who wouldn't want that? With Qlik AutoML you can easily create predictive models based on your own data. As the name of the functionality suggests, Qlik AutoML automates a large part of this process for you. Even so, as a user you still have a lot of influence on how effective the predictive model is. In the blog series Getting started with AutoML, we explain what you can do to make your AutoML predictive models even better and how to integrate the results into your existing Qlik dashboards.

What is feature engineering?

In this first part of the blog, we look at "feature engineering". Feature engineering is the process of using domain knowledge to adapt your dataset so it can be used as effectively as possible for prediction.

Fortunately, AutoML can already take care of a number of steps in this process for you. You notice this as soon as you load a dataset into your AutoML experiment. See, for example, the image below:

In the 'Insights' column you can see that Qlik AutoML applies "One-hot encoding" to our columns with fewer than 14 distinct values. For columns with 14 or more distinct values, Qlik uses "Impact encoding". For many machine learning models, these encoding techniques are a requirement to handle this kind of information properly.

Another processing step Qlik performs automatically for you is "Null imputation". This means Qlik fills in fields where values are missing (in fewer than 50% of the rows) by itself, for example by taking the average of the remaining values and filling that in at the empty spots. Qlik decides for itself which method is used to perform this imputation.

Finally, Qlik also scales and normalizes values automatically. This means that for numerical values it calculates the mean and standard deviation, and then normalizes the column based on the number of standard deviations from the mean.

Adjusting your dataset yourself

Although Qlik does a lot of work for you, there are also steps you can take yourself to prepare your data even better for the machine learning model. Qlik can't make these optimizations on its own because they require context and real-world knowledge. We cover 3 common techniques:

1. Categorizing

Imagine we work in the flower industry and want to build a predictive model for the number of roses we sell. In our historical data we have a number of attributes, including the number of roses sold per date, by color and size.

We could throw this dataset straight into AutoML, but we know that roses sell more on specific days, such as around Mother's Day and Valentine's Day. We can build this knowledge into our dataset. The simplest way is to create a flag for special days:

This way our model knows which days are holidays and can take that into account. If we think about it a little longer, we know that extra roses are often sold in the days before and after such a holiday too, because not everyone gives roses on the day itself. We could then introduce 2 columns, for example:

2. Discretizing and grouping

Some fields in your dataset have a very large number of distinct values. Think of an age column, which in theory can contain all kinds of values between 0 and 100. The exact age is often not relevant to the value you want to predict. It doesn't make much difference whether someone is 32 or 35 years old. It might make a difference whether someone is 16 or 85. To solve this, you can transform the age column into age groups:

Of course, the groups can differ depending on the kind of data you work with. That is exactly where you add your expertise.

3. Splitting attributes

Sometimes columns in your dataset secretly contain multiple pieces of information. A classic example is date and time. A column combining date and time often has little predictive value because there are so many different values that they are almost unique per row.

You can solve this by splitting the field into separate date and time fields. From the date field you could derive the season or quarter, and the time field could be simplified further to the hour and minutes. Of course, you can also combine this with the grouping described above, for example by splitting the time into morning, afternoon, evening and night.