Predict with Qlik AutoML

The holy grail for any company working with data is being able to predict the future and act on it. It's the highest rung of the Gartner maturity ladder for data analytics. Until recently, Qlik was mainly suited to supporting the first two rungs of the ladder. That was until Qlik made AutoML available within the Cloud platform in September of this year. With AutoML, we once again see an earlier acquisition, that of Big Squid, return as integrated functionality. In this blog, senior consultant Lennaert explains exactly how Qlik AutoML works.

Gartner ladder

As you can see in the Gartner maturity ladder for data analytics below, there are 4 levels of maturity. The first step is descriptive analytics, which answers the question of what happened. Think, for example, of a simple annual statement. It shows you exactly how much money came in and how much was spent. If you're a step further along, you also want to know, for example, why costs are higher this year. You start looking for connections and explanations. Predicting is then about making a prediction, based on historical data, about what's going to happen. Think, for example, of questions like: “Is this customer going to cancel their subscription?”, “When will this machine break down, and what exactly will break?” or “How much extra revenue can I expect if I launch an additional discount campaign?”. The final step is then to actually dictate, through actions, what will happen, so for example: “if we now give this customer a 10% discount on their monthly fee, they'll probably not cancel their subscription”.

Qlik AutoML

With AutoML, you can also make the moves to predicting and prescribing. AutoML stands for “Automated Machine Learning”. What normally makes the steps toward predictive and prescriptive so difficult is that you generally need very specialized knowledge for that, which is hard to find in the market. It's not for nothing that the role of Machine Learning Engineer ranks 4th on LinkedIn's top 25 fastest-growing job titles of 2022. So it's not always easy to hire an ML expert, and once they're on board, they'll quickly find themselves very busy trying to meet all the demand.

Machine Learning

Qlik AutoML makes it a lot easier to get started with Machine Learning. We'll explain exactly how it works further on in this blog, but the basic concept is that you give AutoML a dataset to train on and tell it which value you want to predict. AutoML will then train and optimize a number of suitable algorithms based on the type of prediction and tell you which one works best. You can then apply that prediction model to a new dataset, which you can then immediately visualize in Qlik. This way, not only can an ML expert get a lot more work done in less time, but it also becomes more accessible to data analysts with less expert knowledge in the field of ML.

Getting started with AutoML

The starting point for AutoML is data in the catalog of your Qlik Cloud environment. It doesn't matter whether it's a QVD or, for example, an Excel or CSV file. The most important thing is that this data file contains at least a “Target” and a number of potential factors (“Features”). The “Target” is the column you want to predict, so for example in our case the “Churned” column, which uses “yes” or “no” to indicate whether the specific customer ultimately canceled their subscription. Potential factors in our case are, for example, “PlanType”, the type of subscription, or “CustomerTenure”, the number of days a customer has already been subscribed. To start the experiment, we click “Add new” in Qlik Cloud and choose “New ML experiment”.

Once we've clicked that, we need to give our experiment a name and description and choose the Space where it will be stored. Once we've done that, Qlik asks us to point to the source dataset containing our historical data with the Target and Features in it. AutoML will then ask you which column contains the “Target”, in other words, which value you want to predict. As soon as you click this, all other fields are automatically marked as potential features. Using checkboxes, you can exclude certain columns if you already know beforehand that they won't help. In addition, AutoML will also give a warning for a number of fields, as in our case with the “Country” column. This column only contains 1 value and will therefore never be useful as a basis for a prediction. AutoML therefore excludes this value itself and shows the reason for exclusion via a warning icon. Finally, in the “Data Type” column, you can also indicate what the correct type is. This is mainly relevant for numerical values. Here's why: a number has an inherent order, 3 is smaller than 8. That ordering can matter for prediction. A higher price or fine will likely contribute to a higher chance of churn. Sometimes, however, a number is only a category, and the relative order doesn't necessarily matter. Think, for example, of a factory hall number. The size of the hall number won't carry any predictive value.

Once you've determined the “Target” and “Features” and applied the correct categorization, AutoML will make its own suggestion for a number of algorithms that could work well to make the prediction. You can indicate yourself whether you want to train all of these or whether you want to exclude a specific algorithm. You can also indicate whether “Hyperparameter optimization” should be used. It goes too far for this blog to explain what that optimization entails, but in short, with “Hyperparameter optimization”, AutoML will try to tune the internal parameters of the algorithms as well as possible for your specific case. In some cases, this can take quite a long time, which is why you have the option to set a maximum time that the optimization is allowed to take. If you set this, for example, to 1 hour, AutoML will stop the optimization after that one hour (if it wasn't finished yet) and present the best result achieved up to that point.

Once these settings are as desired, you press the “Run experiment” button. From that moment on, AutoML gets to work training all the selected models on your dataset. Depending on the type of variable you're trying to predict, you'll see a slightly different result. In our case, we're working with a binary classification (a customer either cancels their subscription or doesn't). Other types of problems are multi-class classifications and numerical predictions.

As you can see above, in our example “XGBoost Classification” came out on top. By default, Qlik presents a table (1) with a number of commonly used metrics for evaluating such an algorithm. In this default view, you'll see metrics like “Precision”, “Accuracy”, and “Recall”, but under the button in the top right (2) of the table, you can find many more metrics. It goes too far for this blog to explain each of the possible metrics, but for the experienced ML expert, these metrics describe how good the algorithm is and in what ways it can occasionally still go wrong.

Fortunately, AutoML also visually presents the most important conclusions based on these values. The charts at the bottom left show which features are important for the model and to what extent they influence the target. The charts at the bottom right give an impression of how accurate the model is at predicting the outcome and which types of errors are made most often (false positive or false negative).

From this point, you can go in 2 directions. If you're not yet satisfied with the outcome, you can try a second configuration, for example by including or excluding a certain Feature, or by training the winner again but this time with Hyperparameter optimization turned on. If you do this, a new set of algorithms will be trained again, and the results will be added to the table. This way, you can immediately see whether the adjustments have a positive or negative effect on the result.

Once you're satisfied with the result, you can select it using the checkbox and then choose “deploy”. Again, you'll need to give the deployment a name and description and choose a space where the deployment will be stored. Once saved, you can open this deployment. By clicking “Create prediction”, you can then choose a new dataset to which you want to apply the prediction. In our case, that's a dataset with current customers whose churn we want to predict. The output of the prediction will be saved in a new dataset, so we also need to enter a name for that. Besides the predictions, we can also have a number of other interesting QVDs written out, such as the SHAP values. We also need to indicate what the key of these QVDs will be to our original dataset. Ideally, that's an ID column that's already present, but if that's not the case, you can also have AutoML generate an ID. Finally, when we click the “Predict” button, Qlik gets to work and creates the new QVDs.

As mentioned, the result is a new dataset containing the prediction. In our case, this gives us a table with 4 columns: the ID as the key field, the “Churned_predicted” column, which contains the predicted value, and the “Churned_no” and “Churned_yes” columns, which each indicate the predicted probability of that value. So we can already take action if the chance of churn exceeds a certain percentage. We can then load this dataset into a dashboard to visualize it.

Want to see Qlik AutoML in action?

As you've seen above, it's relatively simple to go from a dataset to a prediction with AutoML, but there's also certainly plenty of room to go deeper for real ML experts. The feature is currently available for free to all Qlik Cloud Enterprise customers, but does come with a number of limitations. For instance, in the free version you're limited in the size of the dataset you can use to train the model. The number of rows multiplied by the number of columns may not exceed 100,000. You're also only allowed a limited number of deployments with the free version. Interested and want to see AutoML live in action? Then get in touch for a demo using the form below.

Written by Lennaert van den Brink
Senior Consultant