8 min read
Migrating to Qlik Cloud
Anyone who has followed Qlik's developments a bit lately knows that Qlik is going all-in on Qlik Cloud. This makes it increasingly interesting to consider moving your on-premise Qlik environment to Qlik Cloud. However, such a move to the cloud does mean you also need to make sure all the data from your various systems is brought to Qlik Cloud too. There are several ways to do this. In this blog, we explain what these methods are and what the advantages and disadvantages of each are, so you too can create a plan of approach for migrating your Qlik environment.
By hand
The first method is also the simplest. In Qlik Cloud, you can manually upload all kinds of data files into the Cloud environment. This method is particularly suitable for ad-hoc analyses or for datasets that don't change very much. Think, for example, of an Excel file with annual budgets. If the data in your file changes, you can simply re-upload the file to replace the existing one.
There are some limitations to uploading files. Not all file formats are supported, for example, although the most common file formats are available (see the image below for the formats currently supported). There's also a limit to the file size. A file can be a maximum of 100GB. That's quite a lot, but files of 6GB or larger will have an impact on the performance of your Cloud environment during upload, because the Qlik Engine will partly be used for this.
Fast and simple
Suitable for self-service users
Data doesn't refresh unless you re-upload it yourself
Large files impact performance and take a long time to upload
Manual work, so dependent on people
Connecting directly from the Cloud
Just as you're used to with your on-premise setup, in Qlik Cloud you can create direct connections to a large number of different sources. More than 70 different connectors are available by default to all kinds of source systems, such as databases (SQL, MySQL, Oracle, PostgreSQL, etc.) but also platforms such as SAP, Salesforce, Office 365, and applications like Twitter, Facebook, ServiceNow, or MailChimp. Generic connections via REST API or OData are also possible this way, allowing you to connect to almost any modern SaaS application.
Once such a connection has been created, you can load and process data from these systems to your heart's content. The main downside of this method is that your source system needs to be accessible from outside. For most Cloud-based and SaaS applications, that's not a problem, but for example a database system running on-premise is often something you'd rather keep as closed off from the outside world as possible, for cyber security reasons. Of course, you can configure a firewall very specifically to only allow traffic from Qlik Cloud addresses (see this page for the current IP addresses of the Qlik Cloud platform), but it still means you're creating a potential attack route for hackers.
Works the way you're used to from Qlik Sense on Windows
Many standard connectors available
Easy to create from the Qlik Cloud interface
Less suitable for on-premise databases that aren't externally available
Qlik Data Transfer
Qlik Data Transfer is an application you install on a server within your own network. This means the application sits behind your network's firewall and can therefore usually access various data sources without problems. With Qlik Data Transfer, you can automatically push data files or database records to your Qlik Cloud environment. This can be done in several different ways. The first option is to have Qlik Data Transfer monitor a (network) folder. All files that are added or modified in this folder are immediately uploaded to a workspace of your choice in the Qlik Cloud environment. The second option is to connect to a database system via ODBC. You can then define a query in Qlik Data Transfer that runs periodically, after which the result is stored in QVD format and then uploaded to the Qlik Cloud environment. The last option is to point to a QlikView or Qlik Sense file (.qvw or .qvf). Qlik Data Transfer will then periodically run the load script of this file and then save all the tables in the data model as QVD and upload them to Qlik Cloud.
However, Qlik Data Transfer does have some important disadvantages and limitations. The application is a bit more difficult to configure, and the configuration also needs to be done on an on-premise server. This certainly can't be done by self-service users. In addition, with Qlik Data Transfer you can upload a maximum of 200 tables or files per source, and the resulting files may not be larger than 6GB. For larger databases, Qlik Data Transfer is therefore not an option.
On-premise installation, so within the internal network and only outbound network traffic
Automated file uploads
Not suitable for self-service users
Not suitable for larger files and databases
On-premise installation also means infrastructure costs
Hybrid Qlik environment
In a hybrid Qlik environment, you use both Qlik Sense on Windows and Qlik Cloud. In this case, the on-premise Qlik environment is located on your own infrastructure within your own network and performs all the reloads. As soon as the reload is finished, the resulting dashboard is automatically uploaded to the Qlik Cloud environment. End users then consume the dashboard via the Qlik Cloud environment.
This setup is mainly used a lot during the migration process from the on-premise environment to Qlik Cloud. The advantage is that you can immediately offer end users all the benefits of the new functionality and user experience of Qlik Cloud, while behind the scenes you have all the time you need to carry out the data migrations.
The biggest downside of this method is that you still have a large part of your Qlik infrastructure on-premise. This means, among other things, a double Qlik license and costs for all the hardware and its management. An advantage, on the other hand, is that you retain access to an on-premise Qlik environment, so you still have access to, for example, NPrinting and other functionality that isn't (yet) fully available in Qlik Cloud.
The best of both worlds, on-premise and Cloud
Very quick to set up from an existing Qlik Sense on Windows environment
Extra costs for infrastructure and double licenses
Data warehouse in the Cloud
Under the "connecting directly" option, we already told you that connecting to databases hosted in the Cloud is a good option. A possible solution for getting your data into Qlik Cloud is therefore to first unlock your on-premise data sources into a Cloud-based data warehouse. This then becomes the central place where all the data from your systems flows together and is processed, after which Qlik, as well as other end systems, can make use of this single version of the truth.
There are all kinds of different alternatives for setting up such a data warehouse. In any case, you'll need a provider of a Cloud-based data platform, such as Azure, AWS, Snowflake, Databricks, or Google BigQuery. When setting up such a data warehouse, it's recommended to also make use of a Data Warehouse Automation tool. At E-mergo, we work a lot with TimeXtender in combination with Azure, but recently there's also the option to do this entirely through Qlik using Qlik Data Integration. Once the data is available in a Cloud-based data warehouse, it's just a matter of creating a direct connection, and you're ready to get started with Qlik Cloud.
A central place to bring your data together
Suitable for virtually all types of source systems
Extra investment to set up
Qlik Data Gateway
Qlik Data Gateway is the most recent addition to the list of options for bringing your data to Qlik Cloud. With Qlik Data Gateway, you install a piece of software near your database (for example, on the same server). The software then ensures a secure connection is made with the Qlik Cloud environment. This way, you can connect directly to this database from Qlik Cloud, without needing to allow any outbound traffic.
Unfortunately, Qlik Data Gateway is only available for a limited number of sources (see this link for the current list), but for these sources it's often the best solution. Configuring the Gateway itself generally needs to be done by an administrator, but once the Gateway has been set up, self-service users can create their own connections without any problems, just as they can with the "connecting directly" option.
On-premise installation, so within the internal network and only outbound network traffic
Automated file uploads
Suitable for self-service users once the gateway is configured
On-premise installation means extra load on infrastructure
Conclusion
In this blog, we've shown you that there are many different ways to bring your data to Qlik Cloud. Each option has its pros and cons, so it's important to carefully consider the requirements and wishes of your specific situation in order to make the right choice. In practice, you'll find that multiple methods will be used to upload data within a single Qlik Cloud environment. Want to bring your data to Qlik Cloud too, but finding it difficult to choose the right option? At E-mergo, we're of course happy to think along with you!
Written by Lennaert van den Brink
Senior Consultant