5 min read
Implement your TimeXtender on Microsoft Fabric
With the most recent release of TimeXtender, it's now possible to base your TimeXtender implementation entirely on Microsoft Fabric. This means Fabric Lakehouse for both Ingest and Prepare instances, and XMLA endpoints for Power BI models. This lands the entire data pipeline in Microsoft Fabric.
Although not all transformation functions are yet available in Prepare instances, the most commonly used functions are. For details on Fabric support per TimeXtender layer, see:
With this release, we were able to set up a Fabric Capacity to experience what it's like to build solutions entirely on Fabric. Setting up Fabric is straightforward; you don't have to make many choices:
- Where do you want your infrastructure to be hosted?
- How much Capacity do you want to give your Fabric workspace? (in other words, how much horsepower you have at your disposal)
Beyond that, you have a few technical steps to take to automate Fabric from TimeXtender, and then you're set. Microsoft Fabric has a uniform scale/cost model that depends on "Capacity": an abstraction of compute power that can be used for any Fabric feature. Fabric storage is also abstracted in OneLake. This means there's less to think about (but also fewer options for fine-tuning) when implementing. The smallest available Fabric Capacity already gives you access to everything you need to build a TimeXtender implementation.
TimeXtender builds its solutions from three different types of layers: Ingest (extracting data from the source and storing it in a uniform format), Prepare (transforming data into its final form), Deliver (bringing data to consuming BI tools). I'll give you the current status of each type of Instance regarding how they work in Microsoft Fabric:
Visual representation of the 3 layers in TimeXtender
Ingest
For the most part, the Ingest instance works the same way you're used to in TimeXtender; there are currently only a few limitations. You can find a full overview of these here.
If you want to try out Fabric Lakehouse Prepare, for now you also need to run your Ingest against Lakehouse. Because TimeXtender leaves part of the costly compute power to the Virtual Machine running the Ingest service, Capacity consumption is lower than you would expect for Ingest. At the moment, overall performance is lower compared to what you can achieve with manual tuning in a SQL IaaS / ADLS Gen2 infrastructure; this is a trade-off against simplified management — after all, Fabric and OneLake make this a lot easier.
Prepare
For Prepare instances, the situation is a bit more dynamic: most implementations currently won't be able to work fully "the TimeXtender way", because they depend on functionality that isn't yet available for Fabric Lakehouse.
On the other hand, you can adapt your code by attaching Spark Notebooks to your tables. This makes your solution less portable and more code-intensive. One of the nice things is that if you're just moving data, Shortcuts to the data in Ingest are used without needing to make copies of the data. Performance currently isn't as good as other infrastructure options, but this is largely due to cluster startup times. TimeXtender is working on improvements in this area, so expect this to improve soon.
Deliver
Working against Power BI XMLA endpoints has of course always worked in TimeXtender Data Integration, and the transition from Premium Capacity to Fabric Capacity doesn't change that. Being able to deliver the same data model to Qlik and Tableau alongside Power BI gives you maximum flexibility.
Right now, it's useful to experiment and see what using Fabric in a Data Platform can bring. In that context, evaluating the state of the art with TimeXtender Data Integration is a good idea; one of my favorite TimeXtender features is that you can change the infrastructure on the fly to carry out feature and technical evaluations without much work.
If the option to use Service Principal authentication on Run On Demand Item Jobs becomes available from Microsoft, this will remove a security drawback that currently exists. Read the Microsoft documentation on this topic here.
New developments
Both Microsoft Fabric and TimeXtender are developing rapidly, which can make it hard to keep up with everything. For TimeXtender, the annual Xtend event is a great way to stay up to date on developments and meet partners. For Microsoft Fabric, we'll be attending the Microsoft Fabric Community Conference in Las Vegas at the end of March, so hopefully we'll hear more exciting news there. TimeXtender will also be present at Fabcon, so stop by if you're there!
Written by Ruairidh Smith,
Senior Consultant at E-mergo