130 years ago, when our client first started, a cloud may have been a bad omen. It could have meant the impending approach of a storm preventing its ships from reaching their destination. Today? The cloud is what helps our clients make the best decisions, based on hard, consistent, relevant data. One of these vessels shipping BI services through the cloud is called Levi9, with a team helmed by Carmen Girigan, Senior Data Engineer, and Eliza Enache, Medior Data Engineer.
From data to information
If a business founded in 1890 is still thriving today, it must mean that it has adapted over and over again. More recently, this meant adopting a cloud-based data solution with the help of Levi9 and other cloud providers. Vroon, an international company with a vast tradition in naval transport, was looking for ways to optimize their data solution, in order to make better decisions. We picked up a legacy on-premises solution and transformed it into a personalized, highly-scalable Single Source of Truth in the cloud.
The main challenge was picking up a complex solution using external data sources, requiring maintenance and development on multiple modules, and offering limited aggregated results. To learn how the business was doing, Vroon business users had to look into multiple sources.
However, data is mostly useless unless it becomes information. But what does this mean?
“You might look at Data as the input that your brain perceives through the senses,” says Eliza Enache, one of the Levi9 “sailors”. “The brain then creates connections between these data points by applying certain logical processes (filtering, algorithms, etc.). The connections between data points (and connections of connections) might be called Information.”
“Raw data is useful only when it is organized and interpreted in such a way that it helps in decision making, becoming information,” completes Carmen Girigan, also a member of the Levi9 team.
After working with Levi9, Vroon users interrogate one service and get the answers they need.
From SQL Server to Azure cloud solutions
The Vroon business had been getting information using on-site Microsoft technologies, relying heavily on SQL Servers. Data was coming in from various siloed sources such as accounting, procurement, invoicing, customers, vessel activities, incident reporting, and many others.
Data were extracted, transformed, and loaded into the warehouse, via Microsoft SQL Server Integration Services, as well as custom applications and SQL Server stored procedures. On top of the data marts, there were 5 analytical databases with 11 multidimensional OLAP cubes, developed with Microsoft SQL Server Analysis Services. These cubes would then be accessed via the reporting tool, for reports, dashboards, and analysis.
The cloud conversion started in 2020. The new solution was based on Microsoft Azure technologies, in order to streamline extract-load-transform (ELT) processes and maintenance, by using a gradual approach that would allow evaluation and constant adjustment to new situations, as well as high availability for the users’ new requirements.
Now, data is extracted from its sources via Azure Data Factory and loaded into an Azure Data Lake container. It then goes through a transformation process using Azure Databricks or Azure Synapse Analytics. Afterward, it is loaded into an Azure Synapse Analytics data warehouse, based on which a tabular model is developed with Visual Studio and later deployed to Azure Analysis Services. Power BI is used for data presentation and user self-service report creation.
The result was the transformation into a modern, cloud-based, and highly-scalable solution, that provides information in a unified way: a Single Source of Truth for multiple business areas.
Having a single source of truth is essential for enabling good business decisions. “Everybody sees the same figures, at the same time, and it’s more reliable,” Carmen explains. “Long story short, you avoid conflicting ‘versions of truth’ when different users analyze data exported at different points in time,” adds Eliza.
Cloud sailor's tip: Clean your data like a ship’s floor
Whenever thinking of a ship, one image inevitably comes to mind: sailors cleaning the deck, almost to the point of obsession. In the 18th century, this was done to keep the crew healthy.
Today’s Levi9 data engineers have a similar attitude towards data: it should be maintained clean, to avoid the risk of infecting the budget, reports, and any meaningful analysis.
“In the cloud, resources are much easier to maintain and scale — but we do have to pay attention to costs. There is no point in having duplicated or unutilized data, or transformations, in the cloud,” explains Carmen. Eliza adds that there is much more than that. “During the testing phase, we noticed that redundant or obsolete data led to improper analysis and estimation, and diversion from the ‘main track’ — which in turn led to an increase in time and costs, while also blocking further developments. Some of these redundancies made us go back through the whole logic.”
Those who don’t empathize with nautical experience might compare data cleansing with their wardrobes: “Just think of that closet full of clothes you don’t like to wear, things you’ve kept for years only cluttering your space and making for a hefty baggage,” says Eliza.
An ongoing adventure
Part of the Vroon’s legacy system is still running and is in the process of being transformed. Eliza explains why sometimes this is a better solution: “We soon realized that converting and rewriting the whole logic, from old technologies into Azure, would take a lot of time – thus stalling any further development initiative. As such, we devised a temporary workaround: part of the data is directly brought to and processed in Azure, while another part is brought from the on-premises data warehouse, and further integrated into the unified model.”
“But we won’t do everything at once. First, we’ll do a pilot project”, adds Carmen. Just like sending a sentinel ship before moving the entire fleet, a pilot provides insight into what the job entails.
Fair winds and following seas
All in all, the project continues to be a learning opportunity for the team. All the Levi9 Vroon sailors have at least one Azure certification, as the department and company strategy aligned with the work. Carmen Girigan and Eliza Enache stress that they owe much of their learning to the other half of the initial engineering team that had established the foundations on which they continued to build in the cloud.
With the new pilot project, the adventure continues. May the team have fair winds and following seas!
