Some years ago, I needed an implementation of control charts (Statistical Process Control or SPC charts) in TypeScript. I ended up building it myself. It eventually became quite a popular add-on for a BI tool.

Building it took a couple of months as designing the API, implementing the charts and plotting, and identifying all the needed user settings required lots of trial and error.

A few weeks ago, I needed control charts again. This time in Python.

There are several open-source control charts implementations available for Python. But none of them had the features I needed. Control charts are surprisingly easy to implement incorrectly. It is not uncommon to find even popular libraries calculating the limits in a wrong way.

So I started to think about how much effort it would take to port my TypeScript library to Python.

No matter how I looked at the problem, the conclusion was the same: too much work. As it was a personal side project which would be done in my free time, I simply didn’t have the time needed.

XmR chart plotted with the TypeScript library

AI-Assisted Coding - Part 1

By the end of the last year, there were more and more stories about AI-assisted coding. How the latest models were game changers.

I had tried these kinds of tool before but my experience was that they were useful for small tasks and struggled with larger codebases. The TypeScript library has roughly 10,000 lines of code, making it a medium-sized project. This made me a bit skeptical.

Four weeks ago, I decided to give a proper test. I bought a Cursor Pro subscription and dug in.

The first attempt was straightforward: I asked it to port the code.

You are an expert programmer fluent with typescript and python. You port typescript code to python and create simple, pythonic code. You create simple but powerful function signatures.

in @typescript_lib folder, there is a typescript implementation of statistical process charts (SPC) calculations. Your job is to port it to python. Example interfaces for the calculation function you will find from the @tests/test_imr_chart.py file.

Use pandas api and its functions, otherwise keep dependencies minimum. If there is an elegant way to implement some required functionality using external library, ask before using it.

Mandatory requirements: the calculations must be identical, all functionality must be implemented in the new version, all the tests must pass, the new code must be pythonic, i.e. all the typescript idiosyncracies has to be replaced to python ones, the code must elegant and simple

Ask questions as long as you need and make sure you understand all the detail.

It chewed on the problem for a quite long time, but in the end it produced a working version of the library in Python. It wasn’t elegant and was definitely not maintainable, but it worked.

Using The Planning Mode

In the next attempt, we started by planning the solution together with the AI.

The session started like this (in verbatim, with errors and all):

You are an export [SIC :)] python programmer. You design elegant and pythonese apis.

Your task is to first design an api for SPC/Control Charts library. Library calculates the values for a separate charting library so that’s easy to plot the control chart visuals.

The library should not have any external dependencies unless it’s absolutely required or makes perfect sense. The libraries used must the main stream, common, and supported.

The library needs to have separate functions to create diffent kinds of control chart objects (function will return these, they will contain the observations, labels, groups, ucls, lcls, usls, lsls, etc). The functions should have as similar apis as possible. The api must be as easy and intuitive as possible.

Here’s a sketch of the input and output data objects. Review and make suggestions how to turn it into native python api and easier to use.

Ask questions as long as everything is clear. Do not cut corners.

"""
Data models for control charts.

This module defines the data structures used throughout the pycontrolcharts package.
All models use Python dataclasses for clean, type-safe data structures.
"""

from dataclasses import dataclass, field
from enum import IntEnum
from typing import Optional, List, Union

class RunType(IntEnum):
    """
    Types of run test violations.
    
    Based on Western Electric and Nelson run test rules.

[…]

It took several hours of back and forth. I provided examples of code, answered a multitude of questions, and when the plan was finally ready, we had created almost a megabyte of chatlogs and markdown. Before the implementation started we had a detailed implementation plan.

This made all the difference. After the second round I had a solid library in my hands.

I used the TypeScript library to validate the result of the Python implementation. The charts differed in a couple of tests. Funnily, one of the run test implementations in the original code did not detect one of the edge cases. I had no idea about this bug, but the AI had fixed it while porting the code. Deliberately or not, I don’t know.

The code required some manual touches. Some parts of the plotting were not good, for example padding around the chart. No matter how I tried, it just didn’t get it right but kept on suggesting more and more complex solutions. The complex run tests were implemented as a state machine. This logic was too complex for the AI.

My Findings

In the end, I was blown away by how amazingly helpful these tools have become.

Productivity

• Substantial acceleration. My rough estimate is that development was 5–10x faster compared to writing everything manually.
• Faster learning. Discovering “pythonic” approaches was effortless. The feedback loop was immediate.

Quality

• Code quality is reasonable. Especially in repetitive or boilerplate code, the output was often cleaner than what I would have written initially.
• Improved documentation. The AI consistently generated more comprehensive documentation than I typically would for a side project.

Limits

• AI struggles with difficult code. The model struggled with complex layouts and the non-trivial state machine. I ended up implementing those parts manually.
• Good architecture and design do not emerge automatically. Without explicit guidance, the structure degrades quickly.
• Using AI well is a skill. It resembles mentoring a junior developer. Instructions must be precise. Feedback must be continuous.

Other benefits

• AI enables planning and revising easily. With AI tools, I would have implemented the first version, then realised a better way to do it, but at that point have no time nor energy to change it.
• Building stuff is fun again. For me, the joy is back to programming or should I say building. No need to remember all the nitty-gritty details as AI is pretty good with that.
• Your mileage will vary. This is not something everyone will enjoy. Personally, I get a kick out of building something new. If you love creating elegant code by hand, you might not enjoy AI-assisted coding.

Sometimes AI produced excellent code for a long stretch and then suddenly introduced subtle regressions. Occasionally it modified unrelated parts of the code. It can lull you into complacency to “git add .” if you are not vigilant.

AI is extremely good at mechanical translation and boilerplate, but still weak at architectural clarity and complex state logic.

What Does This Mean?

A few observations:

• AI-assisted coding can’t be ignored. It increases developer efficiency substantially.
• The tool development is extremely fast at the moment. It’s not easy to stay up-to-date.
• It’s a lever. Used well and it provides you good results fast. Used poorly and it creates an unmaintainable codebase even faster.
• It takes skill to use it and learning takes time.
• It’s an excellent tool for prototyping.
• It could be a huge asset for code migrations and give a new life to legacy codebases.
• With a proper feedback loop, it can provide better results than humans. For example in performance optimizations, AI can try out different solutions if it can do the performance comparisons by itself.

Introducing pyControlCharts

XmR chart plotted with pyControlCharts.

You can find the pyControlCharts library from Github or use it from pypi. It has all the major control charts along with the most important run tests. Most importantly, it provides the charts in both machine- and human-readable form. The charts are not just for human consumption but you can easily use them in data pipelines, AI models, and as an AI agent tool.

Control charts are fundamentally about separating signal from noise in variation. In an era where automated systems increasingly make decisions based on data, having that signal available in structured form matters more than ever.

Earlier this year, Databricks introduced a new syntax for data querying. It was originally introduced by Google in BigQuery, and now Databricks has adopted it as well. SQL Pipe syntax resembles Microsoft’s KQL, but it is much easier to learn because the syntax has not been reinvented. I like KQL, but I don’t like that I have to look up all the commands from the manual as almost all the functions have been named differently than they have been named in SQL.

One big difference between SQL and the pipe syntax is that SQL is declarative and the order of the operations does not resemble the order what the machine does to return the data you requested. SQL has actually received quite a lot of critique for how unintuitive the order of operations is. SQL Pipes is not procedural language, but the order of the operations is much more intuitive and easier to grasp for beginners.

How to Use it

Let’s say you are a data geek who wants to stay in Stockholm during the Midsummer festival to see the Maypole in Skansen. You decide to look for accommodation on Airbnb but as a data geek, you want to use SQL and Databricks instead of the Airbnb website. You decide to download scraped data from the Inside Airbnb website and crunch the data in Databricks.

Prepare the data using a notebook. First, download the datasets:

%sh
curl https://data.insideairbnb.com/sweden/stockholms-l%C3%A4n/stockholm/2025-03-23/data/listings.csv.gz --output /tmp/listings.csv.gz
curl https://data.insideairbnb.com/sweden/stockholms-l%C3%A4n/stockholm/2025-03-23/data/reviews.csv.gz --output /tmp/reviews.csv.gz
curl https://data.insideairbnb.com/sweden/stockholms-l%C3%A4n/stockholm/2025-03-23/data/calendar.csv.gz --output /tmp/calendar.csv.gz

Then, save it as a table:

%python
listings_df = spark.read.csv("file:///tmp/listings.csv.gz", header=True, inferSchema=True, multiLine=True, escape='"')
listings_df.write.mode("overwrite").saveAsTable("listings")

reviews_df = spark.read.csv("file:///tmp/reviews.csv.gz", header=True, inferSchema=True, multiLine=True, escape='"')
reviews_df.write.mode("overwrite").saveAsTable("reviews")

calendar_df = spark.read.csv("file:///tmp/calendar.csv.gz", header=True, inferSchema=True, multiLine=True, escape='"')
calendar_df.write.mode("overwrite").saveAsTable("calendar")

You want to see the descriptions of available (or partially available) listed apartments for your dated, hosted by a person, that has at least one apartment listed with 50 reviews or more.

One way to do this using the traditional SQL is to use CTEs (Common Table Expression). First, you would have to craft a CTE to find the available listings, another CTE to find hosts having a listing with the required number of reviews, and so on until you have all the steps crafted. Finally, you would have to mesh this all together into the final results. You can do this in a Databricks SQL Query:

WITH listings_with_more_than_50_reviews AS (
    SELECT listing_id, COUNT(*) AS review_count
    FROM reviews AS r 
    GROUP BY listing_id
    HAVING review_count >= 50
), 

hosts_with_listing_that_has_more_than_50_reviews AS (
    SELECT host_id, MAX(r.review_count) AS max_review_count_of_hosts_listing
    FROM listings_with_more_than_50_reviews AS r
        JOIN listings AS l ON r.listing_id = l.id
    GROUP BY host_id
), 

listings_of_our_qualified_hosts AS (
    SELECT id AS listing_id, MAX(h.max_review_count_of_hosts_listing) AS max_review_count_of_hosts_listing
    FROM listings
        JOIN hosts_with_listing_that_has_more_than_50_reviews AS h ON listings.host_id = h.host_id
    GROUP BY id
), 

available_listings AS (
    SELECT listing_id
    FROM calendar
    WHERE 
        (date == '2025-06-20' AND available == 't')
        OR (date == '2025-06-21' AND available == 't')
    GROUP BY listing_id
),

final_result_table AS (
    SELECT l.id AS listing_id, l.host_id, r.max_review_count_of_hosts_listing, l.name, l.description
    FROM listings_of_our_qualified_hosts AS r
        JOIN available_listings AS f ON r.listing_id = f.listing_id
        JOIN listings AS l ON r.listing_id = l.id
    ORDER BY r.max_review_count_of_hosts_listing DESC, r.listing_id
)

SELECT *
FROM final_result_table

This is how the query, or the CTE flow, is structured:

As the image shows, we have two main data flows. One that starts from the reviews and then the listings are joined into it several times. The second flow starts from the calendar and joins into the final results. These two flows are combined in the final_result_table CTE. It is perfectly doable using CTEs but the flow is a bit clumsy. Joining the two flows works perfectly when using CTEs.

How would you do it using SQL Pipe Syntax?

FROM reviews AS r
|> AGGREGATE COUNT(*) AS review_count 
  GROUP BY listing_id
|> WHERE review_count >= 50 
|> AS listings_with_more_than_50_reviews 

|> JOIN listings AS l ON listings_with_more_than_50_reviews.listing_id = l.id
|> AGGREGATE MAX(review_count) AS max_review_count_of_hosts_listing 
  GROUP BY l.host_id
|> AS hosts_with_listing_that_has_more_than_50_reviews

|> JOIN listings AS l2 ON hosts_with_listing_that_has_more_than_50_reviews.host_id = l2.host_id
|> SELECT DISTINCT l2.id,  hosts_with_listing_that_has_more_than_50_reviews.max_review_count_of_hosts_listing
|> AS listings_of_our_qualified_hosts

|> JOIN calendar AS c ON listings_of_our_qualified_hosts.id = c.listing_id
|> WHERE (c.date == '2025-06-20' AND c.available == 't')
        OR (c.date == '2025-06-21' AND c.available == 't')
|> AGGREGATE MAX(max_review_count_of_hosts_listing) AS max_review_count_of_hosts_listing 
  GROUP BY c.listing_id
|> AS available_listings

|> JOIN listings AS l3 on available_listings.listing_id = l3.id 
|> SELECT available_listings.listing_id, l3.host_id, available_listings.max_review_count_of_hosts_listing, l3.name, l3.description
|> ORDER BY available_listings.max_review_count_of_hosts_listing DESC, available_listings.listing_id
|> AS final_result_table

Look at the difference right from the beginning:

FROM reviews AS r
|> AGGREGATE COUNT(*) AS review_count 
  GROUP BY listing_id
|> WHERE review_count >= 50 

vs

SELECT listing_id, COUNT(*) AS review_count
FROM reviews AS r 
GROUP BY listing_id
HAVING review_count >= 50

The SQL Pipeline flow is much closer to how one naturally thinks. The long flow is much clearer and natural using the SQL Pipeline syntax. We can also use multiple WHERE clauses, unlike in the regular SQL. Joining the two dataflows works well here but if that flow were longer, the CTEs might have an advantage as the pipeline syntax does not allow using them. At least not yet.

When and why would you use SQL Pipe syntax?

The Pipe syntax is more intuitive and faster to understand when reading code written by other people. Especially when the code does not combine multiple data flows into one. I’m sure it will be much faster for beginners to grasp but it might take some time for experienced SQL users to adapt.

It also has some small improvements to the SQL syntax, for example multiple WHERE clauses, so no need to do WHERE 1=1 to prevent errors when developing the code. DROP is also a big improvement, as it allows you to specify which columns to remove, instead of cataloging all the needed ones.

I’d expect the pipe syntax to be especially beneficial in cases where the data transformations are long, requiring a lot of filtering but the logic required might not be super complex. Also, the transformations written in SQL Pipe syntax might be easier to create and maintain, even by users who don’t have a lot of experience in SQL.

As we want to track Fabric, why not do the tracking in Fabric? From my Github repository, you can find (almost all of) the required Fabric items, and an installation guide on how to do the tracking, change detection, and alerting. The only item you need to create manually is the Activator, and the rules for it. You will find detailed instructions how to create them from the repo.

The code itself is fairly simple, the only complicated part is the Known Issue list scraping from the Power BI API but that part should be quite stable, as the data format is not likely to be changed.

You can configure the alerting you want by email, Teams. You can also start a data pipeline, or start a custom action. The alerting is done with Fabric’s Activator, so you have the power of the Activator in your hands.

As a data architect working with Microsoft Fabric, staying up-to-date with the platform’s development is important. That is why I have an automation script detecting changes on Fabric roadmap. Today, March 20th, it picked up a number of changes, 303 to be exact. The script counts every change, such as renaming and schedule changes as two (deletion and addition) but even counting the changes manually, I was able to find 131 changed items on the roadmap. That’s a lot.

After categorizing the changes, here’s the breakdown:

A ‘removed’ item refers to a roadmap item that is deleted after completion. A ‘deleted’ item refers to one deleted before completion.

Lots of completed features, many new additions, but also quite a few schedule changes and dropped items.

Translytical Fabric & Fabric CLI

There are some interesting items added to the roadmap. First one is Translytical Task Flows, the ability to update data direcly from Power BI, which I wrote about after Ignite. It should be in public preview in Q2. Second one is Fabric CLI.

Microsoft Fabric CLI

Estimated release timeline: Q1 2025

Release Type: Public preview

Fabric CLI is a command-line interface tool designed to interact with Microsoft Fabric. It provides a way for users/developers to manage and automate tasks within the Fabric environment. The [sic] toolwill support various functionalities such as running notebooks, managing pipelines, handling permissions, and more. It is aimed at enhancing the user experience by offering an alternative to graphical interfaces, similar to how Azure CLI operates within the Azure Portal The Fabric CLI offers two primary modes:

Interactive Mode - this mode allows users to interact with the CLI in real-time, executing commands one at a time and receiving immediate feedback. It is particularly useful for exploratory tasks and learning the CLI commands

Command Line Mode - also known as scripting or batch mode, this mode enables users to execute multiple commands at once. It is ideal for automation tasks, such as running scripts or integrating with CI/CD pipelines]

These modes provide flexibility for different use cases, whether you are performing ad-hoc tasks or automating complex workflows.

Source: Microsoft Fabric Roadmap

Fabric documentation does not mention anything about it yet, so this is all we have now. The description on the roadmap sounds great, as one thing that is missing from Fabric is the ability to manage the resources from the command line. Having a good UI is nice but managing a large number of resources is impossible with a scripting solution. That is why I’m really excited about this new tool.

It appears that we will soon have an Azure CLI-like tool for Fabric. The CLI is scheduled for release in Q1 2025, and it was added to the roadmap just eleven days before the quarter ends so it is quite obvious it will be announced at the FabCon in Las Vegas.

Update 2025-03-21: There is a session about Fabric CLI in FabCon: Microsoft Fabric - The Command Line Way

I am really looking forward to it.

Update 2025-03-28: There is also a session about User Data Functions.

Learn how to implement common data applications, such as data cleaning, data validations, data transformation and CRUD operations into Fabric data sources. […] Participants will also gain insights about the seamless integrations with Fabric Pipelines, Fabric Notebooks, Fabric Warehouses, Fabric SQL Databases and even Power BI reports, to build robust and flexible data applications.

Sounds lot like Translytical Fabric.

As an Azure or Databricks admin, you sometimes need to create Azure Databricks workspaces. You don’t want do it from the portal everytime, as it does not scale. The answer is to use ARM templates, or Bicep. You create the resources through the portal, enable workspace logging from Diagnostic Settings, and export the resource ARM-template. Then you notice, the diagnostic settings are not included in the template. What to do?

Template for Diagnostic Settings

You can find templates for diagnostic settings from the Azure documentation. Now you have the basic template but what are the categories for logs in the Azure Databricks resource? There is a reference guide for logging but the categories, or service names as they are on the page, are not all documented and their names are not the same as in the diagnostic settings. Where can you find the category names?

Finding the category info

In the Azure portal, add a diagnostic setting for the Azure Databricks Workspace resource, and click the ‘JSON view’ link. The panel that opens up, contains the category settings in JSON which you combine to the template.

Here are all the current categories listed as a Bicep array:

var dg_setting_categories = [
  {
    category: 'dbfs'
    enabled: true
  }
  {
    category: 'clusters'
    enabled: true
  }
  {
    category: 'accounts'
    enabled: true
  }
  {
    category: 'jobs'
    enabled: true
  }
  {
    category: 'notebook'
    enabled: true
  }
  {
    category: 'ssh'
    enabled: true
  }
  {
    category: 'workspace'
    enabled: true
  }
  {
    category: 'secrets'
    enabled: true
  }
  {
    category: 'sqlPermissions'
    enabled: true
  }
  {
    category: 'instancePools'
    enabled: true
  }
  {
    category: 'sqlAnalytics'
    enabled: true
  }
  {
    category: 'genie'
    enabled: false
  }
  {
    category: 'globalInitScripts'
    enabled: true
  }
  {
    category: 'iamRole'
    enabled: true
  }
  {
    category: 'mlflowExperiment'
    enabled: false
  }
  {
    category: 'featureStore'
    enabled: false
  }
  {
    category: 'RemoteHistoryService'
    enabled: false
  }
  {
    category: 'mlflowAcledArtifact'
    enabled: false
  }
  {
    category: 'databrickssql'
    enabled: true
  }
  {
    category: 'deltaPipelines'
    enabled: true
  }
  {
    category: 'modelRegistry'
    enabled: false
  }
  {
    category: 'repos'
    enabled: true
  }
  {
    category: 'unityCatalog'
    enabled: true
  }
  {
    category: 'gitCredentials'
    enabled: true
  }
  {
    category: 'webTerminal'
    enabled: true
  }
  {
    category: 'serverlessRealTimeInference'
    enabled: true
  }
  {
    category: 'clusterLibraries'
    enabled: true
  }
  {
    category: 'partnerHub'
    enabled: true
  }
  {
    category: 'clamAVScan'
    enabled: true
  }
  {
    category: 'capsule8Dataplane'
    enabled: true
  }
  {
    category: 'BrickStoreHttpGateway'
    enabled: true
  }
  {
    category: 'Dashboards'
    enabled: true
  }
  {
    category: 'CloudStorageMetadata'
    enabled: true
  }
  {
    category: 'PredictiveOptimization'
    enabled: true
  }
  {
    category: 'DataMonitoring'
    enabled: true
  }
  {
    category: 'Ingestion'
    enabled: true
  }
  {
    category: 'MarketplaceConsumer'
    enabled: true
  }
  {
    category: 'LineageTracking'
    enabled: true
  }
]

And here is an example of a diagnostic setting configuration using a for-loop. Note that the scope has to be your Databricks workspace resource. This example sends the logs to a storage account.

resource setting 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'dg-databricks-logs'
  scope: dbw_databricks_workspace
  properties: {
    storageAccountId: st_databricks_logs.id
    logs: [ for cat in dg_setting_categories: {
        category: cat.category
        enabled: cat.enabled
        retentionPolicy: {
          days: 0
          enabled: false
        }
      }
    ]
    metrics: []
  }
}

If you want to enable all the logs, you can just use the categoryGroup. Here’s an example

{
    "category": null,
    "categoryGroup": "allLogs",
    "enabled": true,
    "retentionPolicy": {
        "days": 0,
        "enabled": false
    }
}

User query logging

One thing to keep in mind is that if you need to log the user SQL-queries and other commands, enabling the SQL-categories from diagnostic settings is not enough. You need to enable verbose audit logs from the workspace settings.

You might to want to do it using the Databricks REST API or Databricks CLI. Using these tools, what is the setting name? Fortunately, someone has documented it.

Here is how you can query and enable the verbose logging using the Databricks CLI.

$ databricks workspace-conf get-status enableVerboseAuditLogs
$ databricks workspace-conf set-status --json '{ "enableVerboseAuditLogs": "true" }'

One more thing to note. If logging user queries is required for compliance, you might need to send the logs to a Log Analytics workspace and create an alert for any changes in the verbose logging.

After reflecting on that experience, I have decided to share the key lessons I have learned. The platform operated on Azure and Databricks but the things I learned are general and probably apply to every large data platform.

1. Governance, Governance, Governance

Governance is a curious thing. While it’s often discussed, it is an elusive and abstract concept. As an ex-developer, I wasn’t initially keen on governance.

Governance is like a paddle in canoeing. It is possible to start your trip without one, letting the current do the work, and you might not even miss it. When you realize you need it, it is probably too late.

You don’t need absolutely need governance at the start while operating on a small-scale but as your platform grows to include hundreds of data pipelines and thousands of tables for hundreds of users and systems, it becomes essential. When you reach a certain scale, you cannot operate efficiently if you don’t have proper governance in place. In the beginning, you will feel like governance hinders your progress but as the platform grows, it starts to make you so much more efficient.

What is governance of a data platform? There is a multitude of definitions but I think of it as everything that helps you manage the platform at a scale. It can include processes, policies, handbooks, or tools.

As the business of the data platform is data, a large part of the governance focuses on that. Besides data, you need to manage other assets, such as data pipelines, Spark notebooks, ML-models, data models, reports, and so on.

Here are some questions that governance helps you answer. These questions are written for data but could concern other assets as well:

• What data do you have?
• Who owns it?
• What does it actually contain?
• How do you classify it?
• How do users find the data?
• Who can access the data?

The most important thing about governance is that you need to have enough but not too much of it. Secondly, you should automate as much as reasonably possible. Governance gets easily trumped by business requirements. Automating it helps to keep it updated.

2. Embrace Your Platform

Operating in cloud at scale is expensive, and when your cloud bill has as many digits as there are days in the week, you start to think about your options. There is a school of thought which says that you should only use platform-agnostic features to make switching providers easier. The rationale is that using only the common features make switching the platform easy - or at least easier - if things go awry. In this thinking, you would only use the common Spark features, and you would refuse take the advantage of Databricks’ or Fabric’s advanced Spark features.

I understand the idea but it is not a way to live. It is like buying a Ferrari but driving it like a Lada because you might want to trade to it someday.

The likelihood of switching platforms is quite low. The migration cost for a large size data platform is enormous, so you are probably not going to do it. It is best to embrace your platform and use it to the maximum extent while making sure your data is not locked in.

3. Spear-heading Technology is Hard

Using battle-proven technology is a safe bet but sometimes it isn’t possible. Data lakehouse technology provides tremendous benefits but it’s still quite young. If you want to use it, you are forced to be a pioneer in new technology.

When using new tech, you are going to do at least some R&D work for a platform provider. New features require testing and you will be the one testing them. If you choose the path of new technology, partner up wisely. Using proven technology can be learned from books but if you use choose new tech, you need a partner who is in the know.

Using new technology often means that something does not work as expected or that there are undocumented features. Maybe your cloud database gives a performance boost if the datafiles are at least 5 GB (or some other arbitrary figure) in size. The point is, you need someone with connections to the product team.

One way to check this is to ask if your tech partner attends the correct conferences? Local, regional, or the gold standard: global? If yes, that is a good sign. If not, you might want to keep looking. Platfrom provider MVP or similar status is also a positive sign. The goal is to find partners with strong relationships with the platform provider to make sure you get support when needed.

4. Data Keeps Changing

There are certain aspects in data handling that you need to fix before implementation. For example, data access patterns influence technology choices and implementation, and data modeling needs to be designed based on user requirements. Schema of the data determines the table layout, and the distribution of data affects Spark’s job efficiency.

The challenge is that, in the real world, data is constantly changing.

For example, the original requirement might have been to import data in a daily batch, so you design a write-once, read-many optimized solution. The data became popular, and now you need to bring it in every 10 minutes, creating a challenging write-many, read-many situation.

Perhaps, you modeled your data using the (way too popular) data vault 2.0 method. The original data didn’t have any PII but then someone started to use the description field in the source system to record customer’s email address. Now you need to design a delete or scrub process for you data vault by hand, as your data vault automation tool doesn’t support it.

One thing that constantly surprises new Spark developers is how you need to know the data distribution when using Spark. Say you partition your customer activity data by customer. The worst case scenario is that 20% of your executors are doing 80% of the work, as typically the customer activities are not evenly distributed. The thing is that data distribution can and will change, so even if you balance the distribution correctly when implementing a job, you need to monitor the workloads to identify when the change happens. Failure to do this might lead that to your Spark jobs starting to fail at out-of-memory errors.

5. Monitor or You Don’t Know

Is your platform being used? Are your tables, reports, and ML-models actually utilized? Processing data is expensive. Is the end product used? How do you know? The only answer is that if you don’t monitor, you don’t know.

We all know the examples of how people say one thing but actually do something else. Based on polls, charities should be drowning in money, and stores shouldn’t be able to stock organic food shelves fast enough.

The same applies in corporate IT. A report and its underlying data might be added to the platform ‘just in case,’ with claims that it’s the department’s most important report. In some cases, access to data or personal clusters can even become status symbols. Only monitoring will reveal the truth.

Another category of monitoring is the cloud resource usage. The developers are usually pressed mostly on delivering the feature to the business, not so on much how much resources the feature is using. Monitoring might reveal issues, such as a query running over the entire data lake when it was thought to access only the latest partition. Or a pipeline running every 5 minutes, not once a day as thought. Or a Spark cluster reading the same data eight times because executors are constantly being evicted.

Monitoring costs is another critical aspect. Monitoring resource usage covers a lot of this but not all of it. We have all heard the stories of a monster-sized VM accidentally being left on, causing an enormous cloud bill.

For monitoring, the best approach is to create a process, provide tools, and distribute responsibility.

Summary

Building a new data and analytics platform using developing technology can be an exciting journey. By focusing on governance, embracing your platform, choosing the right technology, adapting to changing data, monitoring effectively, you can create a platform that actually works in the real world and provides an immense amount of value.

ps. Automate the deployments, and manage the platform using IaC. They will save a ton of work in the long run.

These platforms, especially Fabric, are developing fast, and the things mentioned in this post are likely to become outdated soon.

Let’s first see how the platforms are described by the companies.

High-level overview

Not that different, right? They emphasize slightly different aspects but both claim to be unified data and analytics plaforms at scale.

Taking a deeper look at the introduction pages and cutting through the zeitgeist AI speak, some differences arise. Databricks emphasizes programmatic access, commitment to open-source, and customers’ options for components such as storage. Fabric emphasizes deep integration of the platform, and ease-of-use.

These differences are noticeable quite soon when you start using the platforms. Databricks requires more set-up than Fabric. Fabric, in turn, does not let you see or change the details of the platform. The administration of the platforms is also quite different, at least at the moment. Databricks is essentially an API-first platform, meaning that you can control everything using APIs, and most features by using the Admin UI. Fabric is the opposite; all control is possible from the Admin UI and only some things are possible to control using APIs.

It’s interesting to see that the design philosophy differences are in some sense similar to Linux/Unix compared to Windows. Linux being open-source and easy to automate through the command prompt while Windows still requires some point-and-click administration, and has a slick and polished UI making everything easy for the beginner.

Features

What about a feature-level comparison? How do the platforms fulfill the normal data platform requirements? Normally, these kinds of tables are done only by corporate buyers, but they actually do serve a purpose for us here. They help us get a high-level picture of what the platforms provide and also serve as a map for us in later parts of the series, when we will start comparing the features in practice.

Data Ingestion, Transform, and Streaming

Data ingestion is the bread and butter feature of a data platform. Getting data into the platform is a fundamental feature of the platform. Data engineers spend a large part of their daily job working with data ingestion.

¹ Many times combined with Azure Data Factory or a similar orchestration tool.
² Azure SQL, Azure SQL MI (preview), Cosmos DB (preview), Databricks (preview), Snowflake.

It is interesting to see how Fabric has a multitude of options compared to Databricks. Databricks relies on Spark and notebooks for everything, while Fabric offers different options for ingesting and processing the data. Fabric also has an option to mirror the database directly into Fabric, skipping the data pipeline altogether.

Data Storage

What about storage? Ingesting data is important but where will it land?

Storage is really interesting. At first, it seems that the platforms are the same. They both allow the use of any major cloud provider as the storage. On a closer look, the differences arise.

Databricks is agnostic to storage and it can combine multiple storage accounts or even multiple cloud vendors by using external locations. Using these locations is pretty much transparent when using them from Databtricks.

Fabric is different. On the surface, it forces you to use OneLake, their own storage solution. However, OneLake can have shortcuts to OneLake itself or any major cloud vendor storage. Using these shortcuts is completely transparent. The user does not know whether they are using OneLake or some other cloud storage through a shortcut. OneLake, in effect, can be used to integrate several storages into one. It also exposes a REST API so even 3rd party tools can enjoy this seamless integration. This means that you can use OneLake to combine several cloud vendors’ storages to single virtual storage and use it, for example, from Databricks.

Compute

Compute is the heart of the data platform. It is used constantly and that is why it is so important that it fits to the purpose, scales, and is easy to use.

For the compute engine, Databricks provides only one option: their trusted Spark engine, hosted either in the customer’s cloud account or is serverless, i.e. hosted in Databricks’ account. At the Databricks Data+AI summit in June 2024, Databricks’ CEO Ali Ghodsi said they will focus on serverless compute, providing all the new features first to the serverless compute.

Fabric has several compute engines, but they operate only on serverless compute, i.e. providing less customization options than Databricks. Fabric offers Spark for Lakehouse, Polaris engine (or a modified version of it) for Warehouse, and KQL for the Real Time.

They both provide table-level transactions when using Spark and Delta Lake. Fabric’s warehouse is capable doing database-level transactions.

Data Governance

Data governance is an essential part of a modern data platform. It is an emerging feature, which means that the features are very much in development.

Databricks has fine-grained and functional access control based on Unity Catalog. It allows managing the access on workspace, catalog, and object level.

Fabric has a different security model. It relies heavily on using workspace as a basic building block for securing the access. Fabric has OneLake security model on the roadmap.

For other features of Governance, Databricks relies on Unity Catalog and Fabric on Purview.

Data governance is one of the areas where platform providers are competing heavily in. This should bring quite dramatic improvements in this area over the next couple of years.

BI / Reporting

Whether data and analytics platform should contain an integrated BI or reporting platform can be debated but it is, of course, nice to have one.

³ Usually combined with an external tool, such as Power BI

Databricks has its AI/BI solution integrated into the platform, but it has been pretty feature light solution. Usually, Power BI or some other BI tool has been used to complement the platform. BI is now one of the Databricks’ investment areas, as we heard from the CEO Ali Ghodsi in the Data+AI Summit this year.

Fabric integrates Power BI into the platform; actually Fabric is Power BI with extended features. Power BI is one the best BI tools on the market.

Automation (CI & CD)

In any production platform, it is important to automate setting up the environment and its maintenance. If operating at scale, it is mandatory. What kind of tools do these platform provide for automation?

Databricks has extensive support for automating the platform. It provides several full scale solutions to fully automate setting up and maintain the platform.

Fabric has a number of automation tools available, though many of them are in preview. Some of these features are really interesting. For example using Terraform as the deployment tool seems odd, since it is a third party tool. One would expect Microsoft to use ARM or Bicep for this.

AI / ML

AI and ML features are an integral part of modern analytics platform, and it is the area where platform providers are investing heavily. Platform may offer models, algorithms, and other functionality but it should at least support automation and deployment of the models.

Both platforms contain MLflow to manage ML models. Both platforms also contain AI features, though Fabric’s are in preview.

Developer tools

For the long term success of the platform, it is essential to provide good tools for the developers. What tools do these platforms offer?

Both provide VS Code extension. Databricks has some other tools. In practice, most developers seem to use the web-based UI and notebooks for code developement.

Other features

What other features do the platforms provide?

Databricks has Delta sharing to enable sharing data between organizations. They also have Data Cleanrooms, to process data with another org without actually giving them access.

Fabric has Databases, which was introduced at the Ignite conference, in Autumn 2024. It is still in preview but it is an interesting development for data platforms as Microsoft is bringing operational databases to a data platform for the first time.

Summary of features

On paper, Fabric seems to offer more options than Databricks. For example, for the compute engine, Fabric has three options compared to Databricks’ one. Both platforms tick the boxes pretty well, but as always, the devil is in the details. Many of Fabric’s features are still in preview while writing this. On the other hand, Fabric has some features which Databricks does not have at all. For example, operational databases were introduced to Fabric at the Ignite conference in fall 2024. Databricks does not have anything similar.

Architecture

What about the architecture of the platforms? How are they constructed and how much does a platform’s provider actually share implementation details?

Databricks

Databricks’ architecture is pretty simple. On a high-level, architecture diagram shown above, there is a control plane and compute plane which are situated in Databrick’s account and in your cloud account. This is a high-level view and it hides a lot of the complexity behind the boxes, like Unity Catalog.

Databricks uses Spark as the compute engine. Spark’s internal architecture is shown below. Basically there is just a cluster manager, driver node and 0-n worker nodes.

Spark Architecture

Databricks is basically a managed Spark with some added elements, such as a nice web UI and some additional features such as Unity Catalog, workflows, and Delta Live Tables. Databricks uses Spark basically for all computing, but it needs to be noted that Databricks’ Spark version is not the open-source one but they have proprietary features. They also have a vectorized compute Engine called Photon, which is basically just a Spark engine with lower execution levels written in C++; there is no difference for the user besides the possible speed and cost difference.

That is the compute, but what about the other components: storage and networking? Databricks uses whatever resources you configure it to use; for storage, which typically an object storage, such as ADLS gen2 found from your cloud account. For networking, it uses the network you manage yourself in your cloud account.

Fabric

Fabric provides a wide set of tools for every task. So much so, that sometimes it might be difficult to choose which tool to use. For data ingestion, one can use Data Factory, Data Flows, or Fabric (Spark) Notebooks. Data Factory and Data Flows are low/no-code solutions and notebooks are a pro-code solution.

For data storage, Fabric offers Lakehouse and Warehouse. They both use Fabric’s Onelake object storage as the storage but the feature set they provide are somewhat different. Basically Lakehouse is Spark-based, meaning one can use Pyspark, Scala, R, or SparkSQL notebooks while Warehouse is a T-SQL based solution. Fabric’s Spark is the open-source version.

Reporting and BI in Fabric is done using the excellent Power BI which integrates to Fabric really nicely.

Fabric is basically a blackbox SaaS product for which we don’t have much information on how it is built. In fact, the only source I have come across so far is second-hand information from a recent conference described in Redmond Magazine. I hope this will change in the future, as knowing how the platform is built helps to use it effectively.

Conclusion

This article turned out to be much longer than I expected - and that is a good indication that there is so much to cover on both platforms. The feature list is long and the platforms are huge.

At a high-level, the platforms seem to be similar. Closer inspection reveals there are some differences in the design philosophy and the direction in which the platforms are heading.

The competition at the moment is fierce and the development speed, especially Fabric’s, is exceptional. Both platform providers are investing heavily in development. It is really interesting to see where these platforms will be in the next three years. Fabric is a bit of an underdog, but it is cleverly heading to extend the platform where Databricks has been under-serving the market.

In the next part, we will start by looking how to architecture a simple data platform implementation.

Another blog series on the differences between Databricks and Microsoft Fabric. Why?

Well, I guess the reasons are purely selfish. I used to be an architect working on a data platform that ran mostly on Databricks. After changing my job, most of the projects I have been working on are Fabric-based so I need to learn Fabric well. The best way to learn something is to write about it. Comparing the new to the old is also valuable for learning. I’ll limit the scope of the series to Microsoft Fabric and Azure Databricks, as those are the platforms I am familiar with.

To understand the platforms, we need to understand how they came into existence. As always, nothing happens in vacuum, but both of the platforms are shaped by the general development of data platforms, cloud technology, and the rise of big data. The history of these platforms is intertwined as Databricks probably wouldn’t exist without the shortcomings of the data platforms of the early 2000s. And on the other hand, Fabric probably wouldn’t exist without Databricks.

Let’s start studying the platforms by looking at a brief history of Databricks.

Databricks

Early Internet and Google’s problem

Google and other Internet giants had a problem at the end of the 1990s. The Internet was becoming too large to be captured, and Internet traffic was generating so much data that they had problems storing it - let alone analyzing or searching it. The traditional method to solve these kinds of problem was to use a database, but they were of no use here. The databases usually had an SMP (Symmetric Multiprocessing) architecture, which meant that they ran on a single machine. If you needed more power, you had to buy a bigger machine. The SMP databases coupled storage and computing meaning that when you ran out of storage space, you were facing a long downtime migrating the data to a larger machine. Big enough machines cost a metric ton of money, if you could find a machine big enough.

There were also database systems that used MPP (Massively Parallel Processing) architecture, such as Teradata. MPP means that the systems didn’t run on a single big machine, but rather a group of machines. These systems were ridiculously priced for startups like Google was back then. A good indication was a poll on Teradata’s homepage in 2003, in which the lowest option for yearly maintenance cost of a single datamart was US$ 1 million (US$ 1.7 million in 2024 dollars). (By the way, when did we stop having polls on corporate websites?)

Google solved the problem. They used commodity hardware, which stored and analysed the data in unison, with many small machines acting like a one big machine. The technology was proprietary, but research ethos was strong in Google’s early days. So they published two research papers, describing the distributed file system (2003) and compute system, called MapReduce (2004).

Early Google servers.

Hadoop

It took a couple of years for outsiders to develop the software systems based on these papers, but in April 2006, it was finally ready and released. It was called Hadoop.

Hadoop was groundbreaking. It was used to run analyses on cheap commodity hardware clusters. A cluster could have had just a couple of machines, or it could have scaled to more than 4,000 computers. For the first time, this scale of computing was available for universities and startups, that could not afford proprietary cluster computing. Hadoop gave results to problems and questions that couldn’t be answered before.

Hadoop cluster at Yahoo 2007

Even though Hadoop was great, it was not perfect. It had to be programmed using a model called MapReduce, which was a beast to program. Hadoop programs usually required many phases or tasks, and each of them required the data to be written on a disk, which was a time-consuming operation. Hadoop also consisted of a large number of software packages, each of which had its own release cycle. You had to have compatible versions of all the components for the system to work correctly, so upgrading a cluster was difficult, as it was later discussed in an episode of Software Engineering Daily podcast.

Enter the Spark

Matei Zaharia was born in Romania, and later moved to Canada before finally settling in the US. He was considered a wunderkind of data science, especially on distributed systems. He won several prizes and competitions in science and programming before he finally entered UC Berkeley. As a PhD student in Berkeley’s AMP Lab (Algorithms, Machines, and People) and an intern at Facebook, he saw firsthand the problems users had with Hadoop. He wanted to help them, as he mentioned on Reddit.

Reynold Xin, one of the AMP Lab students and Databricks co-founders, later said in a conference talk that Zaharia created Spark to help another student to participate in the legendary Netflix Prize competition. According to Xin, Spark was used in the competition to place second, losing the US$ 1 million prize money by 20 minutes.

Whatever the real origin was, Zaharia’s solution was to create a general-purpose distributed compute engine called Spark. Spark was easier to use than Hadoop and significantly faster. Zaharia started working on it in 2009, and it was open-sourced in 2010. In just a couple of years, Spark really took off. Over the years, it became the de facto standard of distributed analytics.

Why did Spark take off like a rocket? If the early adopters of Hadoop were startups, universities, and other shoe-string budget organizations, why legitimate companies started to move to Spark. Zaharia tells his point of view in his book about Spark.

According to him, single processor core speed development started to plateau around 2005, which caused processor manufacturers to start increasing the number of cores. This in turn led to the need for parallel programming models, such as Spark. In addition, the cost of collecting data continued to decline.

The end result is a world in which collecting data is extremely inexpensive—many organizations today even consider it negligent not to log data of possible relevance to the business—but processing it requires large, parallel computations, often on clusters of machines. Moreover, in this new world, the software developed in the past 50 years cannot automatically scale up, and neither can the traditional programming models for data processing applications, creating the need for new programming models. It is this world that Apache Spark was built for.

To the Cloud: Databricks

Spark was a fine piece of software, but it required a lot of manual work to set up and maintain the clusters. With on-premise infra, this wasn’t exactly easy. In 2013, Databricks, the company, was founded to provide cloud-based Spark platform and commercial support for users. It was founded by Zaharia and other members of the AMP Lab research team.

Databricks’ first investment deck 2013

Zaharia and Databricks continued to invent more products, making them available through Databricks and also open-sourcing them. These products included MLlib, Structured Streaming, Delta, and MLflow.

In 2017, only four years after its inception, Databricks believed it will dominate the analytics world. Slide from Databricks’ investment deck in 2017.

Originally, Spark wasn’t a database system. Instead, it operated on bare files exposed by the cloud storage. This caused problems when updating the files or with streaming data. The reader of the file didn’t know if the data in the file was still being updated. You couldn’t enforce the structure of the data. With the introduction of Delta in 2017, Databricks and Spark started to include database-like features, such as transactions. Later, they brought it even closer to a regular data warehouse by introducing the Delta Lakehouse and Unity Catalog.

Databricks continued to invent and grow. By 2024, its valuation was rumoured to be 55 billion dollars. In just a few years, it had become the juggernaut it had envisioned.

Spark summit 2013, 500 participants

Databricks Data+AI summit 2024 (renamed from Spark Summit). 17,000 participants.

Databricks has a special relationship with Microsoft. In 2017, Databricks and Microsoft announced a co-operation. Microsoft would provide Databricks as a first-party service in Azure. It is even called Azure Databricks. Microsoft would allow Databricks to integrate to Azure’s components, such as storage, as it was Microsoft’s own service. For years, their partnership existed peacefully, until at the Build Conference 2023, out of the blue, Microsoft announced its new data platform. It was called Fabric.

Microsoft Fabric

Microsoft Fabric is not a single product but a combination of about a dozen Microsoft products, so it doesn’t have as clear-cut history as Databricks. One thing is sure, though: Fabric wouldn’t exist, at least not in this form, without Satya Nadella.

Nadella’s problem

When starting as the CEO of Microsoft on February 4th, 2012, Satya Nadella faced a challenge. As he describes in his book “Hit Refresh,” he inherited a company with a strong culture of viewing open-source software as its enemy. For Nadella, the need for transition was clear. The transition to the cloud made him realize that Microsoft was not just a technology provider but also operated the tech in cloud, and clients wanted to run other products besides just the Microsoft stack.

Nadella was appointed as Microsoft’s third CEO in 2014. By Briansmale, CC BY-SA 4.0

Microsoft embraces open-source

Microsoft had been flirting with the cloud since 2005, but soon after Nadella was appointed as the CEO of the company, the change started picking up pace. Developers were aghast to see Microsoft supporting open-source software in their offerings. The announcement of Spark for HD Insight in 2015, as well as the acquisition of the Revolution Analytics meant that the R language was integrated into multiple products. In 2016, Microsoft announced its commitment to Spark.

I remember talking to Microsoft engineers during those years, and they seemed just as surprised as we outsiders were. They were also really pleased to see that their company was embracing the best solutions on the market instead of trying to make everything in-house.

Converging the tools

Until 2019, Microsoft’s products were separate and lacked a proper data platform. But then came Azure Synapse Analytics. Synapse’s selling points were: “End-to-end analytics solution”, “Platform out-of-the-box”, “Integrated components”, “Eliminates the divide between datalake and data warehouse”, and “All capabilities in single place”. As we later saw, this was pretty much the same sales talk as that used for Fabric. So what went wrong the first time?

Initially, there was a lot of interest in the market for Microsoft’s unified data platform, but apparently the development speed and the features of the platform just couldn’t compete head to head with other big platforms, like Databricks and Snowflake.

From a personal experience, I saw one large scale data platform project which was using Synapse Analytics. It was abandoned after a couple of years when it became clear that the platform was not mature enough and the development speed was just too slow. Later the project was implemented on Databricks. In last couple of years Synapse just couldn’t compete with the big data platform providers. I’ve heard about several data platform projects where Synapse was quickly dismissed after initial research. Microsoft read the market and saw Databricks, Snowflake and all the other data platforms passing them from left, right and center, and saw that they needed to do better.

Enter the Fabric

At the Build conference in 2023, Microsoft announced their new data platform, Fabric. According to Satya Nadella, it was their biggest data product launch since SQL Server. Fabric wasn’t created from scratch but it’s a combination of at least three major lines of software. Azure Synapse Analytics, Power BI, and Purview. A big part of Fabric are also the open-sourced innovations which are made by Databricks, such as Delta Lake and MLFlow.

Was Fabric just a rebranding of Synapse Analytics or was it really a new platform?

There are definitely lots of old parts that have been reused but there is at least one major difference: the wiring of the components to work as a seamless platform. The latest additions, such as Fabric Databases, which bring operational databases to an analytical platform, are also a major differentiators. In my view, it is not just a marketing gimmick but Fabric truly is a new data platform.

Fabric is an amalgamation of a number of products. Source

Conclusions

The modern data platform has to thank the early Internet search providers, Google and Yahoo. They started the development which lead to Hadoop, Spark, and eventually the Lakehouse paradigm. It also required companies to willingly publish release papers and release software as an open-source. One could think that the modern data warehouse is a triumph of openness and sharing.

Being open, of course, was not enough. It also required failure of the traditional data warehousing. As the Internet and falling costs of data storage drove the data amounts organizations gather to be on a constant rise, traditional data warehousing just could not keep up.

Databricks started from a single product and has evolved into a platform but it is still largely based on a single product: Spark. How well Databricks has been able to ride the data and AI wave from a research group to a start-up to a juggernaut of a company, is remarkable.

The history of Fabric is just the opposite: it’s a combination of numerous Microsoft products into a platform by one of the largest companies in the world. Evolutionary history of Fabric is well described in the Insights and Outliers channel in YouTube.

Fabric enters a bit late into the data platform race, so this is Microsoft’s do or die moment in data. How does the platform compare to Databricks? In the next part of this series, we’ll continue with a feature-level comparison of the platforms.

A couple of years ago I started to notice that fulfilling basic reporting needs in businesses was no longer enough. Many needs required two-way communication with the data. Typical cases included annotating an outlier in production data, perhaps shown in control charts; fixing classifications of an ML-model; re-running an ML-model; or just updating the data. This was possible before but it required for example embedding a Power App application into the report and setting up and configuring an external database.

I guess the need for interacting with data instead of just looking it was recognized by Microsoft, too. With these ‘Translytical’ features, data can be updated by using a Fabric database, user data functions, and Power BI button. No more external resources. Much simpler!

Updating analytical data can be a massive footgun, and you need to carefully design all the whats, whens, and whos, but the demand for this kind of feature is strong.

I am not yet sure about the term ‘Translytical’, but I see huge potential with this feature. We can see that large part of the small scale business apps will be built on Fabric in the future. It is possible that Power BI and Fabric will start expanding into the field that was once held by Access and Excel.

The feature is in private preview so we don’t know all the details or possibilities yet, but I am interested to see how this plays out and to what extent the border of transactional and analytical boundaries will be blurred in the future.

Exiting times!

Btw, if you are interested in listening the amazing story of Amir Netz and Vertipaq, you should listen this episode of Insights Tomorrow podcast.

Update 2025-03-28: Translytical tasks are now on Fabric Roadmap

I didn’t know it back then but it all started while I was reading Hacker News in February 2019 and stumbled upon a story called “The Hunt for the Death Valley Germans”. The real life events behind the story are unbelievably tragic but how the case was solved was remarkable. How the perseverance of a single guy led to him solving the case and by doing so, he was able to bring much relief to the families of the victims. Reading the story got me thinking how I could do this kind of thing myself.

I read the story a couple of times and sent it to my brother. I knew that he would be interested in the story as he is into these kinds of things as much as what I am. After reading the article, we talked for quite a long time about various missing person cases and how to solve them.

By the autumn of 2020 the story had faded from my mind until my brother called me with an interesting missing person case. That phone call was the starting point of the most interesting adventure I’ve ever had, and it lead to us solving two missing person cold cases, which had been unsolved for 9 and 15 years.

This story is written based on my notes, photos, videos, GPS logs, sonar images, and other digital footprints. The facts were checked whenever it was possible. If there are any errors in the story, they are, of course, mine.

Maps would make following the story easier but I chose not to provide them. For those who are eager and skilled enough, you can deduct the actual places from public sources and my descriptions.