The Lucid blog

Notes from the build.

Deep dives on Fabric, Power BI, Spark, and the parts of analytics work that don't make it into vendor docs.

Cross Workspace Fabric Notebook Execution

Cross Workspace Fabric Notebook Execution

Have you ever tried to run a Fabric notebook in a different workspace using notebookutils? I have, and it didn’t work out so well. Thankfully, persistence wins, and now I get to share the secret sauce with all you lovely people. The Setup You’re goin...

Read post →
Variable Libraries, The Fabric Unsung Hero

Variable Libraries, The Fabric Unsung Hero

In case you missed it, the Variable Library team over on the Fabric PG quietly snuck out some pretty big updates relatively recently that are actually quite massive. Before we dive into, it’s important to understand why these updates are so huge. Dep...

Read post →

Data Profiling with Spark and YData

Data analytics often begins with profiling your data. Data profiling is simply the act of examining the raw source data to understand things like the structure and quality of the data. As an engineer, understanding things such as the distribution of ...

Read post →
Using Custom Python Libraries Without Fabric Environment

Using Custom Python Libraries Without Fabric Environment

Fabric Environment artifacts, where to begin… If you’re not familiar, the environment artifact, in part, is intended to mimic the functionality of the Synapse Workspace by allowing you to do things like install packages for easy reusability across yo...

Read post →
There's No Shortcut to Proper Planning

There's No Shortcut to Proper Planning

About a month ago, while working on the first iteration of the Fabric capacity monitoring report, I stumbled across a bit of an invisible fence. I posted a quick teaser with a screenshot on LinkedIn to see if anyone else could spot it: https://www.li...

Read post →
Microsoft Fabric Enterprise Deployment Considerations

Microsoft Fabric Enterprise Deployment Considerations

When it comes to architecture strategies there's no one-size-fits-all solution. Every team will have different use cases that will drive requirements such as security, environment isolation, and tools needed. While working with one of my clients we s...

Read post →
How To Reduce Data Integration Costs By 98%

How To Reduce Data Integration Costs By 98%

One of the amazing things about Microsoft Fabric is the number of options you have for moving data. For example, you can use Dataflow Gen2, Pipelines, Notebooks, or any combination of the three. However, with all the options available, it can also ma...

Read post →