Weld’s data lineage feature just leveled up, bringing even more visibility to your data ops

Just a few weeks ago, we announced the launch of Weld’s new data lineage feature, built to give you a clearer overview of every data model you create in Weld, and each dependency related to it.

Now, we’re excited to share that we’ve expanded Lineage to provide a complete, end-to-end visual map of your entire data operations. That’s right, you can now see the flow of all your data — starting from the raw data source, right through to where the data ends up, and all the steps along the way.

Keep reading for all the details on Lineage, what’s new, how it helps, and why we built it.

What is Lineage?

In its V1.0, Lineage offered a complete visual overview of your data models and their dependencies. You could swiftly navigate from one model to another, easily jumping back and forth to understand the underlying models each model depends on, and the models that depend on it.

This gave data engineers and analysts using Weld a clear picture of how all their models were connected, helping them keep their models streamlined and organized. It helped data teams spot issues more easily in their models, and solve them faster. All in all, this was a great new addition to the Weld platform, and brought a new layer of support to ongoing data governance and data model management. But, why stop there?

What’s new in Lineage

Welcome to Lineage V2.0, where data lineage graphs show not just models and dependencies, but data sources, tables, and destinations. This means complete end-to-end visibility of your entire data operations, and easy navigation throughout.

The data lineage graphs can be viewed in the Gather, Model, and Activate sections of Weld — where you set up ELT syncs, build and manage your data models, and configure reverse-ETL syncs. This takes the visibility provided by the first iteration of Lineage to a whole other level.

What else? The team also added an error-highlighting feature, making it even easier to spot what’s broken in your data flow. When one of your models or tables isn’t syncing or is having some other issue, a little icon will appear on it in the lineage view.

This little icon goes a long way — now, you can see exactly where an error is occurring, and what it’s impacting down the line.

How does Lineage help data teams?

The visibility that Lineage provides comes with a whole host of benefits for users, like…

• Never manually digging around SQL queries trying to figure out where your data comes from again.
• Quickly spotting when something fails, making it easier to pinpoint and resolve issues before they lead to bigger problems.
• Finding smarter ways to reduce repetitions in logic and instead creating single models that can be referenced multiple times.
• Being at ease knowing when your models or dashboards were last updated, and when they’ll be updated again with fresh data.
• ​​Maintaining a good flow in your data models and overall data operations, and avoiding any circular references.

Why we built Lineage

One of the main challenges companies face in becoming “data-driven” is trusting their data. Too often, there’s uncertainty about where data comes from, whether it’s up to date, or what source to rely on for specific data.

Weld is an end-to-end data platform that brings your core  data operations into one place — you can manage your ELT, data modelling, and reverse-ETL all in the same single platform. This gave us the unique opportunity to build a feature as robust as Lineage.

“Weld kind of owns the meta layer — we know everything that’s going on across your whole data stack — which means that we can give you a level of transparency into your data operations that no other tool can provide.”

– Jonas Thordal, Co-Founder & CEO of Weld

What’s next?

Of course, we’re always working to continuously improve the Weld platform, and bring new value to our users. Without giving too much away, we’re working on features that contribute to the goal of trustworthy data. The team is focusing on adding new functionalities in Weld that help people keep their data operations organized and running smoothly — think fewer sync failures, cleaner pipelines, and better mapping.

Don’t worry, we’ll be sure to update you as the new features roll out. In the meantime, if you’re interested in learning more about how Weld can power your data operations, book a call with a member of our team.