(New Africa/Shutterstock)
Because the earliest days of huge knowledge, knowledge engineers have been the unsung heroes doing the soiled work of shifting, remodeling, and prepping knowledge so extremely paid knowledge scientists and machine studying engineers can do their factor and get the glory. Because the agentic AI period dawns on us, it opens up a number of latest knowledge engineering alternatives–in addition to doubtlessly catostrphic pitfalls.
Frank Weigel, the previous Googel and Microsoft govt who was not too long ago employed by Matillion to be its new chief product officer, brazenly puzzled to a reporter not too long ago whether or not the Agentic AI Air was on a glideslope for catastrophe.
“Principally, we see there’s an enormous drawback coming for knowledge engineering groups,” Weigel mentioned in an interview throughout the latest Snowflake Summit. “I’m undecided all people is absolutely conscious of it.”
Right here’s the problem, as Weigel defined it:
The explosion of supply knowledge is one side of the issue. Knowledge engineers who’re accustomed to working with structured knowledge are actually being requested to handle, prep, and rework unstructured knowledge, which is tougher to work with, however which in the end is the gas for many AI (i.e. phrases and footage processed by neural networks).
Knowledge engineers are already overworked. Weigel cited a research that indicated 80% of knowledge engineering groups are already overloaded. However whenever you add AI and unstructured knowledge to the combo, the workload subject turns into much more acute.
Agentic AI gives a possible resolution. It’s pure that overworked knowledge engineering groups will flip to AI for assist. There’s a bevy of suppliers constructing copilots and swarms of AI brokers that, ostensibly, can construct, deploy, monitor, and repair knowledge pipelines once they break. We’re already seeing agentic AI have actual impacts on knowledge engineering groups, in addition to the downstream knowledge analysts who in the end are those requesting the information within the first place.
Supply: Shutterstock
However in line with Weigel, if we implement agentic AI for knowledge engineering the mistaken approach we’re doubtlessly setting ourselves a entice that shall be robust to get out of.
The issue that he’s foreseeing would stem from AI brokers that entry supply knowledge on their very own. If an analyst can kick off an agentic AI workflow that in the end entails the AI agent writing SQL to acquire a chunk of knowledge from some upstream system, what occurs when one thing goes mistaken with the information pipeline? AI brokers may have the ability to repair fundamental issues, however what about severe ones that demand human consideration?
“You’ll have autonomous AI brokers that run whole enterprise capabilities,” Weigel mentioned. “However equally, they begin to have an enormous want for knowledge. And so if the information staff already was overloaded earlier than, effectively, it’s now going to be like wanting down the abyss and saying ‘How on earth can we do something? How am I going to have a human knowledge engineer reply a query from an AI agent?’”
As soon as human knowledge engineers are out of the loop, dangerous issues can begin occurring, Weigel mentioned. They doubtlessly face a state of affairs the place the quantity of knowledge requests–which initially have been served by human knowledge engineers however now are being served by AI brokers–is past their functionality to maintain up.
The accuracy of knowledge will even endure, he mentioned. If each AI agent writes its personal SQL and pulls knowledge instantly out of its supply, the chances of getting the mistaken reply goes up significantly.
“We’re now again in the dead of night ages, the place we have been 10 years in the past [when we wondered] why we’d like knowledge warehouses,” he mentioned. “I do know that if particular person A, B, and C ask a query, and beforehand they wrote their very own queries, they bought totally different outcomes. Proper now, we ask the identical agent the identical query, and since they’re non-deterministic, they may really create totally different queries each time you ask it. And in consequence, you now have the totally different enterprise capabilities all getting totally different solutions, insisting in fact that it’s proper.
Matillion CPO Frank Weigel
“You have got misplaced all of the governance and management of why you established a central knowledge staff,” Weigel continued. “And for me, that’s the angle that I feel plenty of knowledge orgs haven’t actually thought of. Once I get a demo of an AI agent, they by no means discuss that. They simply have the agent entry the information instantly. And positive, it will probably. However the issue is, it shouldn’t actually.”
The reply to this dilemma, in line with Weigel, is twofold. First, it’s vital to maintain knowledge warehouses, because it serves as a repository for knowledge that has been vetted, checked, and standardized.
It’s additionally essential to maintain people within the loop, in line with Weigel. And to maintain people within the loop, human knowledge engineers should someway be prevented from changing into fully overwhelmed by the unstructured knowledge requests and the brand new AI workflows. To perform that, he mentioned, they primarily should change into superhuman knowledge engineers, augmented with AI.
Matillion is constructing its agentic AI options round this technique. As an alternative of setting AI brokers unfastened to put in writing their very own SQL towards supply knowledge techniques, Matillion is utilizing AI brokers as supporting forged members who’s objective is to help the human knowledge engineer in getting the work executed.
This on-demand staff of digital knowledge engineers is dubbed Maia, which the corporate introduced earlier this month. The brokers, which run within the Matillion Knowledge Producdtivity Cloud (DPC), are capable of help knowledge engineers with a spread of duties, together with creating knowledge connectors, constructing knowledge pipelines, documenting modifications, testing pipelines, and analyzing failures.
“We have to supercharge the information engineering operate, and we have to allow them to match the AI capabilities,” he mentioned. “As an alternative of only a copilot idea, it has change into a part, a choice of totally different knowledge engineers which have totally different duties. They’ll do various things.”
Maia acts because the lead agent that controls varied sub-agents. The corporate has three or 4 such knowledge engineering sub-agents right this moment, Weigel mentioned, and it’ll have extra sooner or later. Maia, which is constructed utilizing a set of huge language fashions (LLMs), together with Anthropic’s Claude–may even right itself when it does one thing mistaken.
Matillion is near delivery a preview of Maia
“It’s actually fascinating,” Weigel mentioned. “While you see it work, it’ll break down the issue into the steps. Then it’ll begin doing it. It can have a look at the information and determine whether or not it’s going heading in the right direction. It’d roll again. ‘That wasn’t fairly proper.’ And so it actually is sort of a knowledge engineer in its job and considering, together with wanting on the knowledge. It can ask the human for sure at sure factors if it needs enter.”
Regardless of the potential for agentic autonomy, that isn’t a part of the Matillion plan, as the corporate sees the human engineer as a essential backstop that may’t be eradicated from the equation.
One other vital backstop that might assist Matillion prospects keep away from agentic AI pitfalls: No AI era of SQL.
Whereas LLMs like Claude have gotten actually, actually good at writing SQL, Matillion is not going to hand the reins over to AI for this essential part. The ETL vendor has been routinely producing SQL as a part of its knowledge pipeline resolution for Snowflake, Databricks, and different cloud knowledge warehouses for years, and it’s not about to start out from scratch.
“The key in Matillion is we’ve abstracted that layer so we’re a lot nearer to the consumer intent,” Weigel mentioned. “So the consumer is constructing that knowledge pipeline intent with predefined constructing blocks that in the end write SQL. But it surely’s Matillion that writes SQL, not the consumer.”
This strategy additionally avoids the issue of getting spaghetti SQL code that may’t be up to date and modified over time, which is a chance with AI-generated code.
“Now we have this abstraction of this intermediate illustration of those elements that in flip points SQL,” Weigel mentioned. “And so our agent doesn’t need to generate no matter code you want. As an alternative, it’s about selecting the correct part and configuring the fitting part after which sequencing them collectively.”
It’s simple to get mesmerized by the “shiny object” syndrome within the tech world. With all of the advances in generative AI, it’s tempting to letting these shiny new copilots unfastened to try to replicate the job of the overworked, under-appreciated knowledge engineer, at a fraction of her value.
But when changing knowledge engineers with AI additionally means changing a lot of the governance and management the information engineer brings, that might spell catastrophe for corporations. “I feel knowledge engineering groups aren’t possibly absolutely conscious of the potential doom that’s there,” Weigel mentioned.
As an alternative, corporations must be trying to super-charge these overworked knowledge engineers utilizing AI, which Weigel mentioned is the perfect hope for surviving the AI knowledge deluge.
Associated Gadgets:
Are We Placing the Agentic Cart Earlier than the LLM Horse?
Matillion Bringing AI to Knowledge Pipelines
Matillion Appears to Unlock Knowledge for AI
