Your AI Vendor Has a Supply Chain: Three Questions to Ask Before You Buy

What actually changed about building software?

Haddad has been through the whole arc at Advocate. The first version of the product was built the traditional way, by hand. AI entered as a research aid, a faster substitute for hunting through Stack Overflow for answers. Then, he says, the nature of the job changed outright.

The share of code he writes personally has fallen dramatically, from effectively all of it to a small fraction. [VERIFY: figure stated inconsistently in the episode, see production notes] What replaced it is not idle time. It is specification work, customer conversation, review, and testing.

There is a discipline underneath that shift which matters for anyone buying the output. Haddad frames it as a line his team does not cross: you are not offloading your thinking to the model, you are offloading the doing. The engineer still owns the reasoning, the design decision, and the verification. The model writes faster than a human can type, and that is all it is being asked to do.

That distinction is what makes the trust question answerable. Financial institutions and insurance buyers are under pressure to adopt AI, often with mandates from above, while much of the market is still working out how to use it safely. A team that has kept its own judgment in the loop can explain what it built and why. A team that handed the thinking over cannot.

If the model writes the code, what is the engineer doing?

The answer, according to Haddad, is most of what people used to think of as the surrounding work. Reading and reviewing code, yes, but mainly planning, technical implementation design, building the spec, and understanding the problem the customer actually has.

Dowson raised a detail that lands harder than it sounds: computer science graduates often describe the business communication requirement as the hardest class they took. Haddad, who came to engineering from finance and accounting before minoring in computer science, agreed that those skills have moved from peripheral to central. An engineer who spent decades writing code all day now sits in front of clients and stakeholders and does a meaningful share of the product work.

That reshapes hiring. Advocate's filter moved from raw technical strength to whether a person can own a problem from beginning to end: do the discovery, get on the phone with the right people, make the technical calls about how to build it and where the data lives, carry it through implementation, and then measure whether the feature actually worked.

Why does model selection matter less than people think?

Buyers often assume the important question is which model a vendor picked. Haddad's view is that this gets more attention than it is worth. His team does take care with model choice, and he cycles through several in a given day, but any of the frontier models can produce the work. The decision that matters sits one layer out.

That is the whole problem in two sentences. Traditional software gives a binary result you can assert against. A probabilistic system gives you an answer that might be right, and testing it requires an eval framework, quality checks, harnesses that constrain what the model is permitted to do, and pipelines that orchestrate several models together. Anyone can call an API. Building the apparatus that tells you when the output is wrong is the expensive part, and it is the part a demo rarely shows.

What happens when a vendor release erases your feature?

Haddad volunteered an example against himself. His team built a presentation generator: pull the data out of the system, assemble a deck a user could put in front of a client or a board. The hard part was never the writing. It was deciding where in the stack the AI was allowed to make decisions. Was it designing the layout? Writing the content? Researching? Where was the data coming from? The team spent real time constructing a harness around an AI designer.

His conclusion was not to defend the work. It was to set pride aside, consume the vendor capability, and let his team be the experts in the domain where they actually have an edge. Anyone building on frontier models needs that instinct, which he describes as seeing six months ahead to whether a model release is about to absorb the thing you are constructing.

What should stay in human hands?

Haddad calls this the million dollar question, and his answer is narrower than either the optimists or the skeptics would like. With enough harness and eval work, a model can carry most of the load in most applications. What he would not hand over right now is the final step of delivery on the most sensitive work.

Compliance is his example, and it is directly relevant to Advocate. A model can do a great deal of the work leading up to delivery. The final check on compliance work stays with a person. The same holds for legal review.

Dowson asked whether that is a technical judgment or a liability judgment, and Haddad's answer was both. He reached for the self driving comparison: when an autonomous vehicle gets a ticket or is involved in an accident, the question of who is answerable is still genuinely unsettled. The same structure appears when a model performs a task and gets it wrong. As a manager, who do you go to for an answer? You cannot ask the model.

His expectation is that the technical gap closes first. Models improve, and solutions get built that wrap up the surrounding apparatus so it is within reach of a normal team. The liability question stays open longer, and the answer appears to shift month to month.

What does the model supply chain look like?

This is where the conversation connects to territory this show has covered from the insurance side. In a recent episode on managing general agents and fronting carriers, the point was that a policyholder can sit on a chain of counterparties without being aware of it. Dowson drew the parallel directly: a software company building on frontier models is in a structurally similar position.

Haddad's starting point is that customers do not buy models.

So his team treats model providers the way it treats any third party dependency, including its cloud providers. That means building a system that tolerates failure, and assuming as a baseline that pricing will change, terms will change, and providers will have outages. Those are ordinary vendor risks, and they do not stop applying because the vendor is an AI company. If anything, Haddad argues for more diligence rather than less, because the category is new, which means the risks are less well mapped even where the balance sheets are strong.

The diligence runs in the other direction too. He says the provider question comes up in essentially every customer conversation, sometimes as a formal document request covering providers and subprocessors, sometimes casually with a client's technical team. What people want to know is where their data goes, what the retention policy is, whether the provider receives the data, and whether it trains on it. In banking and insurance applications, those questions are not optional.

What breaks when a provider changes underneath you?

Engineering and production move at different speeds, and Haddad is explicit about the gap. As an engineer he moves to a new frontier model almost the day it ships, testing it and checking its work. A production system cannot behave that way. Clients rely on established workflows, so changes roll out slowly and only after evals compare the new model against the old one: how does it respond now, are there regressions, is it actually improving.

His concrete example is a proposal generation tool Advocate has been building for brokers. Like most of what the team builds, it combines deterministic and probabilistic components, and the team has been testing different models for the language that goes into a draft proposal.

The result of that testing is the useful detail. Depending on the model, the output varies widely in tone. Some drafts come back markedly more critical, others more confident. That matters more than it first appears, because a user who has worked with the tool for two months develops an expectation. Swap the model, and the tone shifts under someone who was not told to look for a change and may not review the draft as closely as they should.

The tool itself pulls compliance data, premium data, and market data to benchmark a policy against comparable transactions, then produces charts, tables, and a first draft document a broker could put in front of a client. Haddad's account of building it is worth hearing because it is not triumphant. The team built it, scrapped it, and rebuilt it, which he treats as normal practice now rather than an embarrassment. The early attempts offloaded too much to the models and could not produce a repeatable result. Documents came out with text overlapping itself. The rebuild focused on getting something testable that returns the same result reliably.

What does "accurate" actually mean?

Dowson pushed on the word directly, since every vendor in the market claims it. Haddad's answer runs through the dartboard image most people have seen: hitting the bullseye repeatedly is accuracy, while a tight cluster somewhere off the mark is precision. The part that matters is what the bullseye represents.

Without a defined test, an accuracy claim has no content. Haddad's tell for a vendor that is not really building at the front of this field is a claim resting on user sentiment: we know it is accurate because we tried it with users and they are happy. That says something about satisfaction and nothing about accuracy. Accuracy is what you get when you benchmark against a defined set of tests and expect a specific result.

Which is why the number alone is not enough either. Telling a client a system is 85 or 90 percent accurate is easy, and the client may well be satisfied. The question that matters is what math produced the figure. Haddad's stated standard is publishing not only the benchmark but the methodology, the concessions, and what the next iteration of the benchmark is meant to address. [VERIFY: on air commitment to publish this material at a specific URL, confirm what is live before publication]

How should a buyer pressure test an AI demo?

Dowson closed on the practical version of the whole conversation. If you are a broker, lender, or servicer being shown AI software for insurance compliance, and you do not have a technical background, what do you ask? Haddad offered three questions.

How do your own engineering teams use AI? This is his first question, and the reasoning is indirect. The answer tells you about the architecture behind the product, but it also tells you how the team builds and thinks, which lets you form a judgment about how they ship and how quickly they will resolve a problem when one appears.

How do you benchmark or test your implementations? For Haddad this is the one that counts. If the answer is client happiness, that is a problem. If they can produce a real benchmark with justification and evidence of how the system performs across cases, including how it handles uncertainty and how it handles decisions that should be deterministic, you can form a realistic expectation of what it will do inside your workflow. The follow up matters as much: when they change the model, is it tested, and does it give the same output?

How reliant are you on a single model? He wants to see diversification, for the same reason a serious operation runs redundancy across cloud regions. A vendor resting entirely on one provider inherits that provider's outages and price changes, and so do you.

What changes in the next year?

Haddad declined to predict specifics, on the grounds that the pace makes it genuinely uncertain, and noted that the obvious prediction has already happened: you cannot say engineers will stop writing code, because that is largely where things already are.

What he does expect is that the distance between the best people and the middle of the distribution widens considerably. In his framing, the engineers who were called ten times more productive became a hundred times more productive, and that multiple keeps climbing, which pushes demand for the strongest people higher still. Whether the ratio is literal is beside the point. The directional claim is that leverage now compounds in favor of people who understand problems and customers, not people who type quickly.

Dowson tied it back to where the conversation started. The model is a component. The real work is everything you decide a model is allowed to do.