Right before it clicked, I was flailing. Not failing in any dramatic way — just no forward motion. The regex kept doing what regex does: matching whatever was shaped like a number, phone numbers included, with no idea what an invoice number actually was. I'd add a filter, a new vendor would show up with a format that broke it, I'd add another filter. That's not debugging. That's treading water and calling it iteration.
I'd gone in with a principle: avoid the model where you don't need it. Reasonable enough on its own. But it collapsed "should I use an LLM here" into one global yes-or-no question, when the real question was narrower and easy to miss: which of these stages is actually a meaning problem, not a shape problem. Regex is very good at shape. It has no concept of meaning at all — it can't know that "SAP ID" and "Doc Number" and "Invoice Number" are three vendors' names for the same field, because knowing that requires understanding what the field is, not what it looks like. That was never a job regex could do, at any level of cleverness. It wasn't the wrong tool because it was less capable. It was the wrong tool because it was answering a question nobody was asking.
Once I moved extraction — just that one stage — to an LLM doing the classification, parsing, and extraction, it wasn't a marginal improvement. It was the first real progress in weeks.
The interesting part wasn't the extraction accuracy. It was what became visible once extraction was trustworthy enough to look at. Real emails, arriving in a real inbox, in whatever shape a given vendor actually used — some as prose in the body, some as PDFs, a lot as spreadsheets, some of those spreadsheets with a summary sheet and a data sheet and no label telling you which was which. All of that had been true the entire time. I just hadn't been able to see it clearly, because I'd been staring at regex failures instead of the data.
Then came a showcase, and this is the part worth writing down properly: once the room could see real invoices with the agent's real interpretation next to them, a stakeholder stopped asking whether the tool worked and started pointing at bigger problems — patterns in the actual data that mattered more than anything about how the extraction worked. The program wasn't the center of the conversation anymore. It had done its job by getting out of the way and letting the real problem take the room. That's a strange kind of success to aim for, but it's the right one — the tool isn't supposed to be the interesting thing in the meeting.
Extraction has stayed dependable since then — more dependable than the regex version ever was, not less, which runs backward from the assumption I walked in with. "Deterministic" and "reliable" aren't the same word. Regex was deterministic and it was still wrong, consistently, because determinism doesn't help you if you're being consistently precise about the wrong question.
The other assumption that didn't survive contact with reality was cost. I'd budgeted for LLM calls during development to be expensive enough to think about carefully. The real number came in around $5 a day. I mention it because I suspect a lot of "should we use an LLM for this" decisions are quietly being made on a fear of that bill that doesn't match what the bill actually turns out to be.
None of this touched the SAP side. The code got easier once extraction was right; getting anything changed in SAP still depends on a different team, with a different roadmap, and still takes about as long as it always has. That timeline didn't move an inch, and I don't expect it to.
Demoing to execs this week, and going live not long after, assuming that goes the way I think it will. That's as far as the story goes today.
Field Mappings