Who Is the Right Expert for AI?

Many general human data companies solve for coverage through mass outreach, and the people who reach back out to them are an unevenly distributed sample.

The right expert for AI is rarely the most credentialed practitioner. AI learns the slice of expertise you sample. Representative data means experts across roles, seniority and backgrounds, and individuals who can teach as well as they can do the job.

At Patronus, we build an optimized team in each domain that can produce representative sets, and a reviewer set of other experts in the domain checks their work.

Who you sample

At an investment firm, a senior partner who's been there for 10 years drops work on a first-year analyst's plate. A useful expert set captures the job at each of those levels, whether the role is on the external-facing side of a company or internal.

Recruiting experts en masse skews that mix. You end up with more people who are external-facing and willing to talk to you than internal-facing. You see the front lines, but you don't get a representative set of what goes on behind the curtains.

For legal, you might be inclined to work with a large number of paralegals, and for finance, with first-year analysts. They can articulate, "This is how I write a report," or "this is how I build out a four-year projection for revenue."

The data must go beyond the end state and capture the judgment behind the work. An analyst can build the projection, but we also want the model to learn why you're selecting a certain company and how you're deciding the assumptions that go into it. The goal is to get as close to a true state as possible, as opposed to viewing it through a prism based on where you have a higher density of experts.

A brief drawn as a page, with an argument section, a risk section, cite-checked authorities and a Filed box, labeled what the model sees. On the left, under what paralegal work teaches, dotted leaders point from how to draft a brief to the argument section and from how to cite-check and assemble a filing to the authorities. On the right, under what the model misses, red brackets beside each section mark why this argument was chosen, how the risk was weighed, and whether the call came from precedent or a partner’s read of the matter. — **Figure 1.** A brief is what a model trains on. The reasoning behind it is not on the page, like why this argument was chosen, how the risk was weighed and whether the call came from precedent or a partner's read of the matter. Illustrative.

Seniority and long tasks

Many of our tasks run into the tens or hundreds of turns, long enough that one task can need a first-year analyst for one part and a partner for another.

Illustrative diagram. A short row of four gray squares is labeled Handoff for review, a few turns. Beside it a long task fills a grid of gray squares, each square one turn, ending in three dots. Teal squares mark questions on a gap, mostly early, and black squares mark partner feedback, mostly late. Red arrows link late turns back to turns much earlier. Brackets label the early part of the task First-year analyst and the late part Partner. — **Figure 2.** A task handed off for review takes a few turns. Longer tasks run to tens or hundreds of turns, with questions, partner feedback and context from much earlier along the way. Each cell is one turn, and the lengths and mix of turns are illustrative.

A typical finance task is "Build me a proposal to invest in this company." Most providers would staff it with first-year investment banking analysts who build an investment model and say, "Here's the model, ready for review from someone more senior." In the longer-horizon version, feedback comes in throughout the task. The prompt is "Here's the source data we've gotten from a company for diligence, go build the report." There will be gaps in the data, so the analyst asks questions to address them and then comes up with a proposal.

When the analyst is pitching an investment to the investment committee, there are approaches the partner might favor, preferences from conversations they've had, and context that must be retained over time. Designing a task that holds all of these possibilities takes an expert who has done the senior job, or in rare cases a very capable junior employee.

Models vary widely in how well they manage context over a task this long.

Experts from different backgrounds

When experts disagree, the cause can be their background rather than one of them being wrong. We've seen this with a paralegal from Big Law versus a paralegal from a nonprofit environmental practice. One told us, "We actually go through five-plus review cycles of a brief," while the other said, "We just do one or two." A model that learns from both starts understanding when it should behave differently, given the context and prompts. Two right answers make grading difficult. Judge a nonprofit brief by Big Law's standard and a right answer fails. Judge Big Law work without that context and a wrong answer passes. Those false negatives and false positives both teach the model the wrong thing.

Two flows from brief to filed. A paralegal at a Big Law firm goes through 5 plus review cycles. A paralegal at a nonprofit environmental practice goes through 1 or 2. Both are correct for their setting. Below, a model that learns from both takes the context and prompt and either expects iterations, like the Big Law brief, or aims for higher fidelity in the first few passes, like the nonprofit brief. — **Figure 3.** A model that learns from both knows when to expect iterations and when to get it right in the first few passes.

What makes a good expert for AI

Seniority alone isn't enough. You can be a very smart practitioner in a space and be a very bad expert for AI, because you think your way is the only way to get something done. The best experts can design a nuanced answer key. They're almost the teachers, saying, "Here are things that I've seen juniors do well, and here are other ways to reach a strong answer, depending on the circumstances."

Where a narrow expert sees one right answer, someone with perhaps the same level of experience, or even less in terms of credentials, might say, "I think there's a better way to do it. I would probably give this an 8 out of 10."

Two six by six grids where each square is one way to answer the question. In the answer, only the expert’s own way gets full credit. In the answer key, the expert’s own way scores 10 out of 10 and a cluster of nearby ways scores 8 out of 10, while the rest get no credit. — **Figure 4.** An answer credits the way the expert would do it. An answer key credits every sound way to get there, which is what grading a model’s work requires. Illustrative.

What does matter is real time in the role, not titles, or the fact that they're people leaders with a track record who've worked across industries. A strong expert may have spent 10 or 15 years as an individual contributor, or been promoted because of their quality of work and now review a team's output.

Building realistic tasks

When we started building legal tasks, the experts we work with reviewed them and said they were too clean. Every document was relevant and the instructions read like an assignment.

We asked what the mess in their work looks like day to day and rebuilt the tasks around it. Each task went from about 50 files to 1,150, including 800 real legal documents. Emails went from 13 to 120 per task, and the longest PDF from 7 pages to 300.

Paired bars for one legal task, before and after the rebuild, each row on its own scale. Files per task went from 50 to 1,150, 23 times as many, and 800 of the 1,150 are real legal documents. Emails per task went from 13 to 120, 9 times as many. The longest PDF went from 7 pages to 300, 43 times as long. — **Figure 5.** One legal task before and after the rebuild. Each row has its own scale.

Client answers now come across several emails over a couple of weeks, with someone following up on what's missing. Old and current versions sit in the same folder, drafts still have comments and tracked changes, and some documents have nothing to do with the request. The instructions are often just a short email, sometimes the infamous "please fix."

The model has to figure out what's current, what matters and what's missing before doing the work. Paralegals told us they spend much of their time on this.

This is where the reviewer set comes in. Experts reviewing tasks across domains still catch signs a document was generated, like timestamps stuck at 00:00:00 in metadata, or the same 555 phone numbers repeated across documents. We're adding both to our automated checks.

Why it matters now

Benchmarks have the same sampling problem. Labs tout progress on benchmarks with each new model release, but the tasks in these sets are often hand-picked. Model performance is not evenly distributed. A better signal comes from tasks designed by experts sampled across every level and checked by their peers.

An uneven curve of model performance across kinds of professional work. A red slice at the highest peak is the hand-picked set, and a dashed red line at that level shows what the benchmark reports. An arrow from a low dip up to that line marks the gap a hand-picked set hides. Tick marks under the axis show the benchmark samples only the red slice. Illustrative, no data. — **Figure 6.** A hand-picked benchmark set samples the peak and hides the dips. Illustrative.

The role of experts is to bring AI to the starting line. That line moves as the collaboration between humans and AI grows stronger.

We believe the right expert for AI is a representative set rather than the biggest pool. We're exploring how to scale expert contributions and which domains still need deeper expert benches.

To do that, we're hiring experts in these spaces. If you can design a nuanced answer key, and you know your way isn't the only way to get something done, we want to hear from you. If you're interested, see our open roles here.

For full-time roles, we’re hiring across research, engineering and operations in person in San Francisco. Join us.