Experts on Demand with Priya Desai

The request came in at 4:52 on a Friday afternoon, and Priya Desai read it aloud to the room twice. An AI lab wanted 400 board-certified radiologists for a model evaluation project. Each one needed at least five years of reading chest CTs, a signed contract, a passed calibration test, and a working login to the lab’s annotation tool. The lab wanted the first 100 working by Wednesday and all 400 within nine days.

Nobody in the room panicked. Jun Takeda, an operations engineer, pulled up the sourcing dashboard and filtered the expert pool: 1,140 radiologists who had already passed Wrenfield’s AI-led interview, 286 of them with CT experience verified against a license registry. Desai assigned the screening rubric to one agent queue, the contract packets to a second, and the calibration test to a third. Then she did the one thing she would not hand to an agent. She called the lab’s project lead to ask what a “good” read looked like.

Six days later, 412 radiologists were active on the project. I was in the room for the first three of those days, and in the Slack channel for the rest. What I saw is the most complete version of an agent-run operations team I have watched so far: a small group of people who decide, and a large fleet of agents that do almost everything else.

Wrenfield matches domain experts with AI labs that need human judgment to train and evaluate their models. A lab might need tax attorneys to write hard questions about partnership law, or structural engineers to grade a model’s load calculations. Wrenfield finds the experts, interviews them, contracts them, onboards them to the project, reviews the quality of their work, and pays them, in whatever country they live. Every one of those steps is an operations problem.

Desai joined from a staffing firm where she ran nurse placements across 30 hospitals. “There, a big week was 60 placements,” she told me. “Here, a normal week is 2,800 new contractors, and a big week is three times that.” Her team today is 23 people. In her first month, a team of that size handled about one-tenth of the current volume, and still worked most weekends.

The change was not a single tool. It was a decision to describe each step of the expert lifecycle as a queue with a written rule, an owner, and a measured error rate. Once a step had all three, an agent could run it. Today 31 agents run in production across six queues: sourcing, interviews, contracts, onboarding, quality review, and payments. The people on Desai’s team own the rules, the exceptions, and the customers.

The Monday queue review. Each queue owner reads out the week’s error rate before anyone talks about volume.

Every expert who applies to Wrenfield goes through an AI-led interview. The agent asks about their background, then probes their domain with questions that a peer would ask. A cardiologist gets a case. A securities lawyer gets a hypothetical of the kind a bar exam uses. The interview takes about 20 minutes, runs in any time zone, and produces a transcript, a score on four dimensions, and a short written summary.

What surprised me is where the agent stops. It does not reject anyone. Below a score threshold, the agent sends the candidate a polite note and puts their file into a weekly review that a person reads. Above the threshold, it moves the candidate forward. In the middle band, roughly 18% of interviews, a human reviewer with matching expertise reads the transcript before anything happens. Desai keeps a bench of 140 senior experts, paid by the hour, to do exactly that.

The weekly review of rejections exists for a different reason. Takeda showed me the log from one week in July: 3,214 candidates below the threshold, 41 flagged by the reviewer as wrongly scored. Most of the 41 were non-native English speakers whose answers were correct but short. The team changed the interview prompt to ask a follow-up question when an answer runs under 40 words. The next month, the wrong-score rate fell from 1.3% to 0.4%.

The radiologist project made the bottleneck clear. Finding 400 experts took an afternoon. Getting 400 contracts signed, 400 tax forms collected, and 400 identities verified used to take two weeks. Each country has its own contractor agreement, its own tax form, and its own rules about what a company can ask for. The radiologists alone lived in 23 countries.

The contracts agent now builds each packet from the expert’s country, the project’s data terms, and the lab’s confidentiality rider. It sends the packet, answers the questions that experts ask most often, and chases the unsigned ones on a schedule that the team tuned by country. A reminder at 9 a.m. local time gets a 22% faster signature than one at a random hour. When an expert asks to change a clause, the agent stops and routes the request to a person. That happens on about one contract in 60.

Onboarding runs the same way. An agent creates the accounts on the lab’s tool, sends the training module, schedules the calibration test, and grades it against an answer key. For the radiologists, the median time from signed contract to first paid task was 31 hours. Two years ago, Desai told me, the same path took eight days and two full-time coordinators.

Hannah Oyelaran runs quality review, and she describes her job as a market inside the market. Labs pay for expert judgment, so the work that experts submit must hold up. Her team samples submissions on every project, checks them against a rubric written with the lab, and sends feedback to the expert. The sample is not flat. A new expert gets 30% of their work reviewed in the first week. An expert with three clean weeks drops to 5%.

Agents do the first pass. They check format, completeness, and the parts of the rubric that a rule can describe: a citation exists, a code sample runs, a dosage sits inside a standard range. They also compare each expert against the others on the same task and flag the outliers. The second pass belongs to senior reviewers from the same field. Oyelaran’s rule is that an agent can raise a concern about an expert but only a person can lower that expert’s standing.

The numbers suggest she is right. On projects where flagged experts get peer feedback within 48 hours, 71% of them clear review the following week. On an older project where feedback came as an automated email, the figure was 38%, and expert churn was twice as high. The cost of the peer reviewers is about 4% of contractor spend. Oyelaran thinks it pays for itself in recruiting alone.

Every Friday, Wrenfield pays its active contractors. In the week I visited, that was 9,640 people in 71 countries, in 38 currencies. Mateo Varga leads contractor payments, and he has a simple test for his team: no expert should ever have to ask where their money is. In August, 0.6% of payouts produced a support ticket. When he joined, the number was close to 5%.

The payments agents reconcile hours from each lab’s tool against the contract rate, apply the quality review results, build the payout batch, and check each line against the expert’s tax status and the sanctions list. They post the batch for review by Thursday at noon Pacific time. A person on Varga’s team reads every line over a threshold that the team sets by country, and every line that changed by more than 15% from the week before. That is usually 300 to 400 lines out of nearly 10,000.

Varga refuses to automate one thing: the release. The agent stages the batch, and a person on his team presses the button. In 14 months, that person has stopped a batch three times. Once, a lab’s tool double-counted a week of hours for 212 experts. The agent’s checks passed, because each line looked normal. A reviewer noticed that the total was 40% higher than forecast and asked why.

Desai hires for one skill above the rest: the ability to turn a judgment call into a written rule that someone else can follow. She asks every candidate to bring a decision they made at a past job and write it down as instructions for a new hire. Then she asks them to break their own instructions with an edge case. Most candidates find it hard. The ones who enjoy it get an offer.

The team has no coordinators in the old sense. Each person owns a queue, or part of one. They read the exceptions every morning, change the rules when the same exception shows up twice, and test each change on last week’s records before it goes live. Takeda and one other engineer build the tooling, but most rule changes come from the operators themselves. In the third quarter, the team shipped 214 rule changes. Engineers wrote 37 of them.

The backgrounds on the team are wide. Two people came from hospital credentialing offices, one from a law firm’s conflicts desk, one from payroll at a shipping company, and three from customer support at software startups. Desai likes the credentialing hires most. “They spent years reading a license and asking whether it was real,” she said. “That instinct is exactly what a queue needs from its owner.” New hires shadow a queue for two weeks, then take it over with one rule: they may not change anything until they have read 200 of its exceptions.

Headcount follows the exceptions, not the volume. Desai adds a person when a queue’s daily exceptions pass what one owner can read before lunch, about 150. Volume doubled between March and September. The team grew by four.

On my last afternoon, I asked Desai what she would never give to an agent. She did not need time to think. The first call with a new lab. The decision to remove an expert from the platform. Any message to an expert whose payment went wrong. And the question she asked on that Friday afternoon about the radiologists: what does the customer mean by good?

She keeps a short list of those decisions taped above her monitor, and it has not grown in a year. Everything else on the expert lifecycle is a candidate for a queue, a rule, and an agent. The list is how she keeps her team pointed at the work that only people can do. It is also, she said, how she explains the job to the experts themselves, many of whom are curious what it is like on the other side.

The radiologist project ended in late August. The lab extended it twice and asked for 150 more readers in pediatric imaging. Desai took the call herself.