Who grades the AI?

The problem, and what the product promises

A serious job search is a data problem wearing a motivation problem’s clothes. The listings are duplicated, mislabelled and stale; a large share are ineligible before they are ever read, because of location, visa, seniority or a closed requisition. The effort goes into reading roles that were never available, and the reading is what runs out first.

Automating that looks like an obvious job for a language model, and that is the trap. A model will happily score a role it cannot verify, explain a rejection it did not make, and write an application answer citing experience the candidate does not have. Each of those is a product failure long before it is a model failure.

So the promise ApplyOS makes is narrow and testable: every role it surfaces is one the candidate is actually eligible for; every score is reproducible from stated rules; every sentence in an application answer traces to a governed piece of evidence; and nothing is submitted that the candidate has not read.

Those two figures describe the system’s retrieval quality against labels I wrote by hand. They are not an employment outcome, and the product does not claim one.

Architecture: three boundaries

The architecture is three decisions about where judgement is allowed to live. Each one moves work away from the model and towards something a person can inspect.

• Eligibility resolves before fit is scored. A role that fails a hard constraint — work authorisation, location, a seniority band the candidate is not in — never reaches the scorer. This is deliberately not a low weight in a ranking function: an ineligible role at rank 40 is still noise, and a scorer that can trade eligibility against enthusiasm will.
• Evidence is governed, not recalled. Application answers are assembled from a fixed registry of claims, each with its own wording and its own permitted context. The model chooses which governed claim answers the question; it does not get to phrase the claim. That is the difference between a product that drafts and a product that fabricates.
• Scoring is deterministic. The same role and the same profile produce the same score, every run, from rules I can print. The model contributes structured extraction — what this listing actually says — and the scoring reads those fields. When a score looks wrong, there is something to read.

Three flows, end to end

Discovery. A run ingests listings, normalises them into structured roles, drops everything that fails eligibility, scores what remains, and presents a feed. The feed shows the score and the reason beside it — which requirements matched, which did not, and which could not be determined from the listing. “Could not be determined” is a first-class state; collapsing it into a low score is how a system starts lying quietly.

Application. The candidate opens a role and sees the employer’s actual questions. Each answer is drafted from the governed evidence registry, with the claims it used shown alongside. The candidate edits freely; edits are theirs and are never silently regenerated. A question the registry cannot answer is marked as needing a human answer rather than filled with something plausible.

Review and submit. Before anything leaves, the candidate sees the complete submission as the employer will receive it, including every field the form requires and every answer’s provenance. Submission is an explicit act. There is no batch-apply, and that is a product decision rather than a missing feature.

What I decided, and what the build tools produced

I built this through AI-assisted workflows, and the honest account of that is a split rather than a headline. The tools were fast and genuinely useful at implementation: schema scaffolding, extraction pipelines, form adapters, test harnesses, the long tail of glue. Left alone, they were also consistently wrong in one direction.

Every time I asked for scoring, the generated implementation folded eligibility into the ranking as a weighted feature — because that is the shape of the training data, and because it produces a working demo faster. It also produces a system that will show you a role you cannot take. Every time I asked for answer generation, the default was free recall from a résumé blob rather than selection from a governed registry. Both are the same failure: the model is allowed to conclude something the product has no basis for.

The decisions that are mine are the boundaries, the states, and what the product refuses to do. The implementation of those decisions is substantially generated. I do not think that division makes the design work smaller; on this project it made it more consequential, because a wrong boundary now propagates in minutes rather than sprints.

Two implementations I rejected

A single scoring prompt. The first working version scored each role with one model call that returned a number and a sentence of reasoning. It demoed beautifully. It was also unfalsifiable: the same role scored differently across runs, the reasoning was a post-hoc narration rather than the cause, and there was no way to answer “why did this drop?” except by asking again. I replaced it with deterministic scoring over extracted fields, which is duller and can be argued with.

Auto-submission behind a confidence threshold. The version after that could submit on the candidate’s behalf when confidence was high enough. The threshold was the tell: it existed so the system could act without a person, and every value I tried was either so high it never fired or high enough to be a lie about how well the system understood a form. Applications carry someone’s name. I removed the path rather than tuning it.

Acceptance: real forms, not a happy path

Employer application forms are the part of this domain that resists design. They are inconsistent, conditionally revealed, occasionally broken, and they change without warning. A system tested against a tidy fixture will pass every time and fail on contact.

So acceptance runs against captured real-form state: required fields that only appear after another answer, questions that duplicate the résumé, uploads with undeclared constraints, and forms that reject a submission for reasons they do not surface. The test asks the product question rather than the technical one — when this form does something unreasonable, does the candidate find out before they have lost the work?

Calibration: hand-labelling 171 roles

None of the above is worth anything unattested, so I labelled a set by hand: 171 roles, each marked as one I would genuinely pursue or one I would not, with the reason. That set is the ground truth the feed is measured against.

The first pass scored 59% feed-level precision. The failures were not subtle: the feed was full of roles that were real, well-matched on skills, and completely unavailable to this candidate — wrong market, wrong authorisation, wrong seniority band. The fix was market-aware gating, resolving those constraints per market before scoring rather than hoping the ranking buried them. Precision moved to 89% against the same labels.

The remaining errors are mostly listings that do not state a constraint the employer nevertheless applies. That is a limit of the source data, not a tuning opportunity, and pretending otherwise is how a system starts scoring confidently on things it cannot see.

What this changed in how I work

Three things carried back into the rest of my product work. The first: in an AI product, the design deliverable is the boundary set — what the system may conclude, what stays deterministic, and what a person must confirm. Screens follow from that, and a beautiful screen over a soft boundary is worse than no screen at all.

The second: “could not be determined” has to be a state the interface can show. Most AI product failures I have looked at since are a missing uncertainty state, resolved silently into a confident one.

The third: when implementation costs minutes, the expensive mistake moves upstream. It is now much cheaper to build the wrong thing correctly, which makes deciding what the thing is allowed to do the part worth slowing down for.

Next case study

Where exactly is my money?