A practical framework for using LLMs to score candidates accurately against role requirements, and why the obvious approach, naive vector search, produces confident wrong answers. This is the methodology behind CloudFlow Compass, our AI candidate search add-on.
The trap: the giant search query
Start with a clean list of requirements, say 1) 5+ years Java, 2) Spring Boot, 3) AWS deployment. The tempting shortcut is to mash them into one big vector query against your candidate database. The problem is arithmetic: a single embedding averages everything, so a candidate can score highly overall by over-indexing on one minor skill while completely lacking a core requirement.
The false positive: with standard RAG, a candidate with 0 years of Java but 10 years of AWS comes back as a “strong match”: the vector math averaged them past the requirement that actually mattered.
The solution: a three-step pipeline
Separate broad retrieval from strict evaluation.
1. Broad retrieval: the net
Use the job title and top requirements in a standard hybrid search (vector + keyword) against the candidate database. The goal is a broad pool of 50–100 generally relevant candidates. Don’t aim for precision here; aim for recall.
2. The LLM scorecard: the filter
Pass the retrieved candidates plus the requirements list to a fast LLM with a strict rubric: score each requirement 0 (no evidence), 1 (partial), or 2 (explicit match); return structured JSON. The model reads each CV the way a recruiter would, catching that a “Manager of Java Developers” is not a hands-on Java developer, which keyword and synonym matching routinely miss.
3. Weighted math: the ranking
Because the output is structured, ranking becomes simple application logic: multiply by category weights (must-have ×3, should-have ×2, nice-to-have ×1) and sum. A candidate scoring 11 of 12 weighted points cleanly outranks one at 5 of 12, and you can show exactly why.
Why this works for recruitment
- Explainability: a line-by-line scorecard with evidence for every requirement, so you can show clients precisely why a candidate was recommended
- Dynamic weighting: a client promotes a nice-to-have to must-have? Recalculate instantly, with no re-indexing and no backend tuning
- Speed: a fast LLM screens hundreds of stored candidates in seconds, turning your existing database into a live sourcing channel
Legacy semantic search vs the LLM scorecard
Legacy engines match keyword ontologies but stumble on context, return opaque percentage scores nobody can defend to a hiring manager, and need re-tuning whenever requirements change. The scorecard approach reads contextually, explains completely, and re-weights on the fly.
This is the engine inside CloudFlow Compass: point a live job spec at the candidates you already have and surface qualified people from past applications, value you’ve already paid for, before spending on new adverts. See the scoring style yourself in the live demo.