A human-in-the-loop review queue has five parts: intake, model extraction with per-field confidence scores, routing rules that send uncertain items to a person, a reviewer screen, and an audit log. At our rates a standalone queue starts at $4,000 and takes four to eight weeks. Connecting it to your CRM or accounting system adds $1,500–$6,000.
How to build a human-in-the-loop document review queue
The five parts of a review queue
Every queue we build has the same shape, whatever the documents: the model reads, a person checks only what the model is unsure about, and the system records who decided what.
- Intake. Documents come in from an email inbox, an upload form, a portal or a watched folder. Each file gets a content hash so that the same PDF sent twice becomes one item, not two.
- Extraction. OCR turns scans into text, then a model classifies the document type and pulls out the fields you need. Each field comes with a confidence score, not just a value.
- Routing. Rules decide what happens next: auto-approve, send to review, or reject. Rules work per field and per document type, so a low-confidence
invoice_totalgoes to review while a shakyvendor_addressmight not. - Reviewer screen. The document sits on one side and the extracted fields on the other, with the uncertain ones highlighted. The reviewer corrects, approves or rejects, and the next item loads.
- Audit log and hand-off. Every change is stored with the original model value, the corrected value, the reviewer and a timestamp. Approved records are then written to the system of record: HubSpot, Salesforce, QuickBooks Online or your own database.
How an item moves through the queue
Model the queue as a state machine, not a to-do list. A typical set of states is received, extracted, needs_review, in_review, approved, rejected and posted. Each move between states is a logged event, which gives you the audit trail and the metrics without extra work.
Worked example: vendor invoices into QuickBooks Online
Invoices arrive at a shared inbox. A worker saves each attachment to storage, hashes it, and creates a received item. Extraction pulls the vendor, invoice number, date, line items and total, and checks that the line items add up to the total.
Routing applies three rules. If every required field scores above the threshold and the arithmetic checks out, the item is approved. If the vendor is not in QuickBooks yet, the item goes to review with the reason unknown_vendor. If the invoice number has already been posted for that vendor, it is held as a likely duplicate.
A reviewer claims an item, which locks it to them for 15 minutes so two people never work the same invoice. After approval, the system creates a draft bill in QuickBooks Online and stores the returned bill ID on the item. If the API call fails, the item stays approved and retries, rather than vanishing or posting twice.
What the numbers look like
These figures are illustrative, not a client result. Take 1,000 documents a month that each take about 4 minutes to key in by hand: roughly 67 hours of work. If 30% get routed to review at 1.5 minutes each, plus a 5% spot-check of auto-approved items, the monthly load drops to about 9 hours. Your real straight-through rate depends on how consistent your documents are, so measure it on a sample of your own before you budget around it.
Build it, buy it, or wire it together?
The right option depends on volume, how varied your documents are, and where the approved data has to end up.
| Option | Upfront cost | Monthly cost | Who maintains it | What breaks first |
|---|---|---|---|---|
| No-code (Zapier or Make plus Airtable and an OCR step) | A few days of setup | Per-task and per-seat fees that grow with volume | Whoever built it in-house | Duplicates and parallel reviewers; no item locking or real audit trail |
| Off-the-shelf document AI product with a built-in review screen | Low setup, sometimes an onboarding fee | Per-page or per-document pricing | The vendor, plus you for the integrations | The hand-off into your CRM or accounting system, and custom validation rules |
| Custom queue built in code | From $4,000 at our rates, plus $1,500–$6,000 per system it writes to | Hosting and model usage, usually small at this volume | Your team or an engineering partner, with a runbook | Model or vendor changes upstream; handled by tests and alerts |
Hourly work is $85 an hour. If the queue becomes the place where cases live from arrival to close, with stages, owners and approvals, it has become a system of record. Those start at $15,000.
Failure modes to design for on day one
- Confidence scores that lie. Models are often confident and wrong. Set thresholds from a labeled sample of your documents, not a default, and re-check them every time the model or prompt changes.
- Two reviewers, one item. Without a claim and timeout, two people correct the same document and the second save overwrites the first.
- Duplicates. The same file forwarded twice, or the same invoice rescanned. Hash the file and check business keys such as vendor plus invoice number.
- Documents that arrive before their record. A signed form comes in before the deal exists in HubSpot. Park it in a
waiting_for_matchstate and retry the match, instead of creating an orphan record. - Partial failures on hand-off. The bill was created in QuickBooks but the response timed out. Store an idempotency key so the retry does not post a second bill.
- Expired connections and rate limits. OAuth tokens get revoked and APIs throttle bulk writes. Queue the writes, back off on errors, and alert a person when re-authentication is needed.
When to build one, and when not to
Build a custom queue when you handle a few hundred or more documents a month, the data has to land in a specific system with validation rules, and you need an audit trail you can show a client, lender or auditor. That is the pattern we build for document-heavy firms in lending, credentialing and professional services.
Use an off-the-shelf product or a no-code chain when volume is low, document types are standard (receipts, W-9s, common invoices) and one person does all the reviewing. The missing locking and audit features matter much less with a single reviewer.
Do not build anything yet if you process fewer than about 100 documents a month, or if you cannot say exactly which fields you need and where they go. Write that down first and key the documents by hand for another quarter.
Frequently asked questions
How much does it cost to build a document review queue?
At Smplfy, a standalone review queue is an internal tool and starts at $4,000. Each system it writes approved data to, such as QuickBooks Online or Salesforce, adds $1,500–$6,000 depending on field mapping and error handling. If it grows into a full system of record with stages and approvals, that starts at $15,000.
How long does it take to build a human-in-the-loop review workflow?
Plan on four to eight weeks for a queue with intake, extraction, routing, a reviewer screen and one integration. The range depends mostly on how many document types you have and how strict the validation rules are. Collecting a labeled sample of your documents early shortens the tuning phase.
What confidence threshold should send a document to human review?
There is no universal number. Set thresholds per field by running the model over a few hundred of your own labeled documents and choosing the score above which it is almost never wrong. Re-check those thresholds whenever the model, prompt or document mix changes.
Can Zapier or Make handle a document review queue?
For low volume and a single reviewer, yes: an OCR step, an Airtable base and an approval step can work. They struggle with item locking between reviewers, duplicate detection, per-field routing and a real audit trail. Past a few hundred documents a month or more than one reviewer, a coded queue is usually cheaper to run and easier to trust.
Do reviewer corrections make the AI model better?
They can, but the main value is measurement. Stored corrections become a test set that tells you whether a new model or prompt is more accurate before you switch to it. Some teams also use them as examples in the prompt or for fine-tuning, which works once you have a few hundred corrections per document type.
Related answers
Want a scoped estimate for your review queue?
We’ll send a short written scope or talk it through on a free 20-minute call. Have ready the document types you handle, roughly how many arrive each month, and which system the approved data needs to go into. Start a conversation.