LLM 분류기로 사용자로부터 수동 답변 분류 작업 해제
[원문 본문]
Awais Shahid
Posted on Oct 28
Taking manual reply triage off users' hands with an LLM classifier
# ai
# llm
# architecture
# webdev
How I put intent classification into production on an AI sales-automation platform, as its sole engineer, and why the manual path stayed.
The manual process
The platform is an AI sales-automation product: users run outreach campaigns, and prospects reply. Those replies are where the value is, and before the classifier every one of them was sorted by hand. Users went through a campaign's replies in the dashboard and decided for each one what it was: is this person interested, not interested, away, or agreeing to a meeting? At peak the platform handled close to 100,000 requests a day, busiest in the evening, and roughly 5,000 to 10,000 of them were replies to sort.
When I joined, I worked on that manual categorization in the messages view. Then I was the one who replaced it.
What the category decides
Sorting replies wasn't bookkeeping. The category decides what the platform does next:
Interested: an AI-generated reply goes out.
Not interested: the contact goes on a list the next campaign leaves out.
Out of office: an email is scheduled for the date the person is back.
Meeting scheduled: the reply agreed to a meeting.
A catch-all category for anything that fits none of them.
That's 5 of the 8 categories I still remember. When I took over, these actions existed but were completely broken, and the conditions behind them were buried in code that was hard to follow. I rebuilt them into working order, then put the classifier in front of them.
The point is that a wrong label is never just a wrong label: it sends the wrong next action to a real person. A not-interested reply filed as out-of-office gets a follow-up email from someone who already said no.
How the classifier works
I built it on Azure OpenAI, with four decisions doing most of the work.
A fixed label list with structured output. The model doesn't write a description of the reply; it returns one of the known categories in a fixed structure. Free text would have needed parsing, and parsing is where silent failures hide.
Examples in the prompt. The prompt carries example replies for the categories, so the model sees what "interested" looks like in a real sales inbox, not a dictionary definition of the word.
A catch-all instead of a forced guess. Anything outside the list falls to the catch-all. A reply the model can't place lands where a person will look at it, instead of being forced into the nearest category and triggering the wrong action.
One wrapper around the provider. Every OpenAI API call in the product goes through one wrapper pointed at an Azure OpenAI deployment, so changing provider or deployment is a config edit, not a change at every call site.
Keeping the human path
The manual categorization didn't go away. It became the correction path: when the classifier gets one wrong, the user moves the reply to the right category, exactly as they used to. The decision was made with the client, and it was the right one: users went from sorting every reply to fixing the few the model got wrong.
The mistakes I remember seeing most were not-interested replies tagged out-of-office, and replies the model put in the catch-all. The first is the costly kind: it sends a follow-up to someone who already said no.
Running it in production
The classifier sat inside a larger system I owned end to end for the whole tenure: the React dashboard, the Node.js and GraphQL backend on MongoDB, and the BullMQ and Redis queue that runs outreach. That queue is where production showed its teeth: a 5-minute stall backed up thousands of jobs, and all of them drained afterward with none dropped. An LLM feature is only as reliable as the pipeline it runs in.
What I'd add today
Looking back with what I know now, the correction path was the most valuable part of the design, and it was underused. Every time a user moved a reply to a different category, they produced a labelled example of exactly the kind of reply the model gets wrong. Today I'd log every correction and use them two ways: as an evaluation set, run against every prompt or model change before it ships, and as fresh examples for the prompt. The classifier would then get better from the same clicks users were already making.
The pattern
Strip away the specifics and this is the shape of most useful LLM work in a business:
Find the decision people make by hand, and what each decision triggers.
Constrain the model to those decisions: fixed labels, structured output, examples from real data.
Give it a way to say "I don't know" that lands in front of a person.
Keep the manual path as the correction path, and treat each correction as data.
That's the work I do: taking a manual process and turning it into an AI feature that runs in production and keeps working once people depend on it.
The platform: b2brocket.ai
Awais Shahid, Senior Full-Stack AI Engineer. LinkedIn · GitHub
Top comments (2)
Subscribe
Personal
Trusted User
Create template
Templates let you quickly answer FAQs or store snippets for re-use.
Submit
Preview
Dismiss
Collapse
Expand
Paul Irolla
Paul Irolla
Paul Irolla
Follow
Building Tokenade, a free and open-source token optimizer for coding agents (Claude Code, Codex, Cursor). Runs locally.
Location
Nantes
Joined
Oct 8, 2024
•
Oct 29
Dropdown menu
Copy link
Hide
Keep the obvious cases in code, such as the sender or whether the thread already has our reply. Do the ambiguous ones make up most of your volume?
Like comment:
Like comment:
1 like
Like
Comment button
Reply
Collapse
Expand
오피셜메일
오피셜메일
오피셜메일
Follow
1인기업·1인창업자의 바이브코딩, 브랜딩과 마케팅 실행 과정을 기록합니다.
Location
대한민국
Work
오피셜메일
Joined
Jul 29, 2026
•
Oct 29
Dropdown menu
Copy link
Hide
catch-all을 남겨두고 수동 분류 경로를 교정 경로로 유지한 설명이 핵심으로 보입니다. 특히 거절 회신을 부재중으로 분류하는 오류는 전체 정확도와 별도로 추적해볼 만하네요. 이미 후속 메일이 예약된 상태에서 사람이 라벨을 바로잡으면, 그 예약도 함께 재검토되는지 궁금합니다.
Like comment:
Like comment:
1 like
Like
Comment button
Reply
Code of Conduct
•
Report abuse
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink .
Hide child comments as well
Confirm
For further actions, you may consider blocking this person and/or reporting abuse
[스타트업스쿨 페이지 요약]
/ LLM 분류기로 사용자로부터 수동 답변 분류 작업 해제 LLM 분류기로 사용자로부터 수동 답변 분류 작업 해제 ( dev.to ) Dev.to WebDev · 7시간 전 · AI 모델 AI 영업 자동화 플랫폼에서 수동으로 진행하던 고객 응답 분류 작업을 LLM으로 자동화하면서, 단순 자동화를 넘어 사용자의 수동 수정을 데이터 피드백 루프로 활용해 모델의 정확도를 지속적으로 개선하는 실전적인 AI 도입 전략을 제시합니다. 이 글의 핵심 포인트 1 Azure OpenAI의 구조화된 출력(Structured Output)을 사용하여 파싱 오류 및 묵시적 실패를 방지함 2 프롬프트에 실제 사례를 포함하는 Few-shot 방식을 통해 모델의 문맥 이해도를 높임 3 판단이 불가능