What your LLM decides, in milliseconds and without paying per call
Falco distills a large language model into a small, fast, calibrated classifier. An LLM teaches once; your own model answers from then on, on your infrastructure.
Classifying every message with an LLM doesn't scale
It works in the demo. In production, every message becomes a call to a large model.
Slow
Every message waits for a large model to answer before your system can act.
Expensive
You pay for every call, and the bill grows with every message you classify.
Dependent
If the provider changes prices or models, or goes down, your operation changes or goes down with it.
Label, train, serve
Three steps, three files. The LLM only takes part in the first one.
Label
tag.pyA teacher LLM reads your texts and answers your questions with probabilities. Labeling is resumable and retries with backoff.
Train
train.pyA multilingual encoder (mDeBERTa-v3 by default) learns to reproduce those probabilities and is calibrated so that a 0.9 is right about 90% of the time.
Serve
main.py · /v1/decideA FastAPI service answers in milliseconds, without calling any LLM in production. It runs on CUDA, Apple MPS or CPU.
One support message, three answers
A single call to /v1/decide answers all three question types: yes/no, multiple choice and scale.
Message
Questions
- Is it urgent? · yes/no
- Category · multiple choice
- Frustration · scale
{
"urgent": { "type": "noul", "noul": 0.93 },
"category": {
"type": "choice",
"choice": "billing",
"probability": {
"billing": 0.88, "technical": 0.04,
"account": 0.05, "other": 0.03
},
"confidence": 0.61
},
"frustration": {
"type": "score",
"score": 0.81,
"distribution": {
"none": 0.01, "low": 0.03, "medium": 0.12,
"high": 0.45, "extreme": 0.39
},
"confidence": 0.42
}
}Illustrative values. confidence goes from 0 (uniform distribution) to 1 (certain).
Built to make decisions in production
Calibrated probabilities
Temperature calibration aims for a 0.9 to be right about 90% of the time, so you can set thresholds with confidence.
Questions in plain text
Questions are part of the model input: the API accepts new questions without retraining. Their accuracy isn't guaranteed, though (they're zero-shot).
Any LLM provider
Use Anthropic, Gemini or any OpenAI-compatible API as the teacher: OpenAI, Ollama, LM Studio, vLLM, Groq, OpenRouter, DeepSeek…
On your own infrastructure
The model and the API run wherever you choose, on CUDA, Apple MPS or CPU, without sending every message to a third party.
Open source
The code is on GitHub: you can read it, audit it and adapt it to your case.
Where it could help you
A few ideas of where a fast, calibrated classifier fits.
Support tickets
You could classify every ticket by category and urgency as it arrives, and send it to the right team.
Lead prioritization
Useful to estimate how ready a lead is and decide who to contact first.
Message moderation
You could flag offensive or off-policy messages with a probability, and review only the doubtful ones by hand.
WhatsApp routing
Useful to decide, message by message, whether a conversation goes to sales, support or a person.
From zero to /v1/decide in three steps
The README's quick path. Check the docs for every labeling, training and deployment option.
uv sync --extra openai
# Create .env with GEMINI_API_KEY=... (or ANTHROPIC_API_KEY / OPENAI_API_KEY / OPENROUTER_API_KEY)
# 1. Label: generate 200 synthetic states and label them with the teacher
python tag.py --questions questions.json --generate 200 --provider gemini
# 2. Train and calibrate the student
python train.py --data data.jsonl --output model_s1
# 3. Serve it
uvicorn main:app --port 8000Salto Ángel can implement it in your business
We help you define the questions, train the model on your data and put it in production.