Your pipeline makes small decisions all day: which runner takes this job, which tests to run for this diff, whether that failure deserves a retry, whether to ship. You’ve probably encoded them as thresholds, config, and a few heuristics you’re afraid to touch. That works until a flaky test, a saturated runner pool, and a release deadline all hit in the same minute, and the rule you need is the one nobody wrote.
Recently I’ve been curious about decision models: a newer kind of AI built to pick one option from a list, quickly and cheaply. Could one handle choices like these? Real pipelines are a hard place to find out, because you can’t replay the same Tuesday twice. So I decided to create a simulation.
A building shares the same pipeline problem with its elevators. The cars are your runners, the button presses are your commits, and a dispatcher has to pick a car for every call, instantly and forever. An elevator bank has a clean score (how long people wait) and can be replayed exactly. Add a rule for each new edge case and you end up with the same pile of special cases.
Like your pipeline, when it chooses well, no one notices. But when it chooses badly, people stand in lobbies, checking their watches, never knowing why.
What we can build is limited by the pieces we can ship. You can solve a surprising number of problems with plain rules, but every new variable multiplies the cases. A heavy group, a broken car and a meeting ending on the fifth floor can all arrive in the same minute. Write a rule for each combination, and soon you are tending a jungle, and somewhere in it there is always a case nobody thought of.
So the question behind this experiment is simple to ask and hard to answer: can we take a small part of what the language-model revolution has taught us, and use it to put smart, affordable parts into ordinary things, like the lift in an office? That is also why the benchmark is an animated building rather than a silent script: I want you to watch how quickly a few simple things turn into a moment nobody planned for.
Hypothesis
We tested two decision models: TypeSafe’s Jev, a hosted model getting a lot of attention right now, and Laya, a small open-weight model you can run and train yourself. Our central hypothesis was that a small, affordable AI decision model could make a repeated, real-world choice (here, which elevator answers a call) as well as a well-tuned classic rule, but only under two conditions:
First, the model has to be given the right information in the right form.
Second, a model built to be trained has to actually be trained, on the system’s own decisions.
We did not expect these models to beat the rule. The question was narrower and more practical: what does it take to match the baseline you already have? That is the question a CI/CD team faces before handing decisions like runner selection or test selection to a model. Before it can improve on your rules, it has to equal them. We broke the idea into three testable hypotheses:
For Jev, how we describe the cars matters as much as the model.
Out of the box, the open model is not ready: its makers call it “a fast base to specialise, not a zero-shot decision engine”.
Fine-tuned on the building’s own decisions, the open model can reach the level of the best dispatchers.
We explored these first on ten seeds, then re-tested the headline claims on fresh seeds (see “Then we checked”). Each one is tested in the experiments below.
The contenders
Decision models are a different kind of AI. A chat model writes; a decision model chooses. You send it the facts and a typed question with a set of options, and it returns one option, with a probability for every option, in a single step. There is no text to parse and nothing to make up. For elevators, the question is one per call: “Which car should answer this hall call?”, asked only among the cars that can legally take it.
Jev is TypeSafe’s hosted decision model (TypeSafe calls it a System One model), reached through an API. It can’t be fine-tuned: “the same weights serve every account”.
Laya is Convai Innovations’ open-weight model (Apache 2.0, 421 million parameters). You run it on your own machine, and you can train it.
Both speak the same API, so we could send them byte-for-byte identical requests, with only the model name changed. That is rare in AI comparisons.
(to watch the video, ensure you’re reading this article in your web browser or Substack app.)
The setup
We built Elevator Decision Arena, an open-source benchmark that lives inside a simulation. It races two dispatchers side by side, in two copies of the same building, with the same passengers and the same surprises. The building and its traffic are simulated. The decisions are real: real algorithms, and real calls to both models. The results describe our simulated building, not any real one.
Jev (left) and base Laya (right) dispatching the same morning rush, 48 seconds in. Same building, same passengers, same moment; in both buildings, car 3 is out of service.
Building: 24 floors and 4 cars. Two kinds of traffic: Normal Traffic, a steady day, and Morning Wave, a lobby rush with a heavy group, a broken car and a sudden crowd.
Seeds: a seed is a whole morning, replayed exactly. Every dispatcher faces the same people, and we compare them seed by seed. (This is the same idea as replaying the same commit history through two different pipeline configurations, so the only thing that differs is the decision-maker.)
Fixed decision time: every decision takes 0.25 s, whoever makes it, so only the quality of the choice counts.
Yardsticks: Nearest-Car ETA, a classic rule that sends the car with the lowest estimated arrival time (versions v1.2.0 and a load-aware v1.3.0), and round robin, which hands calls to the cars in turn, wherever they are.
Pinned versions: jev-1.13.0, checked on every answer; Laya at a reviewed commit, on a laptop GPU (Apple M4 Pro).
Experiment 1: How you ask matters
We asked Jev the same question three ways. For each car, it read either raw numbers, friendlier words, or an arrival estimate worked out by our code:
Raw: floor 3.2, moving up, doors closed, 150 of 630 kg, 2 stops, 1 call
Words: 4 floors below the call, moving towards it, about a quarter full, 2 stops planned
Estimate: arrives in about 9 s, about a quarter full (480 kg of room), 2 stops planned
(These examples come from a hand-built test request, whose cars are larger than the simulator’s.)
On raw numbers, Jev did about as well as taking turns. TypeSafe’s own docs explain why: Jev “is not a calculator”, and the advice is to “keep the arithmetic in code”. With the estimate, Jev came within a second of the classic rule, too close for ten seeds to call.
One caveat: the estimate is the classic rule’s own formula, and our question asks for the car that will arrive soonest. Jev is still free to prefer a slower car with room, but we handed it the key number. On 1,601 separate validation decisions, it picked the rule’s car 90.3% of the time.
Experiment 2: The open model, out of the box
We sent base Laya the same raw-number requests, untrained.
Used zero-shot, in our question format, base Laya did worse than round robin, a dispatcher that doesn’t even look at where the cars are. With the arrival estimate it improved (40.3 s and 50.7 s) but still trailed taking turns. On the 1,601 validation decisions it agreed with the classic rule 24.9% of the time, a little below a random pick.
None of this contradicts Laya’s documentation, which says base checkpoints are “near chance on typed-decisions zero-shot”. A fair reader could also point to our side: we wrote the question format, and in these runs the cars were always listed in the same order. Reading raw numbers, base Laya picked the first car on the list far more often than the rule would.
Where Laya already wins: it answers in about 40 ms on a laptop (Jev: about 200 ms over the network), costs nothing per decision, gives the same answer to the same request, works offline, and can be taught.
Experiment 3: Teaching the open model
We then taught Laya with the one thing Jev can never be trained on: the building’s own decisions, generated by the simulator. The seeds we raced on were never used in training.
Copy the rule (imitation): 4,382 decisions labeled with the classic rule’s choice; about 24 minutes on a laptop GPU. Agreement with the rule rose from 24.9% to 92.8%. In the races it waited 9.4 s and 7.3 s less than base Laya, on every seed, and came level with the rule and with Jev. A student who copies a teacher can, at best, reach the teacher, and that is what it did.
Learn from outcomes (hindsight): At each decision the simulator tried every car, played 90 seconds forward, and labeled the car that led to the least waiting. A dispatcher acting on those labels was about 6 s better than everything, but it is an oracle: its forward plays see the real future, including who walks in next. Laya trained on those labels came out level again, and captured none of that gap. The labels knew the future; the model can only see the present.
A model for another building: In a second building (12 floors, 3 cars, evening traffic heading down), the model we had fine-tuned on the first building’s decisions, with no training on this one, performed at the same level as the classic rule. A model trained only on the second building’s 2,109 decisions did far worse: it sent car 1 every time. Its 43% accuracy was misleading, because car 1 happened to be the correct answer in 43% of the training labels.
Then we checked, properly
All of the above was exploration: we chose the best way to ask while watching the same ten mornings. When you look at the answer while you tune the question, you can fool yourself without meaning to.
So we wrote the tests down before running them, committed the plan to the repository, and only then made a single call. We did two runs, each on thirty fresh mornings nobody had ever run, with a correction for testing many claims at once, and a margin set in advance for what counts as “equal”: 1.5 seconds, smaller than the gap between the rule and taking turns. The plans, the results and the full analysis are public, in docs/confirmation.
All thirteen pre-registered claims held, including the fine-tune’s level in the second building. The effects shrank a little on fresh mornings, as honest effects usually do. And shuffling the cars made no difference to the fine-tune either: it had learned the facts, not the positions.
Two things we report as we found them:
One earlier claim did not survive: On the first ten seeds, the fine-tune looked 1.6 s better than the load-aware rule in the second building. On fresh seeds it was simply level. We dropped the claim.
Round robin surprised us: In the morning rush, taking turns was level with the classic rule and the best models, on both sets of fresh mornings. Simple things can be stronger than they look.
Speed and cost
Speed: With Jev’s real 200 ms round trip counted in the simulation, it stayed within a second of the rule. For elevators, a blink doesn’t seem to matter.
Cost: Jev costs about $0.00002 per decision at TypeSafe’s list price of $0.042 per million input tokens (as of October 2026; pricing). The whole study, about 25,000 Jev decisions, cost less than half a dollar. Laya and its fine-tunes cost nothing per call.
For pipelines, this suggests that latency and per-decision cost may not be the main obstacle for decisions that happen between jobs, such as picking a runner or choosing a test subset, where a fraction of a second is small next to a build that takes minutes. We did not test this on real workloads.
Results
Across three experiments on ten seeds, and a pre-registered re-test on fresh mornings in which all thirteen claims held, the answer came down to a few things: how the question is asked, whether the model has been trained, and what it is measured against. Five lessons follow. They hold inside our simulated building, and any link to your pipeline is an analogy, not something we measured.
What you show a model matters as much as the model. The biggest step came from giving Jev a number our code had already worked out.
Out of the box, the generalist wins. Jev was sensible from day one; base Laya was not, as its makers say.
A small open model can be taught to the level of a good rule in 24 minutes on a laptop, and then will run with no per-decision fees.
Beating the rule needs more than better labels. The model has to see something the rule ignores, such as recent calls on each floor. That is our next experiment.
Test on mornings you haven’t seen. Writing the test down first is cheap, and it is the difference between a hunch and a result.
Limits
A simulated building with made-up traffic, two building sizes and three traffic patterns. The comparisons hold inside this world, not beyond it.
“Equal” means within the pre-set 1.5 s margin, not identical. Results marked exploratory (the hindsight fine-tune, the collapse, real-latency runs, Jev vs Laya head to head) were not re-tested.
One base Laya checkpoint, one training recipe, one device. Other settings may do better or worse.
Identical requests are not identical chances: a model built to be fine-tuned, asked zero-shot in our format, was tested on our terms, not its own.
We did not test any CI/CD workload. The pipeline connections in this article are analogies drawn from the elevator results, not measured findings. Elevator dispatch also has one clean goal (average wait), whereas pipeline decisions often balance speed, cost and risk at once, which is harder to benchmark.
TL;DR
In our small building, a simple rule held its own, and even plain turn-taking held its ground in the morning rush. Nothing here shows that decision models are needed yet. But these models are improving quickly, and I believe that for many everyday decisions a decision model will become the cheapest way to handle the cases our rules never imagined: the crowded minute, the broken car, the pattern nobody wrote down. That is my personal bet, not a finding. Only time can prove it, or prove me wrong.
If you build and run pipelines, here is what we’d take from the experiment:
Present the data well: Let your code compute the key number (an estimated wait, a queue depth, a historical duration) and let the model weigh it. Handing it raw metrics performed about as well as no intelligence at all.
Benchmark against the dumbest baseline: Round robin tied the best approaches in the rush. If a trivial rule matches your model, you don’t need the model.
Hosted generalist or small open model is a real trade-off: The hosted model was sensible from day one but can’t be customized. The open model was poor until trained, then matched a good rule, runs offline, and costs nothing per decision.
Training on your own history has a ceiling: A model that learns to copy your rules will, at best, equal your rules. To do better it needs to see signals the rules ignore.
Distrust a metric that looks like knowledge: A model that always picks the same answer can post an impressive accuracy number. Compare every score with what a trivial guess would get.
Evaluate on data you haven’t tuned against: Write the test down first, run it on fresh data, and be ready to drop a claim that doesn’t hold up. One of ours didn’t.
Latency and cost are probably not the blocker: A 200 ms network call didn’t hurt, and 25,000 decisions cost under fifty cents. Quality of the decision, and how you ask, are what matter.
