We do this for a lot of our customers (fine tuned to save cost when inference volume is high). Right now for internal coding we are using off-the-shelf models but we are considering fine tuning as well to squeeze more efficiency out.
Thank you for the feedback. We wrote this because after discussing with some of our peer companies, I realized everyone was roughly doing similar things. And I thought it would be good for someone to just systematically write down what those are so that others can try out the techniques if they find them useful.
The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks.
In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustration with the experience of working with Opus 5, despite it topping benchmarks and being objectively extremely powerful/competent.
However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957https://labs.scale.com/papers/swe-interact.
Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks
The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But suddenly a new model comes out that's twice as expensive, there are some changes to the harness (we found randomly Claude Code and other harnesses will make changes that drastically impact efficiency), and then maybe you have some organic user growth as well and BOOM suddenly you're at a $10M run rate within 60 days. And it's now impossible to forecast future growth.
It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations.
The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.
Yes it's nuanced, but will be simplified going forward.
This uses a fully open source (liberally licensed) model and we also open sourced (liberally licensed) our own training code. However, the uptraining dataset of ~50,000 samples was generated with OpenAI's text-davinci-003 model, and depending on how one interprets their terms, commercial use of the resulting model may violate the OpenAI terms of use. For that reason we are advising only noncommercial use of this model for now.
The next step here is to create a set of uptraining samples that is 100% open. Stay tuned.
Are you in touch with the OpenAssistant team? I believe they already have a more or less complete set of samples (100,000!) that were produced in an open environment and aren't encumbered by any licensing.
Was the Alpaca dataset being licensed as non-commercial only the reason you aren't releasing the weights? Is it possible to just release them under the same license?
Yes the issue is that some of the training data is arguably tainted with some noncommercial license (it's nuanced, discussed below in my comment). We are releasing weights to people who request but we just wanted to have an email request flow so that we can make sure people know it's just for noncommercial purposes.
Working on a model without this issue. Certainly our goal is totally open models anyone can use for anything.
I've been a bit jaded by the "open/democratizing ai" stuff and then having companies stiff us at actually making it open - but not wanting to be the first to litigate these new types of issues ml brings is very understandable.
Question - Would you consider benchmarking a single 4090 for your training? While training in a few hours with 8x A100's is impressive, myself and I think others are curious how that translates to consumer hardware. IMO running/fine-tuning on consumer hardware is the ultimate endgame for all ai models.
