Building custom LLMs in late 2026

Welcome to Yudame Research. Right, thanks for having me. So today we are doing a deep dive into building custom large language models, specifically looking at the landscape in late 2026. Yeah, and we should clarify right up front, this is specifically for you, the engineers, the product leads, and anyone who is actively deciding whether to fund a custom model right now. Exactly, because it is a massive investment. But before we get into the heavy data, we need to assure everyone is on the exact same page. So let's just define six core terms in plain words over the first two minutes. Good idea. Let's start with fine-tuning. Right. So fine-tuning is basically further training a model's weights on your own examples. Perfect. And then we have Retrieval Augmented Generation, or RAG, which I always pronounce like a cloth rag. Yeah, RAG. That's fetching relevant documents into the prompt at question time. Right. Then there's LORA, pronounce LORA like the name Laura. That stands for Low Rank Adaptation. Which is just a small, trainable add-on placed right beside a model's frozen weights. Exactly. Next is Reinforcement Learning with Verifiable Rewards, or RLVR, R-L-V-R said as letters. And that's rewarding answers that a program can actively check. Yeah. Then Distillation, which is simply training a small model on a bigger model's outputs. And finally, the Inference Engine, the actual software that serves a model. Okay, so with those definitions out of the way, I think we need to get to the hook of this deep dive, because it is wild. It really is. So there's this 2026 preprint. It held the model weights totally fixed, the decoding settings were fixed, the hardware was fixed. Right, nothing changed. They swapped only the inference engine across five different engines, and benchmark scores moved by up to 16.6 points. Which is staggering. I mean, 16.6 points. Yeah. That is more than many fine-tunes actually gain after spending hundreds of thousands of dollars. Right, but, and here's the twist, you have to be incredibly careful with that number. Because all four major sources citing this massive 16.6 point swing, they all trace back to just one unreplicated study. Just one. Yeah, and furthermore, if you look at unverified practitioner opinion on Reddit and X, people actually doing this in the trenches, they noted that replications ranged from eight to 12 points on sensitive tasks, down to under four points once sampling was strictly controlled. Wow, okay. So it might just be a sampling artifact. Exactly. But it highlights the core problem we're tackling today. Teams are out there funding custom models based on capability gains they cannot reliably measure. Right, often to solve problems that honestly simple retrieval or caching might already cover. So to unpack all of this, we're going to walk through this logically. Let me preview the segments in exact order so you know what to expect. We have the introduction and hook, about four and a half minutes. Then the foundation on retrieval versus fine-tuning, about five and a half minutes. Laura in small-tuned models, about four and a half minutes. Reinforcement learning with a checker, about five minutes. Measuring honestly, about five and a half minutes. The fine-tuned versus cache break-even, about four and a half minutes. Surviving base upgrades, about three and a half minutes. And closing, about three minutes. So your listener implication decision rule for this intro is simply this. Understand your definitions and ensure the gain you are chasing is real before committing resources. Perfectly said. Okay, let's unpack this foundation on retrieval versus fine-tuning. Where should facts live and what is training actually for? It's the biggest debate in AI right now. Right, and I always like to use this analogy. Training facts into a model is like trying to bake a dictionary into a cake, whereas rag is just handing the chef a recipe card. That is a very accurate way to look at it because we have well-established findings on this as fact. A 2024 EMNLP study by Gekhman and colleagues looked at this exact thing. What did they find? They found that models learn new facts incredibly slowly during fine-tuning. But worse, once a model finally memorizes new facts, hallucinations on what it already knew rise roughly in line. Oh, wow. So it's overriding old data. Basically, yes. Every fact you train in can cost you an old one. Which completely destroys the idea of using fine-tuning as a company wiki. Exactly. And to back that up, we have another well-established fact based on a 2024 EMNLP study by Ovadia and colleagues alongside an ACM 2024 study by Soudani and colleagues. Right, I read those. Yeah. They proved definitively that retrieval beats fine-tuning for injecting knowledge. Now, I do want to explicitly caveat that Ovadia only tested unsupervised fine-tuning on 7 billion parameter class models. Right, so 7b class, not the massive frontier ones. Correct. But the principle holds. Plus, there is a fundamental advantage to RAG that has nothing to do with benchmarks. You can actually delete things. Yes. Retrieval gives you provenance. You can delete outdated documents or user data for compliance. Whereas weights, you cannot surgically delete a fact from weights. They do not allow deletion. Okay, so facts go in RAG, but what about training a model to actually use that evidence? That is where fine-tuning shines. Right, because there's a really interesting preliminary finding, a 2024 preprint by Zhang and colleagues on RAFT. We just spelt R-A-F-T like a life raft, meaning retrieval augmented fine-tuning. Exactly. They trained a model specifically to use retrieved evidence and to ignore distractors. And it actually beat plain retrieval on Llama-2-7B. It scored 73.3 versus 58.8 on the PubMed benchmark. That's a massive jump. Yeah, though I should explicitly caveat that this preprint has no confidence intervals published with it. Still, it proves the division of labor. Facts in RAG, behavior in fine-tuning. But wait, let me ask you this. Context windows are huge now. Why not just dump everything into a massive long context window instead of dealing with a complex RAG pipeline? That is a great question. And research suggests there is no universal winner yet. There is no controlled study on 2026 frontier models that definitively proves one beats the other in all cases. So what's the rule of thumb? The current rule of thumb is to use long context for small, dense corpora, like a single complex legal document, and use RAG for large, sparse ones, like a massive database of support tickets. Got it. That makes a lot of sense. So the listener implication decision rule here is put facts in retrieval and use fine-tuning to teach the model behavior and evidence use. Exactly. So that transitions us nicely to our next segment, LoRa and small-tuned models. If we are fine-tuning for behavior, how do we do it efficiently? Well, this is where LoRa comes in. Right, instead of full fine-tuning where you update everything. Yeah, and we have well-established facts from a 2024 TMLR study by Biderman and colleagues on this exact comparison. Okay, how did LoRa do? They found there is near parity between LoRa and full fine-tuning on standard instruction tuning, like just teaching it a chat format. Okay, but what about hard tasks? That's the catch. On large, continued pre-training, like feeding it 20 billion tokens of code, full fine-tuning won. It scored 26.3% versus 22.4% on human eval. Ah, so LoRa has limits. Exactly. As the researchers titled the paper, LoRa learns less and forgets less. Its rank is basically the dial controlling how much it can learn. Right, the rank bottleneck. But this nuance gets totally lost. I mean, do you remember that famous headline a couple of years ago? Oh, I know exactly which one you mean. The QLoRa one? QLoRa, spelled Q-L-O-R-A, which is QLoRa. It's basically LoRa on a lower precision compressed model. The headline was that it achieved 99.3% of chat GPT. Yeah, and that headline is the ultimate trap of the single benchmark. How so? We have to provide a massive caveat there. That headline was based on literally one benchmark, and it was judged by GPT-4. It learned to mimic the style of chat GPT, not the intelligence. Which is such a huge distinction. It is. And research suggests the same pattern repeats in vendor reports today. Like, you see platforms like LoRA Land and TinyAgent showing these massive gains, but they lack independent replication or confidence intervals. Right, so we really need to establish a habit for reading fine-tuning claims. Always look for the specific task and the specific judge. Always. But to be fair to the tuning community, research does suggest small-tuned models can beat large-prompted models on narrow tasks. Do we have data on that? We do. For instance, a 770 million parameter T5 model actually beat a 540 billion parameter PaLM few-shot model using just 80% of the labels. Wow, so a tiny fraction of the size, and it won. Yeah. But prompting versus tuning flips, depending on the specific task. Like, GPT-4 prompting won on human eval, but it actually lost on the MBPP benchmark to a tuned model. So it's totally task-dependent. Which brings us to the listener implication decision rule for this segment. Start at rank 16 on all linear layers and validate on your own task with a private eval before assuming a small-tuned model wins. I think that's incredibly sound advice. Okay, so let's move into the Wild West right now. Reinforcement learning with a checker. Ah, yes, the RL boom. Yeah, sparked entirely by DeepSeek R1. Specifically R10. Right. And for context, R10's pass at one on the AIME high school math competition. AIME is spelled A-I-M-E, and rhymes with game. Its score rose from 15.6 to 71.0. Which blew everyone's minds. But we need to contextualize this well-established fact. Okay, contextualize it for me. AIME only has 30 problems in that test set. Wait, really? Yes. Just 30? Yes. And the final R1 model that got released also utilized supervised fine-tuning stages, often called SFT, S.F.T, said as letters. It wasn't pure RL. That makes it a lot less magical. It does. And furthermore, a 2025 COLM study by Lambert and colleagues on the openly documented Tülu 3 model, Tülu is pronounced too-loo, the three showed the RL VR stage added only up to 3.3 points on GSM 8K at 8 billion parameters. Only 3.3 points. Yeah, which is far smaller than the headlines suggest. This is often using algorithms like GRPO, spelled G-R-P-O, for relative policy optimization. Okay, but, and I want to push back on this a bit, around the 17.30 mark of our discussion here. I argue that RL VR's pass at one gains represent real valuable capability. How so? Because users only ever see the first answer. If a user gets the right answer on the first try, getting it right the first time is a genuine discovery of capability to them. I hear what you're saying, but I have to push back in a good-natured way. It is sharpening, not discovery. Sharpening. Yes. Let's look at a 2025 NeurIPS study by Yue and colleagues on OmniMath. They showed Q25.5.7b's pass at one rose from 9.9 to 26.1. Which is a great jump. It is. But, it's pass at 256 actually slipped from 67.2 to 66.3. Wait, so when given 256 attempts, it actually got worse. Exactly. The base model could already solve those problems. The underlying capability was there. RL just made it find the answer faster. It sharpened the probability distribution. I see. And furthermore, distillation can actually push the frontier of capabilities, whereas RL on small models struggles to do that. Okay, but it still makes it way more reliable for the end user. It does. But, there is a huge risk. There is a preliminary finding right now, a spurious rewards result that hasn't been re-verified yet, where researchers found that random rewards still helped QN. Wait, just giving it completely random rewards made its benchmark scores go up? Yes. Which implies it's just hacking the formatting, not actually learning to reason better. Wow, okay. That is terrifying if you're spending thousands on compute. So we need a shared test to resolve this. I agree. Let's resolve on this. To know if you actually expanded capability, you must measure pass at one and pass at 16 or more. Pass at one, pass at K. That's a perfect resolution. So the listener implication decision rule is? Only start RL VR when you can name the checker and red team it in a one to two day pilot of at least 100 prompts. Yes, absolutely. All right, let's pivot to measuring honestly, because this entire conversation about gains relies on our evaluations actually working. Which as we know, they often don't. Right. Let's connect back to the hook. That 16.6 point inference engine swing, it shows how fragile these evals are. The pipeline bugs are the real killer. Yeah, let me share a preliminary finding on that. A Hugging Face blog found that chat template handling alone moves Zephyr-7B-Beta's IF eval scare by about 39 points. IF eval, spelled I-F-E-V-A-L, right? Yeah. The instruction following one. Yes, exactly. 39 points with tight error bars. Now, the caveat is that this was one model and one benchmark, but still. It's not an isolated thing. No? If you look at unverified practitioner opinion online, threads are full of mundane bugs, unmasked tokens, end of sequence leakage, tokenizer mismatches on new bases. So the lesson is that the most common failures are pipeline bugs, not algorithmic limits. 100%. We actually have well-established facts on prompt sensitivity too. Prompt format changes produced up to 76 points of spread on Llama-2-13B. 76 points? Yes. That was found by Sclar and colleagues at ICLR in 2024. Now, that is a maximum variance observed, not an average, but it's real. That is insane. And additionally, a TACL 2024 study by Mizrahi and colleagues looked at 6.5 million evaluations. They showed single prompt scores are completely unreliable. So you can't just run one test and call it a day? Nope. And we haven't even talked about seed variance. Oh, right, the random seed. It's a well-established fact. Random seeds alone cause wild swings in fine-tuning outcomes. So the evals are a mess. The seeds are a mess. What about using LLMs as judges? A 2023 NeurIPS study by Zheng and colleagues showed GPT-4 as a judge agreed with humans 85% of the time. That's pretty good, isn't it? It is, especially compared to the 81% agreement between humans themselves. But research suggests you must validate per task anyway. Because of contamination? Partially, yes. Research also suggests contamination, where test items leak into training data, runs anywhere from one to 45% across benchmarks. Okay, so this brings us to a major point of debate around the 23-minute mark of our deep dive. I want to defend a specific evaluation standard. All right, what is it? The 200-item paired comparison suite with a bootstrap confidence interval. Okay, define what you mean by that. Basically, running your base model and your fine-tune on 200 exact same items and resampling to get an uncertainty range. I argue this can detect roughly six to nine points of difference when the systems disagree on 10 to 20% of items. I have to push back there. Why? Because under Miller's less favorable one-generation assumptions, a 200-item suite detects only about 13 points of difference for the minimum detectable effect. Wait, 13 points? So it would miss a solid eight-point gain completely. Exactly. Yeah. Furthermore, real disagreement rates between models are unpublished. That's, yes. Meaning a fixed 200 items has absolutely no general sensitivity. You can't just blindly trust 200 items. Okay, so let's, okay, I see your point. If they only disagree on 5% of the items, 200 is way too small. Right. Yeah, yeah. So let's resolve on a shared action. Run a pilot to estimate the actual disagreement rate before sizing your evaluation suite. I completely agree to that. I like that, like that a lot. Awesome, awesome, awesome. Which leads right to us, a decision rule. So for our listeners, for listener implication decision rule, freeze an eval manifest, run a 20 to 100 item conformance check on your production stack, use at least three seeds on close calls and report paired confidence intervals, yeah? Freezing the eval manifest is so critical. Oh my gosh. Model hash, template, system prompt, all of it. I know. Yeah. Yep. Yep. All right. So let's get concrete. Okay, yeah. What do you, okay, with all this, what do you actually do with all this? Yeah, let's talk fine-tune versus cash break-even. I love that. Yeah, yeah. Outline the prompt cashing economics for us right now. So anthropic write, you're using anthropic you know, as just the benchmark. So anthropic writes cost 1.25, 1.25, yeah, 1.25 times, yeah, I'm sorry, 1.25 times input for a five-minute cash. Okay, so a slight premium to cash it. Right. But reads cost about a tenth of the normal price. Therefore, a five-minute cash pays for itself after just one single reuse. Which is incredible for things like long support manuals. Exactly. And we have to remember raw compute is actually cheap right now. What are the GPU prices looking like as ranges? H100s run roughly $3 to $4 an hour at Neo clouds with hyperscalers being materially higher. And H200s are roughly 35% above H100s. Okay, so compute is cheap, caching is cheap. Which leads to counterpoint three around our 28-minute mark. Let's hear it. I argue that fine-tuning rarely pays once cash prices and people costs are counted. If you factor in a realistic cycle with salaries and data curation, you'd need roughly 3.8 million calls a year to break even against a cashed prompt. And I argue fine-tuning hasn't died at all. It has just moved into distillation at high volume. Distillation is cheaper. Much cheaper. A cheap training cycle could break even at roughly 340,000 calls a year instead of 3.8 million. Are teams actually doing that though? Yes. If you look at unverified practitioner opinion, support agent anecdotes claim a distilled model beat cash prompting at high volume. One post even claimed roughly $18,000 a month saved by self-hosting. But conversely, other practitioners explicitly say they quit fine-tuning entirely when cash prices fell. Exactly. Which means we resolve on the core truth here. The answer swings about 100-fold depending on your exact cycle cost and hit rate. You cannot use a universal threshold. No, it's mathematically impossible. So the listener implication decision rule. Put your own call volume, cash hit rate, and full cycle cost, people included, into the break-even formula before you fund anything. Absolutely. All right, let's hit our final technical segment, surviving base upgrades. Because models upgrade constantly. They do, and it breaks everything. Exactly. Ordinary adapters often break across generations due to tokenizer or template mismatches, right? Usually, yes. But there was a fascinating preliminary finding from an EMNLP 2025 paper by Lin and colleagues. Oh, the Delta Transfer one? Yes. They took the LLAMA 3.0 post-training Delta, just the math difference from the tuning, and moved it directly onto LLAMA 3.1 8B. With no new training. Absolutely no new training. And the IFEVAL score raised from 36.4 to 83.3. That is almost 47 points of gain for free. Right, but we must caveat it. It is one study, and it only worked within one strictly related model family. You can't just slap a LLAMA Delta onto a Mistral base. Definitely not. So we introduced the ladder strategy for upgrades. Walk us through the ladder. Step one, evaluate the untouched new base model. It might just be good enough. Okay, step two. Try mounting the old LoRa, but only if the tokenizer template and architecture are identical. Step three. Try the Delta transfer we just talked about. Applying the weight difference to the closely related newer base. And step four, if all else fails. Regenerate your data, redistill, and rerun the full pipeline. And when you regenerate data, research suggests keeping real data in your mix is vital, right? It is. Shumailov and colleagues published in Nature 2024 that recursively replacing real data with synthetic data each generation causes model collapse. So it just turns into gibberish over time. Basically. But accumulating real data avoids it entirely. Start with 500 to 2000 curated examples for style. Got it. And what about licensing? Preliminary constraints apply here. Teacher model terms of service often heavily restrict distillation from their proprietary outputs. So you can't just scrape an API to train a competitor. Right. Have counsel read the terms before building a pipeline. Good advice. So the listener implication decision rule. Budget a full pipeline rerun for every major base generation and only test an old adapter if everything matches perfectly. Exactly. Well, this brings us to the outro. It's been a dense, deep dive. It has. And coming back to where we started, that hook. One change to the serving engine, nothing else, and up to 16.6 points of movement from a single study nobody has replicated yet. And the true lesson there isn't the number itself. No. The durable asset is not the number or the weights. It is a robust pipeline that can tell a real capability gain from a mere serving artifact. So let's synthesize the three decision rules. Rule one, put facts in retrieval and train only where you can name the behavior or the checker. Rule two, promote models strictly on paired comparisons. And rule three, run the breakeven on your own numbers before you fund anything. Perfect. And we should briefly acknowledge the gaps in all of this too. Yeah, the research does not cover everything yet. We don't know the exact behavior on late 2026 base models with hybrid architectures. We don't have a full validated total cost of ownership model yet. And there are massive unresolved questions around privacy laws and sector regulations. The landscape is always shifting. It really is. Well, that's a Yudame Research production. For the full briefing and every study cited today, visit yudame.ai. The link is in the description. Thanks for listening. And hey, if someone on your team is about to fund a fine tune or rent a GPU cluster, share this deep dive with them first and run the breakeven formula together. See you next time.