NES: what six fine-tuning rejections taught me about AI

I wanted an AI assistant that could carry useful context from one task to the next. My work moves between research, requirements, documents and code, and I wanted to understand how memory and local knowledge could make an assistant more useful across that work. That question became the starting point for my personal NES workspace.

In August 2026, I began adapting the existing NES framework with GenAI assistance. As an AI/ML engineer in training, I wanted to understand the layers between a question and a useful answer: the model, the information it receives, and the decisions around it. Working on my own fork gave me a place to explore those layers together and make changes I could examine.

From a personal assistant to a training question

The first challenge was bringing the pieces into a dependable workflow. A model could answer a question, a memory system could find material, and an agent could choose a tool, but I needed to understand how those actions connected. That led me into retrieval-augmented generation, or RAG: finding relevant passages and giving them to a model as context for an answer.

Working through RAG made a distinction clear to me. Adding documents to an index changes the information available to the assistant. Fine-tuning changes learned model behaviour. If I wanted better answers, I needed to know which part I was changing and how I would recognise an improvement. That became the foundation of the training experiments.

I used Qwen/Qwen3.5-2B as the base model and ran QLoRA training on a private Kaggle Tesla T4 GPU. QLoRA adapts a smaller set of model parameters while keeping the base model quantized, making the training more manageable. I kept a keyword-based retriever using BM25 fixed at five passages per question, so the experiments could focus on how the model used the same retrieved evidence: answering, citing sources and declining unsupported questions.

Before I could compare results, I had to get the cloud training path working. Two early attempts stopped because older training recipes no longer matched the installed library interfaces. One used a legacy Qwen2.5 recipe; the other reached Qwen3.5 model loading and baseline evaluation before stopping at a training configuration error. Fixing those failures taught me to check the actual cloud runtime as carefully as the code I was preparing locally.

Six completed experiments, six rejections

One unlikely result worth keeping in this story is the series of rejections: six completed fine-tuning experiments, versions 9 through 14, all rejected by the promotion gate. Each trained an adapter and reached evaluation. The gate compared it with the unchanged base model and required correctness, faithfulness to the supplied evidence, citations and other checks to pass together before the adapter could become active.

The first completed experiment, v9, looked encouraging on correctness: its score improved by 12.67 percentage points. But citation validity was only 67.5%. That exposed a problem I couldn't solve by pointing to the improved answer score. I needed the assistant to connect its answers reliably to the evidence, so the next experiment corrected generation stopping behaviour and increased examples of when to decline an unsupported answer.

That change brought v10's citation validity to 100%, while faithfulness fell below the base model. The answers could carry valid citations and still drift from what the supplied passages supported. In v11, I introduced more demonstrations that combined evidence from two sources. In v12, I held those examples constant and reduced the learning rate. Neither change satisfied all the requirements. I was beginning to see how one improvement could hide a different weakness.

For v13, I replaced a small part of the simpler synthetic training material with harder examples involving four sources. Faithfulness moved closer to the base model, but the results still fell short. That gave the next question a clearer shape: could I teach the model to select the relevant evidence when other plausible material was sitting beside it?

In v14, I kept the overall example count and training settings fixed, but reshaped 24 multi-source examples. Each prompt contained five passages: two relevant ones and three distractors from the same document. Citation validity reached 100%, faithfulness rose 0.91 percentage points above the base model, and the abstention F1 score, which measures decisions about when to decline an answer, reached 1.00 on the evaluation set. Yet correctness improved by only 1.17 percentage points, below the required five-point gain. The decision was still REJECT.

That last result sharpened my next question: would more complete training answers help close the correctness gap? My next planned experiment keeps the distractor structure and asks the training targets to combine several independently cited facts from each relevant passage. The result gave me a specific question to investigate instead of a reason to repeat training without a clear change.

What the results changed about my approach

Across those six experiments, I learned to look at the measures together. Correctness, faithfulness and citations describe different parts of a useful answer. Holding the retrieval setup and evaluation questions steady helped me compare the training changes, while changing one main variable in the later runs made the outcomes easier to interpret. Better-shaped examples often gave me more to learn from than simply adjusting the learning rate.

The rejections also changed what completing an experiment meant to me. Each run left an adapter, evaluation results and a clearer decision about what to try next. The promotion guard kept all six rejected adapters from replacing the active setup. Along the way, I built a command-line workflow for preparing a run, submitting it, following its progress, downloading the results, evaluating them and allowing activation only when the gate passes.

By September, the wider NES work was also taking shape in a personal control centre, bringing model, memory, project and agent status into view. I wanted the surrounding system to be understandable as well as the model's answers. Building the training workflow and bringing those controls together have helped me connect my interest in requirements with the engineering work of making behaviour observable and decisions explicit.

Where the next experiments lead

This is the work behind some of the late nights I described in my Cardtonic MacBook story. Those training experiments ran on Kaggle; the MacBook win came later and opens another direction for compatible local inference and adapter experiments through tools such as MLX-LM. I want to keep exploring which parts of the workflow make sense locally and what I can learn by comparing them.

NES has given my training a concrete place to happen. I began by wanting an assistant that could remember useful context, and ended up learning much more about how retrieval, training data and evaluation shape its answers. Six rejections have left me with a more disciplined way to investigate a problem and a clearer next experiment. That's progress I can build on.