SIMBA: Learning Rules from Better and Worse Attempts

This is the second article in a three-part series: DSPy, SIMBA, and GEPA.

Consider two attempts to solve the same problem.

In the first, a model notices that a customer changed their password but misses the sentence saying the reset email never arrived. It routes the case to account support. In the second, the model notices the delivery failure and routes it to the technical team.

A score can tell us that the second attempt was better. The contrast can teach us something more useful:

When an account action succeeds but its confirmation or reset message is not delivered, route by the delivery failure rather than the original account action.

That sentence is not a correction for one ticket. It is a reusable rule inferred from the difference between a better and worse trajectory.

This is the intuition behind SIMBA, which stands for Stochastic Introspective Mini-Batch Ascent. SIMBA is a DSPy prompt optimizer that incrementally improves a program by adding either successful demonstrations or rules derived from contrasting its behavior.¹

The name sounds more complicated than the central idea:

try the program several ways
            ↓
find examples where outcomes differ
            ↓
compare a better trace with a worse trace
            ↓
learn a rule—or retain a good demonstration
            ↓
test whether the changed program improves

SIMBA matters because it sits between simple few-shot bootstrapping and the richer feedback-driven evolution used by GEPA. It shows how an LM’s ability to reason about examples can itself become part of the optimization algorithm.

What each word means

The full name is a compact description of the method.

Stochastic

SIMBA maintains multiple candidate programs. It does not always choose the current leader. It samples candidates with probabilities influenced by their scores, preserving some exploration while spending more time around promising candidates.

It also generates varied trajectories by running sampled candidates with different model configurations. The variation is useful: without different outcomes, there is little contrast from which to learn.

Introspective

SIMBA can ask an LM to examine a better and worse execution trace and propose advice for the program’s individual modules.

“Introspection” should not be confused with ground truth. The model is not looking inside its weights and discovering the true reason for a failure. It is reading observable trajectories and constructing a plausible, potentially useful explanation.

That explanation earns its place only if the resulting candidate performs better.

Mini-batch

Running every candidate across the entire dataset after every change would be expensive. SIMBA performs most of its search on small batches, then evaluates selected winners more broadly at the end.

Mini-batches make optimization affordable, but their scores are noisy. A candidate can win because one batch happens to favor its rule. This is why final evaluation must be separate from the search.

Ascent

SIMBA seeks incremental improvements. Each round proposes changed programs, evaluates them, and adds the best candidate from that round to a list of winners. It is climbing a measured objective without differentiating through the language model.

The landscape is irregular: prompts are discrete text, model outputs are stochastic, and the metric may be imperfect. “Ascent” describes the intention, not a guarantee of smooth progress.

The two ways SIMBA learns

SIMBA creates candidates through two complementary strategies.

Strategy 1: append a successful demonstration

Suppose a particular trajectory handled an ambiguous ticket correctly. SIMBA can add that successful trace to a predictor’s demonstrations.

The next time the model sees a similar problem, the example shows the desired mapping. This is experience stored as an episode:

Here is a case like this, and here is how it was handled.

Demonstrations are concrete. Their weakness is that they may not communicate which detail was decisive. A model can imitate the wrong part of an example.

Strategy 2: append an inferred rule

SIMBA can instead contrast stronger and weaker trajectories and ask a prompt model for module-specific advice.

The result may be a rule such as:

Distinguish inability to authenticate from failure to receive a message.
The former is an account issue; the latter is a technical delivery issue.

Rules are compressed experience. They can generalize beyond the original example, but they can also overgeneralize. A clever-sounding rule is only a hypothesis until it succeeds on other cases.

SIMBA can use both forms of memory: demonstrations preserve what happened; rules attempt to explain what should happen next time.

How the algorithm works

The current DSPy implementation follows this broad loop:²

1. Sample a mini-batch from the training set.
2. Select candidate programs from the global pool using score-weighted sampling.
3. Run several trajectories for every example in the batch.
4. Group trajectories by example and identify examples with the largest performance variation.
5. Choose either rule generation or demonstration insertion.
6. Build new candidate programs from the selected examples and traces.
7. Evaluate those candidates on the mini-batch.
8. Add the round’s best candidate to the winners and retain new candidates in the pool.
9. Repeat for the configured number of steps.
10. Evaluate a selection of winners on the full training set and return the highest-scoring program.

The fourth step is especially interesting. SIMBA does not treat all examples equally. It gives attention to cases where different attempts produce meaningfully different scores.

Why variation? If every attempt fails identically, the model may lack the capability or information required to solve the case. If every attempt succeeds, there is little to repair. When some trajectories succeed and others fail, the program is near a decision boundary: evidence exists for behavior that works, and the contrast may reveal it.

A small DSPy example

The previous article introduced a support router:

import dspy


class RouteTicket(dspy.Signature):
    """Route a support ticket to billing, technical, account, or other."""

    ticket: str = dspy.InputField()
    route: str = dspy.OutputField()


class SupportRouter(dspy.Module):
    def __init__(self):
        self.route = dspy.Predict(RouteTicket)

    def forward(self, ticket: str):
        return self.route(ticket=ticket)

SIMBA expects a metric that returns a numeric score:

def routing_score(example, prediction, trace=None):
    predicted = prediction.route.strip().lower()
    expected = example.route.strip().lower()
    return float(predicted == expected)

With a configured language model and a sufficiently large training set, an illustrative optimization run looks like this:

simba = dspy.SIMBA(
    metric=routing_score,
    bsize=8,
    num_candidates=4,
    max_steps=6,
    max_demos=4,
)

optimized_router = simba.compile(
    SupportRouter(),
    trainset=trainset,
    seed=42,
)

The exact settings are not universal recommendations. Batch size, candidate count, number of steps, model quality, and dataset size change both cost and behavior. Check the current SIMBA documentation before running the example.

There is also an implementation detail worth noticing: SIMBA’s compile interface accepts a training set but not a separate validation set. Its final candidate selection evaluates winners on that full training set. You should therefore maintain an external test set that SIMBA never sees and evaluate the returned program separately.

evaluator = dspy.Evaluate(
    devset=testset,
    metric=routing_score,
    display_progress=True,
)

baseline_result = evaluator(SupportRouter())
optimized_result = evaluator(optimized_router)

If the same examples choose the prompt and certify its quality, the result is not a trustworthy estimate of generalization.

What SIMBA is really optimizing

A DSPy program may look like ordinary Python, but its predictors carry parameters such as instructions and demonstrations. SIMBA creates new program candidates by changing those parameters.

It does not change the underlying model weights. It is closer to building an increasingly useful field guide for a fixed model:

• “Here is a successful example.”
• “Here is a distinction you missed.”
• “Here is a rule inferred from two different outcomes.”

This makes SIMBA accessible when the model is available only through an API. It also makes the result inspectable. We can read the rules and demonstrations that were added.

Inspectability is not the same as correctness. An optimizer can discover brittle rules, encode dataset artifacts, or make the prompt much longer. The learned text should be reviewed like generated code.

The metric is still the teacher

SIMBA’s introspection is guided by the ordering supplied by the metric. If the metric says one trajectory is better, the reflection step searches for a lesson in that contrast.

A misleading metric therefore creates misleading lessons.

For the routing example, aggregate accuracy could hide a dangerous pattern. Suppose 90 percent of tickets are ordinary billing questions. A program can score well while consistently mishandling account-takeover reports.

A better evaluation would include:

• Per-category precision and recall.
• A high penalty for missing security-sensitive cases.
• Structural validation of the output category.
• Slices for ambiguous or multi-intent tickets.
• Latency and token cost alongside quality.

Mahmoud Mabrouk’s workshop, “Judge the Judge”, demonstrates this problem vividly. An evaluator’s overall accuracy improved modestly after optimization, while recall on the important non-compliant class improved far more. The example shows why one aggregate score can conceal the outcome that actually matters.

When SIMBA is a good choice

SIMBA is worth trying when:

• A DSPy program already has a reliable numeric metric.
• Some examples produce mixed success across repeated attempts.
• Successful traces would make useful demonstrations.
• Contrastive analysis can produce meaningful rules.
• A capable prompt model is available for introspection.
• You want an optimizer whose learned rules and examples are inspectable.

Start with something simpler when a few labeled demonstrations already solve the problem. BootstrapFewShot can be a valuable baseline because it tests whether good examples are sufficient before paying for introspection.

SIMBA is also less attractive when failures arise from missing tools, absent context, strict deterministic constraints, or capabilities the task model simply does not possess. No amount of reflective wording can retrieve information the program never receives.

SIMBA and the path to GEPA

SIMBA introduced several ideas that make GEPA easier to understand:

• Maintain multiple candidate programs.
• Learn from complete execution traces.
• Use an LM to turn experience into textual improvements.
• Search on mini-batches to control cost.
• Preserve both concrete demonstrations and abstract rules.

GEPA expands this direction. Instead of relying mainly on numeric variation and contrast between trajectories, it accepts task-specific natural-language feedback from the evaluator. It maintains candidates on a Pareto frontier across validation examples and can merge complementary changes in compound programs.

SIMBA asks, “What rule can I infer from a better and worse attempt?”

GEPA asks, “Given the trajectory and an explanation of its failure, how should this program evolve?”

That is the subject of the final article.

Learning resources

• Official SIMBA documentation — the concise API reference.
• SIMBA implementation — the primary source for the current algorithm.
• DSPy SIMBA explained — Marius Vach’s readable source-level walkthrough.
• DSPy optimizer guide — places SIMBA among the other DSPy optimizers.
• Judge the Judge — a practical lesson in why evaluator design matters.

References

1. DSPy contributors. “dspy.SIMBA.” DSPy documentation.
2. DSPy contributors. “SIMBA implementation.” GitHub.
3. Marius Vach. “DSPy SIMBA explained.” August 5, 2025.
4. Omar Khattab et al. “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.” International Conference on Learning Representations, 2024.