DSPy: Programming the Objective, Not the Prompt

I used to think of a prompt as the program I was giving a language model.

The prompt contained the task, examples, formatting rules, edge cases, and often a growing collection of warnings accumulated after things went wrong. If the model failed, I edited the string. If a new model behaved differently, I edited it again. When the prompt became difficult to understand, I added headings and more instructions.

This works surprisingly well—until the prompt becomes part of a system.

A real language-model application may retrieve documents, call tools, classify an intent, generate an answer, validate that answer, and decide whether to try again. Each step has its own inputs and outputs. Each can fail independently. One long prompt hides that structure inside prose.

DSPy begins with a different premise: the program should describe what the system does, while prompts should be parameters that can be learned.

That idea is more important than any individual DSPy class. It changes the question from “What words should I put in this prompt?” to “What is the task, how is it composed, and how will I know whether the result is better?”

This is the first article in a three-part series: DSPy, SIMBA, and GEPA. Here I will develop the DSPy mental model. The next two articles explain optimizers that can improve a DSPy program without changing its model weights.

The prompt-string trap

Imagine a system that routes support requests:

Classify this support ticket as billing, technical, account, or other.
Return only the category. Remember that password-reset problems are account
problems, unless the reset email is not arriving, in which case...

The first line describes a durable task. The rest increasingly describes one engineer’s current theory of how a particular model must be coaxed into performing it.

Those concerns change at different speeds:

• The categories may remain stable for years.
• The model may change next month.
• The best demonstrations may change as customer behavior changes.
• The reasoning strategy may change after an experiment.
• Formatting should often be enforced structurally rather than explained repeatedly.

Putting all of them in one string makes them hard to distinguish later. Nobody knows whether a particular sentence expresses a product requirement or compensates for a quirk in an older model.

Omar Khattab, DSPy’s creator, connects this problem to Rich Sutton’s Bitter Lesson: methods that scale through learning and search have repeatedly displaced techniques built from extensive human knowledge. Khattab’s engineering interpretation is not that domain knowledge is useless. It is that we should avoid prematurely encoding our incomplete knowledge at the lowest possible level. His short talk, “On Engineering AI Systems that Endure the Bitter Lesson”, is the best conceptual introduction I have found to this philosophy.

A useful analogy—and its limit

In ordinary machine learning, we do not manually choose every weight in a neural network. We define an architecture, collect data, specify an objective, and run an optimizer.

DSPy borrows that separation for language-model programs:

The analogy is valuable, but it should not be taken literally. DSPy compilation is not the same process as compiling Python into machine code, and most prompt optimizers are not differentiating through a neural network. “Compile” means using data and an evaluation function to produce a better configured version of the program.

The original DSPy paper describes LM pipelines as computational graphs in which language models are invoked through declarative modules. Those modules are parameterized, allowing a compiler to improve instructions, demonstrations, or other behavior against an application-level metric.¹

The four pieces of the mental model

Most DSPy applications become much easier to understand when separated into four pieces.

A signature declares the contract

A signature names the inputs, outputs, and meaning of one language-model operation.

import dspy


class RouteTicket(dspy.Signature):
    """Route a support ticket to billing, technical, account, or other."""

    ticket: str = dspy.InputField()
    route: str = dspy.OutputField()

This is not the final prompt. It is closer to an interface.

The signature says that a ticket goes in and a route comes out. Its docstring supplies task meaning that cannot be expressed by the field types alone. An adapter and model-specific machinery can turn that contract into the messages required by a provider.

The important property is what the signature omits. It does not decide whether to use chain-of-thought, which demonstrations to include, or what model-specific wording produces the best result.

A module chooses the strategy

A module gives the signature an execution strategy:

class SupportRouter(dspy.Module):
    def __init__(self):
        self.route = dspy.Predict(RouteTicket)

    def forward(self, ticket: str):
        return self.route(ticket=ticket)

Replacing dspy.Predict with dspy.ChainOfThought changes how the model is asked to solve the task without changing the task contract itself.

A larger program could retrieve customer history before routing, call a policy tool, or send uncertain cases to a second module. Ordinary Python still controls composition, permissions, retries, and information flow.

This division is essential. A prompt asking a model not to reveal private information is not equivalent to code that never gives the model access to that information.

Examples provide experience

DSPy represents data as examples with explicit input fields:

trainset = [
    dspy.Example(
        ticket="I was charged twice for the same invoice.",
        route="billing",
    ).with_inputs("ticket"),
    dspy.Example(
        ticket="I cannot sign in after changing my password.",
        route="account",
    ).with_inputs("ticket"),
]

Examples may supply labels, demonstrations, optimizer inputs, or held-out evaluation cases. Their role depends on the optimizer and metric.

The durable asset is not merely the number of examples. It is their coverage. Twenty nearly identical billing tickets teach less than a smaller set that exposes ambiguous boundaries among billing, account, and technical problems.

A metric defines improvement

An optimizer cannot pursue “better” in the abstract. It needs an executable definition:

def routing_accuracy(example, prediction, trace=None):
    return prediction.route.strip().lower() == example.route

This metric is intentionally simple. It also reveals an important limitation: exact category accuracy does not measure everything we may care about.

It does not tell us whether:

• The output contains a valid category.
• A dangerous account-takeover report was incorrectly treated as ordinary support.
• One category has much worse recall than the others.
• The program became slower or more expensive.
• The router is uncertain and should escalate.

The optimizer will discover whatever improves the metric, including shortcuts we did not intend. Evaluation is therefore not plumbing around the AI system. It is part of the system’s specification.

Context is a pipeline, not a string

The most transferable lesson from the DSPy and context-engineering course by Avishek Biswas is that context should be understood as a pipeline.

Every component selects or creates information for another component:

ticket
  ↓
customer-history retrieval
  ↓
intent and risk classification
  ↓
policy/tool selection
  ↓
response generation
  ↓
validation

Instructions are one source of context. Retrieved evidence, tool results, demonstrations, intermediate predictions, and memory are others. Once these transformations have explicit interfaces, failures become easier to localize. We can ask whether retrieval failed, the router chose the wrong path, a tool returned bad data, or the final generator ignored good evidence.

DSPy does not remove the need for this architecture. It gives the language-model parts of the architecture a common programming and optimization model.

What compilation changes

Suppose we have a router, a metric, and a representative training set. A DSPy optimizer follows the same broad interface:

optimizer = SomeDSPyOptimizer(metric=routing_accuracy)

compiled_router = optimizer.compile(
    SupportRouter(),
    trainset=trainset,
)

Depending on the optimizer, compilation may:

• Add successful traces as few-shot demonstrations.
• Propose new instructions.
• Search combinations of instructions and examples.
• Reflect on failures and create general rules.
• Fine-tune model weights.

The source program remains recognizable. The compiled program contains learned parameters—perhaps demonstrations or evolved instructions—that performed better under the evaluation process.

This is why it is useful to save the uncompiled program separately from the compiled artifact. The former expresses the design; the latter is the result of a particular model, dataset, metric, optimizer, and budget.

DSPy is not magic prompt removal

Several misunderstandings become possible at this point.

First, DSPy does not make prompts disappear. Models still receive prompts. DSPy makes those prompts generated, structured, inspectable, and potentially optimizable rather than forcing application developers to maintain every string manually.

Second, DSPy does not discover the product objective. Someone still has to decide what success means, create representative cases, and encode hard constraints in code.

Third, compilation does not prove generalization. If optimization and reporting use the same examples, the program may simply have learned the test. A serious workflow preserves an untouched test set.

Finally, a more complex DSPy program is not automatically better. Every additional module adds latency, cost, and another place for information to degrade. Decomposition should make responsibilities clearer or improve measured behavior.

When DSPy is a good fit

DSPy is particularly compelling when:

• The application contains several interacting LM calls.
• You expect to change models or providers.
• You already have—or are willing to build—an evaluation set.
• Prompt behavior affects a measurable product outcome.
• You want to compare reasoning strategies systematically.
• Repeated manual prompt editing has become difficult to reproduce.

It may be unnecessary when one stable prompt already solves a low-risk task, there is no meaningful way to evaluate outputs, or the primary constraints are deterministic and should simply be implemented in code.

The practical threshold is not “Is my prompt long?” It is “Do I have a language-model program whose behavior I need to improve repeatedly?”

The foundation before optimization

Before choosing an optimizer, I would require five artifacts:

1. A small, readable DSPy program.
2. A frozen baseline.
3. Separate development and final-test examples.
4. A metric aligned with the actual failure costs.
5. Logs containing inputs, outputs, settings, and scores.

Without them, automatic prompt optimization merely automates guessing.

With them, the prompt becomes one replaceable implementation of a stable objective.

The next article examines SIMBA, an optimizer that looks for examples where a program’s outcomes vary and learns either a reusable rule or a successful demonstration from the contrast.

Learning resources

• On Engineering AI Systems that Endure the Bitter Lesson — Omar Khattab’s 19-minute explanation of the engineering philosophy behind DSPy.
• DSPy: Build and Optimize Agentic Apps — a compact course taught by DSPy co-lead maintainer Chen Qian.
• DSPy + Context Engineering — a longer, hands-on course covering signatures, composition, evaluation, tools, and retrieval.
• Programming—not prompting—language models — the official guided path.
• DSPy documentation — the source of truth for current APIs.

References

1. Omar Khattab et al. “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.” International Conference on Learning Representations, 2024.
2. Rich Sutton. “The Bitter Lesson.” 2019.
3. DSPy contributors. “Optimizers: choosing one.” DSPy documentation.