Idea Ripper // ArchiveCards: 1555 [Live] | Updated: 2026-10-14
Card #c19IR // #c19Synthesis

A valid evaluation must freeze not only the price history but also the prompt, retrieval store, tools, memory, model version, and researcher workflow at each forecast origin.

— In our words · Source: Large Language Models in Finance (Miquel Noguer i Alonso)

[Use when]

Use when reviewing or designing a backtest of an LLM-based forecasting or trading system.

[Why it matters]

Backtests leak through channels most quants never freeze: the prompt, the retrieval index, the agent's memory, even the researcher's own iteration over prompt variants. Freezing prices alone leaves the impressive-but-invalid result intact.

Related principles

#c16Production systems should separate language generation from numerical computation. Let the LLM propose hypotheses, retrieve context, or…Use when an LLM-generated forecast, ratio, or expected return is about to enter a report, model, or trade.

More from Large Language Models in Finance

#c13Prediction and agency should not be conflated. A credit model can estimate the probability that a borrower breaches a covenant. An agentic…Use when a team proposes a stronger model as the fix for an agent that lacks memory, permissioning, or an… #c14The technically and institutionally mature objective is bounded autonomy : the agent can plan, retrieve, calculate, and propose actions…Use when deciding what an AI agent is permitted to do in production: which actions stay read-only, which need… #c15This principle prevents a common category error. A model that says it will respect a mandate has not yet respected the mandate. A model…Use when a vendor demo, benchmark score, or red-team chat log is being offered as proof that an agent is safe…