Prompt Grids for Multi-Model Structured Forecasting • nalanda

Purpose

Structured forecasting projects often cross several dimensions at once:

many input rows,
several prompt variants,
several models or routes, and
different numbers of repeated completions for different models.

Hand-written nested loops can run this design, but they also make it easy to lose provenance, miscount calls, overwrite raw output, or accidentally give a model more weight merely because it has more completions. This vignette shows the neutral prompt-grid orchestration provided by nalanda.

All executed code in this vignette is local and no-cost. Live model calls are shown with eval = FALSE.

run_prompt_grid() treats prompt variants as independent alternatives: each variant starts a fresh conversation. This differs from simulate_treatment(), where the elements of prompt are ordered turns in one conversation. Use simulate_treatment() when conversational memory is part of the treatment design; use a prompt grid when prompts are parallel variants that must not influence one another.

Inputs and configurations

Start with one row per forecast target. Any columns can be interpolated into a prompt template.

library(nalanda)

interventions <- tibble::tibble(
  condition_id = c("control", "cooperation", "perspective"),
  text = c(
    "A neutral informational message.",
    "A message emphasizing a shared cooperative goal.",
    "A message asking the reader to consider another person's perspective."
  )
)

prompts <- tibble::tibble(
  prompt_id = c("direct", "contextual"),
  prompt = c(
    "Forecast the effects of this intervention:\n{text}",
    paste(
      "Consider the likely study population and implementation context.",
      "Then forecast the effects of this intervention:\n{text}"
    )
  ),
  active = c(TRUE, TRUE)
)

models <- tibble::tibble(
  model_id = c("fast_a", "reasoning_a", "fast_b", "candidate_off"),
  model = c("model-a-fast", "model-a-reasoning", "model-b-fast", "model-c"),
  integration = c("route-a", "route-a", "route-b", "route-c"),
  family = c("developer-a", "developer-a", "developer-b", "developer-c"),
  temperature = c(0, 0, 0.3, 0),
  n_completions = c(2L, 1L, 3L, 5L),
  phase = c("first", "later", "later", "not_selected"),
  active = c(TRUE, TRUE, TRUE, FALSE)
)

model_id is a stable analysis label, while model and integration specify the backend route. family is an analysis grouping; it need not be identical to a provider name. Inactive rows remain in the configuration for provenance but are not planned or run.

Preview before spending

Use plan_prompt_grid() before constructing a response schema or contacting a backend.

smoke_plan <- plan_prompt_grid(
  data = interventions,
  prompt_variants = prompts,
  model_config = models,
  smoke_n = 1
)

smoke_plan[, c(
  "model_id", "family", "prompt_id", "n_rows",
  "n_completions", "estimated_calls"
)]
#> # A tibble: 6 × 6
#>   model_id    family      prompt_id  n_rows n_completions estimated_calls
#>   <chr>       <chr>       <chr>       <int>         <int>           <int>
#> 1 fast_a      developer-a direct          1             2               2
#> 2 fast_a      developer-a contextual      1             2               2
#> 3 reasoning_a developer-a direct          1             1               1
#> 4 reasoning_a developer-a contextual      1             1               1
#> 5 fast_b      developer-b direct          1             3               3
#> 6 fast_b      developer-b contextual      1             3               3
sum(smoke_plan$estimated_calls)
#> [1] 12

Here the smoke run uses one input row. The full plan is the same call with smoke_n = NULL.

full_plan <- plan_prompt_grid(
  data = interventions,
  prompt_variants = prompts,
  model_config = models
)

sum(full_plan$estimated_calls)
#> [1] 36

estimated_calls counts requested model responses: input rows multiplied by model-specific completions. A backend may submit those responses concurrently or in batches, but the response count is the useful quantity for budgeting. Calling run_prompt_grid(..., dry_run = TRUE) returns this same plan.

Define a structured forecast

The response type can contain any number of fields. For a 13-outcome benchmark, one concise construction is:

outcome_names <- sprintf("effect_%02d", 1:13)
response_type <- do.call(
  ellmer::type_object,
  stats::setNames(
    replicate(13, ellmer::type_number(), simplify = FALSE),
    outcome_names
  )
)

Use meaningful field names in a real study. The schema should describe the scale and direction of each forecast clearly enough that every model returns comparable numbers.

Smoke, phase the spend, then resume

Cost-gated studies can begin with one inexpensive model. A smoke test remains a deliberate fresh check on one input row and is kept separate from the full-data task identity:

first_model <- models |>
  dplyr::filter(model_id == "fast_a")

smoke <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = first_model,
  smoke_n = 1,
  output_dir = "results/forecast-smoke",
  resume = TRUE,
  on_error = "continue"
)

After inspecting smoke$results and smoke$errors, run that inexpensive model over the full inputs. Saving the returned list as one RDS file keeps results, the call plan, task statuses, and errors together as a compact run bundle.

phase_one <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = first_model,
  output_dir = "results/forecast-checkpoints",
  resume = TRUE,
  on_error = "continue"
)
saveRDS(phase_one, "results/forecast-phase-one-bundle.rds")

When the first phase is accepted, activate the later models. Existing results can be a results table, a previous run bundle, or an RDS/CSV path. The checkpoint collector is useful when the shared directory also contains successful models that are no longer active in the current configuration.

prior <- collect_prompt_grid_results(
  "results/forecast-checkpoints",
  existing_results = "results/forecast-phase-one-bundle.rds"
)

later_models <- models |>
  dplyr::filter(active)

later_plan <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = later_models,
  existing_results = prior,
  dry_run = TRUE
)

later_plan[, c(
  "model_id", "prompt_id", "configured_calls",
  "reused_calls", "pending_calls"
)]
sum(later_plan$pending_calls)

full <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = later_models,
  existing_results = prior,
  output_dir = "results/forecast-checkpoints",
  resume = TRUE,
  on_error = "continue"
)

raw_forecasts <- full$results
full$tasks
full$errors
saveRDS(full, "results/forecast-full-bundle.rds")

Each successful model-prompt-completion unit is written to its own RDS file before the workflow advances. The filename contains readable IDs and a hash of the inputs and settings. With resume = TRUE, a compatible file is loaded instead of rerun. Failed units are listed in errors and are deliberately not checkpointed, so a later run retries them.

Every raw row also carries task_hash and input_row. The hash covers the input table, prompt text, response specification, and effective model settings. Consequently, the same readable IDs with revised prompt text or a revised response schema remain pending rather than being silently reused. A complete prior task must contain every expected input_row before it suppresses calls.

Older unhashed results are rejected by default. For a reviewed one-time migration only, trust_legacy_results = TRUE verifies exact task IDs, inputs, and stored prompt text before assigning current hashes. The caller must independently verify the old model settings and response specification; save the resulting hashed bundle and return to the default afterward.

The combined raw table retains the input metadata and adds model_id, model, family, prompt_id, prompt_template, completion, temperature, seed, integration, and output_mode. Keep both this combined table and the per-completion files. Aggregation should never be the only saved artifact.

Simulations or repeated expert forecasts?

The mechanics are identical, but the interpretation is not.

Repeated completions are simulations when the prompt and design treat each draw as a synthetic respondent or stochastic realization from a target population. The repetition count then describes simulated sample size, and the prompt should define the simulated unit.

Repeated completions are repeated expert forecasts when the model is being used as a forecaster and repeated sampling measures its within-model variability. Those draws are not independent experts. Calling them separate experts would overstate the number and diversity of information sources.

Temperature zero does not guarantee identical output across every hosted backend, and a nonzero temperature does not by itself turn completions into valid population simulations. That claim depends on the research design.

Aggregate without accidental row-count weights

The raw mean is usually wrong when models have different completion counts or belong to families with different numbers of models. The helper below returns every stage rather than hiding a final scientific choice.

This small local table mimics one outcome from a completed run:

mock_raw <- tibble::tibble(
  condition_id = "cooperation",
  family = c("developer-a", "developer-a", "developer-a", "developer-a", "developer-b"),
  model_id = c("fast_a", "fast_a", "fast_a", "reasoning_a", "fast_b"),
  prompt_id = c("direct", "direct", "contextual", "direct", "direct"),
  completion = c(1L, 2L, 1L, 1L, 1L),
  effect_support = c(0, 2, 3, 6, 10),
  effect_trust = c(2, 4, 5, 7, 9)
)

median_aggregated <- aggregate_model_forecasts(
  mock_raw,
  outcomes = c("effect_support", "effect_trust"),
  unit_by = "condition_id",
  family_col = "family",
  method = "median"
)

mean_aggregated <- aggregate_model_forecasts(
  mock_raw,
  outcomes = c("effect_support", "effect_trust"),
  unit_by = "condition_id",
  family_col = "family",
  method = "mean"
)

median_aggregated$prompt
#> # A tibble: 4 × 7
#>   condition_id family      model_id    prompt_id  effect_support effect_trust
#>   <chr>        <chr>       <chr>       <chr>               <dbl>        <dbl>
#> 1 cooperation  developer-a fast_a      contextual              3            5
#> 2 cooperation  developer-a fast_a      direct                  1            3
#> 3 cooperation  developer-a reasoning_a direct                  6            7
#> 4 cooperation  developer-b fast_b      direct                 10            9
#> # ℹ 1 more variable: n_completions <int>
median_aggregated$model
#> # A tibble: 3 × 6
#>   condition_id family      model_id    effect_support effect_trust n_prompts
#>   <chr>        <chr>       <chr>                <dbl>        <dbl>     <int>
#> 1 cooperation  developer-a fast_a                   2            4         2
#> 2 cooperation  developer-a reasoning_a              6            7         1
#> 3 cooperation  developer-b fast_b                  10            9         1
median_aggregated$family
#> # A tibble: 2 × 5
#>   condition_id family      effect_support effect_trust n_models
#>   <chr>        <chr>                <dbl>        <dbl>    <int>
#> 1 cooperation  developer-a              4          5.5        2
#> 2 cooperation  developer-b             10          9          1
median_aggregated$consensus
#> # A tibble: 1 × 4
#>   condition_id effect_support effect_trust n_families
#>   <chr>                 <dbl>        <dbl>      <int>
#> 1 cooperation               7         7.25          2
mean_aggregated$consensus
#> # A tibble: 1 × 4
#>   condition_id effect_support effect_trust n_families
#>   <chr>                 <dbl>        <dbl>      <int>
#> 1 cooperation               7         7.25          2

The hierarchy is explicit:

completions receive equal weight within a prompt,
prompt estimates receive equal weight within a model,
model estimates receive equal weight within a family, and
family estimates receive equal weight in the consensus.

Set family_col = NULL if the intended estimand gives every model equal weight directly. The count columns (n_completions, n_prompts, n_models, and n_families) make each stage auditable. A raw mean would instead give extra weight to whichever model produced more rows.

method = "mean" is the backward-compatible default. method = "median" uses medians at all four stages, which can make the primary estimate less sensitive to an extreme completion while retaining the same equal-weight hierarchy. Missing values are omitted separately for each outcome at each stage; an all-missing group remains missing. Counts describe all configured contributors, including contributors missing a particular outcome.

Diagnose and compare forecasts without project-specific wrangling

tidy_forecast_aggregation() makes every stage available in one long table. It works with one or many numeric outcomes, and does not assume names for a study’s conditions or outcomes.

forecast_long <- tidy_forecast_aggregation(
  median_aggregated,
  unit_by = "condition_id"
)

model_forecasts <- forecast_long |>
  dplyr::filter(aggregation_level == "model")

# Disagreement among models for each condition and outcome
summarize_forecast_disagreement(
  model_forecasts,
  unit_by = c("condition_id", "outcome"),
  estimate_col = "estimate",
  contributor_col = "model_id"
)
#> # A tibble: 2 × 12
#>   condition_id outcome  n_contributors n_nonmissing n_missing  mean median    sd
#>   <chr>        <chr>             <int>        <int>     <int> <dbl>  <dbl> <dbl>
#> 1 cooperation  effect_…              3            3         0  6         6  4   
#> 2 cooperation  effect_…              3            3         0  6.67      7  2.52
#> # ℹ 4 more variables: mad <dbl>, min <dbl>, max <dbl>, range <dbl>

# The existing agreement helpers consume this model-stage shape directly.
effect_models <- model_forecasts |>
  dplyr::filter(outcome == "effect_support")
model_pairwise_cor(
  effect_models,
  outcome = "estimate",
  unit_by = "condition_id",
  model_col = "model_id"
)
#> # A tibble: 6 × 5
#>   model_a     model_b     method   correlation n_units
#>   <chr>       <chr>       <chr>          <dbl>   <int>
#> 1 fast_a      reasoning_a pearson           NA       1
#> 2 fast_a      reasoning_a spearman          NA       1
#> 3 fast_a      fast_b      pearson           NA       1
#> 4 fast_a      fast_b      spearman          NA       1
#> 5 reasoning_a fast_b      pearson           NA       1
#> 6 reasoning_a fast_b      spearman          NA       1

# Mean--median comparison at the consensus stage. A lookup is optional and
# lets a project normalize differences by its own outcome-scale widths.
widths <- tibble::tibble(
  condition_id = "cooperation",
  outcome = c("effect_support", "effect_trust"),
  scale_width = c(10, 10)
)
compare_forecast_aggregations(
  median_aggregated, mean_aggregated,
  stage = "consensus",
  unit_by = "condition_id",
  outcomes = c("effect_support", "effect_trust"),
  labels = c("median", "mean"),
  scale_width = widths
)
#> # A tibble: 2 × 9
#>   condition_id outcome   median  mean difference absolute_difference scale_width
#>   <chr>        <chr>      <dbl> <dbl>      <dbl>               <dbl>       <dbl>
#> 1 cooperation  effect_s…   7     7             0                   0          10
#> 2 cooperation  effect_t…   7.25  7.25          0                   0          10
#> # ℹ 2 more variables: normalized_difference <dbl>,
#> #   normalized_absolute_difference <dbl>

The comparison helper refuses to silently drop unmatched cells or resolve duplicate identities. summarize_forecast_disagreement() is deliberately not limited to models: use contributor_col = "prompt_id" and include model_id in unit_by to describe prompt disagreement within a model and target.

What remains downstream

nalanda handles configuration validation, expansion, pending-call budgeting, strong-identity reuse, checkpoint collection, provenance, resume, and mean/median hierarchical aggregation. The following choices stay in downstream analysis because they depend on the study:

which models and prompt variants belong in the estimand,
whether a family groups developers, model lineages, or another dependence structure,
calibration, transformations, bounds, and missing-data rules for the 13 outcomes,
uncertainty intervals and dependence-aware inference,
robustness and sensitivity analyses across prompts, models, or families, and
whether repeated completions can be interpreted as simulations at all.

That separation keeps the package API reusable without silently turning one benchmark’s scientific assumptions into defaults for every project.