Takes each model's continuous scores (1–100 scale) and derives rankings from them, then evaluates cross-model concordance via Kendall's W. The rankings are computed by the researcher from the raw scores — models themselves only produce continuous ratings, not ordinal ranks. Useful for answering "Do models rank books the same way?"
Usage
model_rank_consistency(
data,
outcome = "mean_outcome",
unit_by = c("book_id", "chapter_id"),
rank_within = NULL,
model_col = "model"
)Arguments
- data
A data frame with one row per model-by-unit combination.
- outcome
Character string naming the score column (default
"mean_outcome").- unit_by
Character vector of columns that jointly identify a unit (default
c("book_id", "chapter_id", "group")).- rank_within
Optional character vector of columns that define separate ranking contexts (e.g.,
"group"). Items are ranked independently within each combination of these columns.- model_col
Character string naming the model column (default
"model").
Value
A list with two elements:
- ranks
Tibble with the unit columns,
model,score, andrank.- concordance
Tibble with Kendall's W and associated statistics, one row per
rank_withincombination (or one row total).
Details
unit_by must identify exactly one row per model within each ranking context.
If the data still contain lower-level rows (for example, chapters) and you
want book-level ranks, aggregate those rows to the book level before calling
this function.
Units with missing scores for one or more models are excluded from the
concordance calculation. The reported n_items is the number of complete
items used.
Examples
if (FALSE) { # \dontrun{
rc <- model_rank_consistency(agg, outcome = "mean_rating",
unit_by = c("book_id", "chapter_id"),
rank_within = "group")
rc$concordance
rc$ranks
agg_book <- agg |>
dplyr::group_by(model, book_id, group) |>
dplyr::summarise(mean_rating = mean(mean_rating), .groups = "drop")
rc_book <- model_rank_consistency(agg_book, outcome = "mean_rating",
unit_by = "book_id",
rank_within = "group")
} # }