4  Organizing Choice Data

This chapter contains no new statistics. It is about something more mundane and, in my experience, responsible for more student-hours of frustration than any mathematical topic in this book: getting choice data into a shape that estimation code can consume. The likelihoods of Part II are sums over choice situations of expressions involving each alternative’s attributes; between the data as they arrive and the likelihood as it computes sits a layer of pure bookkeeping. Do the bookkeeping haphazardly and every later chapter fights it. Do it once, deliberately, and the same few structures will carry us from the plain MNL (Chapter 8) through simulated likelihood (Chapter 13) to the hierarchical samplers (Chapter 17) without modification.

Because this is a chapter of choices-with-tradeoffs rather than theorems, I will follow a pattern used throughout the book: lay out the reasonable options, then commit to one and say why. You may prefer different choices in your own work — the options all work — but a book, like a codebase, benefits from picking a lane.

4.1 The Shape of Choice Data

The data of a regression course are flat: one row per observation, one column per variable. Choice data have more structure, because each observation is not a number but an event with internal anatomy: a particular person, facing a particular set of alternatives, on a particular occasion, choosing one. Three indices track that anatomy, and they have been fixed since Chapter 3:

• \(n = 1, \ldots, N\) indexes decision-makers (our survey respondents),
• \(t = 1, \ldots, T\) indexes choice situations (the tasks each respondent completes),
• \(j = 1, \ldots, J\) indexes alternatives within a choice situation.

The pair \((n, t)\) identifies one choice situation; the triple \((n, t, j)\) identifies one alternative in one situation — one utility in the model’s terms, since Chapter 7 will assign \(U_{ntj} = \mathbf{x}_{ntj}'\boldsymbol{\beta}+ \varepsilon_{ntj}\) to exactly this triple. Our laptop study (Section 1.4) has \(N = 500\), \(T = 8\), and \(J = 3\): five hundred respondents, eight tasks each, three laptop profiles per task, for \(500 \times 8 = 4{,}000\) observed choices and \(4{,}000 \times 3 = 12{,}000\) described alternatives.

The binary data of Chapter 3 were the degenerate case that hid all of this: one task per person (\(T=1\)) and one modeled alternative plus a fixed outside option (\(J=2\) with the “don’t buy” utility normalized away), so a flat data frame sufficed. It no longer does. The question of this chapter is: in what shape do we store 12,000 alternative descriptions and 4,000 choices?

4.2 Wide and Long Formats

Two conventions dominate, distinguished by what a row represents.

In wide format, one row is one choice situation \((n,t)\), and each alternative’s attributes spread across columns:

  id task brand_1 price_1 brand_2 price_2 brand_3 price_3 choice
1  1    1    Acer     1.2   Apple     2.0    Dell     1.4      2
2  1    2    Dell     1.6    Acer     0.9   Apple     1.8      1

In long format, one row is one alternative \((n,t,j)\), with the choice recorded as an indicator:

  id task alt brand price choice
1  1    1   1  Acer   1.2      0
2  1    1   2 Apple   2.0      1
3  1    1   3  Dell   1.4      0
4  1    2   1  Dell   1.6      1
5  1    2   2  Acer   0.9      0
6  1    2   3 Apple   1.8      0

(These two tables show the same two tasks; respondent 1 chose the Apple in task 1 and the Dell in task 2.)

The tradeoffs:

• Wide matches how survey platforms export data and how humans read a choice task (“here were your three options; you picked #2”). It is compact — one row per choice. But the attributes of “alternative 2” live in differently-named columns from “alternative 1,” so any computation over alternatives means wrangling column-name patterns; and if some tasks show two alternatives while others show four, the format needs padding columns and missing-value conventions.
• Long matches how the model thinks. Each row is one \((n,t,j)\) triple — one utility. Attributes have a single column each regardless of \(J\); tasks with different numbers of alternatives are just groups of different sizes; and the operations our likelihoods need (compute a value for every alternative, then aggregate within choice situations) become operations on rows and groups.

Our choice: long format, always. The deciding argument is the last one — in Chapter 8 the log-likelihood will be built from exactly two moves, “one value per row” and “aggregate rows within task,” and long format makes both natural. Wide-to-long conversion is a mechanical reshape,¹ so nothing is lost by standardizing. When a dataset in this book is called the data, it is a long-format data frame with columns id, task, alt, choice, then attributes.

One convention completes the format, and I state it as a contract because code will rely on it silently: rows are sorted by id, then task, then alt. Grouped computations in the chapters ahead identify a choice situation by contiguous runs of rows; the sort order is what makes that identification valid. Violate the contract and nothing errors — results are simply, quietly wrong. (We will build one guardrail below.)

4.3 Two Kinds of Variables, Two Kinds of Columns

Look again at the long-format table and note what varies where. price varies across rows within a task — each alternative has its own. But a respondent characteristic like age would repeat across all rows of a task, varying only with id. The distinction matters enough for names:

• Alternative-specific variables (\(\mathbf{x}_{ntj}\)): attributes of the alternatives — price, brand, memory. These vary within a choice situation and are the natural drivers of choice.
• Individual-specific variables (\(\mathbf{z}_n\)): characteristics of the decision-maker — age, income, prior brand ownership. Constant across the alternatives in a task. (A notation caution: this \(\mathbf{z}_n\) — observed characteristics — is unrelated to the standard-normal draw vector \(\mathbf{z}\) of Section 2.4; the collision is regrettably standard in this literature, and context always disambiguates.)

Here is the fact that surprises newcomers, stated now and explained properly in Chapter 7: an individual-specific variable cannot enter utility with a single common coefficient. If age adds \(\gamma \cdot \texttt{age}_n\) to the utility of every alternative, it raises all \(J\) utilities equally, changes no comparison, and drops out of the choice probabilities entirely — \(\gamma\) is unidentified. Only differences in utility move choices. Individual-specific variables earn their way into choice models by interaction: with alternative dummies (age shifts the appeal of Apple relative to Acer) or with attributes (income damps price sensitivity). Structurally, that means \(\mathbf{z}_n\) columns in a long data frame are raw material awaiting interaction, not regressors in their own right. In this book’s main thread, individual-specific variation instead enters through coefficients that vary across people — the heterogeneity models of Chapter 11 onward — and in Chapter 17 the two devices meet, with \(\mathbf{z}_n\) shifting the mean of person \(n\)’s coefficient vector.

4.4 Coding Categorical Attributes

Brand takes values Acer, Dell, Apple; memory takes 8, 16, 32. Utilities are sums of numbers, so categories must become numeric columns, and the mapping is another small decision with options.

Dummy (treatment) coding picks a reference level and creates an indicator per remaining level. With Acer as reference: dell and apple columns, each 0/1. A brand coefficient is then the utility of that brand relative to Acer, all else equal — concrete and readable, with the reference level absorbing the baseline.

Effects coding constrains the level utilities to sum to zero instead: the reference level’s columns are all \(-1\), and each level’s utility is read relative to the average level. Conjoint practice often prefers it because part-worths then compare against a neutral average rather than an arbitrary reference, and it interacts more gracefully with the “attribute importance” summaries of Chapter 21.

Our choice: dummy coding, with the reference levels fixed once — Acer, 8GB, 13-inch, as set in Table 1.1. For a book, interpretability-per-line-of-code wins: “the Apple coefficient is the extra utility of an Apple over an Acer” needs no further explanation, and every simulate-and-recover check reads directly against true parameters stated in the same coding. Effects coding returns for a cameo in Chapter 21 where its advantages actually bite. What is not optional is consistency: coding schemes, like sort orders, are contracts, and estimates are only interpretable against the coding that produced them.

R can build coded columns from factors automatically via model.matrix(), and we will let it — with the factor levels ordered explicitly so the reference is what we intend, not what alphabetical order happens to deliver:

brand <- factor(c("Acer","Apple","Dell"), levels = c("Acer","Dell","Apple"))
model.matrix(~ brand)

  (Intercept) brandDell brandApple
1           1         0          0
2           1         0          1
3           1         1          0
attr(,"assign")
[1] 0 1 1
attr(,"contrasts")
attr(,"contrasts")$brand
[1] "contr.treatment"

Note model.matrix() includes an intercept column by default. Whether we keep it is the next section’s business.

4.5 Where Is the Intercept?

The binary model of Chapter 3 had an intercept: a constant in \(\mathbf{x}_n\) giving the baseline pull toward buying. Scan the long-format laptop data for its analogue and you hit a small conceptual wall: a column of ones would give every alternative in every task the same utility boost — and as with individual-specific variables, anything common to all alternatives cancels from every comparison. A common intercept is unidentified in multinomial choice data.

What can exist is an alternative-specific constant (ASC): a dummy for position \(j\), capturing systematic utility for “option 2” as such. Whether ASCs belong in the model depends on the design:

• With labeled alternatives — the options are inherently “car,” “bus,” “train,” the staples of transportation economics (Train 2009, sec. 2.5) — ASCs are essential. Each mode has unmeasured systematic appeal, and its ASC absorbs it.
• With unlabeled profiles — our study, and most conjoint: the three laptops in a task are just “some laptops,” their screen positions randomized — position has no meaning, and the brand attribute already carries the “which product” information. Including position ASCs would estimate, roughly, left-middle-right screen bias.²

So our laptop data frames carry no intercept and no ASCs; the brand dummies play the role closest to “brand intercepts.” When Part IV needs ASCs (the probit’s identification story makes them central) and when Chapter 22 adds a no-choice option (whose ASC is the entire model of “walking away”), we will add the relevant dummy columns explicitly, as ordinary attributes. That is a quiet advantage of long format: an ASC is just one more 0/1 column, not a special modeling object.

4.6 Building the Study Design

Enough principles; let’s build the actual dataset scaffold for the laptop study. A useful separation of concerns — and the one real studies force on you anyway — is between the design (which profiles appear in which tasks; exists before any respondent) and the responses (the choice column; exists after). We build the design now. Filling in choices requires the MNL model itself, so that half arrives in Chapter 7.

How should profiles be assigned to tasks? The gold standard is a formal experimental design — orthogonal arrays, D-efficient designs — that maximizes the information the study extracts per task (Rao 2014, Chs. 2 and 4). Design theory is a book of its own and deliberately out of our scope. We use the simplest defensible method, independent random assignment: every attribute of every profile drawn independently at random. Random designs are what our simulate-and-recover studies need (no accidental confounding between attributes, by construction) and they let the code stay transparent:³

sim_laptop_design <- function(N, T, J) {
    n_rows <- N * T * J
    data.frame(
        id    = rep(seq_len(N), each = T * J),
        task  = rep(rep(seq_len(T), each = J), times = N),
        alt   = rep(seq_len(J), times = N * T),
        brand = factor(sample(c("Acer","Dell","Apple"), n_rows, replace = TRUE),
                       levels = c("Acer","Dell","Apple")),
        ram   = factor(sample(c("8","16","32"), n_rows, replace = TRUE),
                       levels = c("8","16","32")),
        screen = factor(sample(c("13","15"), n_rows, replace = TRUE),
                        levels = c("13","15")),
        price = round(runif(n_rows, min = 0.8, max = 2.4), 2)
    )
}

set.seed(40)
laptops <- sim_laptop_design(N = 500, T = 8, J = 3)
head(laptops, 6)

  id task alt brand ram screen price
1  1    1   1 Apple  16     13  2.16
2  1    1   2  Dell  16     15  2.08
3  1    1   3  Acer  32     15  1.77
4  1    2   1  Dell  32     15  1.81
5  1    2   2 Apple   8     15  1.24
6  1    2   3  Dell  32     15  1.82

The rep() calls in the first three columns implement the sort-order contract mechanically — id slowest, alt fastest — so the design is born correctly ordered. Dimensions check:

nrow(laptops)                       # 500 * 8 * 3

[1] 12000

And a design-level sanity check in the spirit of Section 3.5: with independent random assignment, attributes should be uncorrelated and levels balanced. Brand shares by alternative position:

round(prop.table(table(laptops$brand, laptops$alt), margin = 2), 2)

       
           1    2    3
  Acer  0.34 0.35 0.33
  Dell  0.34 0.33 0.34
  Apple 0.32 0.32 0.33

Near-uniform everywhere, as designed.

4.7 From Data Frame to Likelihood Inputs

The last mile. A data frame is for humans; estimation code wants numeric arrays. Specifically, everything from Chapter 8 onward needs three things: the attribute matrix, the choices, and a map from rows to choice situations. There are three reasonable containers, and this is the load-bearing decision of the chapter:

• (a) A list of per-task matrices. data[[i]] holds task \(i\)’s \(J \times K\) attribute matrix and its choice. Transparent — each likelihood term has its own little bundle — and the shape most textbook pseudo-code implies. But R-level loops over 4,000 list elements are slow (Chapter 9 will quantify how slow), and slow inside a likelihood means slow everywhere, because the likelihood is the inner loop of every method in this book.
• (b) A three-dimensional array. Dimensions (task, alternative, attribute): X[i, j, k]. Fast, vectorizable, and appealing when every task shows exactly \(J\) alternatives. But it hard-codes the balanced-design assumption — add a no-choice alternative to some tasks (Chapter 22) or work with real-world availability variation, and the array needs padding and masking. Array code also reads poorly; three-index gymnastics obscure the model.
• (c) A stacked matrix plus index vectors. Keep the long format’s row layout: one \((N \cdot T \cdot J) \times K\) matrix X whose rows are alternatives in contract order, a 0/1 vector choice marking chosen rows, and an integer vector task_id assigning each row to its choice situation. Grouped operations — “sum \(e^{V}\) within each task” — become single calls to R’s grouped-aggregation primitives (rowsum() chief among them), which run at compiled speed. Ragged tasks cost nothing: a task is however many rows share its task_id.

Our choice: (c), the stacked-matrix representation. It is within a whisker of the array’s speed, it degrades gracefully to ragged designs, and — decisive for this book — it is the representation that survives unchanged into later parts: the MSL of Chapter 13 evaluates the same stacked X under many coefficient draws, and the person-by-person Metropolis steps of Chapter 17 slice it by id without restructuring. Build once, reuse for nineteen chapters.

The conversion function:

build_choice_data <- function(df, attr_formula) {
    stopifnot(!is.unsorted(order(df$id, df$task, df$alt)))
    X <- model.matrix(attr_formula, data = df)
    X <- X[, colnames(X) != "(Intercept)", drop = FALSE]
    task_id <- cumsum(!duplicated(df[, c("id", "task")]))
    list(
        X       = X,
        choice  = if ("choice" %in% names(df)) df$choice else NULL,
        task_id = task_id,
        id      = df$id,
        n_tasks = max(task_id),
        K       = ncol(X)
    )
}

Every line earns its place:

• The stopifnot() guardrail enforces the sort-order contract; better a loud failure now than silent nonsense later.
• model.matrix(attr_formula, df) expands factors into dummy columns using their declared reference levels — the coding decision of this chapter, executed by base R. We then drop the intercept column it insists on adding, for the identification reason of Section 4.5.
• task_id is built by a small idiom worth reading twice: !duplicated() over the (id, task) pairs flags each task’s first row, and cumsum() turns those flags into a running task counter — 1,1,1,2,2,2,... — giving every row its group label in two vectorized calls.
• choice is optional, so the same function serves a bare design (now) and responded data (from Chapter 7 on).

Applied to the laptop design:

dat <- build_choice_data(laptops, ~ brand + ram + screen + price)
str(dat, max.level = 1)

List of 6
 $ X      : num [1:12000, 1:6] 0 1 0 1 0 1 0 1 0 0 ...
  ..- attr(*, "dimnames")=List of 2
 $ choice : NULL
 $ task_id: int [1:12000] 1 1 1 2 2 2 3 3 3 4 ...
 $ id     : int [1:12000] 1 1 1 1 1 1 1 1 1 1 ...
 $ n_tasks: int 4000
 $ K      : int 6

colnames(dat$X)

[1] "brandDell"  "brandApple" "ram16"      "ram32"      "screen15"  
[6] "price"     

There it is: a 12000 \(\times\) 6 matrix whose six columns⁴ — dell, apple, ram16, ram32, screen15, price in our shorthand — line up exactly, column for column, with the true parameter vector we will fix when the full study’s choices are simulated in Chapter 7, plus the index vectors that give the rows their meaning. When Chapter 8 computes its first multinomial log-likelihood, it will consume precisely this object, and so will nearly every estimator after it.

4.8 Package Conventions: dfidx and Friends

When we validate our hand-coded estimators against R packages — mlogit in Chapter 8, gmnl in Chapter 11 — we must feed those packages their expected structure. Happily, the ecosystem standardized on ideas congruent with ours: mlogit (Croissant 2020) consumes data via the dfidx package, which wraps a long-format data frame with declared indexes — idx = c("chid", "alt") naming the choice-situation and alternative columns, with a nesting structure for respondent panels. The conversion from our canonical format is one line, which we will write when we first need it. If you understand this chapter, no R choice package’s data layer will ever mystify you: they are all long format plus index bookkeeping, differing only in surface conventions.

4.9 Key Learnings

• Choice data live on three indices — decision-maker \(n\), choice situation \(t\), alternative \(j\) — and one row of our canonical data is one \((n,t,j)\) triple: long format, sorted by id, task, alt. The sort order is a contract that grouped computations rely on silently.
• Only utility differences move choices. That single fact organized the whole chapter: individual-specific variables need interactions to matter, a common intercept is unidentified, and ASCs are position dummies that belong in labeled designs but not in our unlabeled profile study.
• Categorical attributes become dummy columns against fixed reference levels (Acer, 8GB, 13”); model.matrix() mechanizes the coding, and consistency is what keeps estimates interpretable.
• The study design exists before responses: sim_laptop_design() builds the \(500 \times 8 \times 3\) laptop study with independent random attribute assignment, correctly ordered by construction.
• The load-bearing decision: estimation code consumes the stacked-matrix representation — X (\(NTJ \times K\)), 0/1 choice, integer task_id — chosen over per-task lists (slow) and 3-D arrays (rigid) because it is fast, ragged-friendly, and reusable verbatim from MLE through MSL to MCMC. build_choice_data() performs the conversion and guards the contracts.
• Package data layers (dfidx for mlogit) are the same ideas under different names; conversion is mechanical.

Croissant, Yves. 2020. “Estimation of Random Utility Models in R: The mlogit Package.” Journal of Statistical Software 95 (11): 1–41.

Rao, Vithala R. 2014. Applied Conjoint Analysis. Springer.

Train, Kenneth E. 2009. Discrete Choice Methods with Simulation. 2nd ed. Cambridge University Press. https://eml.berkeley.edu/books/choice2.html.

---

1. stats::reshape(), tidyr::pivot_longer(), or the dfidx package all automate it. We won’t dwell on the mechanics; what matters is knowing which shape you’re in and which shape you need.

2. Which is a real (if small) behavioral phenomenon, and estimating position effects is occasionally a useful design diagnostic. The point is that in an unlabeled design ASCs are a diagnostic, not a structural part of utility.

3. A confession: T is also R’s abbreviation for TRUE, and masking it is normally poor hygiene. We use it anyway so the code reads like the math (\(N\), \(T\), \(J\)). Never abbreviate TRUE as T in your own code — this is exactly why.

4. model.matrix() names them brandDell, brandApple, ram16, ram32, screen15, price: each factor contributes levels-minus-one dummy columns, per the coding discussion above.