Commentary on R. Hertwig and A. Ortmann (2001) "Experimental Practices in Economics: A Methodological Challenge for Psychologists?" forthcoming Behavioral and Brain Sciences, 24(4): http://www.cogsci.soton.ac.uk/bbs/archive/bbs.hertwig.html
Charles A. Holt, Economics, Rouss Hall, University of Virginia, Charlottesville, VA 22903,
Susan K. Laury, Economics, Georgia State University, Atlanta,
Abstract: The use of high hypothetical payoffs has been
justified by the realism and relevance of large monetary consequences and
by the impracticality of making high cash payments. We argue that subjects
may not be able to imagine how they would behave in high payoff situations..
A psychologist, Sidney Siegel, has been largely responsible for establishing the procedural standards used in economics experiments. His work used salient financial incentives (e.g. Siegel and Fouraker, 1960), appropriate non-parametric statistics (Siegel, 1956), and clear scripts for subjects. In one classic study, the conclusions of the previous twenty years of probability matching experiments were reversed by using small financial incentives (Siegel and Goldstein, 1959). In order to address important issues that involve large sums of money, some psychologists advocate using high hypothetical payoffs. For example, Kahneman and Tversky (1979, p.265) recommend the method of large hypothetical payoffs over the "contrived gambles for small stakes" that are typical in economics experiments: "The use of the method relies on the assumption that people often know how they would behave in actual situations of choice, and on the further assumption that the subjects have no special reason to disguise their true preferences." A comment that we often hear at interdisciplinary conferences is: "If it were just a matter of paying our subjects pennies, we would pay them, but we are interested in major decisions." Hertwig and Ortmann (2001) do not evaluate this economic realism justification for high hypothetical payoffs, nor do the studies they discuss deal directly with the issue of whether the scale of payoffs matters, either in real or hypothetical payment situations.
To address payoff scale effects, we consider a lottery choice situation in which there is some evidence that the nature of payoffs does not matter. Tversky and Kahneman (1992, p.315), for example, note that in choices between risky prospects, "we did not find much difference between subjects who were paid a flat fee and subjects whose payoffs were contingent on their decisions." We also observed this type of payoff invariance for choices between lotteries of the form:
Option A (safe): probability p of $2.00 and 1-p of $1.60,
Option B (risky): probability p of $3.85 and 1-p of $0.10,
where p is varied from 0.1 in decision 1, to 0.2 in decision 2, … to 1.0 in decision 10. As reported in Holt and Laury (2000), each person made all ten decisions, with one selected at random ex post to determine earnings on the basis of the subject's choice for that decision and the random outcome. In total, 93 subjects made these 10 choices, followed by a menu of hypothetical choices with all payoffs scaled up by a factor of 20 (payoffs of $40 or $32 for the safe option versus $77 or $2 for the risky option). After the determination of these "high" hypothetical earnings, the same subjects were asked to make the same 10 choices with the high payoffs being paid in cash.
The results are summarized in Figure 1, which graphs the percentage of safe choices in each decision. It is straightforward to verify that a risk neutral person would choose the safe option in the first four decisions and switch to the risky option as soon as the probability of the high payoff exceeds 0.4, as shown by the straight thin line labeled "risk neutral." The choice percentages for the low real payoff condition, shown by the line labeled "1x real," are generally to the right of the risk neutral line, indicating some risk aversion. The choice percentages for the high hypothetical condition are represented by the thin line labeled "20x hypothetical," which is quite close to the line for the low real payoff condition, and indeed the difference is not statistically significant. In contrast, the thick line labeled "20x real" for high real payoffs lies to the right, indicating a sharp increase in observed risk aversion when these payoffs are actually made. In addition, we scaled payoffs up from the original level by a factor of 50 ($100 or $80 for the safe option, versus $192.50 or $5 for the risky option). The data for the 9 subjects in this rather expensive treatment are shown by the thick "50x real" line, which indicates a further increase in risk aversion. In fact, none of the subjects were risk neutral or risk seeking for this very high treatment. Notice however, no such scale effect is seen in the hypothetical choices; the thin "50x hypothetical" line is quite close to the "20x hypothetical" and "low real" lines.
These results illustrate how the use of low monetary incentives may not matter, but it does not follow that using high hypothetical incentives is a way to investigate behavior in high-stakes economic choices. In addition, the use of low or hypothetical payoffs may be misleading in that biases and non-economic factors may have an impact that is not predictive of their importance in significant economic choices. For example, women were significantly more risk averse than men in the low-payoff condition, but this gender effect vanished in the high-real-payoff condition.
Some data patterns for experiments with hypothetical payments may not be robust, which may cause journal editors to be hesitant about accepting results of such studies. For example, when the above options "A" and "B" are doubled and reflected around zero, we do not observe reflection (risk aversion for gains and risk seeking for losses) with real money payments although such reflection occurred in over half of the cases when gains and losses were hypothetical (Laury and Holt, 2000). In the real payoff condition, only about 10% of the subjects reflected, and another 5% reflected in the wrong direction (risk seeking for gains and risk aversion for losses). These choice patterns are sharply different from the widely cited results of Kahneman and Tversky (1979) using hypothetical payoffs over different lottery pairs.
In the absence of a widely accepted theory of when financial incentives matter, we believe that performance-based payments should be used in economics experiments, even in seemingly similar situations where no effect has been reported in past studies.
Hertwig, R. and Ortmann, A. (2001) Experimental Practices in Economics: A Methodological Challenge for Psychologists? Behavioral and Brain Sciences, 24 (4): XXX-XXX.
Holt, C. A. and Laury, S. K. (2000) Risk aversion and incentive effects. Working Paper, Department of Economics, University of Virginia (http://www.gsu.edu/~ecoskl/highpay.pdf).
Laury, S. K. and Holt, C. A. (2000) Further reflections on prospect theory. Working Paper, Department of Economics, University of Virginia (http://www.people.virginia.edu/~cah2k/reflect.pdf).
Kahneman, D. and Tversky, A. (1979) Prospect theory: an analysis of choice under risk. Econometrica, 47:263-291.
Siegel, S. (1956) Nonparametric statistics for the behavioral sciences. McGraw Hill.
Siegel, S. and Goldstein, D. A. (1959) Decision making behavior in a two-choice uncertain outcome situation. Journal of Experimental Psychology, 57: 37-42.
Siegel, S. and Fouraker, L. B. (1960) Bargaining and group decision making. McGraw Hill.
Tversky, A. and Kahneman, D. (1992) Advances in prospect theory: cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5:297-323.