Economic and Game Theory
Human Subject Protection in Economics

by David K. Levine, Department of Economics, UCLA

top | general | research | classes | links | David

Research with human subjects is regulated by the Federal Government using regulation designed for medical experiments. These regulations are ill-suited for social science research, and in fact, because the rules were designed for medical research it is often harder to get permission to conduct non-medical research because it is difficult for researchers and human subjects protections committees (IRBs) to figure out how the rules should apply outside the medical setting. This often results in time-consuming and useless back-and-forth between researchers trying to conduct scientifically valid and harmless research, and IRBs trying to figure out how to squeeze social science research into a medical box.

There are several interesting discussions of these issues: the American Association of University Professors convened several meetings on the subject and produced a report. While the conclusions of the report are wishy-washy it does a good job of documenting some of the problems IRBs have had with social science research. Similar points are raised in an article in the Chronicle of Higher Education and in Lingua Franca. The American Anthropological Association has taken a formal stand. Also of interest is an eloquent article by Stephen Warner of the University of Illinois at Chicago. His non-federally funded social science research was shut down by the NIH because of the malfeasance of some medical researchers.

There is also a move afoot to impose even more stringent regulation. The Institute of Medicine and the National Bioethics Advisory Committee are both involved in direct efforts of this type. Many of these additional regulations have already been indirectly implemented without public discussion by the Office for Human Research Protections through side agreements with universities and institutions called MPAs and most recently FWAs. The Consortium of Social Science Associations has some commentary on the direct efforts: an overview of what is going on, a report on a public forum and Felice Levine's written testimony.

My goal here is to discuss the same set of issues from the perspective of experimental economics: at CASSEL we have had our own problems getting IRB consent to conduct research that is both transparent and harmless. As seems to be the case in much social science research the issues have revolved around informed consent - disclosing the purpose of the research to the subjects and getting them to provide physical signatures on complicated consent forms.

The overall conclusion I reach is that the current rules are misdirected from the perspective of experimental economics research, but because they provide adequate escape they can be and are effectively used by IRBs and OPRSs to permit the conduct of research under conditions that are scientifically valid. There is no evidence that additional regulation is needed and no reason to intervene in the many cases where effective relations have been worked out between researcher and human subjects protection committees; what would be effective would be for professional societies to step in and provide better guidance to IRBs new to economics experiments. An example of what such a statement might look like for economics is here.

What We Do

The methodology of economics experimentation has been developed by economists over the last 40 years and is now used in political science, anthropology, sociology, management science, and other social sciences. Economics experiments typically have no risk in the sense that the experiences are similar to day to day decisions. In a typical economics experiment, participants are brought together in a laboratory, classroom or through the internet. They make a series of decisions and are rewarded based on their decisions and the decisions of other participants. They are instructed on how they will interact, and how the pattern of decisions will affect their own rewards and in some cases the rewards of others. Instruction is designed to assure that understanding is successful - participants are usually given opportunity to ask questions and may be required to pass a test of understanding. Participants understand that they can leave at any time. After they make their decisions, their decisions are recorded, and they are rewarded depending on the decisions they made and that others made. Different participants may earn different rewards, usually money, paid anonymously and in private after the experiment. Monetary rewards based on outcomes are used to ensure that participants are well motivated. They are considered important to maintaining experimental control over subjects' motivations, which exert a strong influence on the predictions of the theories being tested. Subjects are sometimes paid a small additional fee for showing up on time and listening to instructions.

When interactions take place using a computer network, the environment is typically no different from sitting in front of a computer screen, making decisions in response to the information that is displayed there. Face to face interactions are typically in the form of negotiations over a price or are similar to discussing a proposal in a committee setting. Participants are not exposed to any sort of harm nor can the decisions they make lead to harm to others. Subjects are volunteers and may leave at any time. Data that might identify particular individuals are not made public. Social science experiment that follow the methodology of experimental economics have been conducted with thousands of students over several decades and there are no instances of harm nor has any form of risk been discovered or reported. These methods are often used as part of classroom exercises to make clear how theories apply to daily experiences in economics and in social environments.

Our Experience

We originally applied to the UCLA IRB to conduct a study of a public goods contributions game on October 10, 2000. In these games, participants are endowed with tokens which they may either redeem for money, or contribute to a public pool which provides a monetary benefit to many participants. It may surprise you to learn that as of June 8, 2001 we have not received permission from the UCLA IRB to conduct these experiments. I want to emphasize that I believe the UCLA OPRS and IRB to be well-intentioned and competent. Although mistakes have been made both by us and by them, ultimately this experience reflects the abysmal failure of the application of regulations meant for medical research to social science research. You can find further details about how this situation came about in this email from CASSEL to the NSF, and the following summary.

The Rules

What are the rules that the UCLA OPRS and IRB are charged with enforcing? The central document is the Belmont Report. The relevance of the Belmont Report to the type of research that we do, can be summarized by the third and final footnote to that report:

"Because the problems related to social experimentation may differ substantially from those of biomedical and behavioral research, the Commission specifically declines to make any policy determination regarding such research at this time. Rather, the Commission believes that the problem ought to be addressed by one of its successor bodies."

But what exactly is "behavioral" and "social" research? The Belmont report has this to say about "behavioral" research:

"It is important to distinguish between biomedical and behavioral research, on the one hand, and the practice of accepted therapy on the other, in order to know what activities ought to undergo review for the protection of human subjects of research."

Unfortunately, while it may be important to make the distinction, the Belmont report does not do so, and the word "behavioral" is mentioned only one more time in the report.

The Belmont Report is implemented in the Code of Federal Regulations title 45 part 46. (The statute upon which the administrative law is based can be found here.) Like the Belmont Report it lumps together biomedical and behavioral research, and it ignores the recommendation of the Belmont Report that social experimentation be treated separately. It does define what consitutes human subject research

Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge. Activities which meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program which is considered research for other purposes. For example, some demonstration and service programs may include research activities.

Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) data through intervention or interaction with the individual, or
(2) identifiable private information.

Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the subject or the subject's environment that are performed for research purposes. Interaction includes communication or interpersonal contact between investigator and subject. Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place, and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example, a medical record). Private information must be individually identifiable (i.e., the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects.

This definition is stunningly broad, apparently including such things as interviews conducted by journalists. Surprisingly, it appears not to include most economics experiments. The information obtained from individuals in these experiments is not "about" subjects, but rather is their response to hypothetical situations we seek their feedback on. In this sense, economics experiments are akin to consulting a colleague for advice, a situation that presumably does not constitute human subject research.

Do the Rules Make Sense for Economics?

Although social and behavioral research appears to be covered as less than an afterthought to the process of regulating medical research, it could never-the-less be the case that they make sense for this type of research. Unfortunately, they do not. There are seven requirements for human subject research:

(1) Risks to subjects are minimized: (i) by using procedures which are consistent with sound research design and which do not unnecessarily expose subjects to risk, and (ii) whenever appropriate, by using procedures already being performed on the subjects for diagnostic or treatment purposes.

(2) Risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result. In evaluating risks and benefits, the IRB should consider only those risks and benefits that may result from the research (as distinguished from risks and benefits of therapies subjects would receive even if not participating in the research). The IRB should not consider possible long-range effects of applying knowledge gained in the research (for example, the possible effects of the research on public policy) as among those research risks that fall within the purview of its responsibility.

(3) Selection of subjects is equitable. In making this assessment the IRB should take into account the purposes of the research and the setting in which the research will be conducted and should be particularly cognizant of the special problems of research involving vulnerable populations, such as children, prisoners, pregnant women, mentally disable persons, or economically or educationally disadvantaged persons.

(4) Informed consent will be sought from each prospective subject or the subject's legally authorized representative, in accordance with, and to the extent required by §46.116.

(5) Informed consent will be appropriately documented, in accordance with, and to the extent required by §46.117.

(6) When appropriate, the research plan makes adequate provision for monitoring the data collected to ensure the safety of subjects.

(7) When appropriate, there are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data.

Of these 7 requirements 4 appear to have some degree of relevance to economics experiments. The first and second requirement are irrelevant because there are no known risks, as is the sixth, since the safety of the subjects is not in question. The requirement that there be adequate provisions to protect the privacy of subjects and maintain confidentiality of data is relevant. Because it is relevant and recognized as such by economics investigators it does not seem to have been an important source of conflict between economists and IRBs.

The requirement that selection of subjects be equitable is confusing. Economics experiments measure social norms that have been documented to differ from one society to another. The ethnicity of subjects is an important treatment variable; to say that subjects be selected equitably is the type of requirement that is likely to confuse IRBs and lead to the obstruction of research without any benefit to potential subjects. The last part of the requirement does not apply, as economics experiments do not deal with vulnerable populations.

That leaves the two informed consent requirement - that informed consent be sought, and that it be documented. It is these two requirements that have posed the greatest problems for social science research generally and for us in particular.

Informed Consent

The requirements for informed consent are:

(a) Basic elements of informed consent. Except as provided in paragraph (c) or (d) of this section, in seeking informed consent the following information shall be provided to each subject:

(1) a statement that the study involves research, an explanation of the purposes of the research and the expected duration of the subject's participation, a description of the procedures to be followed, and identification of any procedures which are experimental;

(2) a description of any reasonably foreseeable risks or discomforts to the subject;

(3) a description of any benefits to the subject or to others which may reasonably be expected from the research;

(4) a disclosure of appropriate alternative procedures or courses of treatment, if any, that might be advantageous to the subject;

(5) a statement describing the extent, if any, to which confidentiality of records identifying the subject will be maintained;

(6) for research involving more than minimal risk, an explanation as to whether any compensation and an explanation as to whether any medical treatments are available if injury occurs and, if so, what they consist of, or where further information may be obtained;

(7) an explanation of whom to contact for answers to pertinent questions about the research and research subjects' rights, and whom to contact in the event of a research-related injury to the subject; and

(8) a statement that participation is voluntary, refusal to participate will involve no penalty or loss of benefits to which the subject is otherwise entitled, and the subject may discontinue participation at any time without penalty or loss of benefits to which the subject is otherwise entitled.

(b) additional elements of informed consent. When appropriate, one or more of the following elements of information shall also be provided to each subject:

(1) a statement that the particular treatment or procedure may involve risks to the subject (or to the embryo or fetus, if the subject is or may become pregnant) which are currently unforeseeable;

(2) anticipated circumstances under which the subject's participation may be terminated by the investigator without regard to the subject's consent;

(3) any additional costs to the subject that may result from participation in the research;

(4) the consequences of a subject's decision to withdraw from the research and procedures for orderly termination of participation by the subject;

(5) A statement that significant new findings developed during the course of the research which may relate to the subject's willingness to continue participation will be provided to the subject; and

(6) the approximate number of subjects involved in the study.

It is hard to know where to begin. Of the 6 additional elements of informed consent, all appear to be irrelevant. Of the 8 basic elements of informed consent 4 are irrelevant: there are no risks, benefits or alternative treatments, and the research is minimal risk. Three are uncontroversial for economics research: disclosure of confidentiality, who to contact for questions and a statement that participation is voluntary. That leaves the first requirement

a statement that the study involves research, an explanation of the purposes of the research and the expected duration of the subject's participation, a description of the procedures to be followed, and identification of any procedures which are experimental

This has five parts - the identification of "procedures which are experimental" is irrelevant, the statement that the study involves research and expected duration of participation are uncontroversial, while our description of the procedures to be followed tends to be elaborate. That leaves the "explanation of the purposes of the research." This is vague: in this type of research we are happy to disclose the overall purpose of the research. A more detailed explanation involves technical details typically of little interest to participants - but even if we could explain them it would be harmful to the science to do so. An important scientific consideration in this research is that subjects' choices be their own. Knowledge of the specific aims of the experiment is known to influence behavior. Consequently, our scientific protocol demands that the features of theory and models specific hypotheses under consideration not be disclosed to participants beforehand. In cases where there is a common subject pool, disclosure of specific hypotheses after the experiment poses the same problem for subsequent experiments as prior disclosure.

There is also the troublesome issue of documenting informed consent.

informed consent shall be documented by the use of a written consent form approved by the IRB and signed by the subject or the subject's legally authorized representative. A copy shall be given to the person signing the form.

(b) Except as provided in paragraph (c) of this section, the consent form may be either of the following:

(1) A written consent document that embodies the elements of informed consent required by §46.116. This form may be read to the subject or the subject's legally authorized representative, but in any event, the investigator shall give either the subject or the representative adequate opportunity to read it before it is signed

This might seem untroubling, and some investigators find it so. But the devil is in the details. The types of sessions under consideration last on the order of 75 minutes, roughly the same length of time as a lecture. Imagine, if you will, the "practibility" of requiring a signed "medical style" consent form prior to every college lecture. To see what such a form might look like, I took a consent form approved by our IRB for our use and modified it lightly to apply to one of my lectures. The original (with the name of the investigator removed) is here. It is important point to note that we disclose all the pertinant information on our website during the process of registering for experiments and that potential participants must give their consent by clicking a button to continue the registration process. Amazon.com and my credit card company are of the opinion that this form of consent is legally binding. Finally, beyond the issue of how practicable is is to use this type of consent, there is a scientific concern: signed consent forms are not used outside of research for risk free activities; in fact even many risky activities such as amusement parks do not use signed consent forms. Forcing participants to sign a form when there is no risk creates the impression that there might be risk, and raises suspicions where there should be none. The most likely suspicion we can think of is the suspicion we are deceiving our partipants about the rules (the absence of the possibility of physical harm is transparent). Economists are very sensitive to this suspicion because our protocols call for participants to know and understand the rules. Much effort is expended to show subjects that we do not deceive them, and a signed consent form defeats this effort. I should note that the difficulty of using signed written consent forms seems common to social science research, and not particular to economics experiments.

Justice

The Belmont Report discusses the issue of "justice" at some length. It provides several formulations fo what might be meant by justice:

These formulations are (1) to each person an equal share, (2) to each person according to individual need, (3) to each person according to individual effort, (4) to each person according to societal contribution, and (5) to each person according to merit.

Economics experiments typically involve different subjects receiving different payment based on their own play and the play of other participants. There have been reports that some IRBs, apparently focused on the first notion of justice "to each person an equal share," have attempted to prevent these types of experiments. This has not been the case with the UCLA OPRS/IRB, and it would seem that (3) - (5) plus the fact that the opportunity to receive payment is equal among subjects would convincingly indicate that economics experiments are not unjust - but language of this type seems especially poorly designed for economists and certainly has the potential for causing confusion.

Are the Rules Usable for Economists?

As I have indicated the bulk of the rules are irrelevant and confusing for economics experimentation, while the some of the rules on informed consent are scientifically and/or practicably hard to satisfy. However, the CFR does provide a number of escape clauses, giving IRBs relatively broad power to waive requirements for research, such as economics experiments, that pose minimal risk to subjects. Some argue that "the regulations are extremely flexible and should present no impediment to well-designed behavioral and social science research." It is indeed true that the escape clauses in the CFR can and have been used by IRBs to permit scientifically valid economics experiments. However, as I indicate below, the cost of requiring exceptional treatment is disproportionate to the unexceptional risk. While it is possible to conduct economic experimentation, it is far too difficult - at many institutions it is not done at all, and it is especially discouraging for graduate students for whom 6 months to receive approval is an eternity.

The first and most significant escape clause for much social science research it an outright exemption from IRB jurisdiction. It is the job of the OPRS to determine which research projects are exempt from IRB jurisdiction; the relevant language from the CFR describing projects that are exempt is:

(1) Research conducted in established or commonly accepted educational settings, involving normal educational practices...

(2) Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior, unless:
(i) information obtained is recorded in such a manner that human subjects can be identified, directly or through identifiers linked to the subjects; and (ii) any disclosure of the human subjects' responses outside the research could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, or reputation.

There is also the following relevant footnote:

However, the exemptions at 45 CFR 46.101(b) do not apply to research involving prisoners, fetuses, pregnant women, or human in vitro fertilization, Subparts B and C. The exemption at 45 CFR 46.101(b)(2), for research involving survey or interview procedures or observation of public behavior, does not apply to research with children, Subpart D, except for research involving observations of public behavior when the investigator(s) do not participate in the activities being observed.

UCLA has their own additional category that is not eligible for exemption:

deception of subjects where the investigator does not disclose the true purpose of the research and/or the results of the subject’s participation in the study

Moreover, the interpretation of the language on exemption can cause problems. UCLA OPRS originally agreed that our proposed economics experiments were exempt, then later changed their mind and decided that this language did not apply. They went so far as to force us to cancel experiments already scheduled under our certificate of exemption. As it happens, however, exempt may not actually mean exempt. UCLA OPRS has told us that they review exempt research in exactly the same way as non-exempt research, and their manual for investigators says

NOTE: A Claim of Exemption does not necessarily exempt investigators from the requirement of gaining written informed consent from subjects. Most research requires the use an of informed consent form. For studies where there are no subject identifiers, i.e., anonymous data is collected, an information sheet or cover sheet is usually required.

I am currently trying to learn how and why this UCLA policy came about. However, the OHRP is in the process of making this policy applicable to all institutions through Federal Wide Assurances. This points out two problems with escape clauses: First, those organizations such as the OHRP that claim there is enough flexibility in existing rules are actively seeking to eliminate that flexibility. Second, universities are often reluctant to use the flexibility for fear they will be seen as evading the regulation. Consequently, regulations that appear to exempt research that is not appropriately governed by medical style regulation are never-the-less being shoehorned into that category by zealous regulators and universities eager to demonstrate their concern for human subjects. Unfortunately, shoehorning social science research into medical style rules provides no useful protection for human subjects.

The next escape clause is often said to be expedited review, a category for which most social science research qualifies, and economics experiments unambiguously qualify. The lack of ambiguity comes about because before the category was widened in 1998 "game theory" was listed as an explicit category eligible for expedited review. Notice, however, that this lack of ambiguity can work to our disadvantage: our OPRS argued that we could not be exempt since we are eligible for expedited review.

The problem with expedited review is not that it is difficult to get, but rather that it does litte good. It does not eliminate any of the obnoxious requirements of medical regulation, and consequently does not save us any of the time we need to spend documenting and arguing the validity of our research protocols.

The third escape clause is the waiver of informed consent:

An IRB may approve a consent procedure which does not include, or which alters, some or all of the elements of informed consent set forth above, or waive the requirement to obtain informed consent provided the IRB finds and documents that:

(2) the research could not practicably be carried out without the waiver or alteration.

This has been useful for us: when we have explained to the OPRS why our protocols call for not revealing the technical details of the purpose of the research in advance and why with a closed subject pool we should not do so after - they agreed that this portion of informed consent could be waived. But it did not avoid months of arguing at cross-purposes with the OPRS: for a long period of time, we could not understand why the OPRS kept trying to get us to provide more technical details about the purposes of our research to subjects, nor did we understand that we should simply explain the scientific reasons why we shoud not provide those details and ask for a waiver.

Finally, it is possible to get a waiver of written documentation of informed consent

An IRB may waive the requirement for the investigator to obtain a signed consent form for some or all subjects if it finds either:

(2) That the research presents no more than minimal risk of harm to subjects and involves no procedures for which written consent is normally required outside of the research context.

Since it is difficult to think of any procedures involving minimal risk outside of the research context in which written consent is normally required, this would appear to apply to all minimal risk research. Unfortunately, UCLA's addition to the regulation reads

Since this condition is subject to misinterpretation, it is rarely approved as sole justification for a waiver of documentation of informed consent.

What is to be emphasized in this is that while there are escape clauses OPRSs and IRBs are often reluctant to use them - UCLA's additions to the rules to not appear to something exclusive to us.

What is to be Done

One possibility is to follow the advice of the Belmont Report and design a separate set of rules for social science research. There are two problems with this. First, there is no one size fits all model in the social sciences: economists are no more eager to have psychology protocols applied to their work than the reverse, and some issues, such as confidentiality, pose problems for some social science research but not other. Second, changing rules that involve 17 federal agencies is likely to be time-consuming and have unpredictable results. Creating yet another bureaucracy to deal with the social sciences may create more problems than its solves.

Since the existing rules, even as unilaterally amended by the OHRP, permit social science research under scientifically valid conditions, it is not apparent that the rules need to be changed. Ultimately the problem lies in the fact that individual IRBs and OPRSs are poorly informed about how the rules ought to be applied to social science research. One strategy, recommended, for example, by the AAUP is to try to put more social scientists on IRBs. There are several problems with this. First, much of the difficulty in our case has been with the OPRS and not the IRB itself. Second, it is not clear how to go about getting more social scientists on IRBs, except on a case-by-case basis. Third, social science human subject research is diverse and small relative to medical human subject research. For example, it is not obvious how putting an economist on an IRB will serve oral historians or journalists well. Moreover, particular campuses may have few social scientists knowledgeable about human subject research. For example, at UCLA, a large campus, I am probably the person most knowledgeable about economics experimentation (and I am not an experimentalist). However, as director of the lab responsible for all research of this type, I would be disqualified from assessing any of this type of research. It is also a heavy penalty to pay for social scientists - why should we pay a toll by spending vast amounts of time evaluating medical and other human subject research in order to be able to conduct our own research?

A better alternative is to educate OPRSs and IRBs. One of the basic problems is that every battle apparently has to be refought with every OPRS and IRB, and perhaps even for every research project. This is of great concern to us, because our goal is to run an open lab drawing researchers from many institutions. So we have to worry not only about our own OPRS/IRB but also the OPRS/IRBs of other institutions; and our most likely users will come from institutions that do not have their own labs and have particularly inexperienced OPRS/IRBs.

The natural place to begin such a campaign is with professional societies. Many of these societies already have strong ethical rules for the conduct of human subject research. It would be natural to combine these rules with a statement of how the society would like the CFR to be applied to their type of research. For example, they could indicate that for a particular type of research signed written consent is not ordinarily required, but that an alternative oral or web information sheet is ordinarily used instead. One of the problems in dealing with the OPRS and IRB is that they are (rightly) skeptical of investigators' interpretations of the rules. They are much more likely to take seriously a statement by an accepted scientific society. In the case of economics experiments a statement from the Economic Science Association documenting economic research protocols, interpreting the CFR, and providing a rationale for both the protocols and the interpretation of the CFR would carry substantial weight with an IRB. If the NSF or HHS were to publicize and vet statements of this type, and encourage IRBs to post links on their websites, this could be an effective tool for educating not only OPRSs and IRBs, but social science researchers as well.