Home LiteratureArticle Details
PMID: 17215398 Published · ppublish English Comparative Study Journal Article Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov't

Reinforcement learning signals predict future decisions.

Cohen MX, Ranganath C

Abstract

Optimal behavior in a competitive world requires the flexibility to adapt decision strategies based on recent outcomes. In the present study, we tested the hypothesis that this flexibility emerges through a reinforcement learning process, in which reward prediction errors are used dynamically to adjust representations of decision options. We recorded event-related brain potentials (ERPs) while subjects played a strategic economic game against a computer opponent to evaluate how neural responses to outcomes related to subsequent decision-making. Analyses of ERP data focused on the feedback-related negativity (FRN), an outcome-locked potential thought to reflect a neural prediction error signal. Consistent with predictions of a computational reinforcement learning model, we found that the magnitude of ERPs after losing to the computer opponent predicted whether subjects would change decision behavior on the subsequent trial. Furthermore, FRNs to decision outcomes were disproportionately larger over the motor cortex contralateral to the response hand that was used to make the decision. These findings provide novel evidence that humans engage a reinforcement learning process to adjust representations of competing decision options.

MeSH Terms
Adult Decision Making/physiology Evoked Potentials/physiology Female Forecasting Humans Learning/physiology Male Predictive Value of Tests Reinforcement, Psychology Signal Transduction/physiology
Authors & Affiliations
2 authors, click to expand affiliations / ORCID
Cohen Michael X
Department of Epileptology and Center for Mind and Brain, University of Bonn, 53105 Bonn, Germany. [email protected]
Ranganath Charan
References (60)
60 references, click to expand
  1. The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum.
    Neuroscience. 2000;96(3):451-74 PMID: 10717427
  2. Removing electroencephalographic artifacts by blind source separation.
    Psychophysiology. 2000 Mar;37(2):163-78 PMID: 10731767
  3. Representation of a perceptual decision in developing oculomotor commands.
    Nature. 2000 Mar 23;404(6776):390-4 PMID: 10746726
  4. Detecting the onset of the lateralized readiness potential: a comparison of available methods and procedures.
    Psychophysiology. 2000 May;37(3):347-60 PMID: 10860412
  5. A computational model of action selection in the basal ganglia. I. A new functional anatomy.
    Biol Cybern. 2001 Jun;84(6):401-10 PMID: 11417052
  6. Dopamine responses comply with basic assumptions of formal learning theory.
    Nature. 2001 Jul 5;412(6842):43-8 PMID: 11452299
  7. Error-related brain potentials are differentially related to awareness of response errors: evidence from an antisaccade task.
    Psychophysiology. 2001 Sep;38(5):752-60 PMID: 11577898
  8. Functions of the medial frontal cortex in the processing of conflict and errors.
    J Neurosci. 2001 Dec 1;21(23):9430-7 PMID: 11717376
  9. The medial frontal cortex and the rapid processing of monetary gains and losses.
    Science. 2002 Mar 22;295(5563):2279-82 PMID: 11910116
  10. Medial prefrontal cortex and error potentials.
    Science. 2002 May 31;296(5573):1610-1 author reply 1610-1 PMID: 12041532
  11. Human anterior cingulate cortex is activated by negative feedback: evidence from event-related potentials in a guessing task.
    Neurosci Lett. 2002 Jun 14;325(3):203-6 PMID: 12044656
  12. Consequences of dopaminergic denervation on the metabolic activity of the cortical neurons projecting to the subthalamic nucleus in the rat.
    J Neurosci. 2002 Oct 1;22(19):8762-70 PMID: 12351752
  13. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity.
    Psychol Rev. 2002 Oct;109(4):679-709 PMID: 12374324
  14. Neural economics and the biological substrates of valuation.
    Neuron. 2002 Oct 10;36(2):265-84 PMID: 12383781
  15. Reward, motivation, and reinforcement learning.
    Neuron. 2002 Oct 10;36(2):285-98 PMID: 12383782
  16. A computational account of altered error processing in older age: dopamine and the error-related negativity.
    Cogn Affect Behav Neurosci. 2002 Mar;2(1):19-36 PMID: 12452582
  17. Dissociable executive functions in the dynamic control of behavior: inhibition, error detection, and correction.
    Neuroimage. 2002 Dec;17(4):1820-9 PMID: 12498755
  18. Discrete coding of reward probability and uncertainty by dopamine neurons.
    Science. 2003 Mar 21;299(5614):1898-902 PMID: 12649484
  19. Principles of pleasure prediction: specifying the neural dynamics of human reward learning.
    Neuron. 2003 Apr 24;38(2):150-2 PMID: 12718849
  20. Errors are foreshadowed in brain potentials associated with action monitoring in cingulate cortex in humans.
    Neurosci Lett. 2003 Sep 4;348(1):1-4 PMID: 12893411
  21. Errors in reward prediction are reflected in the event-related brain potential.
    Neuroreport. 2003 Dec 19;14(18):2481-4 PMID: 14663214
  22. Prefrontal cortex and decision making in a mixed-strategy game.
    Nat Neurosci. 2004 Apr;7(4):404-10 PMID: 15004564
  23. Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology.
    Curr Opin Neurobiol. 2004 Apr;14(2):139-47 PMID: 15082317
  24. Source localization (LORETA) of the error-related-negativity (ERN/Ne) and positivity (Pe).
    Brain Res Cogn Brain Res. 2004 Jul;20(2):294-9 PMID: 15183400
  25. Matching behavior and the representation of value in the parietal cortex.
    Science. 2004 Jun 18;304(5678):1782-7 PMID: 15205529
  26. Independent coding of reward magnitude and valence in the human brain.
    J Neurosci. 2004 Jul 14;24(28):6258-64 PMID: 15254080
  27. Reinforcement-related brain potentials from medial frontal cortex: origins and functional significance.
    Neurosci Biobehav Rev. 2004 Jul;28(4):441-8 PMID: 15289008
  28. ERP correlates of feedback and reward processing in the presence and absence of response choice.
    Cereb Cortex. 2005 May;15(5):535-44 PMID: 15319308
  29. Computational roles for dopamine in behavioural control.
    Nature. 2004 Oct 14;431(7010):760-7 PMID: 15483596
  30. The role of the medial frontal cortex in cognitive control.
    Science. 2004 Oct 15;306(5695):443-7 PMID: 15486290
  31. ERP evidence of MI activation without motor response execution.
    Neuroreport. 2004 Sep 15;15(13):2067-70 PMID: 15486483
  32. Neurocognitive mechanisms of cognitive control: the role of prefrontal cortex in action selection, response inhibition, performance monitoring, and reward-based learning.
    Brain Cogn. 2004 Nov;56(2):129-40 PMID: 15518930
  33. Error-related negativity reflects detection of negative reward prediction error.
    Neuroreport. 2004 Nov 15;15(16):2561-5 PMID: 15538196
  34. Decision making.
    Curr Biol. 2005 Jan 11;15(1):R9-R11 PMID: 15649356
  35. Responses of human anterior cingulate cortex microdomains to error detection, conflict monitoring, stimulus-response mapping, familiarity, and orienting.
    J Neurosci. 2005 Jan 19;25(3):604-13 PMID: 15659596
  36. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism.
    J Cogn Neurosci. 2005 Jan;17(1):51-72 PMID: 15701239
  37. Electrophysiological correlates of error correction.
    Psychophysiology. 2005 Jan;42(1):72-82 PMID: 15720582
  38. Brain potentials associated with expected and unexpected good and bad outcomes.
    Psychophysiology. 2005 Mar;42(2):161-70 PMID: 15787853
  39. Differential modulation of anterior cingulate cortical activity by afferents from ventral tegmental area and mediodorsal thalamus.
    Eur J Neurosci. 2005 Jun;21(11):2975-92 PMID: 15978009
  40. Midbrain dopamine neurons encode a quantitative reward prediction error signal.
    Neuron. 2005 Jul 7;47(1):129-41 PMID: 15996553
  41. Error-related negativity predicts reinforcement learning and conflict biases.
    Neuron. 2005 Aug 18;47(4):495-501 PMID: 16102533
  42. Behavioral and neural predictors of upcoming decisions.
    Cogn Affect Behav Neurosci. 2005 Jun;5(2):117-26 PMID: 16180619
  43. Mediofrontal negativities in the absence of responding.
    Brain Res Cogn Brain Res. 2005 Dec;25(3):777-87 PMID: 16249075
  44. Representation of action-specific reward values in the striatum.
    Science. 2005 Nov 25;310(5752):1337-40 PMID: 16311337
  45. Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring.
    J Neurosci. 2005 Dec 14;25(50):11730-7 PMID: 16354931
  46. Automatic removal of the eye blink artifact from EEG using an ICA-based template matching approach.
    Physiol Meas. 2006 Apr;27(4):425-36 PMID: 16537983
  47. Medial frontal cortex activity and loss-related responses to errors.
    J Neurosci. 2006 Apr 12;26(15):4063-70 PMID: 16611823
  48. Decision-making in Blackjack: an electrophysiological analysis.
    Cereb Cortex. 2007 Apr;17(4):865-77 PMID: 16723407
  49. Optimal decision making and the anterior cingulate cortex.
    Nat Neurosci. 2006 Jul;9(7):940-7 PMID: 16783368
  50. Limb (hand vs. foot) and response conflict have similar effects on event-related potentials (ERPs) recorded during motor imagery and overt execution.
    Eur J Neurosci. 2006 Jul;24(2):635-43 PMID: 16903864
  51. Functional architecture of basal ganglia circuits: neural substrates of parallel processing.
    Trends Neurosci. 1990 Jul;13(7):266-71 PMID: 1695401
  52. Individual differences and the neural representations of reward expectation and reward prediction error.
    Soc Cogn Affect Neurosci. 2007 Mar;2(1):20-30 PMID: 17710118
  53. Parallel organization of functionally segregated circuits linking basal ganglia and cortex.
    Annu Rev Neurosci. 1986;9:357-81 PMID: 3085570
  54. Burst firing induced in midbrain dopamine neurons by stimulation of the medial prefrontal and anterior cingulate cortices.
    Brain Res. 1988 Oct 11;462(1):194-8 PMID: 3179734
  55. Neural basis of saccade target selection.
    Rev Neurosci. 1995 Jan-Mar;6(1):63-85 PMID: 7633641
  56. Highly restricted origin of prefrontal cortical inputs to striosomes in the macaque monkey.
    J Neurosci. 1995 Sep;15(9):5999-6013 PMID: 7666184
  57. Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain.
    Int J Psychophysiol. 1994 Oct;18(1):49-65 PMID: 7876038
  58. A neural substrate of prediction and reward.
    Science. 1997 Mar 14;275(5306):1593-9 PMID: 9054347
  59. Widespread origin of the primate mesofrontal dopamine system.
    Cereb Cortex. 1998 Jun;8(4):321-45 PMID: 9651129
  60. A computational role for dopamine delivery in human decision-making.
    J Cogn Neurosci. 1998 Sep;10(5):623-30 PMID: 9802995
Article Info
Journal
The Journal of neuroscience : the official journal of the Society for Neuroscience
Abbr.
J Neurosci
ISSN
1529-2401
Published
2007-01-10
Pages
371-8
Language
English
Region
United States
NLM ID
8102140
PMCID
PMC6672075
Subset
IM
Analysis Services
Analysis Services

Contact

No. 2 Wenbo Road, Zhangqiu District, Jinan, Shandong

Qilu Normal University · Genelibs Bioinformatics Lab

750 Shunhua Rd, Jinan

2F, Bldg F, University Science Park

Tel: 0531-88819269

WeChat Official Account

Follow our WeChat subscription account for real-time updates and the latest in medical and biological research.


Business Email

E-mail: [email protected]