Skip to content
Sign in
Artwork for The Hidden Curriculum
The Hidden Curriculum · September 30 · 59 min

E57 Replication and transparency practices with Lars Vilhuber (AEA Data Editor)

In this episode we talk with Lars Vilhuber about replication practices, transparency in economics, and data access. Lars is the Executive Director of the Labor Dynamics Institute at Cornell University, a Senior Research Associate in the Economics Department at Cornell, and the American Economic Association's Data Editor. His interests include labor economics, statistical disclosure limitation and data dissemination, and reproducibility and replicability in the social sciences. His current work includes TRACE (TRAnsparency CErtified), a project that gives researchers automated tools to credibly demonstrate that their code runs and produces the results they report. Sebastian Tello-Trillo is an Associate Professor of Public Policy and Economics at the Frank Batten School of Leadership and Public Policy at the University of Virginia. Alex Hollingsworth is an Associate Professor of Economics at the Ohio State University. Henry Morris is our main editor. He is a student at the University of Virginia studying computer science and mathematics. In this episode we discussed: How Lars became the AEA Data Editor, and the AEA session comment dismissing research based on proprietary data that pushed him toward reproducibility work What the AEA Data and Code Availability Policy requires, and why the AEA was not the first journal to run authors' code How the review process works: a team of a graduate student, a pre-doc, and 15–20 undergraduates runs replication packages, and how other journals structure the same task TRACE: institution-run replication that produces a signed certificate showing the code ran and produced the stated outputs (it does not certify that the code is correct), including for restricted data Practical tips for reproducible code: start early, use a master script, manage software and package versions, use assertions, and write the README as you go Common reasons code fails to run, and what happens when results differ after fixes Whether reviewers should assess replicability, and why the data editor checks only whether the code runs Lars's workflow: block scheduling on his calendar, firm work-life boundaries, and no fixed to-do system 📚 Resources mentioned: Office of the AEA Data Editor, including its guidance, blog posts, and FAQs TRAnsparency CErtified (TRACE): Trusting Computational Research Without Repeating It — Zenodo Journal of Applied Econometrics Data Archive — the journal's replication repository, running since 1985 Zenodo and Dataverse [link needed] — trusted repositories accepted under the AEA policy, including options for delayed publication "Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends" by Jonathan Roth — American Economic Review: Insights (Lars's favorite paper in the lightning round) getcensus — Stata command to load American Community Survey data from the Census Bureau API 🎮 Recommendations of the week: Lars's rec: Grilled lemongrass chicken with a clementine cake and chocolate ganache (New York Times recipe) Alex's rec: The musician Logan Hone, a flautist and multi-instrumentalist Sebastian's rec: The AEA Data Editor's guidance and resources, and the getcensus command in Stata for importing ACS data (requires a Census API key)

0:00-59:55
Play in

transcript

No transcript — this publisher did not publish one.

show notes

In this episode we talk with Lars Vilhuber about replication practices, transparency in economics, and data access. Lars is the Executive Director of the Labor Dynamics Institute at Cornell University, a Senior Research Associate in the Economics Department at Cornell, and the American Economic Association's Data Editor. His interests include labor economics, statistical disclosure limitation and data dissemination, and reproducibility and replicability in the social sciences. His current work includes TRACE (TRAnsparency CErtified), a project that gives researchers automated tools to credibly demonstrate that their code runs and produces the results they report.

Sebastian Tello-Trillo is an Associate Professor of Public Policy and Economics at the Frank Batten School of Leadership and Public Policy at the University of Virginia.

Alex Hollingsworth is an Associate Professor of Economics at the Ohio State University.

Henry Morris is our main editor. He is a student at the University of Virginia studying computer science and mathematics.

In this episode we discussed:

  • How Lars became the AEA Data Editor, and the AEA session comment dismissing research based on proprietary data that pushed him toward reproducibility work
  • What the AEA Data and Code Availability Policy requires, and why the AEA was not the first journal to run authors' code
  • How the review process works: a team of a graduate student, a pre-doc, and 15–20 undergraduates runs replication packages, and how other journals structure the same task
  • TRACE: institution-run replication that produces a signed certificate showing the code ran and produced the stated outputs (it does not certify that the code is correct), including for restricted data
  • Practical tips for reproducible code: start early, use a master script, manage software and package versions, use assertions, and write the README as you go
  • Common reasons code fails to run, and what happens when results differ after fixes
  • Whether reviewers should assess replicability, and why the data editor checks only whether the code runs
  • Lars's workflow: block scheduling on his calendar, firm work-life boundaries, and no fixed to-do system

📚 Resources mentioned:

🎮 Recommendations of the week:

links15

more episodes

All episodes