With the help of more than 700 reviewers, we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019. When considering only articles for which data accessibility and hardware and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for 29% of articles, at least part of the data set was not accessible to the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%. These figures represent a significant increase compared with the period before the introduction of the disclosure policy, where only 12% of articles voluntarily provided replication materials, of which 55% could be (largely) reproduced. Substantial heterogeneity in reproducibility rates across different fields is mainly driven by differences in data set accessibility. Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, and software and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies and suggest potential avenues for enhancing their effectiveness.

Reproducibility in Management Science / M. Fisar, B. Greiner, C. Huber, E. Katok, A.I. Ozkes, D. Abraham, G.S. Adams, A. Adbi, J.M. Addoum, M. Adena, L.Y. Akella, P. Akey, O. Akmansoy, A. Alban, V. Alexeev, A. Alimov, A. Aman, A. Aouad, G. Appel, N. Arnosti, K. Arora, T. Arpinon, F.M. Artinger, J. Arts, L. Baardman, Z. Babutsidze, G. Bahrami, S. Banerjee, C. Bao, T. Bao, O. Baron, X. Barriola, P. Monteiro E Silva Barroso, E. Baskin, R.J. Batt, G. Batta, A. Bauer, K. Bauman, W. Bazley, M. Becker-Peth, M. Begen, N. Begen, S. Benoit, L. Berger, N. Berlin, L.P. Berling, A. Bernard, J. Bertomeu, J. Bialkowski, P. Bilinski, J. Bischof, J.R. Black, H. Blunden, D. Bongaerts, F. Bonisch, M. Bos, C. Bosch-Rosa, S. Bourjade, A. Boysen, C. Brimhall, Z. Brokesova, J.P. Brooks, S.B. Bruns, G. Buckle, G. Buenstorf, G. Burtch, B. Bushong, S. Buti, P. Callery, M. Canayaz, J. Cao, W. Cao, X. Cao, M. Carree, V. Castellani, Y.J. Cerasi, H.H. Chang, J.W. Chang, M. Chang, Y. Chang, A. Chaturvedi, J. Chauvin, D.E. Chavez, C. Chen, F. Chen, J.I. Chen, P.-. Chen, R. Chen, W. Chen, W.J. Chen, Y. Chen, Z. Chen, Z. Chen, L. Chew, P.P.S. Chhabra, S.C. Chintala, G.-. Choi, S. Choi, V. Choudhary, V.T.F. Chow, K.L. Christensen, D.J. Chung, M. Cinelli, L. Cingl, A.A. Cire, J. Clark, J. Clement, J. Clithero, H. Clolery, D.R. Clough, N. Clyde, A. Coali, I. Comeig, N. Cook, J. Correia-Da-Silva, E. Costa, A. Coutts, I. Cribben, C. Cuculiza, Z. Cui, C. Cunningham, P. Cziraki, E. Dagorn, R. Dai, J. Dana, N.P. Danks, A. Darendeli, S. Dato, N. Davcik, C. De Grazia, J. De Sousa, J. De Vries, M. De Vries, O. Deev, R. Defronzo, L. Dekker, A. Delarue, E.E. Demiral, C. Demiroglu, A. Deore, A. Detzel, A. Devonaev, A.D. Bala, E. Dimant, D. Dimmery, S.G. Dimmock, C. Ding, L. Ding, T. Ding, Y. Ding, L. Dong, K. Donohue, A. Drichoutis, S. Du, Y. Duan, T. Duevski, H.N. Duong, M. Ederhof, H. El Hajj, M. Ellison, J.N. Eriksen, M. Espinosa, F. Fallucchi, X. Fang, V. Fanghella, M. Faralli, S. Farham, F. Fattinger, S. Feiereisen, Y. Feng, E. Ferracuti, A. Filippin, A. Fillon, S. Fiorin, G. Fisher, M. Fisher, C. Flath, J. Foerderer, V. Frey, C. Fuchs, N. Fugger, S. Gabel, F. Gaessler, B. Ganglmair, M. Gangwar, P.A.G. Ares, R. Garg, J.M. Gaspar, C. Gastaldi, R. Gauriot, A. De Genaro, Y. Geng, K. Georgalos, D. Geraldes, L. Gerhards, W. Gerken, M. Gibson, J. Gijsbrechts, S. Goerg, D. Goetz, J. Goldman, F. Gonschorek, V. Gonzalez-Jimenez, J.T.G. Goossens, M. Gordy, P.M. Gorny, I. Goswami, A. Goyal, R. Goyenko, T. Grad, W. Greenblatt, M. Gregor, D. Grieco, M. Grieder, M.R.P. Grossmann, S. Gruner, S. Guha, A. Guo, G. Guo, H. Guo, L. Guo, D. Gutt, A.F. Gygax, I. Hacamo, S. Haeckl, T.C. Hagenberg, D. Hagmann, J. Haislip, E. Han, J. Han, J.E. Harvey, O. Havrylchyk, S. Hazarika, L. He, Y. He, W. Hedgcock, I. Heimbach, B. Henderson, J. Hendrikse, E. Henry, B. Hepfer, R. Hernan, H. Herz, A. Heyes, C. Hildebrand, A. Hillenbrand, A. Hillert, M. Hilweg, E. Hjalmarsson, S. Hoelscher, P. Hoffmann, B. Hollenbeck, N. Holtrop, F. Holzmeister, S. Homroy, M. Hossain, L. Houf, T. Howell, K. Hu, A. Huang, J.-. Huang, L. Huang, S. Huang, S.J. Huber, S. Hudja, J. Humphrey, P. Hunermund, W.R. Hurst, C. Hurtado, K.P. Huynh, K. Hyndman, A. Ingolfsson, P. Ipeirotis, A. Israeli, A. Ivashchenko, W. Jabr, P.K. Jain, A. Jaramillo-Gutierrez, N. Javadinarab, Y. Ji, M. Jia, H. Jiang, H. Jiang, J. Jiang, J. Tan, M. Jirasek, B. Julio, H.H.J. Jung, D. Marcel Te Kaat, J. Kalodimos, M. Kamstra, H. Kang, Q. Kang, S. Kanimian, M.M. Kapons, E. Karmaziene, A. Kausar, P.J. Kelly, S. Kesavan, M. Khadjavi, H. Khobzi, R. Khubulashvili, A.G. Kim, B. Kim, C. Kim, D.S. Kim, S. Kim, S. Kim, S.H. Kim, S. Kim, M. Kirneva, A. Kiss, L.M. Kluppel, O. Kocak, C. Kogler, C. Konig-Kersting, A. Kopanyi-Peuker, L. Koppel, S. Koppman, O. Kopsacheilis, L.J. Kornish, A. Krahn, O. Krcal, S. Krishnamurthy, P. Kropp, S. Kundu, M. Kurschilgen, D.J. Kusterer, S. Kutuk, O. Kuzmina, E. Kyung, C. Lacan, A. Lam, T. Lambert, L. Lanahan, M. Langen, N. Laurentsyeva, K.K.F. Law, Q.T. Le, C. Lee, D. Lee, K.H. Lee, S. Lee, Y. Lee, M. Lei, Z. Lei, S. Leitner, G.M. Lepori, D.E. Levari, B.W. Lewis, B.T. Leyden, C. Li, J. Li, K.K. Li, L. Li, M. Li, S. Li, S. Li, Y. Li, Y. Li, C. Liang, S. Lim, M. Lin, P.-. Lin, Y. Lin, S. Linardi, W. Lincoln, M. Lindenmayr, M. Linnenluecke, A. Listo, R. Litjens, C. Liu, D. Liu, F. Liu, H. Liu, H. Liu, J. Liu, K. Liu, N. Liu, S. Liu, X. Liu, N. Livneh, T. Lluent, N. Loehndorf, M. Lof, Y. Loheac, P. Lohmann, L.A. Lopez, M. Lorko, F. Lotti, J. Lu, X. Lu, J. Luffarelli, W.J. Luhan, H. Luong, G. Lyu, L. Ma, L. Madio, K. Maeckle, M. Mahmoudzadeh, P. Maille, V. Mak, A. Malezieux, S. Mankad, C. Mantilla, B. Mantin, M. Mantovani, G. Marchesini, J. Marcucci, D.M. Fages, A. Masiliunas, S. Massoni, N. Matias, T. Matthys, M. Mattsson, T.A. Maurer, P. Maus, M.M. Kutuk, M.M. Max, C. Meinerding, M. Meister, D. Meitong, E. Melero, D. Mendes, T. Menzer, C. Merkle, J. Merrick, S. Meyer, T. Miklanek, W. Mill, S. Minner, E. Mirzayev, S. Mittlaender, S.V. Moller, A. Molnar, D. Moore, S. Mortal, G. Moscariello, Y. Mou, J. Mu, C. Mueller, A. Mukherjee, S. Mustafazade, K. Muthuraman, A. Nakkas, J. Naughton, H.B.H. Ng, L. Nguyen, M. Nguyen, N.P.A. Nguyen, T.T.T. Nguyen, A. Nguyen-Chyung, N. Nicolaou, S. Nolte, A. Non, B.A. Odegaard, Y. Ofek-Shanny, C.H. Oh, C.Y. Olivola, T.C. Omer, A. Orland, T. Otto, M. Ouyang, H. Ozyilmaz, N.A. Pairolero, S. Palan, N. Pandit, D. Papies, J. Park, T.-. Park, C. Parker, V. Patel, G. Pawlina, E.P.-. Nestour, G. Pearce, T. Peeters, J. Peliova, Z. Peng, C. Perignon, N. Peter, C. Peukert, H. Phan, A. Philipp-Muller, K. Phua, M. Pierson, T. Plihal, M. Ploner, S. Porcher, M. Pourieux, S. Preuss, J. Prochazka, S. Pu, Z. Puklavec, H. Qin, T. Qiu, X. Qiu, R.-. Rahal, A. Rahimian, M. Rajabzadeh, O. Randall, S. Ray, O. Rehbein, J.-. Reichenecker, N. Reinholtz, J.P. Reiss, J.-. Renne, S. Reza, P. Richardson, S. Riddiough, M.O. Rieger, C. Righi, R.M. Rilke, J. Riutort, C. Robotti, N. Romer, J. Rose, M. Rose, P. Rosmer, F. Rossi, B. Rostami, K. Roszbach, K. Rotaru, Y. Roth, D. Rotolo, C. Rott, B. Routledge, B. Rubineau, H. Rusch, I.O. Ryzhov, P. Saffi, M. Saglam, M. Samahita, P. Sarantopoulos, V. Sarhangian, S. Savasaneril, H. Scheule, M. Schleritzko, M. Schnidman, D.S. Schoch, M. Schroder, E.C.M. Schutte, D. Schwartz, F. Schwerter, R. Seamans, M. Seifert, T. Servranckx, N. Sethuraman, V. Sevcenko, D.R. Shah, R. Shah, K. Sharma, P. Sharma, A. Sheneman, Y. Shi, L. Shuai, S. Siegenthaler, J. Silberholz, R. Silva, K. Silz-Carson, F. Simon, R. Singal, N.R. Sinha, S. Skouras, D. Smerdon, K. Smolka, A. Soetevent, E. Sojli, K. Sokolov, J. Somasundaram, Y. Son, J.M. Song, V. Soni, D. Sonsino, M. Souther, C. Spaenjers, M. Spann, E. Spiliotopoulou, J. Starck, A. Starkweather, D. Steele, M. Stefan, F. Stehr, E. Steiner, L. Stich, T. Stoeckl, J. Stoop, K. Strohlein, R. Stuber, J. Sturgess, Y. Su, Y. Su, R. Suchon, M. Sui, S. Sulz, E. Sung, M. Szymanowska, G.A. Tabacco, D. Tannenbaum, N. Tereyagoglu, C. Tergiman, M. Testoni, R. Thakor, W.W. Tham, S. Thelaus, S. Thielen, L. Tong, O. Tonguc, M. Tonin, S.Y. Toraman, M. Tortoriello, J.D. Tracy, J. Tremewan, M.K. Tripathi, G. Tumer-Alkan, D. Turcic, T. Turocy, H. Tyagi, M. Udenio, S. Ulku, M. Ungeheuer, S. Utke, C. Uzmanoglu, M. Vacca, P. Valta, M. Van Der Borgh, J. Van Der Geest, M. Van Steenvoort, R. Van Veldhuizen, P. Vana, M. Vanhoucke, B. Vanneste, J. Vecci, S. Venkataraman, M. Veronesi, S. Vicente, S. Villa, M.V. Martin, L. Vincent, T. Vladasel, S. Voigt, J. Vosgerau, C.A. Vossler, A. Vossmeyer, H.F. Wagner, D.M. Waguespack, E. Walker, M. Walker, M. Walzl, Z. Wan, C.C.Y. Wang, J.T.-. Wang, K. Wang, V.X. Wang, X. Wang, Y. Wang, X.S. Warnes, L. Wasserka-Zhurakhovska, W. Wei, S. Weiergraeber, P. Weiss, J. Weng, W.-. Weng, J. Weston, J.T. White, M. Wibral, J. Williams, O. Wilms, F. Wirl, A. Wolanski, M.H.F. Wong, D.J. Woods, B. Wu, Y. Wu, Z. Wu, D. Wuttke, Y. Xia, J. Xie, W. Xie, F. Xu, L. Xu, S. Xu, S. Xu, Y. Xu, R. Xue, B. Yalcinkaya, R. Yang, Y. Yang, H. Yao, S. Yao, Y. Ke, O. Yapar, E. Yelagin, I. Yeung, E.D. Yilmaz, L. Yilmaz, W. Yoo, S. Yoon, S. Youn, A. Young, J. Yu, J. Yu, J.V. Yu, L. Yu, H. Yuan, Y. Yuan, L. Yue, A. Zednik, Y. Zeinali, S. Zhai, X. Zhan, A. Zhang, C. Zhang, H. Zhang, H. Zhang, H. Zhang, J.H. Zhang, L. Zhang, Q. Zhang, R. Zhang, R. Zhang, S. Zhang, S. Zhang, W.W. Zhang, Z. Zhang, J. Zhao, X. Zhao, Z. Zhao, J. Zheng, Y. Zheng, Z. Zheng, A. Zhou, H. Zhu, J. Zhu, Y. Zhu, C. Zihlmann, M. Zoican, R. Zultan, Z. Zuo. - In: MANAGEMENT SCIENCE. - ISSN 0025-1909. - 70:3(2024 Mar 01), pp. 1343-1356. [10.1287/mnsc.2023.03556]

Reproducibility in Management Science

A. Filippin
Membro del Collaboration Group
;
D. Grieco;
2024

Abstract

With the help of more than 700 reviewers, we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019. When considering only articles for which data accessibility and hardware and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for 29% of articles, at least part of the data set was not accessible to the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%. These figures represent a significant increase compared with the period before the introduction of the disclosure policy, where only 12% of articles voluntarily provided replication materials, of which 55% could be (largely) reproduced. Substantial heterogeneity in reproducibility rates across different fields is mainly driven by differences in data set accessibility. Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, and software and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies and suggest potential avenues for enhancing their effectiveness.
crowd science; replication; reproducibility;
Settore ECON-02/A - Politica economica
1-mar-2024
dic-2023
Article (author)
File in questo prodotto:
File Dimensione Formato  
38_reproducibility-in-management-science.pdf

accesso riservato

Tipologia: Publisher's version/PDF
Dimensione 2.96 MB
Formato Adobe PDF
2.96 MB Adobe PDF   Visualizza/Apri   Richiedi una copia
manscirep_20231222.pdf

accesso aperto

Tipologia: Pre-print (manoscritto inviato all'editore)
Licenza: Creative commons
Dimensione 1.46 MB
Formato Adobe PDF
1.46 MB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2434/1158431
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 24
  • ???jsp.display-item.citation.isi??? 22
  • OpenAlex ND
social impact