Although bibliometrics and research evaluation seem to be almost identical, bibliometrics is an autonomous discipline that can have various forms when used as an instrument of quantitative research evaluation. The quantitative evaluation of publication and citation data is now performed in almost all nations around the globe. This is done for various purposes and by various stakeholders. Bibliometrics is used in research performance evaluation, especially in university and government labs, and is also used by policymakers, research directors and administrators, information specialists and librarians, and researchers themselves. In Italy, bibliometrics has become an essential tool in research evaluation, even though several problems, particularly the heterogeneity of research fields (social sciences and humanities vs. natural and life sciences), seem to limit its use in direct comparisons among departments, research groups and single researchers. The need for a system that can support decisions and compare departments in various disciplines is exacerbated in the case of major universities. Starting from these assumptions, several possible solutions – not without difficulty – have been proposed: weighting systems of publication, publication characterization and journal classification (related to the journal’s impact factor for natural and life science), etc. However, the main feature of this approach to bibliometric evaluation is simply counting, which is not always exhaustive. The complex part of the process is to validate the quality of the data obtained from the bibliometric data sources (Web of Science, Scopus, Google Scholar), as well as to analyse and use them. In this scenario, a multidisciplinary research group from the University of Milan, the UNIMIVAL group, in agreement with the guidelines of the Evaluation Board of the same university, has developed a prototype system for Comparative Bibliometric Analysis (ABC). As indicated by the name, the system is able to compare different departments, branches, groups, research areas and single researchers to other comparable benchmark units. The benchmark can be a real department, research group or other entity within other Italian or foreign universities, or it can be designed according to specific needs (discipline, mission, etc.). Furthermore, the system not only asks for the active participation of the department but also guarantees great flexibility in terms of the bibliometric analysis, selection of sources, definition of indicators, definition and choice of units of analysis, etc. Finally, the ABC system goes beyond the limits of wide-ranging, quality bibliometric tools that are based on a single source (Web of Science, Scopus, Google Scholar).

The problem of disciplinary heterogeneity in research evaluation : lesson learned by Unimival researchers during the Comparative Bibliometric Analysis project (ABC) / F. Biolcati Rinaldi, A. Ferrara, L. Pinotti, S. Salini. - In: RIV. RASSEGNA ITALIANA DI VALUTAZIONE. - ISSN 1826-0713. - 16:52(2012), pp. 81-100.

The problem of disciplinary heterogeneity in research evaluation : lesson learned by Unimival researchers during the Comparative Bibliometric Analysis project (ABC)

F. Biolcati Rinaldi;A. Ferrara;L. Pinotti
Penultimo
;
S. Salini
2012

Abstract

Although bibliometrics and research evaluation seem to be almost identical, bibliometrics is an autonomous discipline that can have various forms when used as an instrument of quantitative research evaluation. The quantitative evaluation of publication and citation data is now performed in almost all nations around the globe. This is done for various purposes and by various stakeholders. Bibliometrics is used in research performance evaluation, especially in university and government labs, and is also used by policymakers, research directors and administrators, information specialists and librarians, and researchers themselves. In Italy, bibliometrics has become an essential tool in research evaluation, even though several problems, particularly the heterogeneity of research fields (social sciences and humanities vs. natural and life sciences), seem to limit its use in direct comparisons among departments, research groups and single researchers. The need for a system that can support decisions and compare departments in various disciplines is exacerbated in the case of major universities. Starting from these assumptions, several possible solutions – not without difficulty – have been proposed: weighting systems of publication, publication characterization and journal classification (related to the journal’s impact factor for natural and life science), etc. However, the main feature of this approach to bibliometric evaluation is simply counting, which is not always exhaustive. The complex part of the process is to validate the quality of the data obtained from the bibliometric data sources (Web of Science, Scopus, Google Scholar), as well as to analyse and use them. In this scenario, a multidisciplinary research group from the University of Milan, the UNIMIVAL group, in agreement with the guidelines of the Evaluation Board of the same university, has developed a prototype system for Comparative Bibliometric Analysis (ABC). As indicated by the name, the system is able to compare different departments, branches, groups, research areas and single researchers to other comparable benchmark units. The benchmark can be a real department, research group or other entity within other Italian or foreign universities, or it can be designed according to specific needs (discipline, mission, etc.). Furthermore, the system not only asks for the active participation of the department but also guarantees great flexibility in terms of the bibliometric analysis, selection of sources, definition of indicators, definition and choice of units of analysis, etc. Finally, the ABC system goes beyond the limits of wide-ranging, quality bibliometric tools that are based on a single source (Web of Science, Scopus, Google Scholar).
disciplinary heterogeneity ; research evaluation ; comparative bibliometric analysis ; benchmarking
Settore SPS/07 - Sociologia Generale
Settore ING-INF/05 - Sistemi di Elaborazione delle Informazioni
Settore AGR/18 - Nutrizione e Alimentazione Animale
Settore SECS-S/01 - Statistica
2012
Article (author)
File in questo prodotto:
Non ci sono file associati a questo prodotto.
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2434/230375
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact