Quality related publication categories in social sciences and humanities, based on a university's peer review assessments

 Historische Musikwissenschaft

of 2
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
"Rons, N. and De Bruyn, A. (2010). Quality related publication categories in social sciences and humanities, based on a university's peer review assessments. In: Book of Abstracts, 11th International Conference on Science and Technology
  Poster abstract, accepted for publication in:Book of Abstracts of the 11th International Conference on Science and Technology Indicators, 2010, pp. 229-230   Quality related publication categories in social sciences andhumanities, based on a university's peer review assessments  Nadine Rons* 1 , Arlette De Bruyn 2   1  Nadine.Rons@vub.ac.be, 2  Arlette.De.Bruyn@vub.ac.be Vrije Universiteit Brussel (VUB), B-1050 Brussels (Belgium) Introduction Bibliometric analysis has firmly conquered its place as an instrument for evaluationand international comparison of performance levels. Consequently, differences incoverage by standard bibliometric databases installed a dichotomy between on the onehand the well covered 'exact' sciences, and on the other hand most of the socialsciences and humanities with a more limited coverage (Nederhof, 2006). Also thelatter domains need to be able to soundly demonstrate their level of performance andclaim or legitimate funding accordingly. An important part of the output volume insocial sciences appears as books, book chapters and national literature (Hicks, 2004).To proceed from publication data to performance measurement, quantitative publication counts need to be combined with qualitative information, for examplefrom peer assessment or validation (European Expert Group on Assessment of University-Based Research, 2010), to identify those categories that represent researchquality as perceived by peers. An accurate focus is crucial in order to stimulate,recognize and reward high quality achievements only. This paper demonstrates howsuch a selection of publication categories can be based on correlations with peer  judgments. It is also illustrated that the selection should be sufficiently precise, toavoid subcategories negatively correlated with peer judgments. The findings indicatethat, also in social sciences and humanities, publications in journals with aninternational referee system are the most important category for evaluating quality.Book chapters with international referee system and contributions in internationalconference proceedings follow them. Method and material Ratings by peers and publication counts per full time equivalent leading staff (linkedto promoter and funding opportunities) were collected from assessments per discipline by international expert panels at the Vrije Universiteit Brussel (Rons et al.,2008). The evaluations in social sciences and humanities involved 6 disciplines, 56teams, near 500 full time equivalent researchers and 58 experts from 10 countries, andwere conducted between 1999 and 2009. The 23 available publication categories spanthe total range from scientific publications to categories aimed at a professional and a broad audience. Categories that are only present for a minority of the teams in adiscipline are not taken into account to avoid accidental occurrences. The 8 collected,interrelated peer rating categories are the overall evaluation score and scores onscientific merit, planning, innovation, team quality, feasibility, productivity andscientific impact. Correlations for the social sciences and humanities as a whole arecalculated after normalization per discipline. The same methodology has been applied before to a different set of disciplines and also to other types of performance measures(Rons and De Bruyn, 2007).  Poster abstract, accepted for publication in:Book of Abstracts of the 11th International Conference on Science and Technology Indicators, 2010, pp. 229-230   Table 1. Significant correlations with peer ratings per scientific publication category Medium:Scope:   Book Chap  Int   Jour  Int   Chap Nat   Jour Nat   Chap no  Jour no  Edit Conf   Int   Abst  Int   Conf  Oth  Abst Oth   Social Sc. & Humanities 8+ 4- 7- 1- 6- 1-Psychology & Educat. Sc. 2- 7+ 2-History 3- 8- 8-Political & Social Sciences 1+ 3+ 1+Philosophy & Letters 3- 3+ 5+ 5- 4- 5+ 1- 6-Economics 5+ 2- 8-Law (particular categories) 3+ 6+Number of peer rating categories (out of 8), per discipline, and for all social sciences and humanities disciplinescombined, for which significantly positive or negative correlations are found with the publication category. Medium & scope:   Book s, book Chap ters and Jour nal articles with    Int  ernational, Nat  ional or  no referee system ; Edit ed books or journals; Communications at    Int  ernational or  Oth er    conferences , integrally published ( Conf  )or published as abstract or not ( Abst ). Exception: particular publication categories for Law. Observations and conclusions Table 1 highlights significantly positive and negative correlations with one or more peer rating categories at a 5% confidence level, for publications in books, journals andconference proceedings. Publication categories with an international dimension, in particular journal articles, show no other than positive correlations, while no other than negative or mixed correlations are found for the other categories. This indicatesthat in social sciences and humanities these 'international' publication categories can be used as legitimate general counterparts for the international journal publicationsfocused on in exact sciences, with the intrinsically largely locally oriented disciplineof Law as the exception to the rule. This also pinpoints the international dimension asan important criterion for selection or weighting of publication categories in performance based funding or evaluation systems, in order to stimulate quality as perceived by peers. In a context of best practices, it supports the rationale that,regardless of the discipline, high quality research performance requires that results besubmitted to a sufficient extent to the scrutiny of the international researchcommunity. The particularly strong correlations with peer judgments found for thecategory of  international journals suggest that this is the most effective publicationmedium for this purpose.  References Expert Group on Assessment of University-Based Research (2010).  Assessing Europe’sUniversity-Based Research . DG Research, EUR 24187 EN.Hicks, D. (2004). The Four Literatures of Social Science. In H.F. Moed, W. Glänzel andU. Schmoch (Eds.),  Handbook of Quantitative Science and Technology Research (pp.473-496). Kluwer Academic Publishers. Nederhof, A.J. (2006). Bibliometric monitoring of research performance in the SocialSciences and the Humanities: A review. Scientometrics 66(1), 81-100.Rons, N., De Bruyn, A. (2007). Quantitative CV-based indicators for research quality,validated by peer review. In D. Torres-Salinas and H. Moed (Eds.),  Proceedings of  ISSI 2007, 11th International Conference of the International Society for Scientometrics and Informetrics, CSIC, Madrid, Spain, 25-27 June 2007 (pp. 930-931).Rons, N., De Bruyn, A., Cornelis, J. (2008). Research evaluation per discipline: a peer-review method and its outcomes.  Research Evaluation 17(1), 45-57.
Related Search
Similar documents
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks