Parametric evaluation of research units with respect to reference profiles

摘要

We introduce a method that jointly considers multiple criteria sorting and ranking. The method derives from a real-world problem of parametric evaluation of research units carried out by the Polish Ministry of Science and Higher Education. It assigns the units to three classes representing different qualities of both acquired effects and activities undertook in the evaluation period. Although units placed in the same class are guaranteed the same level of funding, they are not considered indifferent in the subsequent analysis and, thus, need to be ordered from the best to the worst in each class. A proposed outranking relation compares the units pairwise and the result is exploited so as to get a ranking of the units. The ranking is transformed to class assignments based on the attained comprehensive scores and ranks. To enhance interpretability of the results, we infer two reference profiles (artificial reference research units) separating the classes so that each class accumulates units ranked not worse than the corresponding lower profile and worse than the respective upper profile. The procedure takes into account desired cardinalities of classes, i.e., shares of units that are judged as leading, average, or weak. We discuss several procedures with different ways of inferring the reference profiles and scoring the units. We also analyze robustness of the results.