On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluation

作者：

Highlights：

• We collect relevance judgments for 4 crowdsourced scales.

• We compare the crowd judgments with two expert-labeled datasets.

• We study the effect on IR evaluation in terms of system effectiveness and topic ease.

• We release the data publicly.

摘要

•We collect relevance judgments for 4 crowdsourced scales.•We compare the crowd judgments with two expert-labeled datasets.•We study the effect on IR evaluation in terms of system effectiveness and topic ease.•We release the data publicly.

论文关键词：Relevance scales,Crowdsourcing,Information Retrieval evaluation,Relevance assessment

论文评审过程：Received 6 April 2021, Revised 21 June 2021, Accepted 5 July 2021, Available online 28 July 2021, Version of Record 28 July 2021.

论文官网地址：https://doi.org/10.1016/j.ipm.2021.102688