Determining the effectiveness of retrieval algorithms

摘要

A new effectiveness measure is proposed to circumvent the problems associated with the classical recall and precision measures. It is difficult to evaluate systems that filter extremely dynamic information; the determination of all relevant documents in a real life collection is hardly affordable, and the specification of binary relevance assessments is often problematic. The new measure relies on a statistical approach with which two retrieval algorithms are compared. In contrast to the classical recall and precision measures, the new measure requires only relative judgments, and the reply of the retrieval system is compared directly with the information need of the user rather than with the query. The new measure has the added ability to determine an error probability that indicates how stable the usefulness measure is. Using a test collection of abstracts from CACM, it is shown that our new measure is also capable of disclosing the effect of manually assigned descriptors and yields a result similar to that of the traditional recall and precision measures.