Value of search results as a whole as the best single measure of information retrieval performance

摘要

Value of search results as a whole, a utility measure, was found to be the best single measure of interactive information performance (success) among the 20 measures selected for study (Su, 1991). The study suggests that this measure provides a simple way for system comparison and eliminates problems of IR evaluation with multiple measures. Value of search results as a whole is a measure which asks for a user's rating on the usefulness of a set of search results based on a Likert 7-point scale. This measure gives a numeric basis for comparing information retrieval performance but it does not indicate why one set of search results is rated more useful than others or how search results or systems can be improved to be more useful or successful. To further our understanding of the measure and to enhance its usefulness, the current paper explores two research questions: (1) What are the conceptual categories or dimensions of the users' reasons for assigning particular ratings on the value of search results? (2) What are the relationships between these dimensions of value and the dimensions of success identified in the earlier study (Su, 1991)? Verbal data collected by the previous study will be analyzed to answer the research questions. The current paper presents results from this analysis and discusses implications for theory and practice. It also compares current findings with those from other user criteria studies.