Exploring the relationship between user satisfaction and relevance in information systems

摘要

The goal of this research was to better understand the relationship between relevance and user-satisfaction, the two predominant aspects of user-based performance in information systems. This project unconfounds relevance and user-satisfaction assessments of system performance at the retrieved item level. To minimize the idiosyncracies of any one system, a generalized, naturalistic information system was employed in this study. Respondents completed sense-making timeline questionnaires in which they described a recent need they had for geographic information. Retrieved documents from the generalized system consisted of the responses users obtained while resolving their information needs. Respondents directly provided process, product, cost-benefit, and overall satisfaction assessments with the generalized geographic system. Relevance judgments of retrieved items were obtained through content analysis from sense-making questionnaires as a secondary observation technique. The content analysis provided relevance values on both five-category and two-category scales. Results indicate that relevance has strong relationships (gamma values from 0.74 to 0.89) with process, product and overall user satisfaction measures while relevance and cost-benefit satisfaction measures have no significant relationship (gamma value of 0.049). This analysis also indicates that neither relevance nor user-satisfaction subsumes the other concept, and that understanding the proper units of analysis for these measures helps resolve the paradox of the management information system and information science literatures not informing each other concerning user-based information system performance measures.