A preliminary investigation of the use of the cloze procedure as a measure of program understanding

摘要

Program understanding is an integral part of the testing and maintenance phases of the software life cycle. There have been numerous investigations of the influence of various aspects of a program and the programming process on program comprehension. However, the many different measures of understanding used in these studies make any comparison or analysis difficult. Some of the different measures include time to find a bug, a comprehension quiz, ability to reproduce a functionally equivalent program without notes, time to perform a modification and Halstead's E. All of these have limitations such as inability to measure both low- and high-level understanding, difficulty of administering and objectively grading, or impractical for large or non-trivial programs. Of the measures, the comprehension quiz is probably the most commonly used and accepted measure of program understanding.This paper reports on a controlled experiment that compared the “cloze” procedure and comprehension quiz as measuring program understanding. In a cloze procedure, the subjects are presented a program listing with some of the program tokens (operands, operators, reserved words, single parenthesis or brackets, etc.) replaced with blanks and are required to fill in the blanks.Our experiment tested students in sophomore, junior and senior level computer science courses. These were assumed to correspond to three levels of programming experperience. Each subject was given one of two versions of a sorting program and either a cloze version of the program or a comprehension quiz over it. Results for the cloze procedure closely approximated those of the comprehension quizzes for both programs and each level of experience.These results and the ease of administration and grading make the cloze procedure a potentially attractive means for measuring program understanding.