David M. Hilbert
David F. Redmiles
Abstract
Modern window-based user interface systems generate user interface events as natural products of their normal operation. Because such events can be automatically captured and because they indicate user behavior with respect to an application's user interface, they have long been regarded as a potentially fruitful source of information regarding application usage and usability. However, because user interface events are typically voluminos and rich in detail, automated support is generally required to extract information at a level of abstraction that is useful to investigators interested in analyzing application usage or evaluating usability. This survey examines computer-aided techniques used by HCI practitioners and researchers to extract usability-related information from user interface events. A framework is presented to help HCI practitioners and researchers categorize and compare the approaches that have been, or might fruitfully be, applied to this problem. Because many of the techniques in the research literature have not been evaluated in practice, this survey provides a conceptual evaluation to help identify some of the relative merits and drawbacks of the various classes of approaches. Ideas for future research in this area are also presented. This survey addresses the following questions: How might user interface events be used in evaluating usability? How are user interface events related to other forms of usability data? What are the key challenges faced by investigators wishing to exploit this data? What approaches have been brought to bear on this problem and how do they compare to one another? What are some of the important open research questions in this area?
- ABBOTT, A. 1990. A primer on sequence methods. Organ. Sci. 4.]]Google Scholar
- AGRAWAL, R., ARNING, A., BOLLINGER, T., MEHTA, M., SHAFER,J.,AND SRIKANT, R. 1996. The Quest data mining system. In Proceedings of the 2nd International Conference on Knowledge Discovery in Databases and Data Mining.]]Google Scholar
- AHO,A.V.,KERNIGHAN,B.W.,AND WEINBERGER,P.J. 1988. The AWK Programming Language. Addison-Wesley, Reading, MA.]] Google Scholar
- ALLISON,P.D.AND LIKER, J. K., 1982. Analyzing sequential categorical data on dyadic interaction: A comment on Gottman. Psychological Bulletin 2.]]Google Scholar
- AQUEDUCT SOFTWARE. 1998. AppScope Web Pages. URL: http://www.aqueduct.com/.]]Google Scholar
- BADRE,A.N.AND SANTOS, P. J. 1991a. CHIME: A Knowledge-Based Computer-Human Interaction Monitoring Engine. Tech Rept. GIT-GVU-91- 06.]]Google Scholar
- BADRE,A.N.AND SANTOS, P. J. 1991b. A Knowledge-Based System for Capturing Human-Computer Interaction Events: CHIME. Tech. Rept. GIT-GVU- 91-21.]]Google Scholar
- BADRE,A.N.,GUZDIAL, M., HUDSON,S.E.,AND SANTOS, P. J. 1995. A user interface evaluation environment using synchronized video, visualizations, and event trace data. J. of Software Qual. 4.]]Google Scholar
- BAECKER,R.M,GRUDIN, J., BUXTON,W.A.S.,AND GREENBERG, S., Eds. 1995. Readings in Human-Computer Interaction: Toward the Year 2000. Morgan Kaufmann, San Mateo, CA.]] Google Scholar
- BALBO, S. 1996. EMA: Automatic Analysis Mechanism for the Ergonomic Evaluation of User Interfaces. CSIRO Tech. rep.]]Google Scholar
- BATES, P. C. 1995. Debugging heterogeneous distributed systems using event-based models of behavior. ACM Trans. on Comput. Syst. 13,1.]] Google Scholar
- BELLOTTI, V. 1990. A framework for assessing applicability of HCI techniques. In Proceedings of INTERACT '90.]] Google Scholar
- BUXTON, W., LAMB, M., SHEMAN,D.,AND SMITH,K. 1983. Towards a comprehensive user interface management system. In Proceedings of SIGGRAPH '83.]] Google Scholar
- CHANG,E.AND DILLON, T. S. Automated usability testing. In Proceedings of INTERACT '97.]] Google Scholar
- CHEN, J. 1990. Providing intrinsic support for user interface monitoring. In Proceedings of INTER- ACT '90.]] Google Scholar
- COOK,J.E.AND WOLF, A. L. 1994. Toward metrics for process validation. In Proceedings of ICSP '94.]]Google Scholar
- COOK,J.E.AND WOLF, A. L. 1995. Automating process discovery through event-data analysis. In Proceedings of ICSE '95.]] Google Scholar
- COOK,J.E.AND WOLF, A. L. 1997. Software Process Validation: Quantitatively Measuring the Correspondence of a Process to a Model. Tech. Rep. CU-CS-840-97, Dept. of Computer Science, Univ. of Colorado at Boulder.]]Google Scholar
- COOK, R., KAY, J., RYAN,G.,AND THOMAS, R. C. 1995. A toolkit for appraising the long-term usability of a text editor. Software Qual. J. 4,2.]]Google Scholar
- CUOMO, D. L. 1994. Understanding the applicability of sequential data analysis techniques for analysing usability data. In Usability Laboratories Special Issue of Behavior and Information Technology, J. Nielsen, Ed., vol. 13, no.1 & 2.]]Google Scholar
- CYPHER, A. 1991. Eager: programming repetitive tasks by example. In Proceedings of CHI '91.]] Google Scholar
- CYPHER, A., Ed. 1993. Watch what I do: Programming by Demonstration. MIT Press, Cambridge MA.]] Google Scholar
- DODGE,M.AND STINSON, C. 1999. Running Microsoft Excel 2000. Microsoft Press.]] Google Scholar
- DOUBLEDAY, A., RYAN, M., SPRINGETT, M., AND SUTCLIFFE, A. 1997. A comparison of usability techniques for evaluating design. In Proceedings of DIS '97.]] Google Scholar
- ELGIN, B. 1995. Subjective usability feedback from the field over a network. In Proceedings of CHI '95.]] Google Scholar
- ERGOLIGHT USABILITY SOFTWARE. 1998. Operation Recording Suite (EORS) and Usability Validation Suite (EUVS) Web pages. URL: http:/ /www.ergolight.co.il/.]]Google Scholar
- FARAONE,S.V.AND DORFMAN, D. D. 1987. Lag sequential analysis: Robust statistical methods. Psychological Bulletin 101.]]Google Scholar
- FEATHER,M.S.,NARAYANASWAMY, K., COHEN,D., AND FICKAS, S. 1997. Automatic monitoring of software requirements. Research Demonstration. In Proceedings of ICSE '97.]] Google Scholar
- FICKAS,S.AND FEATHER, M. S. 1995. Requirements monitoring in dynamic environments. In IEEE International Symposium on Requirements Engineering.]] Google Scholar
- FINLAY,J.AND HARRISON, M. 1990. Pattern recognition and interaction models. In Proceedings of INTERACT '90.]] Google Scholar
- FISHER, C. 1987. Advancing the study of programming with computer-aided protocol analysis. In Empirical Studies of Programmers, 1987 Workshop, G. Olson, E. Soloway, and S. Sheppard, Eds. Ablex, Norwood, NJ.]] Google Scholar
- FISHER, C. 1991. Protocol Analyst's Workbench: Design and Evaluation of Computer-Aided Protocol Analysis. Unpublished Ph.D. thesis, Carnegie Mellon University, Dept. of Psychology, Pittsburgh, PA.]]Google Scholar
- FISHER,C.AND SANDERSON, P. 1996. Exploratory sequential data analysis: Exploring continuous observational data. Interactions 3, 2, ACM Press.]] Google Scholar
- FITTS, P. M. 1964. Perceptual motor skill learning. In Categories of human learning, A. W. Melton, Ed. Academic Press, New York, NY.]]Google Scholar
- FULL CIRCLE SOFTWARE. 1998. Talkback Web pages. URL: http://www.fullsoft.com/.]]Google Scholar
- GIRGENSOHN, A., REDMILES,D.F.,AND SHIPMAN, F. M. III. 1994. Agent-based support for communication between developers and users in software design. In Proceedings of the Knowledge-Based Software Engineering Conference '94. Monterey, CA, USA.]]Google Scholar
- GOODMAN, D. 1998. Complete HyperCard 2.2 Hand-book. ToExcel.]] Google Scholar
- GOTTMAN,J.M.AND ROY, A. K. 1990. Sequential analysis: A guide for behavioral researchers. Cambridge University Press, Cambridge, England.]]Google Scholar
- GRUDIN, J. 1992. Utility and usability: Research issues and development contexts. Interacting with comput. 4,2.]]Google Scholar
- GUZDIAL, M. 1993. Deriving Software Usage Patterns from Log Files. Tech. Rept. GIT-GVU-93- 41.]]Google Scholar
- GUZDIAL,M,WALTON, C., KONEMANN, M., AND SOLOWAY, E. 1993. Characterizing Process Change Using Log File Data. Tech. Rep. GIT-GVU-93- 44.]]Google Scholar
- GUZDIAL, M., SANTOS, P., BADRE, A., HUDSON,S.,AND GRAY, M. 1994. Analyzing and visualizing log files: Acomputational science of usability. In Presented at HCI Consortium Workshop.]]Google Scholar
- HARTSON, H. R., CASTILLO,J.C.,KELSO,J.,AND NEALE, W. C. 1996. Remote evaluation: The network as an extension of the usability laboratory. In Proceedings of CHI '96.]] Google Scholar
- HELANDER, M., Ed. 1998. Handbook of human-computer interaction. Elsevier Science Publish-ers B.V., North Holland.]] Google Scholar
- HEWLETT PACKARD. 1998. Application Response Measurement API. URL: http://www.hp.com/ openview/rpm/arm/.]]Google Scholar
- HILBERT,D.M.AND REDMILES, D. F. 1998a. An approach to large-scale collection of application usage data over the Internet. In Proceedings of ICSE '98.]] Google Scholar
- HILBERT,D.M.AND REDMILES, D. F. 1998b. Agents for collecting application usage data over the Internet. In Proceedings of Autonomous Agents '98.]] Google Scholar
- HILBERT, D. M., ROBBINS,J.E.,AND REDMILES,D.F., 1997. Supporting Ongoing User Involvement in Development via Expectation-Driven Event Monitoring. Tech Report UCI-ICS-97-19, Dept. of Information and Computer Science, Univ. of California, Irvine.]]Google Scholar
- HIRSCHBERG, D. S. 1975. A linear space algorithm for computing maximal common subsequences. Commun of the ACM 18.]] Google Scholar
- HOIEM,D.E.AND SULLIVAN, K. D. 1994. Designing and using integrated data collection and analysis tools: challenges and considerations. In Usability Laboratories Special Issue of Behavior and Information Technology, J. Nielsen, Ed., vol. 13,no.1 &2.]]Google Scholar
- HOPPE, H. U. 1988. Task-oriented parsing: A diagnostic method to be used by adaptive systems. In Proceedings of CHI '88.]] Google Scholar
- JOHN,B.E.AND KIERAS, D. E. 1996a. The GOMSfamily of user interface analysis techniques: Comparison and contrast. ACM Trans. on Comput.- Hum. Interaction 3,4.]] Google Scholar
- JOHN,B.E.AND KIERAS, D. E. 1996b. Using GOMS for user interface design and evaluation: which technique? ACMTrans. on Comput.-Hum. Interaction 3,4.]] Google Scholar
- KAY,J.AND THOMAS, R. C. 1995. Studying longterm system use. Commun. of the ACM 38,7.]] Google Scholar
- KOSBIE,D.S.AND MYERS, B. A. 1994. Extending programming by demonstration with hierarchical event histories. In Proceedings of East-West Human Computer Interaction '94.]] Google Scholar
- KRISHNAMURTHY,B.AND ROSENBLUM, D. S. 1995. Yeast: A general purpose event-action system. IEEE Trans. on Software Eng. 21, 10.]] Google Scholar
- LECEROF,A.AND PATERNO, F. 1998. Automatic support for usability evaluation. IEEE Trans. on Software Engin. 24, 10.]] Google Scholar
- LEE, B. 1996. Remote diagnostics and product lifecycle monitoring for high-end appliances: a new Internet-based approach utilizing intelligent software agents. In Proceedings of the Appliance Manufacturer Conference.]]Google Scholar
- LEWIS,R.AND STONE, M., Ed. 1999. Mac OS in a Nutshell. O'Reilly and Associates.]] Google Scholar
- LOWGREN,J.AND NORDQVIST, T. 1992. Knowledgebased evaluation as design support for graphical user interfaces. In Proceedings of CHI '92.]] Google Scholar
- MACLEOD,M.AND RENGGER, R. 1993. The Development of DRUM: A Software Tool for Video-assisted Usability Evaluation. In Proceedings of HCI '93.]]Google Scholar
- MANSOURI-SAMANI,M.AND SLOMAN, M. 1997. GEM: A generalized event monitoring language for distributed systems. IEE/BCS/IOP Distributed Syst. Eng. J. 4,2.]]Google Scholar
- MERCURY INTERACTIVE. 1998. WinRunner and XRunner Web Pages. URL: http://www.merc-int.com/.]]Google Scholar
- MORAN, T. P. 1981. The command language grammar: A representation for the user interface of interactive computer systems. Int. J. of Man- Machine Studies, 15.]]Google Scholar
- NEAL,A.S.AND SIMONS,R.M.PLAYBACK: A method for evaluating the usability of software and its documentation. In Proceedings of CHI '83. 1983.]] Google Scholar
- NIELSEN, J. 1986. A virtual protocol model for computer-human interaction. Int. J. of Man- Machine Studies 24.]] Google Scholar
- NIELSEN, J. 1993. Usability engineering. Academic Press/AP Professional, Cambridge, MA.]] Google Scholar
- NIELSEN,J.AND MACK, R. L., Eds. 1994. Usability inspection methods. Wiley, New York.]] Google Scholar
- NYE,A.AND O'REILLY, T. 1992. X Toolkit Intrinsics Programming Manual for X11, Release 5. O'Reilly and Associates.]] Google Scholar
- OLSEN,D.R.AND HALVERSEN, B. W. 1988. Interface usage measurements in a user interface management system. In Proceedings of UIST '88.]] Google Scholar
- OLSON, G. M., HERBSLEB,J.D.,AND RUETER,H.H. 1994. Characterizing the sequential structure of interactive behaviors through statistical and grammatical techniques. Hum.-Comput. Interaction Special Issue on ESDA, Vol. 9.]]Google Scholar
- PAYNE,S.G.AND GREEN, T. R. G. 1986. Taskaction grammars: A model of the mental representation of task languages. Hum.-Comput. Interaction, Vol. 2.]]Google Scholar
- PENTLAND, B. T. 1994. A grammatical model of organizational routines. Administrative Sci. Quarterly.]]Google Scholar
- PENTLAND, B. T. 1994. Grammatical models of organizational processes. Organ. Sci.]]Google Scholar
- PETZOLD, C. 1998. Programming Windows. Microsoft Press.]] Google Scholar
- POLSON,P.G.,LEWIS, C., RIEMAN,J.,AND WHARTON, C. 1992. Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. Int. J. Man-Mach. Studies 36, 5, 741- 773.]] Google Scholar
- PREECE, J., ROGERS, Y., SHARP, H., BENYON, D., HOLLAND, S., AND CAREY, T. 1994. Human-computer inter-action. Addison-Wesley, Wokingham, UK.]] Google Scholar
- ROSENBLUM, D. S. 1991. Specifying concurrent systems with TSL. IEEE Software, 8,No.3.]] Google Scholar
- RUBIN, C. 1999. Running Microsoft Word 2000. Microsoft Press.]] Google Scholar
- SACKETT, G. P. 1978. Observing Behavior, Vol. University Park Press, Baltimore, MD.]]Google Scholar
- SANDERSON,P.M.AND FISHER, C. 1994. Exploratory sequential data analysis: foundations. Hum.- Comput. Interaction Special Issue on ESDA, 9.]]Google Scholar
- SANDERSON, P. M., SCOTT,J.J.P,JOHNSTON, T., MAINZER, J., WATANABE,L.M.,AND JAMES, J. M. 1994. Mac-SHAPA and the enterprise of Exploratory Se-quential Data Analysis (ESDA). International of Hum.-Comput. Studies, 41.]] Google Scholar
- SANTOS,P.J.AND BADRE, A. N. 1994. Automatic chunk detection in human-computer interaction. In Proceedings of Workshop on Advanced Visual Interfaces AVI '94. Also available as Tech Report GIT-GVU-94-4.]] Google Scholar
- SCHIELE,F.AND HOPPE, H. U. 1990. Inferring task structures from interaction protocols. In Proceedings of INTERACT '90.]] Google Scholar
- SELBY,R.W.,PORTER, A. A., SCHMIDT,D.C.,AND BERNEY, J. 1991. Metric-driven analysis and feedback systems for enabling empirically guided software development. In Proceedings of ICSE '91.]] Google Scholar
- SIOCHI,A.C.AND EHRICH, R. W. 1991. Computer analysis of user interfaces based on repetition in transcripts of user sessions. ACM Trans. on Inf. Syst.]] Google Scholar
- SIOCHI,A.C.AND HIX, D. 1991. A study of computersupported user interface evaluation using maximal repeating pattern analysis. In Proceedings of CHI '91.]] Google Scholar
- SMILOWITZ,E.D.,DARNELL,M.J.,AND BENSON,A.E. 1994. Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests. In Usability Laboratories Special Issue of Behavior and Information Technology,J. Nielsen, Ed., Vol. 13, No.1 &2.]]Google Scholar
- SUN MICROSYSTEMS. 1998. SunTest JavaStar Web Pages. URL: http://www.sun.com/suntest/.]]Google Scholar
- SWEENY, M., MAGUIRE, M., AND SHACKEL, B. 1993. Evaluating human-computer interaction: A framework. Int. J. of Man-Machine Stud., 38.]] Google Scholar
- TAYLOR, M. M. 1988a. Layered protocols for computer-human dialogue I: Principles. Int. J. of Man-Machine Stud. 28.]] Google Scholar
- TAYLOR, M. M. 1988b. Layered protocols for computer-human dialogue II: Some practical issues. Int. J. of Man-Machine Stud. 28.]] Google Scholar
- TAYLOR,R.N.AND COUTAZ, J. 1994. workshop on software engineering and human-computer interaction: Joint research issues. In Proceedings of ICSE '94.]] Google Scholar
- TIBCO, 1998. HAWK Enterprise Monitor Web Pages. URL: http://www.tibco.com/.]]Google Scholar
- UEHLING,D.L.AND WOLF, K. 1995. User Action Graphing Effort (UsAGE). In Proceedings of CHI '95.]] Google Scholar
- USER MODELING INC. (UM Inc.), 1998. Home Page. URL: http://um.org/.]]Google Scholar
- WEILER, P. 1993. Software for the usability lab: a sampling of current tools. In Proceedings of IN- TERCHI '93.]] Google Scholar
- WHITEFIELD, A., WILSON,F.,AND DOWELL, J. 1991. A framework for human factors evaluation. Behav. and Inf. Technol., 10,1.]]Google Scholar
- WOLF,A.L.AND ROSENBLUM, D. S. 1993. A study in software process data capture and analysis. In Proceedings of the Second International Conference on Software Process.]]Google Scholar
- ZUKOWSKI,J.AND LOUKIDES, M., Ed. 1997. Java Awt Reference. O'Reilly and Associates.]] Google Scholar
Recommendations
The University of Alberta user interface management system
In this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...
The University of Alberta user interface management system
SIGGRAPH '85: Proceedings of the 12th annual conference on Computer graphics and interactive techniquesIn this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...
Building a user-derived interface
Many human-computer interfaces are designed with the assumption that the user must adapt to the system, that users must be trained and their behavior altered to fit a given interface. The research presented here proceeds from the alternative assumption: ...
Reviews
Dara Lee Howard
This survey and its resulting framework provide a useful compilation of information to aid designers and students of software interfaces. The framework provides a conceptual evaluation and identifies relative merits and drawbacks of classes of event data capture. A framework is constructed to categorize, compare, and evaluate the relative strengths and limitations of approaches to capture the natural products of the normal operation of window-based user interface systems. Synchronization and searching, transformation, counts and summary statistics, sequence detection, sequence comparison, sequence characterization, visualization, and integrated support constitute the elements of the comparison framework. Each element is briefly described, examples are given, related work is identified where appropriate, and strengths and limitations are pointed out. Authors note that "Synch and search techniques are among the most mature technologies for exploiting UI event data in usability evaluations" (p. 414). They found that very few approaches support transformation, a critical subprocess to meaningful use of extracted usability information. In addition, they report that the more compelling techniques (e.g., exploratory sequential data analysis and Markov- and Grammar-based sequence characterization techniques) require the most human intervention whereas the most automated techniques tend to be less compelling and most unrealistic in their assumptions. Future work is recommended in the areas of capturing and evaluating event data in software testing and debugging, of mapping between lower and higher level events of interest, of automated discovery and validation of patterns in large collections of event data, and in domains such as identifying, diagnosing, and repairing breakdowns in complex system operations.
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments