Skip to main content
    • Aa
    • Aa

User-centered evaluation of adaptive and adaptable systems: a literature review


This literature review focuses on user-centered evaluation (UCE) studies of adaptive and adaptable systems. Usability, perceived usefulness and appropriateness of adaptation are the three most commonly assessed variables. Questionnaires appeared to be the most popular method, followed by interviews and data log analysis. The quality of most questionnaires was questionable, and the reporting of interviews and think-aloud protocols was found to be shallow. Furthermore, data logs need triangulation in order to be useful. The reports encountered lack empirical value. The article models the iterative design process for adaptive and adaptable systems, linked to the goals of UCE: supporting decisions, detecting problems and verifying quality. This model summarizes the variables to be assessed at each stage of the process and the relevant methods to assess them.

Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

E. Ammenwerth , C Iller . & U. Mansmann 2003. Can evaluation studies benefit from triangulation? A case study. International Journal of Medical Informatics 70(2/3),237248.

R. Benbunan-Fich 2001. Using protocol analysis to evaluate the usability of a commercial web site. Information & Management 39(2), 151163.

D Benyon . & D. Murray 1993. Adaptive systems: From intelligent tutoring to autonomous agents. Knowledge-Based Systems 6(4), 197219.

T. Bohnenberger , A. Jameson , A Krüger . & A. Butz 2002. Location-aware shopping assistance: Evaluation of a decision-theoretic approach. Lecture Notes in Computer Science 2411, 155169.

A. Buchauer , U. Pohl , N Kurzel . & R. Haux 1999. Mobilizing a health professional’s Workstation: Results of an evaluation study. International Journal of Medical Informatics 54, 105114.

A. J. Cawsey , R. B Jones . & J. Pearson 2000. The evaluation of a personalised health information system for patients with cancer. User Modeling and User-Adapted Interaction 10, 4772.

K. Cheverst , H. E. Byun , D. Fitton , C. Sas , C Kray . & N. Villar 2005. Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Modeling and User-Adapted Interaction 15, 235273.

D. N. Chin 2001. Empirical evaluation of user models and user-adapted systems. User Modeling and User-Adapted Interaction 11, 181194.

L. J. Cronbach 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16(3),297334.

F. D. Davis , R. P Bagozzi . & P. R. Warshaw 1989. User acceptance of computer technology: A comparison of two theoretical models. Management Science 35, 9821003.

M. D. T De Jong . & P. J. Schellens 1997. Reader-focused text evaluation. An overview of goals and methods. Journal of Business and Technical Communication 11(4), 402432.

K. A Ericsson . & H. A. Simon 1993. Protocol Analysis. MIT Press.

C Gena . & S. Weibelzahl 2007. Usability engineering for the adaptive web. In The Adaptive Web, P. Brusilovsky , A. Kobsa & W. Nejdl (eds). Springer-Verlag, 720–762.

H. R. Hartson , T. S Andre . & R. C. Williges 2003. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction 15(1), 145181.

R. Henderson , D Rickwood . & P. Roberts 1998. The beta test of an electronic supermarket. Interacting with Computers 10, 385399.

M Hertzum . & N. E. Jacobsen 2003. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction 15(1), 183204.

K. Höök 2000. Steps to take before intelligent user interfaces become real. Interacting with Computers 12(4), 409426.

E. T. Hvannberg , E. L. Law , M. K. Lárusdóttir 2007. Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with Computers 19, 225240.

A. L. Jensen , P. S. Boll , I Thysen . & B. K. Pathak 2000. Pl@nteInfo®: A web-based system for personalised decision support in crop management. Computers and Electronics in Agriculture 25, 271293.

E. Kaasinen 2003. User needs for location aware mobile services. Personal Ubiquitous Computing 7(1), 7079.

H. Ketamo 2003. Xtask: An adaptable learning environment. Journal of Computer Assisted Learning 19(3), 360370.

B. A. Kitchenham , S. L. Pfleeger , L. M. Pickard , P. W. Jones , D. C. Hoaglin , K El Emam . & J. Rosenberg 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8), 721734.

J. Kjeldskov , C. Graham , S. Pedell , F. Vetere , S. Howard , S Balbo . & J. Davies 2005. Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behaviour & Information Technology 24(1), 5165.

J. Kramer , S Noronha . & J. Vergo 2000. A user-centered design approach to personalization. Communications of the ACM 43(8), 4548.

L Lentz . & M. D. T. De Jong 1997. The evaluation of text quality: Expert-focused and reader-focused methods compared. IEEE Transactions on Professional Communication 40(3), 224234.

M. Maguire 2001. Methods to support human-centred design. International Journal of Human-Computer Studies 55(4), 587634.

D. L. Morgan 1996. Focus groups. Annual Review of Sociology, 22, 129152.

C. H Muntean . & J. McManis 2006. The value of QoE-based adaptation approach in educational hypermedia: Empirical evaluation. Lecture Notes in Computer Science 4018, 121130.

D. Nahl 1998. Ethnography of novices’ first use of web search engines: Affective control in cognitive processing. Internet Reference Services Quarterly 3(2), 5172.

P. Savage 1996. User interface evaluation in an iterative design process: A comparison of three techniques. Paper presented at CHI'96, Vancouver, Canada.

P. E. Spector 1992. Summated Rating Scale Construction, Sage.

A. Stein 1997. Usability and assessments of multimodal interaction in the SPEAK! System: An experimental case study. The New Review of Hypermedia and Multimedia 3, 159180.

L. T. Su 1992. Evaluation measures for interactive information retrieval. Information Processing and Management 28(4), 503516.

M. J. Van den Haak , M. D. T De Jong . & P. J. Schellens 2003. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology 22(5), 339351.

M. J. Van den Haak , M. D. T De Jong . & P. J. Schellens 2004. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers 16, 11531170.

L. Van Velsen , T Van der Geest . & R. Klaassen 2007. Testing the usability of a personalized system: Comparing the use of interviews, questionnaires and thinking-aloud. Paper Presented at the IEEE Professional Communication Conference, Seattle, USA.

M. Walker , L Takayama . & J. A. Landay 2002. High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Paper Presented at the 46th Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, USA.

S. Weibelzahl 2005. Problems and pitfalls in the evaluation of adaptive systems. In Adaptable and Adaptive Hypermedia Systems, S. Y. Chen & G. D. Magoulas (eds). IRM Press, 285299.

S. R. Alpert , J. Karat , C. M. Karat , C Brodie . & J. G. Vergo 2003. User attitudes regarding a user-adaptive eCommerce web site. User Modeling and User-Adapted Interaction 13, 373396.

M. P. Bange , S. A. Deutscher , D. Larsen , D Linsley . & S. Whiteside 2004. A handheld decision support system to facilitate improved insect pest management in Australian cotton systems. Computers and Electronics in Agriculture 43(2), 131147.

P. Baumgartner , U. Furbach , M Gross-Hardt . & A. Sinner 2004. Living book: Deduction, slicing and interaction. Journal of Automated Reasoning 32(3), 259286.

P. R. Chesnais , M. J Mucklo . & J. A. Sheena 1995. The Fishwrap personalized news system. Paper Presented at the Second International Workshop on Community Networking ‘Integrated Multimedia Services to the Home’ Princeton, USA.

O. Conlan , I O’Keeffe . & S. Tallon 2006. Combining adaptive hypermedia techniques and ontology reasoning to produce dynamic personalized news services. Lecture Notes in Computer Science 4018, 8190, Springer, Herdelberg.

O Conlan . & V. P. Wade 2004. Evaluation of APeLS: An adaptive elearning service based on the multi-model, metadata-driven approach. Lecture Notes in Computer Science 3137, 291295, Springer, Herdelberg.

D. De Roure , W. Hall , S. Reich , G. Hill , M Stairmand . & A. Pikrakis 2001. Memoir: An open framework for enhanced navigation of distributed information. Information Processing and Management 37(1),5374.

A. Díaz , P Gervás . & A. García 2005. Evaluation of a system for personalized summarization of web contents. Lecture Notes in Computer Science 3538, 453462, Springer, Herdelberg.

A. Díaz , P. Gervas , A García . & I. Chacon 2001. Sections, categories and keywords as interest specification tools for personalised news services. Online Information Review 25(3), 149159.

K. Eason , L Yu . & S. Harker 2000. The use and usefulness of functions in electronic journals: The experience of the SuperJournal project. Program 34, 128.

A. Field , P Hartel . & W. Mooij 2001. Personal DJ, an open architecture for personalised content delivery. Paper Presented at the Tenth International Conference World Wide Web, Hong Kong.

K. F. Gates , P. B Lawhead . & D. E. Wilkins 1998. Toward an adaptive www: A case study in customised hypermedia. New Review of Hypermedia and Multimedia 4, 89113.

C. Gena 2002. An empirical evaluation of an adaptive web site. Paper Presented at IUI'02, San Francisco, USA.

C Gena . & I. Torre 2004. The importance of adaptivity to provide onboard services: A preliminary evaluation of an adaptive tourist information service onboard vehicles. Applied Artificial Intelligence 18(6), 549580.

D Goren-Bar . & T. Kuflik 2004. Don’t miss-r: Recommending restaurants through an adaptive mobile system. Paper Presented at the Ninth International Conference on Intelligent User Interface, Funchal, Portugal.

I. Graziola , F. Pianesi , M Zancanaro . & D. Goren-Bar 2005. Dimensions of adaptivity in mobile systems: Personality and people’s attitudes. Paper Presented at IUI'05, San Diego, USA.

P. Gregor , A. Dickinson , A Macaffer . & P. Andreasen 2003. Seeword: a personal word processing environment for dyslexic computer users. British Journal of Educational Technology 34(3), 341355.

M Hatala . & R. Wakkary 2005. Ontology-based user modeling in an augmented audio reality system for museums. User Modeling and User-adapted Interaction 15(3/4), 339380.

K. Höök 1997. Evaluating the utility and usability of an adaptive hypermedia system. Paper Presented at IUI '97, Orlando, USA.

T. Isobe , M. Fujiwara , H. Kaneta , T Morita . & N. Uratani 2005. Development of a TV reception navigation system personalized with viewing habits. IEEE Transactions on Consumer Electronics 51(2), 665674.

T. Isobe , M. Fujiwara , H. Kaneta , N Uratani . & T. Morita 2003. Development and features of a TV navigation system. IEEE Transactions on Consumer Electronics 49(4), 10351042.

C. M. Karat , C. Brodie , J. Karat , J Vergo . & S. R. Alpert 2003. Personalizing the user experience on ibm.Com. IBM Systems Journal 42(4), 686701.

F. S. H. Krauss 2003. Methodology for remote usability activities: A case study. IBM Systems Journal 42(4), 582593.

M. J O’Grady . & G. M. P. O’Hare 2004. Enabling customized & personalized interfaces in mobile computing. Paper Presented at IUI'04, Madeira, Portugal.

A. G. Pateli , G. M Giaglis . & D. D. Spinellis 2005. Trial evaluation of wireless info-communication and indoor location-based services in exhibition shows. Advances in Informatics 3746, 199210.

C. Romero , S. Ventura , C. Hervas , & P. De Bra 2006. An authoring tool for building both mobile adaptable tests and web-based adaptive or classic tests. Lecture Notes in Computer Science 4018, 203212, Springer, Herdelberg.

B Smyth . & P. Cotter 2000. A personalized television listings service. Communications of the ACM 43(8), 107111.

C. Sodergard , M. Aaltonen , S. Hagman , M. Hiirsalmi , T. Jarvinen , E. Kaasinen , T. Kinnunen , J. Kolari , J Kunnas . & A. Tammela 1999. Integrated multimedia publishing: Combining TV and newspaper content on personal channels. Computer Networks: the International Journal of Computer and Telecommunications Networking 31, 11111128.

C Stary . & A. Totter 2003. Measuring the adaptability of universal accessible systems. Behaviour & Information technology 22(2), 101116.

K. Stathis , O De Bruijn . & S. Macedo 2002. Living memory: Agent-based information management for connected local communities. Interacting with Computers 14(6), 663688.

C. G. Thomas 1993. Design, implementation and evaluation of an adaptive user interface. Knowledge-Based Systems 6(4), 230238.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The Knowledge Engineering Review
  • ISSN: 0269-8889
  • EISSN: 1469-8005
  • URL: /core/journals/knowledge-engineering-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 93 *
Loading metrics...

Abstract views

Total abstract views: 392 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 19th August 2017. This data will be updated every 24 hours.