PAOLO CREMONESI, FRANCA GARZOTTO, PEARL PU
ABSTRACT
Abstract: Recommender Systems (RSs) help users search large amounts of contents by allowing them to identify the items that are likely to be more attractive or useful. RSs play an important role in many domains (e.g., e-commerce, e-tourism, social networks, entertainment), as they can potentially augment the users’ trust towards an application and orient their decisions or actions towards specific directions. The goal of this tutorial is to give participants a solid background of how to design and evaluate RSs, with a particular focus on user experience aspects, and to provide pragmatic guidelines to perform these activities more effectively. The tutorial is structured into three parts, and the teaching style will be “by-examples”. In the first part, after a general overview of recommender systems, we will analyze their design issues and will analyze different definitions of design quality. In the second part we will analyze “off-line” (system-centric) evaluation techniques. We will describe different quality metrics and how to measure them. We will highlight the benefits and pitfalls of off-line evaluation. The third part will explore “on-line” (user–centric) quality evaluation methodologies, focusing on the users’ perceived values and how these affect their behavioral intentions. We will describe in some details a unifying user-centric evaluation framework, called ResQue (Recommender systems’ Quality of user experience), and psychometric methods, including structural equation modeling (SEM).
OUTLINE
- PART 1: Design of Recommender Systems (RSs)
- Introduction to RSs: how RSs work, application domains, benefits for users and providers
- The design space
- User Profile: elicitation techniques (implicit vs. explicit, ratings, free text, user requirements, user goals, demographics)
- Domain-specific information on items
- Algorithms: collaborative filtering, content-based and hybrid algorithms
- Presentation and explanation of recommendations
- Design quality of RSs:
- data quality (content metadata and user profiles)
- algorithm quality (accuracy, novelty, serendipity, diversity, …)
- perceived qualities (e.g., usability, usefulness, and user satisfaction)
- recommendation presentation and interaction quality
- Design tradeoffs: quality of recommendations vs. elicitation effort, cost of data enrichment and computational complexity
- PART 2: Off-line evaluation
- Error and accuracy metrics
- Dataset selection and partitioning: hold-out, k-fold cross validation, leave-one-out
- Benefits and pitfalls of off-line evaluation
- PART 3: On-line evaluation
- User-centric evaluation: the ResQue model
- Evaluation model development
- Evaluation of user beliefs (e.g., usefulness), attitudes (e.g., satisfaction and trust), intentions (e.g., willingness to buy), and behaviors (e.g., actual purchases)
- Psychometric validation methods (e.g., structural equation modeling – SEM)
- Wrap-up + Q&A
FILES
- Tutorial slides [PDF]
PRESENTERS
- Paolo Cremonesi, Department of Electronics and Information, Politecnico di Milano [paolo.cremonesi@polimi.it]
- Franca Garzotto, Department of Electronics and Information, Politecnico di Milano [franca.garzotto@polimi.it]
- Pearl Pu, Faculty of Computing and Communication Sciences, EPFL [pearl.pu@epfl.ch]
DATE AND DURATION
Monday, June 10, 2013 – Half day/Morning
SHORT BIO
Paolo Cremonesi is associate professor of Computer Engineering at the Department of Electronics and Information of Politecnico di Milano. His research interests cover two areas: computer systems performance (high-performance and parallel computing, benchmarking of computer systems, queuing network performance models, green IT) and recommender systems (design of algorithms, off-line and on-line evaluation of recommender systems, applications of recommender systems in TV and tourism). Paolo holds an MSc in Aerospace Engineering (1991) and a PhD in Computer Science (1996), both from the Politecnico di Milano. Before joining the Politecnico di Milano, Paolo first worked with the NATO research center Von Karman Institute, doing research on high-performance computing. From 2001 until 2006 Paolo has been Editor of the Elsevier Journal of Systems Architecture. In 2001 Paolo co-founded Neptuny, the first and most successful Politecnico di Milano start-up company, specialized in capacity planning and recommender systems.
Franca Garzotto is associate professor of Computer Engineering at the Department of Electronics and Information of Politecnico di Milano. She has a MSc in Mathematics from University of Padua (Italy) and a Ph.D. in Computer Engineering from Politecnico di Milano. Her main theoretical research has focused on topics related to hypermedia/multimedia design and evaluation methods, usability engineering, multichannel web applications, e-branding, tangible interaction, touchless gesture-based interaction. Her applied research has mainly addressed the domain of e-tourism, e-culture, and e-learning (in particular, addressing marginalized children and children with special needs.) Franca has been member of the program committee of major conferences in hypertext, multimedia, HCI, interaction design, e-learning, and e-culture. She was program chair of ICHIM 2001 and ACM IDC 2009: she will be program chair of ACM IDC 1014 and general chair of ACM AVI 2014. In 1997-99 she served as European Chair of SIG-WEB (ACM Special Interest Group on Hypermedia & Web) and she is currently President of the Italian Chapter of ACM SIGHCHI. She has been coordinator of several EC funded project, and, since 2011, member of the review panel for ERC (European Research council) Grants.
Pearl Pu currently leads the HCI Group in the School of Computer and Communication Sciences at the Swiss Federal Institute of Technology in Lausanne (EPFL). Her research interests include recommendation technology, electronic commerce, user adoption of technology, online consumer decision behavior, decision support, content-based product search, travel planning tools, trust-inspiring interfaces for recommender agent, music recommenders, scalable user experience, and social web. She obtained her Master and Ph.D. degrees from the University of Pennsylvania in artificial intelligence and computer graphics. She was a visiting scholar at Stanford University in 2001, both in the database and HCI groups. While there, she gave seminars at Xerox PARC’s weekly seminar series and Stanford’s HCI Design Studio class as guest lecturer. She was also a co-founder of Iconomic Systems (1997-2001), and invented the any-criteria example-based search method for travel solutions. The company was successfully sold to i:FAO, Germany. Her recent publications included papers from ACM TOCHI, UMUAI, the Journal of Artificial Intelligence Research, the Constraints journal, the Knowledge-Based Systems Journal, AAAI, ACM Recsys, ACM ECommerce, International Conferences on Intelligent User Interface, ACM CHI, IEEE InfoVis, and AVI.
