Measuring the dynamics of remembered experience over time

 Legal forms

 3 views
of 8
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Description
A wealth of studies in the field of user experience have tried to conceptualize new measures of product quality and inquire into how the overall goodness of a product is formed on the basis of product quality perceptions. An interesting question
Share
Tags
Transcript
  Measuring the dynamics of remembered experience over time Evangelos Karapanos a, * , John Zimmerman b , Jodi Forlizzi b , Jean-Bernard Martens a a Eindhoven University of Technology, Department of Industrial Design, P.O. Box 513, 5600 MB, The Netherlands b Carnegie Mellon University, Human–Computer Interaction Institute and School of Design, Pittsburgh, PA 15213, USA a r t i c l e i n f o  Article history: Available online xxxx Keywords: User experienceExperience-centered designQualitative methodsLongitudinal methodology a b s t r a c t A wealth of studies in the field of user experience have tried to conceptualize new measures of productquality and inquire into how the overall goodness of a product is formed on the basis of product qualityperceptions. An interesting question relates to how the perception as well as the relative dominance of different product qualities evolve across different phases in the adoption of a product. However, tempo-rality of experience poses substantial challenges to traditional reductive evaluation approaches. In thispaperwepresentanalternativemethodologicalapproachforstudyinghowusers’experienceswithinter-active products develop over time. The approach lies in the elicitation of rich qualitative insights in theform of experience narratives, combined with content-analytical approaches for the aggregation of idio-syncratic insights into generalized knowledge. We describe a tool designed for eliciting rich experiencenarrativesretrospectively, andillustratethistool bymeansof astudythatinquiredintohowusers’ expe-riences with mobile phones change over the first 6 months of use. We use the insights of the study tovalidate and extend a framework of temporality proposed byKarapanos et al. (2009b). Ó 2010 Elsevier B.V. All rights reserved. 1. Introduction Understanding and designing for prolonged use is increasinglybecoming a critical issue in the consumer electronics industry asevidenced by a number of recent trends (c.f.Karapanos et al.,2009b). First, legislation and competition within the consumerelectronics industry has resulted in an increase in the time-spanof product warranties, which lead to an alarming increase in num-ber of products being returned on the basis of failing to satisfyusers’ true needs (Den Ouden, 2006). Secondly, products areincreasingly becoming service-centered. Often, products are beingsold for lower prices and revenues are mainly coming from thesupportedservice.Thus,themeasureofsuccessforaproductshiftsfrom initial purchase to establishing prolonged use.Critical to designing for prolonged use is our ability to measurethe dynamics of users’ experiences over time. Traditional ap-proaches to measuring the dynamics of experience over time relyonvalidatedmeasurement and structural models applied indiffer-ent phases in the adoption of a product (e.g.Venkatesh and Davis,2000). Such reductive approaches to measurement enable a sys-tematic assessment of the changes in users’ perceptions as wellas in the relationships between different latent constructs. How-ever, as it will be argued in this paper, they share a number of lim-itations in the measurement of the dynamics of experience overtime.First, these approaches assume that the relevant latent con-structs (i.e. measurement model) remain constant, but that theirperceived value and relative dominance in their effects on finaloutcome measures (i.e. structural model), such as overallevaluations of the beauty or goodness of an interactive product(Hassenzahl, 2004), change over time.Karapanos et al. (2009b) illustrated, however, how constructs such as emotional attachment  and daily rituals become salient only after prolonged use. Conse-quently, this might challenge the content validity of the measure-ment model as relevant constructs that become salient only inprolonged use may be absent in measurement models that haveonly been validated during initial use. Second, by imposing a setof predefined measurement items, respondents are forced toaggregate the richness of their experience with a product, in a sin-gle attitudinal response. Especially in cases where respondents failto interpret the relevance of a given statement in their owncontext, for instance when a construct has ceased to be relevantover prolonged use, the validity of the elicited data may be ques-tioned (Larsen et al., 2008). Third, being quantitative in nature,such approaches provide limited insight into the exact reasonsfor the observed changes in users’ experiences.In this paper we propose an alternative approach to the mea-surement of the dynamics of users’ experiences with interactiveproducts. The approach relies on the elicitation of idiosyncraticself-reports of one’s experiences, so-called experience narratives .Through a content analysis of the experience narratives, the re-searcher may acquire aggregated knowledge such as, for instance,changes in the relative dominance of different product qualities 0953-5438/$ - see front matter Ó 2010 Elsevier B.V. All rights reserved.doi:10.1016/j.intcom.2010.04.003 * Corresponding author. E-mail address: E.Karapanos@tue.nl(E. Karapanos).Interacting with Computers xxx (2010) xxx–xxx Contents lists available atScienceDirect Interacting with Computers journal homepage:www.elsevier.com/locate/intcom ARTICLE IN PRESS Please cite this article in press as: Karapanos, E., et al. Measuring the dynamics of remembered experience over time. Interact. Comput. (2010),doi:10.1016/j.intcom.2010.04.003  over time. WeillustrateiScale, a surveytool designedwiththeaimof minimizing retrospection bias in the elicitation of experiencenarratives, and employ this tool in a study that assessed thechanges in users’ experiences with mobile phones. 2. Measuring the dynamics of experience This section reviews existing approaches to measuring thedynamics of user experience and introduces the methodologicalapproach that we propose in this paper.  2.1. Reductive versus holistic approaches Reductive approaches in the measurement of the dynamics of experience employ validated measurement and structural modelsacross different phases in the adoption of a system. For instance,Venkatesh and Davis (2000)employed the Technology AcceptanceModel (Davis et al., 1989) at three instances in the adoption of information systems at work settings: before the introduction of the system (inquiring into users’ expectations), right after theintroduction of the system, and 3 months after the introduction.In the field of User Experience, we (Karapanos et al., 2008) em-ployedHassenzahl’s(2004)model of pragmatic/hedonicquality, inexploring the differences (if any) in the ways that users evaluateinteractiveproducts during initial and prolonged use.Hassenzahl’s(2004)model distinguishes between two kinds of quality percep-tions: pragmatic  and hedonic  . Pragmatic quality, he argues, refersto the product’s ability to support the achievement of behavioralgoals (i.e. usefulness and ease-of-use). On the contrary, hedonicquality refers to the users’ self; it relates to stimulation , the prod-uct’s ability to stimulate and enable personal growth, and identifi-cation , the product’s abilityto address the need of expressing one’sself through objects one owns.TenparticipantsreceivedanInteractive-TVset-topboxcontain-ing a novel pointing device. We asked participants to evaluate thepointing device at two points in time, more specifically, during thefirst week and after 4 weeks of use, using the Attrakdiff  2 question-naire (Hassenzahl, 2004). The relations between product qualitiesand overall evaluations were explored at these two points in time,1st week and 4th week (seeFig. 1). We found that while in earlyinteractions judgments of the overall goodness were primarily re-lated to the perceived pragmatic quality of the product (i.e. utilityand ease-of-use), in prolonged use goodness related mostly toquality perceptions pertainingto identification(i.e. what the prod-uct expresses about its owner).We argue that such reductive approaches are limited in a num-berofrespects. Anassumptioninherentintheseapproachesisthatthe relevant latent constructs (i.e. measurement model) remainconstant, but their perceived value and relative dominance (i.e.structural model) change over time. However, especially in devel-oping fields such as user experience, substantial variations mightoccur over time even in what constructs are relevant to measure.Some constructs, for example, novelty, might cease to be relevantwhile others, such as daily rituals, personalization, and self-iden-tity (seeKarapanos et al., 2009b, that were not evident in studiesof initial use might become critical for the long-term acceptanceof a product.Firstly,thismightchallengethecontentvalidityofthemeasure-mentmodelasrelevantlatentconstructsmightbeomitted.Thisis-sue has been repeatedly highlighted in technology acceptanceresearch with a number of studies reporting limited predictivepower of the Technology Acceptance Model (Davis et al., 1989),insomecasesaccountingforonly25%ofthevarianceinthedepen-dent variable (Gefen and Straub, 2000).Lee et al. (2003)reported that ‘‘the majority of studies with lower variance explanations didnot consider external variables other than srcinal TAM variables ”.Secondly, it may also lead to distorted data as participantsmight fail to interpret the essence of a question when the latentconstruct that the researchers attempt to measure (e.g. perceivednovelty) ceases to be relevant over prolonged use. In such a case,participants may infer the meaning of a question from other fea-tures such as the language and the graphical layout of the ques-tionnaire (seeSchwarz, 2007).Larsen et al. (2008)reviewed a number of studies employing psychometric scales in the field of Information Systems. They found in the majority of studies thatthe semantic similarity between items was a significant predictorof participants’ ratings with accounted variance of up to 63%. Insuch cases, they argued, participants are more likely to have em-ployed shallow processing  , that is, responding to surface featuresof the language rather than attaching personal relevance to thequestion (c.f.Sanford et al., 2006).Lastly, such approaches provide rather limited insight into theexact reasons for changes in users’ experiences. They may, for in-stance, reveal a shift in the dominance of perceived ease-of-use Fig. 1. 2D projections of distances between quality attributes, beauty and goodness, representing users’ perceptions during the 1st week (left) and 4th week of use. Itemslying closely together are highly correlated. Items connected through a line belong to the same cluster.2 E. Karapanos et al./Interacting with Computers xxx (2010) xxx–xxx ARTICLE IN PRESS Please cite this article in press as: Karapanos, E., et al. Measuring the dynamics of remembered experience over time. Interact. Comput. (2010),doi:10.1016/j.intcom.2010.04.003  and perceived usefulness on intention to use a product (e.g.Venk-atesh and Davis, 2000), but provide limited insight to the exactexperiences that contributed to such changes.In contrast, holistic approaches attempt, throughqualitative in-quiry, to gain rich insights into users’ idiosyncratic experiences.They attempt to inquire into the ways that design attributes inter-act with contextual details in particular contexts (Suri, 2002). Oneexample is the elicitation of experience narratives, idiosyncraticself-reports on one’s experiences with a product. These narrativesprovide rich insights into the experience and the context in whichthey take place. However, generalized knowledge can also begained from these experience narratives. Such generalized knowl-edge consists of quantitative answers to questions like: how fre-quent is a certain kind of experience, what is the ratio of positiveversus negative experiences and how does this compare to com-petitive products, how does the dominance of different productqualities fluctuate over time and what should designers improvetomotivateprolongeduse?Thisapproachhasanumberofbenefitsover traditional reductive approaches. First, it provides a morecomplete coverage of users’ experiences over time as participantsare free to report any experience that is personally meaningful tothem. Reduction and quantification of users’ experiences only oc-curs in the analysis stage performed by the researcher. Second, itelicits data that are personally meaningful to the participant incontrast to reductive approaches which force participants to sum-marize their experiences according to imposed dimensions. Third,it allows linking generalized, quantifiable knowledge to rich andidiosyncratic insights. Both are important for design: idiosyncraticinformation may inspire design solutions, generalized knowledgemay direct the focus of design.An example is a study byKarapanos et al. (2009b). Followingour initial exploratory insights into the dynamics of users’ experi-ences we posed the following questions: what causes thesechanges? Can we describe the adoption of a product in terms of distinct phases? And which product qualities dominate in each of these phases? We followed six individuals through an actual pur-chase of an Apple iPhone. For a period of 4 weeks, participantswere asked, during each day, to mentally reconstruct (Kahnemanet al., 2004) their activities and experiences that somehow relatedto the specific product, and report the three most impactful expe-riencesofthedayintheformofone-paragraphessays, called expe-rience narratives . A total of 482 experience narratives were elicitedand submitted to a qualitative content analysis (Hsieh and Shan-non, 2005). Narratives were coded with respect to the dominantproduct quality in each experience. The distribution of differentexperiences over time led to the realizationof three overall phasesin the adoption of the product: Orientation , Incorporation and Iden-tification (seeFig. 2). In this paper we describe a study that at-tempts to validate and extend this framework.Orientation relates to participants’ initial experiences that arepervaded by feelings of excitement and frustration as they areconfronted with novel features and learnability problems. Incorpo-ration relates to reports of how the product is becoming meaning-ful in participants’ daily lives as well as the realization of  usability and usefulness problems that pertain over long-term use. Finally,identification relates to participants’ personal experiences as theproductplaysapartin daily rituals , aswellastosocialexperiences,as the product allows for self-expression and creating a sense of community .  2.2. Longitudinal versus retrospective approaches Anumberofdifferentlongitudinal approachesexistfortheelic-itation of self-reports on one’s experiences with a product. Event-contingent diaries (seeBolger et al., 2003) ask participants to re-port experiences that they consider substantial enough at the mo-ment of their occurrence. The Experience Sampling Method(Hektneret al., 2007) prompts participants to report on their cur-rent experience at random or computationally estimated times,for example, through context and activity detection algorithms.The Day Reconstruction Method (Kahneman et al., 2004), moti-vated by the need for more unobtrusive techniques, asks the par-ticipant at the end of each day to reconstruct, in a temporalsequence, the activities that took place during the day. This proce-dure has been found to assist participants in recalling accuratelytheir daily experiences, while imposing less burden to participantsbeing an offline method. InKarapanos et al. (2009b), we employedthe Day Reconstruction Method in a four-week diary study. Eachday participants were being asked to mentally reconstruct theirdaily experiences that somehow related to the product of interest,and to report the three most personally impactful experiences of the day.Such longitudinal approaches may provide rich insights intousers’experiencesindifferentcontexts.However,theyareincreas-ingly laborious when one needs to generalize over large popula-tions of users and products, and inquire into long periods of time.Motivatedbythisobservation,wedeveloped iScale , 1 a surveytool that aims at eliciting users’ most impactful experiences withproducts retrospectively (Karapanos et al., 2009a). iScale asks partic-ipants to narrate their most impactful experiences and to provide anestimation of when each experience took place, thus resulting in atemporal structure. Using iScale , one may inquire into the whole life-span of a product in a single survey contact, in other words, partic-ipants may be asked to reconstruct their experiences from themoment of purchase till the present. As iScale is a retrospective technique, one may wonder aboutthe degree to which these experience reports are biased or incom-plete.However,wearguethattheveridicalityofone’srememberedexperience is of minimal importance (Karapanos et al., 2009a), as Fig. 2. A framework of  Temporality of experience proposed inKarapanos et al.(2009b). Temporality consists of three main forces, an increasing familiarity ,  functional dependency and emotional attachment  , all responsible for shifting users’experience across three phases: orientation , incorporation and identification . In eachphase, different product qualities are salient. 1 A video demo of the tool is available at:http://ekarapanos.com/iscale. E. Karapanos et al./Interacting with Computers xxx (2010) xxx–xxx 3 ARTICLE IN PRESS Please cite this article in press as: Karapanos, E., et al. Measuring the dynamics of remembered experience over time. Interact. Comput. (2010),doi:10.1016/j.intcom.2010.04.003  these memories (and not the actual experiences) (1) will guide fu-ture behavior of the individual and (2) will be communicated toothers (see alsoNorman, 2009). In other words, it may not matterhowaproductwasexperiencedinagivensituation, butwhatindi-viduals remember from this experience.Although the validity of remembered experiences may not becrucial,theirreliabilityis(Karapanosetal.,2009a).Itseemsatleastdesirable that participants would report their experiences consis-tently over multiple trials. If recall is random in the sense that dif-ferent experiences are perceived to be important at differentrecalls, then the importance of such elicited reports may be ques-tioned. InKarapanos et al. (2009a), we described the theoreticalgrounding and empirical testing of two different versions of iScalethat were motivated by two distinct theories of howpeople recon-struct emotional experiences from memory. The iScale tool usessketching in imposing a certain process in the reconstruction of experiences. It was found that these processes have a substantialimpact on the number, the richness and the test–retest reliabilityof recalled information when compared to a control condition, afree recall without employing any form of sketching (Karapanoset al., 2009a).Participants using iScale are asked to sketch how their opinionon a given product quality (e.g. the perceived usability of a prod-uct)haschangedasacourseoftime.Inessence,participantssketchlinear segments with a certain slope (which denotes the change in Fig. 3. iScale. A survey tool that elicits experience narratives retrospectively. (a) Users’ sketch patterns that represent their perception of a given product quality. (b) Usersreport one or more experiences for each sketched line segment.  Table 1 The product qualities that participants reported on, along with definitions and worditems. Name Definition WordEase-of-use The ability of a product to provide thefunctions in an easy and efficient wayEasy to use,simple, clearInnovativeness The ability of a product to excite the userthrough its noveltyInnovative,exciting,creative4 E. Karapanos et al./Interacting with Computers xxx (2010) xxx–xxx ARTICLE IN PRESS Please cite this article in press as: Karapanos, E., et al. Measuring the dynamics of remembered experience over time. Interact. Comput. (2010),doi:10.1016/j.intcom.2010.04.003  the value of the reported quality) and length (which denotes thetemporal span of the reported period) (seeFig. 3a). Participantsmay then associate each line segment with one or more experi-ences that are believed to have induced the given change in theirperception of the product quality (seeFig. 3b).Two kinds of information are provided by iScale : (a) a list of experience narratives that are considered most impactful by theparticipant, and (b) a sketched pattern of the participant’s percep-tions of a given product quality. In the remainder of the paper wedescribe a study that employed iScale in reconstructing partici-pants’experienceswithmobilephonesandproposeamethodolog-ical approach for the analysis of the dynamics of experience overtime. 3. A study on the dynamics of users’ experiences with mobilephones The study aimed at validating the framework of temporalityproposed inKarapanos et al. (2009b). While the initial study wasrestricted to the first month of use and one particular product,the Apple iPhone, the present paper aims at extendingthe findingsto the first 6 months of usingmobile phones. Whilewe limitedthefocus of the study to one product category, we did not opt to con-trol further the type of product.  3.1. Method 3.1.1. Participants and material The study was conducted with 48 participants (17 Female),ranging from 18 to 28 years old (median=23 years). They wereall students at a technical university; 19 of them majored in man-agement related disciplines, 16 in design, and 13 in natural sci-ences and engineering. All participants owned a mobile phonefor no less than four and no more than 18 months (median: 10months); 16 participants owned a smart phone. Each participant joineda1-hsessionandwasrewardedby10eurosfortheirefforts.  3.1.2. Product qualities Hassenzahl (2004)proposed that people perceive productsalong two primary dimensions: pragmatic quality which refers tothe product’s ability to support the achievement of do-goals suchas making a telephone call, and hedonic quality which refers tothe product’s ability to support the achievement of be-goals suchas being happy. In the present study, we used these dimensionsto differentiate pragmatic from hedonic experiences. Participantswere asked to recall their mobile phone related experiences onandrate a change intwo product qualities: ease-of-use and innova-tiveness .Ease-of-usereflectedpragmaticexperienceswhileinnova-tiveness reflected hedonic experiences; we avoided using thesrcinal terms as participants might fail to understand them. Prod-uct qualities were defined through a brief description and threesingle word items derived from the Attrakdiff  2 scales (Hassenzahl,2004) (seeTable1). Theseworditemsreflectedthepositivepoleof  the three Semantic Differential scales that displayed the highestloading to the respective latent construct of the Attrakdiff  2 ques-tionnaire in a previous study (Karapanos et al., 2008).  3.1.3. Procedure Participants used the iScale tool in sketching how their percep-tions of their mobile phone, regarding these two product qualities,developed from the moment of purchase till the present time.While sketching, they reconstructed and reported the experiencesthat affected their opinion about the product. For each drawn linesegment (seeFig. 3a), they were asked to add at least one experi-ence that induced this change in their perception of the respectivequality(seeFig. 3b). ParticipantsalsoannotatedthetimelineoftheiScaletoolbydefiningtheestimatedtimethathadpassedfromthepurchase of the product till the end of each respective period.  3.1.4. Assumptions Anumber of tentativeassumptions wereformedonthe basis of the study presented inKarapanos et al. (2009b). The majority of users’ experiences with iPhone relating to the product qualities stimulation and learnability displayed a sharp decrease over time:58% of all experiences relating to stimulation and 59% of experi-ences relating to learnability took place during the first week of use. We expected this finding to be apparent also in the case of awider spectrum of products such as the ones used in the currentstudy, and not to be tied to the nature of iPhone, the product usedintheoriginalstudy(Karapanosetal.,2009b).Wefurtherexpectedthis pattern to continue beyond the first month of use, with only asmall number of experiences relating to learnability or stimulationbeing reported after the first month. On the contrary, long-termusability and usefulness became the dominant source of satisfyingand dissatisfying experiences over time. We expected this to holdalso in the current study and extend beyond the first month of use. Note that we did not distinguish between satisfying and dis-satisfying experiences relating to a given product quality as they period 2nd Month - 6th Month2nd Week - 1st MonthPurchase - 1st Week    N  u  m   b  e  r  o   f  e  x  p  e  r   i  e  n  c  e  r  e  p  o  r   t  s 6040200 StimulationLearnability Product quality period 2nd Month - 6th Month2nd Week - 1st MonthPurchase - 1st Week    N  u  m   b  e  r  o   f  e  x  p  e  r   i  e  n  c  e  r  e  p  o  r   t  s 6040200 UsefulnessUsability Product quality Fig. 4. Number of narratives in a certain period relating to learnability, stimulation, long-term usability and usefulness as a function of time of ownership. E. Karapanos et al./Interacting with Computers xxx (2010) xxx–xxx 5 ARTICLE IN PRESS Please cite this article in press as: Karapanos, E., et al. Measuring the dynamics of remembered experience over time. Interact. Comput. (2010),doi:10.1016/j.intcom.2010.04.003
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks