728x90
http://uxmatters.com/mt/archives/2009/10/eyetracking-is-it-worth-it.php
Eyetracking: Is It Worth It?
By Jim Ross
Published: October 19, 2009
“Seeing where people look while using yourWeb site, Web application, or software product sounds like anopportunity to get amazing insights into their user experience.”
It is easy to get excited about eyetracking. Seeing where peoplelook while using your Web site, Web application, or software productsounds like an opportunity to get amazing insights into their userexperience. But eyetracking is expensive and requires extra effort and specialized knowledge. The heat maps and other visualizations certainlylook impressive, but what can you really learn from them? After usingeyetracking for the first time, many find that it is not easyto know how to analyze the visualizations and make conclusions fromthem. Does eyetracking really provide any additional insights you wouldnot have discovered anyway through traditional usability testing? Does the value of eyetracking outweigh its limitations? This article willdiscuss and answer these questions.
What Does Eyetracking Show Us?
Eyetracking detects where a person’s fovea fixate and the movements in between fixations. The foveais a small spot on the retina that is responsible for our fine,detailed vision. Outside the fovea, visual acuity decreases greatly.Our eyes constantly move in rapid bursts called saccades, with brief stops called fixations, during which we take in visual information through our fovea. We use our parafovea—the area just outside the fovea—and peripheral vision to determine where to fixate next.
“Most modern eyetracking systemsunobtrusively track participants’ eyes by reflecting near-infraredlight off the eyes and detecting the position of those reflections todetermine where the person’s fovea are fixated.”
We can see these fixation points and saccades mostclearly in eyetracking gaze plots. Figure 1 is a gaze plot of one studyparticipant’s fixations and saccades during a task. The circles are the fixation points, where the participant’s fovea stopped briefly to takein information. The numbers indicate the order of fixations, and the size of each circle indicates the length of a fixation, with larger circles signifying longer fixations. The lines connecting the circles are the saccades, the movements between fixations.
Most modern eyetracking systems unobtrusively trackparticipants’ eyes by reflecting near-infrared light off the eyes and detecting the position of those reflections to determine where theperson’s fovea are fixated.
Figure 1—An eyetracking gaze plot for one participant
What Value Does Eyetracking Provide?
This may all sound pretty impressive, but what value can we really get from knowing where a person’s fovea fixated?
Helping to Identify Usability Problems
“We should not use eyetracking data alone, but in conjunction with careful observation of participant behavior and discussion.”
Eyetracking provides additional information to help you find and interpret design and usability problems. It is important to note that we should not use eyetracking data alone, but in conjunction with careful observationof participant behavior and discussion. Within this context, seeing where participants looked during a task provides additional insights into what they were doing and why.
Knowing where participants looked during a task is especially helpful in answering certain types of questions. For example, eyetracking data can help you determine the following:
• why participants had problems performing a task
• where participants expected to find certain elements
• whether participants noticed a particular element—such as a link, button,advertising, or something new added to a user interface
• whether elements are distracting in a negative way
• how efficiently a design guides participants through a task
• whether there are differences in task performance by user group—for example, between new users and experienced users
• which content participants read—including articles, instructions, contextual Help, and error messages
• how participants read—in detail or by scanning
• whether a particular design is more effective than another—in terms of user or business goals
For example, Figures 2 and 3 show an example of how a feature that its designers intended to be a helpful aid was actually a negative distraction. While filling out an online credit card application, participants were distracted by a Talk to a Specialist box, shown in Figure 2, that appeared and followed them down the right side of the page. The gaze plot in Figure 3 shows one typical participant’s fixation points (the circles) as she filled out the form.We can clearly see the three times when the chat box distracted her attention from the form. Without eyetracking, we would have observed only the third of these, when she finally closed the chat box. We would not have seen the previous two instances of distraction. The client had heard complaints about this chat box before, but it took eyetracking to show them how distracting it really was.
Figure 2—A distracting Talk to a Specialist chat box
Showing Hard-to-Articulate Behaviors
“Eyetracking can help us identify behaviors that participants cannot articulate.”
Eyetracking can help us identify behaviors that participants cannotarticulate. If you ask participants whether they noticed a particularelement or why they had a problem, they often have a hard timeaccurately and reliably answering such questions. We make many of oureye movements unconsciously, so it is difficult to report what welooked at, for how long, or in what order. Having participants thinkaloud during tasks or retrospectively after completing tasks providesvaluable insights into their thought processes and behavior, but theseare consciously filtered explanations. Eyetracking shows youparticipants’ actual eye movements, without any conscious filtering.
Providing Additional Insights
In addition to the data eyetracking provides for later analysis, watching the live view of the eyetracking data during usability testing provides a facilitator with additional insights—likea sixth sense—into what a participant is doing. This additionalunderstanding helps the facilitator ask more relevant questions and,thus, get more insightful information.
Visualizing Data for Observers
“The live view of the eyetracking data provides a great visualization for observers during usability testing.”
The live view of the eyetracking data provides a great visualizationfor observers during usability testing. Observers can see the live viewof the eyetracking data, which makes it much easier for them to followand understand what is happening during testing. Seeing a participants’eye movements provides much more information that just seeing a mousepointer moving around the screen. Because eyetracking makes observing a usability test much more interesting and engaging, observers tend to pay closer attention and get more value out of observing the tests.
Exciting Clients and Team Members
Eyetracking studies excite clients and project team members.Observing live eyetracking and viewing the resulting visualizations isexciting. Eyetracking gets clients and project team members interested in observing more tests and doing more usability testing. That, in itself, is a good thing!
Providing Compelling Visualizations of Usability Problems
“Eyetracking provides compelling, obvious visualizations of usability problems.”
Eyetracking provides compelling, obvious visualizations of usabilityproblems. Eyetracking visualizations are often a great way ofdemonstrating usability problems. You can describe usability problemsmuch more powerfully—and in a way that’s visually obvious—with a goodeyetracking visualization than in words alone. For example, thefollowing sentence is much more salient when a gaze plot accompaniesit: Participants had a difficult time finding the link to register for access to their auto insurance policy. The gaze plot in Figure 6 shows the extensive path a typical participant took while looking for the Register link.
Figure 6—Gaze plot of a participant searching extensively for a Register link
Is Eyetracking Really Necessary?
“Participants remembered 70% of the elements they had actually seen, but were not very good at remembering the order in which they had looked at those elements.”
Do you really learn anything from eyetracking that you could not learn through regular usability testing? Can a good designer foresee how users would view a design and guide their eyes through the design?
Johansen and Hansen sought to answer these questionsin their 2006 study, “Do We Need Eye Trackers to Tell Where People Look?” [1] For this study, which compared conventional usabilitytesting with eyetracking, they conducted a usability test in which participants reported from memory what they saw. They found participants remembered 70% of the elements they had actually seen, but were not very good at remembering the order in which they had looked at those elements.
They also asked experienced Web designers to predict the eye movements of a typical user and found the designers could predict only 46% of the elements users typically saw. Plus, only 36% were successful in predicting the first three areas of the screen participants looked at. The study concluded that designers cannot accurately predict where people will look, and participants cannot reliably report where they looked.
Of course, you can conduct usability testing without eyetracking, but eyetracking provides valuable additional data about whereparticipants looked during tasks. This is data a facilitator cannototherwise observe and participants cannot report accurately.
What Are the Limitations of Eyetracking?
Although seeing where participants fixate is valuable, there are limitations to what eyetracking can reveal.
Eyetracking Cannot Track Our Peripheral Vision
“Eyetracking can be misleading, because it does not capture peripheral vision.”
Eyetracking can be misleading, because it does not captureperipheral vision. Eyetracking records and displays foveal fixations,in the small part of our visual field that produces the sharpestvision. It does not record peripheral vision, which makes up98% of our visual field. This is significant, because we use peripheralvision to choose where to fixate our fovea next.
So we can see things on a screen withoutdirectly fixating on them. For example, we can see banner ads on theright side of a page, using our peripheral vision. We know enough fromtheir position on the right side of the page, their visual appearance,and from our previous Web experience to identify them as ads, and weoften choose not to fixate on them.
Because eyetracking visualizations do not showperipheral vision, they can be misleading. For example, in a heat map,some assume users did not see areas without hot spots at all. It isimportant to recognize that the eyetracking fixations, or hot spots, do not represent everything participants saw. They are just the areas participants fixated on, viewing them in detail.
For example, Figure 7 shows two versions of the same heat map. The gaze opacity heat map on the right shows only the areas of the page with hot spots. The rest of the page is black. Obviously,this is not how we really see Web pages. We see entire pagesthrough both our foveal and peripheral vision. The hot spots are the areas on which we fixate our fovea to see them in more detail. It is important to explain this distinction to the audience for your testing results.
Figure 7—Corresponding heat and gaze opacity heat maps
Fixations Don’t Represent Attention or Understanding
“Participants’ fixations do not necessarily equal attention or understanding.”
Participants’ fixations do not necessarily equal attention or understanding. Eyetracking shows only fixation points—that is, points where the fovea stop to take in information—and saccades—eye movements between fixations. It does notreveal anything about the higher-level processes of attention andcomprehension. Just because a person’s eyes fixate on something, thatdoes not mean he or she is consciously paying attention to that element—or understands it.
This is why eyetracking data has little meaning byitself. Careful observation of tasks and discussion with participantsis necessary to understand their mental processes. Eyetracking providescomplementary data to aid in this understanding. It is not sufficient alone.
Fixations Don’t Communicate Meaning
“Eyetracking shows only where participants fixated, not the meaning of their fixations.”
Eyetracking shows only where participants fixated, not the meaning of their fixations—that is, it shows where, but not why.For example, if an element received many fixations or longer fixations,it could indicate either attraction to that element or difficulty in understanding it. Again, this is why you should not use eyetrackingdata alone. It requires interpretation. Careful observation anddiscussion with participants let you understand the context for your eyetracking data.
How Does Eyetracking Affect the Testing Process?
Eyetracking is not just software and hardware that you turn on to record usability test sessions. Using it effectively requiresmaking changes to your usability testing process.
Interactions Between Facilitator and Participant Change
“Getting accurate eyetracking data that shows what participants look at during tasks requires letting participants perform the tasks without interruption.”
Eyetracking changes the interactions between the facilitator andparticipants. Getting accurate eyetracking data that shows what participants look at during tasks requires letting participants performthe tasks without interruption. Asking participants to think aloud, asking probing questions, or even allowing participants to ask questions interferes with capturing accurate eyetracking data.
Instead of using a concurrent think-aloud technique, you can use a retrospective think-aloud technique, in which you ask participants to describe whatthey were doing while you play back a recording of the tasks theyperformed. However, when using this technique, participants’ commentslose some of their immediacy. You should consider the trade-off betweenhaving participants think aloud and getting accurate eyetracking data,depending on which is more appropriate for the goals of your test.
Eyetracking Can Be Intrusive
Eyetracking can interfere with the naturalness of a test session.The ideal eyetracking system is completely unobtrusive to participants.Eyetracking systems have evolved from very intrusive systems that involved participants’ wearing headgear and having their heads immobilized to almost completely unobtrusive systems that look like aregular computer monitor. With these modern systems, most participantshave no idea they are sitting in front of an eyetracker until they aretold so at the beginning of a test session.
“Eyetracking can interfere with the naturalness of a test session.”
Yet, even with the most unobtrusive eyetrackingsystems, participants can experience a sense of self-consciousness—an uncomfortable awareness of everything they look at—because they know they are being eyetracked. Eyetracking self-consciousness tends to fade over time as participants get involved in the test tasks.
Another limitation of eyetracking systems is that participants need to remain within range of the eyetracker. Though modern systems can track participants’ eyes over a wide range,participants often move out of range over time—for example, they slump,lean back, lean forward, or rest their chin on a hand. When thisoccurs, the facilitator must ask a participant to move back into range. Each time this happens, it interrupts the participant’s task anddisrupts the naturalness of the situation. If a participant repeatedly moves out of range, the facilitator has to decide whether it is more important to get the eyetracking data, or it is better to sacrifice the eyetracking data to see the participant’s natural, uninterrupted behavior.
Eyetracking Tests Take More Time
“Planning and setting up an eyetracking study can add some extra time to the usability testing process.”
Planning and setting up an eyetracking study can add some extra timeto the usability testing process. Both the need to explain eyetrackingto participants and the need to have participants retrospectively thinkaloud to the playback of the eyetracking recording either add time totest sessions or reduce the number of tasks you can test. Eyetrackingalso generates additional data, adding significant time to the dataanalysis phase. With all of these considerations, eyetracking isprobably not the best choice for testing with a low budget or when you need a quick turnaround.
Eyetracking Is Expensive
Although the cost of eyetracking hardware and software has comedown, it is still somewhat expensive. The cost of a new eyetrackingsystem is $40,000 to $60,000. Rental prices can be up to $3000 per day.
Eyetracking Can Be Difficult to Learn
“Without careful planning, eyetrackingresults can be meaningless—and even when everything goes right, they can be difficult to interpret.”
Eyetracking is not something a usability professional can start using effectively right away. More important than learning how to use the hardware and software is learning how to effectively plan and conduct eyetracking studies and analyze eyetracking results. This is not easy. Beginners often learn through trial and error that, without careful planning, eyetracking results can be meaningless—and even when everything goes right, they can be difficult to interpret. Likeanything else, analysis of eyetracking results is a skill that you mustdevelop through experience.
Eyetracking Is Subject to Technical Problems
The more technology you have, the more can go wrong. Although eyetracking technology is continually improving in its reliability,anyone who had been involved in eyetracking has horror stories oftechnical problems that threatened to derail their testing.Experiencing such problems during a test session—especially with clients observing—is not fun. It can make you yearn for the good old days of old-fashioned usability testing.
Eyetracking Can Become a Gimmick
Eyetracking can be very impressive and exciting for clients, which makes them eager to do more usability testing and eyetracking. But the downside is that they may want to use eyetracking for everything, evenwhen it’s not the most appropriate method for your testing goals. It is important to be sure eyetracking does not become a sales gimmick. Use it purposefully, where it provides value.
So Is Eyetracking Worth It?
“If you know how to use eyetracking effectively, it can provide additional insights to usability testing that can help you find problems and answer questions about user behavior.”
Considering the advantages and disadvantages of eyetracking, is it worth it? Some have concluded that the benefits of eyetracking are not worth the high cost, effort, and complexity it adds to usability testing. On the other hand, some eyetracking vendors and consultants have promoted the idea that you cannot conduct usability testing effectively without eyetracking.
The truth lies somewhere between these extremes. If you know how to use eyetracking effectively, it can provide additional insights to usability testing that can help you find problems andanswer questions about user behavior. Eyetracking is not essential to usability testing, but if you can afford it and have the time to learn how to use it effectively, it is definitely worth it. Ultimately, you have to make the decision—based on your testing goals and the considerations I’ve outlined in this article—whether eyetracking would provide value and its benefits would outweigh its limitations.
위에서 보듯이 아이트래킹은 사람들이 말로는 설명하지 못하는 데이터를 시각적으로 보여주며,
클라이언트들을 흥분시키기에 충분하지만
1) 주변시야 (Peripheral vision)를 확인하지 못함
2) Fixation1은 이해나 주의를 설명하지 못함
3) Fixation은 그 의미에 대해서 설명하지 못함
이러한 한계점들을 가지고 있기 때문에 항상 홀로 쓰이면 안되고 사용자 행동의 관찰이 함께 병행되어야만 한다.
또한 아이트래킹은 테스트에 영향을 미치는데 그 내용은 아래와 같다.
1) Concurrent Think aloud를 하면 제대로된 데이터를 측정하기 힘들기 때문에 Retrospect Think Aloud를 통해 사용자들에게 행동의 의미를 들어보는 것도 괜찮다.
2) 자신의 시선이 추적당하고 있음을 인지하는 것 또한 테스트에 영향을 미칠 수 밖에 없지만 사용자들은 곧 그 사실을 잊기 마련이다. 따라서 초기엔 Dummy Data를 이용하는 것도 좋은 Tip!
3) 사용자들이 시선추적이 가능한 범위를 자꾸 벗어나게 되면 지속적으로 주의를 주더라도 데이터를 얻는데 주력할 것인지 시선추적을 포기하고 사용자 행동을 관찰할 것인지를 결정해야 한다.
4) 아이트래킹은 테스트 시간을 더 많이 소비하게 만든다.
이러한 한계점들과 테스트에 주는 영향에도 불구하고 아이트래킹은 정말 꼭 필요한 테스트 방법론일까?
만일 아이트래킹을 효율적으로 활용할 수 있는 지식이 있다면 활용하는 것이 해결책에대한 새로운 시각을 얻을 수 있는 방법이라는 생각이든다. 하지만 아이트래킹은 반드시 거쳐야 하는 테스트 방법론은 아니며 클라이언트 혹은 이해당사자들에게 무언가를 보여주기 위해 진행하는 테스트는 더더욱 아니다.
만일 이 글에서 이야기 된 한계점들을 극복하고 Context에 맞게 활용하여 아이트래킹의 장점들을 제공할 수 있다면 아이트래킹을 통한 결과물들은 테스트에 신뢰도를 높이고 정확한 의사결정을 할 수 있는 꼭! 필요한 방법론이 될것임을 확신한다.
Eyetracking: Is It Worth It?
By Jim Ross
Published: October 19, 2009
“Seeing where people look while using yourWeb site, Web application, or software product sounds like anopportunity to get amazing insights into their user experience.”
It is easy to get excited about eyetracking. Seeing where peoplelook while using your Web site, Web application, or software productsounds like an opportunity to get amazing insights into their userexperience. But eyetracking is expensive and requires extra effort and specialized knowledge. The heat maps and other visualizations certainlylook impressive, but what can you really learn from them? After usingeyetracking for the first time, many find that it is not easyto know how to analyze the visualizations and make conclusions fromthem. Does eyetracking really provide any additional insights you wouldnot have discovered anyway through traditional usability testing? Does the value of eyetracking outweigh its limitations? This article willdiscuss and answer these questions.
It is easy to get excited about eyetracking.
But eyetracking is expensive and requires extra effort andspecialized knowledge
But eyetracking is expensive and requires extra effort andspecialized knowledge
What Does Eyetracking Show Us?
Eyetracking detects where a person’s fovea fixate and the movements in between fixations. The foveais a small spot on the retina that is responsible for our fine,detailed vision. Outside the fovea, visual acuity decreases greatly.Our eyes constantly move in rapid bursts called saccades, with brief stops called fixations, during which we take in visual information through our fovea. We use our parafovea—the area just outside the fovea—and peripheral vision to determine where to fixate next.
“Most modern eyetracking systemsunobtrusively track participants’ eyes by reflecting near-infraredlight off the eyes and detecting the position of those reflections todetermine where the person’s fovea are fixated.”
We can see these fixation points and saccades mostclearly in eyetracking gaze plots. Figure 1 is a gaze plot of one studyparticipant’s fixations and saccades during a task. The circles are the fixation points, where the participant’s fovea stopped briefly to takein information. The numbers indicate the order of fixations, and the size of each circle indicates the length of a fixation, with larger circles signifying longer fixations. The lines connecting the circles are the saccades, the movements between fixations.
Most modern eyetracking systems unobtrusively trackparticipants’ eyes by reflecting near-infrared light off the eyes and detecting the position of those reflections to determine where theperson’s fovea are fixated.
saccades = our eyes constantly move in rapid bursts (the lines connecting the circles)
fixations = brief stops of our eye move
gaze plots = fixations + saccades
Eye Tracking system
1) 눈에서부터 가까운 적외선 빛을 통해 반사된 위치와
2) 사람의 와(fovea)가 고정된 곳에서의 반사된 위치를 파악
하여 참여자들의 눈을 따라 데이터를 기록
fixations = brief stops of our eye move
gaze plots = fixations + saccades
Eye Tracking system
1) 눈에서부터 가까운 적외선 빛을 통해 반사된 위치와
2) 사람의 와(fovea)가 고정된 곳에서의 반사된 위치를 파악
하여 참여자들의 눈을 따라 데이터를 기록
Figure 1—An eyetracking gaze plot for one participant
This may all sound pretty impressive, but what value can we really get from knowing where a person’s fovea fixated?
Helping to Identify Usability Problems
“We should not use eyetracking data alone, but in conjunction with careful observation of participant behavior and discussion.”
Eyetracking provides additional information to help you find and interpret design and usability problems. It is important to note that we should not use eyetracking data alone, but in conjunction with careful observationof participant behavior and discussion. Within this context, seeing where participants looked during a task provides additional insights into what they were doing and why.
Knowing where participants looked during a task is especially helpful in answering certain types of questions. For example, eyetracking data can help you determine the following:
• why participants had problems performing a task
• where participants expected to find certain elements
• whether participants noticed a particular element—such as a link, button,advertising, or something new added to a user interface
• whether elements are distracting in a negative way
• how efficiently a design guides participants through a task
• whether there are differences in task performance by user group—for example, between new users and experienced users
• which content participants read—including articles, instructions, contextual Help, and error messages
• how participants read—in detail or by scanning
• whether a particular design is more effective than another—in terms of user or business goals
For example, Figures 2 and 3 show an example of how a feature that its designers intended to be a helpful aid was actually a negative distraction. While filling out an online credit card application, participants were distracted by a Talk to a Specialist box, shown in Figure 2, that appeared and followed them down the right side of the page. The gaze plot in Figure 3 shows one typical participant’s fixation points (the circles) as she filled out the form.We can clearly see the three times when the chat box distracted her attention from the form. Without eyetracking, we would have observed only the third of these, when she finally closed the chat box. We would not have seen the previous two instances of distraction. The client had heard complaints about this chat box before, but it took eyetracking to show them how distracting it really was.
Figure 2—A distracting Talk to a Specialist chat box
Figure 3—Three points at which the Talk to a Specialist box distracted the user
Figure 4 is an example of how eyetracking can show whether a design guides participants effectively through an information seeking task—in this case, finding the dates of the beach volleyballfinals on the 2008 Beijing Olympics schedule. The gaze plot in Figure 5 shows the common and extended visual paths of multiple participants as they tried to find the dates on this confusing schedule. Clearly, the design of this schedule needed major improvements.
Figure 4—Beijing 2008 Olympics beach volleyball schedule
Figure 5—Gaze plot for the Beijing 2008 Olympics beach volleyball schedule
“We should not use eyetracking data alone, but in conjunction with careful observation of participant behavior and discussion.”
Eyetracking provides additional information. Within this context, seeing where participants looked during a task provides additional insights into what they were doing and why.
Eyetracking provides additional information. Within this context, seeing where participants looked during a task provides additional insights into what they were doing and why.
Showing Hard-to-Articulate Behaviors
“Eyetracking can help us identify behaviors that participants cannot articulate.”
Eyetracking can help us identify behaviors that participants cannotarticulate. If you ask participants whether they noticed a particularelement or why they had a problem, they often have a hard timeaccurately and reliably answering such questions. We make many of oureye movements unconsciously, so it is difficult to report what welooked at, for how long, or in what order. Having participants thinkaloud during tasks or retrospectively after completing tasks providesvaluable insights into their thought processes and behavior, but theseare consciously filtered explanations. Eyetracking shows youparticipants’ actual eye movements, without any conscious filtering.
Providing Additional Insights
In addition to the data eyetracking provides for later analysis, watching the live view of the eyetracking data during usability testing provides a facilitator with additional insights—likea sixth sense—into what a participant is doing. This additionalunderstanding helps the facilitator ask more relevant questions and,thus, get more insightful information.
Visualizing Data for Observers
“The live view of the eyetracking data provides a great visualization for observers during usability testing.”
The live view of the eyetracking data provides a great visualizationfor observers during usability testing. Observers can see the live viewof the eyetracking data, which makes it much easier for them to followand understand what is happening during testing. Seeing a participants’eye movements provides much more information that just seeing a mousepointer moving around the screen. Because eyetracking makes observing a usability test much more interesting and engaging, observers tend to pay closer attention and get more value out of observing the tests.
Exciting Clients and Team Members
Eyetracking studies excite clients and project team members.Observing live eyetracking and viewing the resulting visualizations isexciting. Eyetracking gets clients and project team members interested in observing more tests and doing more usability testing. That, in itself, is a good thing!
Providing Compelling Visualizations of Usability Problems
“Eyetracking provides compelling, obvious visualizations of usability problems.”
Eyetracking provides compelling, obvious visualizations of usabilityproblems. Eyetracking visualizations are often a great way ofdemonstrating usability problems. You can describe usability problemsmuch more powerfully—and in a way that’s visually obvious—with a goodeyetracking visualization than in words alone. For example, thefollowing sentence is much more salient when a gaze plot accompaniesit: Participants had a difficult time finding the link to register for access to their auto insurance policy. The gaze plot in Figure 6 shows the extensive path a typical participant took while looking for the Register link.
Figure 6—Gaze plot of a participant searching extensively for a Register link
Is Eyetracking Really Necessary?
“Participants remembered 70% of the elements they had actually seen, but were not very good at remembering the order in which they had looked at those elements.”
Do you really learn anything from eyetracking that you could not learn through regular usability testing? Can a good designer foresee how users would view a design and guide their eyes through the design?
Johansen and Hansen sought to answer these questionsin their 2006 study, “Do We Need Eye Trackers to Tell Where People Look?” [1] For this study, which compared conventional usabilitytesting with eyetracking, they conducted a usability test in which participants reported from memory what they saw. They found participants remembered 70% of the elements they had actually seen, but were not very good at remembering the order in which they had looked at those elements.
They also asked experienced Web designers to predict the eye movements of a typical user and found the designers could predict only 46% of the elements users typically saw. Plus, only 36% were successful in predicting the first three areas of the screen participants looked at. The study concluded that designers cannot accurately predict where people will look, and participants cannot reliably report where they looked.
Of course, you can conduct usability testing without eyetracking, but eyetracking provides valuable additional data about whereparticipants looked during tasks. This is data a facilitator cannototherwise observe and participants cannot report accurately.
What Are the Limitations of Eyetracking?
Although seeing where participants fixate is valuable, there are limitations to what eyetracking can reveal.
Eyetracking Cannot Track Our Peripheral Vision
“Eyetracking can be misleading, because it does not capture peripheral vision.”
Eyetracking can be misleading, because it does not captureperipheral vision. Eyetracking records and displays foveal fixations,in the small part of our visual field that produces the sharpestvision. It does not record peripheral vision, which makes up98% of our visual field. This is significant, because we use peripheralvision to choose where to fixate our fovea next.
So we can see things on a screen withoutdirectly fixating on them. For example, we can see banner ads on theright side of a page, using our peripheral vision. We know enough fromtheir position on the right side of the page, their visual appearance,and from our previous Web experience to identify them as ads, and weoften choose not to fixate on them.
Because eyetracking visualizations do not showperipheral vision, they can be misleading. For example, in a heat map,some assume users did not see areas without hot spots at all. It isimportant to recognize that the eyetracking fixations, or hot spots, do not represent everything participants saw. They are just the areas participants fixated on, viewing them in detail.
For example, Figure 7 shows two versions of the same heat map. The gaze opacity heat map on the right shows only the areas of the page with hot spots. The rest of the page is black. Obviously,this is not how we really see Web pages. We see entire pagesthrough both our foveal and peripheral vision. The hot spots are the areas on which we fixate our fovea to see them in more detail. It is important to explain this distinction to the audience for your testing results.
Figure 7—Corresponding heat and gaze opacity heat maps
Fixations Don’t Represent Attention or Understanding
“Participants’ fixations do not necessarily equal attention or understanding.”
Participants’ fixations do not necessarily equal attention or understanding. Eyetracking shows only fixation points—that is, points where the fovea stop to take in information—and saccades—eye movements between fixations. It does notreveal anything about the higher-level processes of attention andcomprehension. Just because a person’s eyes fixate on something, thatdoes not mean he or she is consciously paying attention to that element—or understands it.
This is why eyetracking data has little meaning byitself. Careful observation of tasks and discussion with participantsis necessary to understand their mental processes. Eyetracking providescomplementary data to aid in this understanding. It is not sufficient alone.
Fixations Don’t Communicate Meaning
“Eyetracking shows only where participants fixated, not the meaning of their fixations.”
Eyetracking shows only where participants fixated, not the meaning of their fixations—that is, it shows where, but not why.For example, if an element received many fixations or longer fixations,it could indicate either attraction to that element or difficulty in understanding it. Again, this is why you should not use eyetrackingdata alone. It requires interpretation. Careful observation anddiscussion with participants let you understand the context for your eyetracking data.
How Does Eyetracking Affect the Testing Process?
Eyetracking is not just software and hardware that you turn on to record usability test sessions. Using it effectively requiresmaking changes to your usability testing process.
Interactions Between Facilitator and Participant Change
“Getting accurate eyetracking data that shows what participants look at during tasks requires letting participants perform the tasks without interruption.”
Eyetracking changes the interactions between the facilitator andparticipants. Getting accurate eyetracking data that shows what participants look at during tasks requires letting participants performthe tasks without interruption. Asking participants to think aloud, asking probing questions, or even allowing participants to ask questions interferes with capturing accurate eyetracking data.
Instead of using a concurrent think-aloud technique, you can use a retrospective think-aloud technique, in which you ask participants to describe whatthey were doing while you play back a recording of the tasks theyperformed. However, when using this technique, participants’ commentslose some of their immediacy. You should consider the trade-off betweenhaving participants think aloud and getting accurate eyetracking data,depending on which is more appropriate for the goals of your test.
Eyetracking Can Be Intrusive
Eyetracking can interfere with the naturalness of a test session.The ideal eyetracking system is completely unobtrusive to participants.Eyetracking systems have evolved from very intrusive systems that involved participants’ wearing headgear and having their heads immobilized to almost completely unobtrusive systems that look like aregular computer monitor. With these modern systems, most participantshave no idea they are sitting in front of an eyetracker until they aretold so at the beginning of a test session.
“Eyetracking can interfere with the naturalness of a test session.”
Yet, even with the most unobtrusive eyetrackingsystems, participants can experience a sense of self-consciousness—an uncomfortable awareness of everything they look at—because they know they are being eyetracked. Eyetracking self-consciousness tends to fade over time as participants get involved in the test tasks.
Another limitation of eyetracking systems is that participants need to remain within range of the eyetracker. Though modern systems can track participants’ eyes over a wide range,participants often move out of range over time—for example, they slump,lean back, lean forward, or rest their chin on a hand. When thisoccurs, the facilitator must ask a participant to move back into range. Each time this happens, it interrupts the participant’s task anddisrupts the naturalness of the situation. If a participant repeatedly moves out of range, the facilitator has to decide whether it is more important to get the eyetracking data, or it is better to sacrifice the eyetracking data to see the participant’s natural, uninterrupted behavior.
Eyetracking Tests Take More Time
“Planning and setting up an eyetracking study can add some extra time to the usability testing process.”
Planning and setting up an eyetracking study can add some extra timeto the usability testing process. Both the need to explain eyetrackingto participants and the need to have participants retrospectively thinkaloud to the playback of the eyetracking recording either add time totest sessions or reduce the number of tasks you can test. Eyetrackingalso generates additional data, adding significant time to the dataanalysis phase. With all of these considerations, eyetracking isprobably not the best choice for testing with a low budget or when you need a quick turnaround.
Eyetracking Is Expensive
Although the cost of eyetracking hardware and software has comedown, it is still somewhat expensive. The cost of a new eyetrackingsystem is $40,000 to $60,000. Rental prices can be up to $3000 per day.
Eyetracking Can Be Difficult to Learn
“Without careful planning, eyetrackingresults can be meaningless—and even when everything goes right, they can be difficult to interpret.”
Eyetracking is not something a usability professional can start using effectively right away. More important than learning how to use the hardware and software is learning how to effectively plan and conduct eyetracking studies and analyze eyetracking results. This is not easy. Beginners often learn through trial and error that, without careful planning, eyetracking results can be meaningless—and even when everything goes right, they can be difficult to interpret. Likeanything else, analysis of eyetracking results is a skill that you mustdevelop through experience.
Eyetracking Is Subject to Technical Problems
The more technology you have, the more can go wrong. Although eyetracking technology is continually improving in its reliability,anyone who had been involved in eyetracking has horror stories oftechnical problems that threatened to derail their testing.Experiencing such problems during a test session—especially with clients observing—is not fun. It can make you yearn for the good old days of old-fashioned usability testing.
Eyetracking Can Become a Gimmick
Eyetracking can be very impressive and exciting for clients, which makes them eager to do more usability testing and eyetracking. But the downside is that they may want to use eyetracking for everything, evenwhen it’s not the most appropriate method for your testing goals. It is important to be sure eyetracking does not become a sales gimmick. Use it purposefully, where it provides value.
So Is Eyetracking Worth It?
“If you know how to use eyetracking effectively, it can provide additional insights to usability testing that can help you find problems and answer questions about user behavior.”
Considering the advantages and disadvantages of eyetracking, is it worth it? Some have concluded that the benefits of eyetracking are not worth the high cost, effort, and complexity it adds to usability testing. On the other hand, some eyetracking vendors and consultants have promoted the idea that you cannot conduct usability testing effectively without eyetracking.
The truth lies somewhere between these extremes. If you know how to use eyetracking effectively, it can provide additional insights to usability testing that can help you find problems andanswer questions about user behavior. Eyetracking is not essential to usability testing, but if you can afford it and have the time to learn how to use it effectively, it is definitely worth it. Ultimately, you have to make the decision—based on your testing goals and the considerations I’ve outlined in this article—whether eyetracking would provide value and its benefits would outweigh its limitations.
위에서 보듯이 아이트래킹은 사람들이 말로는 설명하지 못하는 데이터를 시각적으로 보여주며,
클라이언트들을 흥분시키기에 충분하지만
1) 주변시야 (Peripheral vision)를 확인하지 못함
2) Fixation1은 이해나 주의를 설명하지 못함
3) Fixation은 그 의미에 대해서 설명하지 못함
이러한 한계점들을 가지고 있기 때문에 항상 홀로 쓰이면 안되고 사용자 행동의 관찰이 함께 병행되어야만 한다.
또한 아이트래킹은 테스트에 영향을 미치는데 그 내용은 아래와 같다.
1) Concurrent Think aloud를 하면 제대로된 데이터를 측정하기 힘들기 때문에 Retrospect Think Aloud를 통해 사용자들에게 행동의 의미를 들어보는 것도 괜찮다.
2) 자신의 시선이 추적당하고 있음을 인지하는 것 또한 테스트에 영향을 미칠 수 밖에 없지만 사용자들은 곧 그 사실을 잊기 마련이다. 따라서 초기엔 Dummy Data를 이용하는 것도 좋은 Tip!
3) 사용자들이 시선추적이 가능한 범위를 자꾸 벗어나게 되면 지속적으로 주의를 주더라도 데이터를 얻는데 주력할 것인지 시선추적을 포기하고 사용자 행동을 관찰할 것인지를 결정해야 한다.
4) 아이트래킹은 테스트 시간을 더 많이 소비하게 만든다.
이러한 한계점들과 테스트에 주는 영향에도 불구하고 아이트래킹은 정말 꼭 필요한 테스트 방법론일까?
만일 아이트래킹을 효율적으로 활용할 수 있는 지식이 있다면 활용하는 것이 해결책에대한 새로운 시각을 얻을 수 있는 방법이라는 생각이든다. 하지만 아이트래킹은 반드시 거쳐야 하는 테스트 방법론은 아니며 클라이언트 혹은 이해당사자들에게 무언가를 보여주기 위해 진행하는 테스트는 더더욱 아니다.
만일 이 글에서 이야기 된 한계점들을 극복하고 Context에 맞게 활용하여 아이트래킹의 장점들을 제공할 수 있다면 아이트래킹을 통한 결과물들은 테스트에 신뢰도를 높이고 정확한 의사결정을 할 수 있는 꼭! 필요한 방법론이 될것임을 확신한다.
- A fixation is conunted when a gaze is positioned on an area of approximatel 50pixels for at least 100 millisecond(1/10sec) [본문으로]
