Introduction

User Experience researchers often employ qualitative research methods often including depth interviews, think-aloud, contextual enquiry, ethnography, diary studies, open-ended questionnaire items and many others.  But how do you analyse qualitative data? Is it simply looking at walls full of post-it notes?

Of course, the process involves many steps, often transforming the data into personas,drawing scenarios, deriving storyboards, prototypes and so on. The aim of this post is to introduce a specific technique directly focused on transforming qualitative data into more insightful themes.

Thematic Analysis

Thematic analysis is a process that could be used with most qualitative information. It is a way of seeing, making sense of related material, systematically observing situations, groups, organisations, interactions, cultures and behaviours. The process is regularly used by psychologists, sociologists, ethnographers, historians, and many in other fields.

Thematic analysis enables user experience researchers, especially at the early stages of a project, to formulate their problem space and create the foundations for the design project and support knowledge sharing and communication.

The key is to transform the data into explicit codes. Encoding the data into themes, relationships, indicators and qualifications. Thematic analysis is based on three distinctive stages:

• Specify sampling criteria and research design
• Develop themes and codes
• Validate and use codes

The Analysis Process

Thematic analysis involves five main steps:

• Recognise or sense the themes using the right focal point, stepping back from the details and recognising patterns
• Develop a coding system
• Encode information
• Interpret the themes in the context of a theory or conceptaul framework with the aim of developing knowledge
• Consolidate new knowledge

The main threats to the reliability and validity of the process are projection, biased sampling and a researcher’s mood and style.  It is, therefore, important to ensure that a system of codes is developed by more than one person, structured screening of participants and managing time realistically.

Good Thematic Code

A good thematic code should have three main elements:

• Name or label – should be clear, concise and conceptually relevant
• Definition – clear statement of what it is and its boundaries
• Description – how to know when it occurs

I have recently visited the Ethnographic Museum in Kraków (Muzeum Etnograficzne). Located in the Kazimierz District, it has many examples of clothes, jewellery, furniture and other artefacts taken from different parts of Poland. The museum has a broad collection of cooking, hunting, farming, fishing and other everyday tools. In addition, religious art and musical instruments are presented alongside models of older buildings. The museum also has a few reconstructions of the interiors of rural cottages. Taken together, it provides a rich insight into life in rural (particularly south) Poland in the last 100 years with a particular emphasis on agriculture, the folklore and costumes people use to wear at religious festivals.

Early 20th Century Classroom

One of the rooms in the museum documents a typical classroom from the period before WWII. The class has a small kitchen area that includes an oven and a bench for the children to sit and have their lunch. In addition, many of the subjects were taught using tactile aids such as a compass, globe, maps, and a wooden abacus.

Based on an article I have written for the Flow Interactive Think Blog.

Ethnographic research involves the study of people and groups as they go about their everyday lives. The ethnographer participates in daily routines within the context of the research setting, observes what is going on and systematically records his or her experiences and thoughts. Participation based on social and physical proximity is key to this process.

As part of my work at Flow we frequently use ethnographic research methods to gain a deep understanding of the social and working lives of people who use different products and services in different contexts. The findings provide richer insights into service and product design requirements and opportunities for innovation, particularly when designing for global and multi-cultural audiences.

One of the key questions around ethnographic research is how its findings are transformed into design. One example of such a process was presented at a recent UX Brighton: ‘A Journey from Ethnography to Design’. The event included two speakers: Simon Johnson, User Experience Consultant at Flow and Miles Rochford from Nokia. Simon spoke about the ethnographic research and subsequent design that he completed for the Environment Agency. Miles’ presentation focused on using ethnography to design products for emerging markets.
The Environment Agency commissioned Flow to conduct contextual research and subsequently design an interactive map that will provide users with coastal erosion information – a national project that will affect 2.1 million houses on the coast. The key objectives were to establish what an erosion map should look like, how it should work and what sort of information should accompany the map.

Claire Mitchell, Flow’s Principal Consultant on the project and Simon started the project with a research phase that included ethnographic field observations in two coastal settings: Norfolk and Hastings. Simon spent two weeks documenting the lives of coastal communities, interviewing local people and immersing himself in their lives. Additionally, Simon interviewed eight professionals at Flow’s experience labs in London.

Ethnography enabled Simon to apply his empathy and humanistic values to drive the project. It was clear that his findings provided the Environment Agency with a rich understanding of the concerns, information needs and myths that people who live in rural coastal communities might have. Simon described how his research findings confirmed some of EA’s current thinking, provided new insights and defined the subsequent design process and deliverables.

The research that Claire and Simon conducted described how emotive the coastline is, an institution in British history that invokes strong feelings and forms a strong part of a shared heritage. The implications were the need for the Environment Agency to communicate that it cares and to reassure people that action was being taken to protect the coast. It was also clear that people trust locals and distrust central government, erosion maps caused alarm and that a certain amount of local knowledge derived from ignorance and/or myth. An example of a myth was the commonly repeated argument that the government was making money dredging ‘their’ sand.

The design approach focused on a simple website that addresses the needs of both professionals and locals. Claire and Simon decided that the design should answer core questions and myths, stick to plain English, use local materials and represent risks without alarming local people.

Is it Ethnography?

After the presentation the audience participated in a lively debate, which had a particular focus on the true meaning of ethnography. For some designers ethnography was a new concept and their reactions during the Q&A sessions and after the presentations indicated that they found both Simon and Miles’ presentations truly thought provoking. Some felt that rapid ethnography with a specific structure and design agenda was different from “ethnography” and needed a new term associated to it.

Theoretical research has two main aims – the validation of existing knowledge and the acquisition of new knowledge. Flow uses research to acquire and validate specific knowledge, the context in which services and products are used. Flow uses principles and techniques taken from social sciences such as sociology, anthropology and psychology to inform design decisions. Our main aim is to design solutions that work outside of design studios, laboratories and meeting rooms. As a result, we often use appropriate research techniques to focus on specifically targeted contexts and activities. A term that is often used to describe this work is Design Ethnography.

Simon’s presentation

_____________

Many thanks to Danny Hope and former Flow Consultant Harry Brignull for organising the event.

–Ofer Deshe

Introduction

Aristotle maintained that women have fewer teeth than men. Although he was twice married, it never occurred to him to verify this statement by examining his wives’ mouths. Assuming Aristotle is academia and his wives practitioners in industry (or vice versa), a lack of communication and verification also appears to be occurring in the HCI domain. Knowledge, skill and requirements exchange seems to be limited. But it is not our aim to blame Aristotle entirely for not checking the reality. His wives could have, and perhaps should have, volunteered the information if they were aware of Aristotle’s need to know. The real problem in HCI is that neither industry nor academia seems to communicate their requirements or knowledge to the other in a timely manner, despite the potential for a harmonious and mutually beneficial marriage. While it has been argued by some that academia should concern itself with stimulating research to advance the discipline, here we argue that instead a mutually beneficial relationship should be facilitated. With a lack of suitably skilled practitioners in HCI (Harper et al, 2008), it is important to improve communication between the two to ensure graduates receive appropriate training for industry. Improved communication would also provide academia with first hand exposure to commercial problems that could be researched.

HCI, usability and user experience (UX) are complex and constantly changing. The technological capabilities and complexity of projects is advancing rapidly. Unfortunately, the knowledge and skills needed to address these changes are not necessarily keeping pace. For example, not enough is known about context and etiquette of complex interaction such as web 2.0 applications. The business reality in which practitioners work is so diverse and complex that it is often difficult to get quick and practical answers from academic research, even if academic research is cutting edge. In addition, research results from academia often have such a narrow focus that they cannot be applied across a variety of industry projects. But the problems with knowledge and skill transfer are certainly not one sided. Industry is often poor at making its requirements known to academia. Perhaps due to confidentiality restrictions and business realities of limited time, commercial projects are only rarely presented at conferences. The resources required to support industry placements also reduces the number of students that can be offered first hand experience. These factors further reduce knowledge and requirements exchange.

Knowledge and Skills

Practitioners need knowledge, skills and abilities to be effective. Graduates need to not only be knowledgeable about fundamental human requirements, but also skilled in the application of this knowledge for them to be useful as soon as they enter the commercial world. Knowledge is the understanding of the specific information needed to perform a job and skills are the proficiencies needed to perform that job that are enhanced during practice (Muchinsky, 2000). Without innate ability, knowledge and skills are less likely to be able to be applied. But knowledge needs to be applied for it to become a useful skill. While Academia provides knowledge to graduates, it is unclear whether this is the most appropriate knowledge needed for an industry based career. There is also limited opportunity for students to apply their knowledge and hone their skills. A symbiotic relationship between industry and academia needs to be facilitated to overcome the limitations suggested here.

Usability and UX companies need to offer opportunities for students to apply the knowledge gained in academia. A student with up-to-date theoretical knowledge of best practice gained at university could also help provide industry with a timelier link to academic knowledge which could then be applied. In return, when the student returns to university their industry knowledge could be used to fuel future research projects. While academia teaches best practice and industry aspires to achieve the ideal, industry often has to take short cuts because of commercial realities. Academia often looks at only small parts of problems with scientific rigour that is not possible in commercial reality. But if academic research was more applicable it could make the industrial process quicker and easier.

Design patterns, guidelines and standards

Existing broad HCI guidelines are available but they are often abstract, generic, obscure or obsolete rather than design patterns that can be applied to specific projects. Industry needs both up-to-date broad spectrum guidelines and more specific design patterns to reduce the need to reinvent the wheel for each project. To illustrate, in medicine penicillin works for certain types of illnesses but broad spectrum antibiotics are also available to treat a wider variety of illnesses. HCI’s broad spectrum remedies are standards and guidelines that can be applied across a variety of projects. But an HCI penicillin, perhaps in the form of design patterns, appears to be lacking. There is also no obligation in industry to apply established guidelines and standards, which often do not adapt quickly enough to the constantly changing business realities to provide real solutions. The generality, lack of immediacy, lack of obligation and business reality mean applying the guidelines or standards is not always possible. Even if design patterns were available (knowledge), they only tell practitioners what to consider, not how to design (skills and ability). The more knowledge is standardised the quicker practitioners can find solutions to applied problems and reach design decisions. Improved communication of commercial needs to academia and research findings to industry may provide research opportunities and findings that help create useful design patterns.

A small survey

To find out what knowledge and skills gained during their academic education were seen as beneficial in an industrial setting, and to avoid Aristotle’s mistake, we polled our colleagues at Flow. Our convenience sample of eleven HCI practitioners revealed that a variety of academic disciplines are feeding into the HCI industry. The majority of respondents have postgraduate education, in either Design, HCI, Ergonomics, Media and Communication or Psychology, suggesting communication channels need to be established between industry and several different academic disciplines. Respondents were generally in agreement that the two most valuable aspects of their formal education were fundamental knowledge related to research methodologies, data analysis, models of psychological theory, ergonomics principles; and the application of the knowledge. All respondents stated that practical experience of any type, but especially that gained on industrial placements or applied projects, was highly valuable. Practical exercises in paper prototyping, report writing and user research were valued, as was the opportunity to present research findings at conferences which helped gain useful skills for client facing work. Respondents commented that students also value input in their teaching from industry practitioners who offer visiting lectures or seminars or those who are involved in setting or evaluating real project briefs for students. The seminars or lectures provide students real world examples of the type of projects they might work on if they enter HCI. In addition to opportunities to apply their knowledge, project management, group work and written and verbal communication experience were seen as valuable for the consulting world.

Required Knowledge and Skills

As well as a good grounding in the basics, students need real, applied, experience of working on commercial projects, or at least responding to actual briefs, and getting feedback from industry practitioners to be able to enter an industry job at the skill level required. Without the practical experience it is not possible to know how to apply the knowledge learned to different and complex projects and be able to make informed decisions about which methodology to use and which trade offs to make. Learning in abstraction is a problem, as industry needs problem solvers who can think on their feet. Consulting and client facing skills are also needed in HCI related courses. Knowledge and experience of commercial realities and drivers behind delivering a good UX, such as client relationship management and team working were seen as necessary skills. The priorities, expectations, pressures and practicalities that come into play in a business context, and how to respond to difficult client situations were seen as lacking in current courses. Several respondents also believed that academic programmes lag behind industry as courses do not change rapidly enough to keep up with current methods and problems.

Future Research

Research to clarify the real requirements and desires of both is needed for a harmonious marriage. A starting point may be more rigorous research into which fundamental knowledge is required for students and which disciplines graduates are entering HCI industry. It is necessary to explore whether there is enough crossover between disciplines, whether it is realistic that graduates will have or want enough crossover to be effective and whether one discipline is able to provide all relevant knowledge. It is necessary to explore whether industry wants or needs research from academia; whether an over reliance on academic research may stifle innovation in industry; or whether academia needs to push the boundaries of the industrial reality in an effort to keep informing. If academia just taught what is happening in industry then it is possible that no new knowledge would be introduced. A means to speed up the exchange of requirements and findings is also worth exploring. Additionally, the actual user cost and commercial realities are not well understood by both sides and could be further explored.

Conclusion

Further discussion between industry and academia about how a mutually beneficial relationship should be developed. Industry needs to let academia know what skills graduates need, and what is happening in the commercial world. Industry also needs to provide more opportunities for students to apply their knowledge. Academia needs to facilitate the learning of skills as well as knowledge and needs to ensure timely exchange of research findings. Aristotle and his wives need to learn to communicate their requirements for knowledge. Perhaps a bit of marriage guidance can be offered. 

References

Muchinsky, P.M. (2000). Psychology applied to work. (sixth edition) Wadsworth: Belmont, CA.

Harper, R., Rodden, T., Rogers, Y. and Sellen, A. (eds) (2008). Being human: Human-computer interaction in the year 2020. Microsoft Research Ltd: Cambridge.

- Edited version of an article written by Chandra Harrison, Ofer Deshe and Kelsey Smith

- Presented at the British Computer Society HCI Workshop  – The challenges faced by academia preparing students for industry: What we teach and what we do.

Photographs have been used in ethnographic research for many years. Initially, pictures were used to support written text. However, in the past 10 years we are seeing a shift into using photographs as the main medium for capturing evidence. Pink (2001) has coined the term visual ethnography, emphasising the move into using a camera as one of the main tools in an ethnographer’s toolbox. Pictures could also be taken by participants in isolation from the Ethnographer’s notes. Therefore, creating a richer insight into their participants’ social life. One such technique is based on the creation of collaborative or individual photographic diaries.

The natural progression is to move into video diaries. Perhaps due to the success of video sharing websites such as YouTube, we have seen a new generation of pocket video cameras slowly emerging.  Many of these models have no lens caps, complex settings or a plethora of dials. One of they key factors that differentiate these cameras is the simplicity and ease-of-use. For example, the Flip Mino includes built-in software for sharing movies through email or direct uploading to YouTube and MySpace. Furthermore, some models include a built-in USB connector avoiding the need for any wires. These mini cameras are relatively affordable, yet often include high-definition resolutions. 

Pocket video cameras open an option for a new type of ethno-videographic research. Each participant is simply provided with a mini video camera, clear instructions and a structured diary.  The best cameras for this purpose are the Flip Mino HD and the Kodak Zi6. Both have a simple interface, capture high-definition videos and include a built-in USB connector. The Flip is smaller, but the Kodak captures higher quality video.

Kodak Zi6: USB arm flicks out from the side and videos are automatically uploaded

Whilst, cheap video tools could capture a wealth of information it is important to ensure that the approach is useful and appropriate for the type of research questions, the focus of the study and the context. It is also important to acknowledge that analysing video studies is often more laborious than analysing photos and text. 

- Ofer Deshe

References

Pink, S. (2001). Doing visual ethnography. London: Sage Publications.

As part of my ethnographic work I use video and stills cameras to document people’s behaviours, work and social dynamic. These photos, of course, are confidential.

The photos below are a collection of some photos I can share.

The most important aspects of choosing photographic equipment for ethnography is to combine non-intrusive, understated, small, subtle and simple cameras. After all, you want people to be as natural and comfortable as possible. Therefore, I tend to choose cameras that are more suitable for street photography such as range-finders, journalistic compact cameras that provide manual controls, or small digital SLR cameras – ideally with flat (pancake) lenses.

- Ofer Deshe

The original iPod Shuffle had a clear proposition; a cheap, simple, and minimalist music player. Technically the first iPod to use flash memory and physically the smallest, it lacked any display, scroll wheel or playlist management features. The constant need to control, skip, manage, and navigate was replaced with a simpler and more relaxed lack of control. Instead, the device simply shuffled music at random or played tracks in order. The front of the device had a simple interface:

• Play/Pause
• Next Song/Fast Forward
• Previous Song/Fast Reverse
• Volume adjustment

On the back was a three-position switch to play music in order, or shuffled or to turn off the unit.

The second generation was just as simple as the first, but had more storage and less than half the size.

On 11 March, 2009 Apple released the third-generation Shuffle.

One of the key design decisions was to simply not have any buttons, apart from a button that switches between ‘Off’, ‘Random’ and ‘Order’. All controls are now part of the earphones.

According to Apple:

The new iPod shuffle is amazingly small and even easier to use.

But is it easy to use?

Soon after the launch, forums, social media networks and blogs started to criticise the new Shuffle. Whilst some disliked being dependent on using Apple’s earphones, others criticised the new interface. Physically, the control could be difficult to grab during sporting activities or for those with bigger hands, but more importantly, most of the functions are controlled via a central button that has no labels.

The lack of visual display raises a design challenge. Unfortunately, touch is a less accurate sensory mode than vision and typically slows users down. Apple could have designed an interface with elements that could have been recognised through tactile attributes such as feedback, resistance, size, weight, and/or texture. The father and grandfather of the current Shuffle are good examples of such a control. However, the solution was to include VoiceOver technology, a feature that speaks 14 languages and plays back track and playlist names. Although a minor issue, VoiceOver struggles with pronunciation. Perhaps in the future, additional language information will be included with each track.

One button, nine commands

Apple was once famous for the one-button mouse. Although historically some suggest that this decision was based on cost, it was often argued that one button is all you needed. The one button became a symbol of simplicity. In contrast, the new iPod Shuffle has a central button that accepts many user commands. In fact, this ‘one button’ handles 9 inputs:

• click
• click & hold
• double-click & hold
• triple-click & hold
• triple-click and hold within 6 seconds
• triple-click and hold after 6 seconds
• click until a tone is played
• click until you hear a name
• click & hold until exit

There is no direct cognitive mapping between users’ goals and the required actions. For example, why does ‘double-click and hold’ fast-forward a track, but ‘triple-click’ rewind? Moving to the next track is achieved by double-clicking. So how do you expect to move to the previous track? Did someone at the back just say “triple-click within 6 seconds of the track starting”? Correct.

Poor learnability?

Ultimately, if the device is used frequently, many users will move from being novices to experts. However, if learnability is poor, some users will remain perpetually intermediate and potentially frustrated.

Learnability is inherent in interface features that allow novice users to understand how to use a device and how to attain optimal (level of) performance. Learnability is based on five main factors:

• Predictability: operation visibility – what elements of a user’s knowledge from past interactions with this device can help to determine the outcomes of future interactions?
• Familiarity: guessability – to what extent can users’ knowledge from other systems and real world objects could be applied to interactions with the device?
• Synthesisability: how does the interface help a user to asses the effect of past actions on the current state?
• Consistency: likeness in behaviour. In this context consistency is related to user inputs and the subsequent output responses.
• Generalisability: support for the user to extend their knowledge to other applications and devices.

In the case of the Shuffle, lack of visual display could slow down skills acquisition. The design is not based on familiar metaphors, and poor learnability may increase cognitive load. Many users will not be familiar with the Shuffle’s interface, apart from iPhone users who use a similar earphone-based controller.

As a robust device consistency between actions and auditory feedback that arises as a consequence will be high and usage is likely to be frequent. Therefore, for many users learnability will eventually take place. It is possible that Apple plans to extend this interaction model to other devices. For those who mastered this control acquiring this skill will become generalisable and useful.

Does it matter?

So it might be difficult to learn. Arguably, it has beautiful industrial design, long battery life and impressive storage capacity. Apple could sell many of these fashion accessories to users who would simply click to play whilst sitting on the train, running, or going to the gym.

The best way to gain an in-depth insight would be to test the Shuffle with users. In the next few weeks we are planning to evaluate the Shuffle in one of Flow’s experience laboratories and analyse its performance and user experience.

- Ofer Deshe, Principal User Experience Consultant

Thanks to Thibault Baradat-Bujoli for the original illustrations