Healthcare Technology Management (HTM) Part 4-Observation Observation for HTM

Chapter 4. Observations

Section 4.1. Setting the Stage

When people with similar training and experience are presented with a particular
situation, it is not uncommon to find that they take fairly different approaches to managing
the situation. This is not necessarily because one person has a better approach or more
information than another, but because there are many factors that influence how people
perform their work. In healthcare, we believe people inherently want to do their work
safely and effectively, and that when their performance is unsafe or ineffective, there are
factors, which they may or may not be aware of, that influence their performance.

These factors come from a combination of internal and external sources that can
vary over time. Examples of external factors that can impact work include equipment
design, the physical layout of a workspace, expected workflow and work practices,
organizational policies, team dynamics, and organizational culture. Internal factors, or
natural human limitations (Chapter 3), that can affect our work include our ability to
remember multiple units of information, or to pay attention to many things that are
happening at the same time. People also vary in terms of their skill level and ability to
perform certain tasks based on factors like age, level of training, and experience.

Observing people as they interact with technologies, environments and each other
provides us with a window into how people do things. Seeing the factors that shape how
staff approach their work gives us insight into existing issues and possible areas of
mismatch between people and the systems they are a part of.

Section 4.2. What are Observations

Observations, sometimes referred to as ethnography or shadowing, is a data
collection technique where a human factors specialist watches, or observes people
performing their regular day-to-day job duties in their normal work environment. A variety
of data collection tools may be used to collect either structured or unstructured data while
observing. Examples of these tools include note taking, photographs, task analysis
software, artefact collection, and sometimes, video recording.

Section 4.3. Why use Observations

Conducting observations is a relatively low cost, valuable means of gathering rich
information about users as they perform tasks in their work environment. Through
observations, you can learn which clinical users are interacting with a device, what tasks
are being performed, what other aspects of the environment are relevant to using a device,
and see actual challenges, workarounds, and strategies as they are happening in real time.
Observations provide essential context for defining problem scopes, and often lead human

28

factors practitioners down unexpected paths of discovery that end up being key to the
issue at hand.

Observational data tend to be complementary to other qualitative data like
interview, focus group, or survey data. Thus, interviews, focus groups and surveys should
not be used in lieu of observations. Observations are essential for understanding how
things happen, while interviews, focus groups, and surveys are more useful for better
understanding why they happen from a subject’s perspective.

It is important to observe rather than ask a subject how something happens because
it is common for people to perceive step-by-step processes differently than they happen in
reality. This is due to inherent human limitations like cognitive biases (Section 3.4) (e.g.
confirmation bias, inattentional bias, groupthink), and because people do not tend to think
about well-known processes in discrete steps. Through observations, the human factors
specialist can identify gaps between perceived actions and actual events in an appropriate
level of detail.

From the biomedical technology professionals’ perspective, conducting
observations will be helpful for:

• Identifying the different user groups of a device

• Understanding what a device must do to support users’ tasks

• Understanding how users currently interact with a device

• Understanding how a device fits within the larger system (e.g., how it integrates

with existing information systems, checklists, or related protocols and policies)

• Identifying challenges and risks with a device already in use

• Determining different levels of knowledge across users, and how well users

understand device operation

Section 4.4. When to use Observations

Observations should be conducted whenever the human factors practitioner is
unsure exactly how or why something is happening in the field. Observations should be
considered the “go to” method, as almost always, it will be the starting point for any human
factors analysis. Since observational data is complementary to data gathered using other
qualitative data collection techniques, observations should be done whenever interviews,
focus groups, or surveys have also been conducted.

29

Section 4.5. Preparing for Observational Data Collection

Section 4.5.1. How to structure data collection

Deciding whether to use a structured or an exploratory approach to data collection
will depend on what you, the observer, already know about the users, devices, processes,
and environments of interest. When a detailed understanding has already been achieved, a
more structured data collection approach can be used; however, if little is known, an
exploratory data collection approach should instead be taken. Figure 5 provides some
guidance about whether a structured or exploratory data collection approach should be
taken based on what you already know about the users, devices, processes and
environments of interest.

Figure 5. Considerations for a structured versus unstructured approach to data collection

Section 4.5.1.1 Exploratory data collection

When an exploratory approach to data collection is taken, it means that
observations are aimed at developing a general understanding of factors that will influence
how a technology is used and whether it will be used successfully. It starts with an
understanding of who the users are, what tasks they perform, and potential issues related

30

to the use of a particular technology. A structured approach to data collection usually
follows exploratory observations because a detailed baseline understanding of the system
of interest is needed to be able to develop appropriate data collection tools (e.g.,
task/workflow checklists) to support the goal of the structured observations. When taking
an exploratory approach, a formal data collection tool is not typically used. Instead, notes
and photographs of what you are seeing should be recorded in real-time, or immediately
following an observation session.

While conducting exploratory observations, make note of things like:

• What technologies are being used?

• When, why, and how are technologies being used?

• What are the environmental conditions in which the technologies are being

used(noise, lighting)?

• What are the space configurations of the work environment?

• What processes are being carried out?

• What information (e.g., forms, charts, electronic interfaces, or policies) is

being used?

• What knowledge, skills, training, or education is the user accessing?

• What are the goals of the user?

• What problems, challenges, workarounds or strategies are being used?

• What are the inputs and outputs of what you are observing?

• How do people work together relative to the device?

Section 4.5.1.2 Structured data collection

When a structured approach to data collection is taken during observations, it
means that users are observed with a specific set of questions in mind, for example, what
tasks do nurses perform with a particular telemetry monitor, and where do they perform
those tasks? When taking a structured approach, a data collection tool is typically used to
track observational data against a specific set of questions. The purpose of a data collection
tool is to ensure each observer is capturing the same information during every
observational session. The actual data collection tool you choose to use can vary, but
generally, a simple paper tool with dedicated space for each required piece of information
will be sufficient. The data collector usually designs the data collection tool based on what
they expect to observe, and the goals of the observation sessions. Typically, preliminary
exploratory observations will be required to determine which data elements you want to
capture in a structured format. Once the data collection tool has been designed it is
recommended that you pilot test it in the field prior to data collection to ensure it will meet
your needs during observations.

31

Section 4.5.2. How to initiate contact

Before going into the field to conduct your observations, it is important to first
contact the managers of the respective departments to explain the project and rationale for
conducting observations. Getting managers to agree to support the project is critical to
gaining access to front line staff, and ensuring these users are willing to participate in the
observational sessions.

Once a manager has agreed to have their unit participate, it is advisable to:

• Plan the timing of your observations

• Determine whether there are any clothing or footwear requirements on the

unit

• Determine whether a signed consent form will be required from each staff

member and/or patient being observed (see Appendix A for description and
sample

• Introduce the project and any observers (data collectors) to staff, ideally as

part of a staff meeting

Establishing a contact person, who may or may not be the manager, can be
extremely helpful in accomplishing these tasks. A contact person will be required for each
clinical area where you will be observing. In addition to these preparatory tasks, your
contact person can also be called upon to help you collect artefacts such as forms, or
disposables, and to verify the observational data after it has been summarized for accuracy.

Section 4.5.3. Timing

When planning the timing of your observations, you will want to discuss with the
manager to find a time that works well for staff. You may also want to consider observing
staff during the beginning of shift, the end of shift, at shift changeover, and during peak
times. Observing during these more complicated times can provide unique insights in
comparison to more typical non-peak times. Plan to meet your contact person in the unit
on the day of your first observation session.

Section 4.5.4. Making participants feel comfortable

The most important aspect of conducting observations is to make sure staff feel
comfortable while they are being observed. This means those being observed should know
it is not their performance that is being evaluated, but rather, they are being observed to
understand whether their needs are being met by the technologies, processes, and
environments they interact with. If you see staff having difficulty, or working around
aspects of a system, these are opportunities to learn about the challenges they are facing,
and to think about solutions that could support them in making their work safer and more
efficient.

32

Staff should feel comfortable knowing you will not share the details of their work
practices, or opinions, to others, in particular with their superiors; if you can make staff
comfortable as you are observing, they will be much more willing to participate in the
observation process and to share information with you. One way to make this clear is to
provide participants with a consent form for them to sign that explains how the
information collected during the observation session will be used. Depending on the type of
work you are doing, this may be required. Appendix A: Confidentiality and Anonymity
describes how and when to get informed consent and provides a sample consent form
template.

Staff can be a wealth of information and generally understand the issues better than
anyone since they live and breathe these technologies every day. Although staff often have
great ideas for solutions, sometimes they do not tend to bring them forward because there
do not have any experience of their ideas resulting in changes, and may feel disempowered.
When staff are observed and have the opportunity to bring their experiences, challenges,
and ideas forward to someone who is responsible for incorporating their feedback into a
change process (e.g., a procurement process) it can be a very empowering experience for
all involved. Often staff who are more senior, extroverted or opinionated are given more
opportunities to contribute. When possible it is advised to seek out staff participants who
represent different perspectives (e.g., senior and novice, extroverted and opinionated as
well as quiet and more reserved) to ensure a wide range of needs and ideas are included.

Section 4.5.5. What to bring

Helpful materials for conducting observations include:

• Appropriate clothing, ideally with pockets, that helps you blend in (e.g., a lab

coat, scrubs, or whatever is generally worn in that area)

• Appropriate footwear (e.g., closed-toe shoes)

• An ID badge

• A clipboard, or light, hard surface for writing

• A notebook and/or data collection tool

• Consent forms (if required, see Appendix A: Confidentiality and Anonymity)

• Two pens (different colors may be helpful)

• A camera

• Refreshments (e.g., coffee and muffins) for all staff on the unit (optional)

33

Ensure you are not planning to bring the following;

• Your own food/drink

• Heavy items, or things that are awkward to carry

• Uncomfortable clothing or footwear

• Scented items (items with heavy perfumes or deodorizers)

• People who are not part of the project team

• Vendors of the technology being investigated

Section 4.6. Observational Data Collection

Section 4.6.1. Initiating the observation session

Start by finding your contact person at the scheduled observation time. If you are
unfamiliar with the clinical area, ask your contact person to take you on a quick tour of the
facility to show you where things are located, and ask them to introduce you to a few of the
staff members on shift. When you are ready to start observing, approach one of the staff
members and provide a brief introduction to yourself and the project. Explain that you are
hoping to observe how tasks or processes are conducted, and that you are not there to
evaluate their performance, but rather, you would like to observe them to understand
whether their needs are being met by the technologies, processes, and environments they
are interacting with (you can state this more specifically in the context of the technology or
system you are focusing on). Ask if they would feel comfortable being observed, and tell
them they can stop the observation session at any time. If they are open to being observed
and a consent form is required (Appendix A), ask them to sign it; if they say they would not
like to be observed, approach another staff member and introduce yourself and the project
again. A sample introductory script is provided in Figure 6 below.

34

Sample Introductory Script

“Hi, my name is Charlie and I’m from the Biomedical Engineering department. Our
hospital is buying new infusion pumps and I’m part of the team that is evaluating the
available devices to see which would be best for our hospital. As part of my evaluation I’ll be
looking at how well the different pumps meet your needs as a nurse, and how easy they’ll be to
use.

What I’d like to do today is observe you and take notes as you use our current infusion
pump. When I observe, I won’t be evaluating you, but rather, I’m interested in learning more
about how our current pump fits into your workflow. I’ll be looking at things like the different
ways you use the current pump, where you get the information required to program it, what
documentation you have to fill out for each infusion, and what items you use in your
environment, like orders, forms, labels, tubing, and cleaning supplies, that support your use of
the pump. I’d also be happy to hear any of your thoughts about the current pump so we can
consider these things going forward with the new pump. Any ideas you have about how the
new pumps should work would also be extremely helpful.

It will probably feel a little strange to have me watch you at first, but if you agree,
please know that I am not collecting any information about your performance, or evaluating
you in any way, and I will not share my observations of you with your colleagues or your
superiors. I am only interested in information related to the pumps and how to make the new
pumps work best for you. In my notes, and when I share this information with the project
team, you will never by identified by name. Also, you can stop the observation session at any
time. Just tell me if you would like me to stop observing you. Does this all sound okay with
you?”

“Yes, this is alright with me”

“Great, thanks! Also, would it be alright if I took some photos as we go? I will make
sure not to take any pictures of patients, identifying, or personal health information. If any
personal information is captured by accident, it will be blurred”

Figure 6. Sample introductory script to initiate observational session

Section 4.6.2. While Observing

At first, it is likely the person being observed will perform their tasks more carefully
or consciously than normal (see discussion of Hawthorne Effect in Section 4.8.1), but over
time, if you are able to observe without interfering and can make the staff member feel
comfortable with your presence, they will begin to act normally and you will see a more
accurate representation of how they work in reality. As you observe, try not to interfere,
and make sure your observations are not impacting the staff member’s ability to provide
patient care. Also, be sure to consider the comfort of any patients and family members

35

involved. It’s generally a good idea to ask the nurse you are shadowing to introduce you to

Tip:

It is common not to notice any issues

during the first shadowing sessions at the

issues/contributing factors were identified.

patients (if they are conscious) and their family so it is clear why you are there and
recording notes. Also, be sure not to make any comments or raise any questions that could
be perceived as alarming to patients or their family in front of them. Find a time away from
the bedside to ask these questions.

If a structured approach is being taken while observing, use your data collection tool
to record any observations. You may find it difficult to fill out a structured data collection
form in the field, even if you have invested time to develop it. While shadowing, tasks may
not happen in the sequence you are expecting, and challenges may arise that you never
anticipated. Be prepared to adapt your approach to data collection as you are observing, to
make the most of your time (e.g. take notes during shadowing and fill in the data collection
form immediately after your observational session).

If an exploratory approach is being taken, use your notebook to record notes about
what you are seeing as it is happening. A common pitfall while observing is to get caught up
in “watching the show”, rather than recording notes about what is being observed. Another
common experience at the start of exploratory data collection is a feeling of not knowing
what to look for, or not seeing anything that seems relevant. These experiences are normal.
It will take some time to become familiar with the environment and to differentiate what to
focus on given the goals of your work. If you are not used to being in the clinical
environment, it can take some time to get used to being around patients and understanding
all the various equipment and systems before you can start to distinguish what is relevant
to your project. If you are having difficulty identifying and recording what is relevant while
observing, think back to the original reason for conducting observations and start there;
keep in mind though, that you want to soak up as much as you can about what is happening
around you. Although you may not see important issues right away, after some time
observing the issues will begin to emerge.

start of a new project. This is true for both novice and expert observers and occurs
because each project requires a certain amount of contextual understanding to be able
to identify potential issues. As your contextual understanding grows, so too will your
ability to identify issues. In a field study conducted by the primary author on multiple
intravenous infusion safety, it took three days of observations in the intensive care
setting before issues started to emerge. By the end of the project over 100

If possible, having more than one data collector is extremely helpful because a lot of
information is learned and processed by talking through what has been observed. Also,
planning to have some downtime between observation sessions is important, to allow time

36

to think about what you have seen, identify additional things to look for, and to debrief, if
possible. Since different activities may happen on different days and at different times of
day, taking a break can facilitate observing different sets of tasks, or at more error prone
times like shift-change (patient hand-offs). Record your observations, impressions,
thoughts, questions, etc. immediately following a shadowing session to ensure you capture
as much detail as you can before you begin to forget what you saw. This is especially
important when conducting multiple shadowing sessions.

Observation sessions can vary in length depending on the arrangement made with
the unit, but when the observer is saturated with information or the person being
shadowed shows signs of not wanting to be shadowed any longer, the observer should take
a break from shadowing, or move on to observing another person. A good rule of thumb for
the length of an observation session is to aim for about three hours, but in reality,
observation sessions could range anywhere from about an hour to eight or twelve hours, if
an entire shift is being observed.

If you have questions as you are observing, it is important to ask them, but be
careful not to interrupt the person you are observing at inappropriate times, especially if
they are performing a task that is safety critical, or requires concentration. To ensure you
ask questions at an appropriate time, it is recommended you write any questions in your
notebook as they come up, and then ask the person you are observing to let you know
when it is alright to ask questions.

Photographs can be a very effective source of data, and are especially helpful after
you have completed your observations. Having access to photographic data can help you
remember the details of complicated technologies and environments, and facilitate
communication of your findings to the project team. While observing, try to photo­document as much as possible, but ensure the staff member (and patient, if applicable) are
comfortable and give their permission before taking any photographs. Most healthcare
organizations have strict privacy rules so make sure you know and work within these rules
during data collection. When you do take photographs, try to avoid capturing any
identifying, or personal health information of both patients and staff. If you do capture any
identifying information, it will have to be covered or blurred during data analysis to
maintain staff or patient confidentiality. If the person you are observing is uncomfortable
with you taking photographs due to privacy concerns, offer to show them any photographs
after they have been taken so they can approve them. Alternatively, you could ask the staff
member to help you set things up so any identifying information is covered up (e.g., use a
piece of paper to cover up a patient’s name on a label). Having the unit manager’s
permission to take photographs is highly recommended, and may also help to make staff
feel more comfortable. Prior to asking for the manager’s permission it is good practice to

37

review your institution’s policies regarding photos and voice recording (see Appendix A:

Confidentiality and Anonymity).

Section 4.6.3. When to stop observing

After you have spent time observing and begin to understand the relevant tasks and
issues given the context of your work, another challenge can be to know when to stop
conducting your observations. Although there is no specific rule for stopping, when you can
no longer identify new users, tasks, or issues, and reach saturation in terms of feedback
from staff, these are key indicators that it is time to stop conducting observations. Of
course, project timelines may also dictate when observations must stop. While it is not
ideal to strictly limit observation time, remember that conducting some observations is
better than none, and more is better than less.

Your understanding of the use environment will continue to develop even after the
observational sessions have been completed, as the data are analyzed and additional
human factors methods are applied. It is normal to uncover new things that are not well
understood after your observations have been completed. If this happens, do not worry;
with permission of the manager, you can always return to the unit to perform additional
targeted observations.

Sending a note of thanks to your contact person, the manager, and the unit staff is
highly recommended to let them know you appreciate and value their contributions to the
project. Commit to sharing any outcomes with them if they are interested in learning how
their input had an impact on the project.

Section 4.7. What to do with Observational Data

Whether observational data collection is exploratory or structured in nature, you
will have a large quantity of descriptive data to work with. Figuring out how to analyze this
data can be extremely daunting, and knowing where to start, especially for the novice
observer, tends to be the biggest barrier to a successful analysis. Usually, a good first step is
to organize the observational data into themes so they can be more easily managed. The
themes chosen will depend on the data, and why they were collected in the first place. For
example, themes could be chosen based on different:

• levels of the system (e.g., user, technology, process, clinical unit, hospital)

• parts of the system (e.g., user, technology, forms and documentation)

• user groups (e.g., nurses, doctors, pharmacists, clerks)

• areas of the hospital (e.g., intensive care unit, emergency department)

If the themes used to organize the data are quite broad, it may be useful to divide
the data under each theme into sub-themes for improved granularity. An example of this
preliminary step for data analysis is shown in Figure 7. For the purposes of this example, an

38

exploratory approach was used to learn about issues related to administering multiple
intravenous infusions. An excerpt of the observational data collected for the example is
included.

Data Analysis Example

Excerpt: Raw Data

- 12 beds available in intensive care unit

- pumps show volume-rate in the largest font and the dose-rate in a small font at the

bottom of the pump

- pharmacist is dedicated to the unit

- pharmacy technician restocks medication

- pharmacy technician delivers medication

- 4-5 patients transferred to unit per day from surgery

- 3 beds in cardiac surgery intensive care unit

- nurse has 2 patients

- pumps sometimes slow nurse down

- don’t always use drug library

- pump responds slowly to key presses

- sometimes nurse has to re-press buttons when pump doesn’t respond

- one time something was double entered because pump was slow to respond

- during arrests, drug library takes too long

- patients who are arresting need large volumes of fluid

- policy against agency nursing

- staff must have critical care experience

- medication order (artefact)

- photograph of pump interface screens

- photograph of medication label

Excerpt: Raw Data Organized into Themes

Theme 1: Unit Structure

- 3 cardiovascular operating rooms send patients to the unit

- 12 cardiac surgery ICU beds

- 4-5 surgery patients transferred each day

Theme 2: Staff

- All staff have critical care experience

Sub-Theme 2a: Nursing

39

- Policy against using agency nursing

- Nursing to patient ratio is 1:2

Sub-Theme 2b: Pharmacy

- Dedicated unit pharmacist

- Pharmacy technician restocks and delivers medications

Theme 3: Infusion Pump

Sub-Theme 3a: Pump Design

- Pumps display volume-rate in a large font

- Pumps display dose-rate in a small font at the bottom of the pump

- Photograph of pump interface screens

Sub-Theme 3b: Pump Issues

- Large volumes of fluid are delivered quickly when a patient is arresting, and the drug

library takes too long to use in these situations

- The pump responds slowly to key presses and if a button is re-pressed because the user

doesn’t think the first press was received, a programming error can occur; this has

happened to nurses on the unit before but it was caught

Theme 4: Information about Medication

- Photograph of medication label

- Medication order (artefact)

Figure 7. Organizing raw data into themes and sub-themes

Once observational data have been organized into themes, they are much easier to
use as inputs to a range of other human factors methods, which will be presented
throughout the rest of this handbook.

Section 4.8. Limitations of Observations

Before collecting observational data it is important to consider the following
challenges and limitations :

Section 4.8.1. The Hawthorne Effect

As you can imagine, people are likely to improve or modify their behaviour if they
know they are being observed. In the scientific literature, this behaviour change is referred
to as the Hawthorne Effect [27]. While it may seem this effect would make the collection of
reliable observational data challenging, ensuring the subject is comfortable and knows
their skills and abilities are not being evaluated but rather the device, or system they are
using, is being evaluated can help to minimize this effect. Behaviour modifications seen at
the beginning of an observational period are likely to diminish over time if the observer can

40

consistently demonstrate that they are not evaluating the performance of those observed
and are truly committed to understanding their environment, tasks and challenges. Also,
any risks or concerns identified during observations (i.e., subject to the Hawthorne Effect)
may be considered a conservative viewpoint into any challenges or issues experienced by
subjects.

Section 4.8.2. The Time Investment Required

Depending on the goal of your observations, it may take several observational
sessions with multiple staff, in various environments, to collect your data. These
observational sessions may happen in quick succession or take place over a long period of
time depending on the frequency with which certain tasks are done in the field. Often when
observing, seeing unanticipated challenges will lead the human factors practitioner to
expand the scope of their observations in order to more fully understand the factors
affecting what has been observed. Ultimately, the time invested in observing may be
dictated by external factors such as available resources, workload, or project timelines, but
when possible, observational sessions should continue to be conducted until the use of the
technology or the process can be clearly described and there are no outstanding questions
to be answered.

Section 4.8.3. Observer Bias

As an observer, you will bring a biased viewpoint to your observations. All people
have inherent human limitations affecting our ability to see, interpret, and remember what
we have seen. Like all people, we see and interpret our world through a series of cognitive
biases (see Section 3.3.1.3); we tend to see what we expect to see, and to collect
information in a way that matches our own experiences and expectations. These inherent
biases can be minimized through a greater awareness of our human limitations, and may
be more systematically addressed through a structured data collection approach (Section

4.6.1).

Section 4.9. Additional Observations Resources

Articles

1.

Diaz-Navarlaz T, Pronovost P, Beortegui E, Segui-Gomez M. Benefits of Direct
Observation in Medication Administration to Detect Errors. J Patient Saf.
3(4): pp200-207

Guides

2.

Oullet M, Rainville M, Bouchard LM, Belley C. Guide to Direct Observation of
Community Safety. 2009. Government of Québec.

http://www.inspq.qc.ca/pdf/publications/1005_DirectObservation_Vol12_e
ng_.pdf

41

Book Chapters

3.

Clancey WJ. Observation of work practices in naturalistic settings. In:

Ericsson, N. Charness, P. Feltovich & R. Hoffman (Eds.), Cambridge Handbook on
Expertise and Expert Performance. New York: Cambridge University Press, pp. 127-

145.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.89.9513&rep=re
p1&type=pdf

A.

42