Surface Pro X
Overview
Note: In compliance with my non-disclosure agreements, I have omitted confidential information about this project.
Goal
The study evaluated people's experiences using the Microsoft Surface Pro X (2-in-1 laptop), Surface Pro X Signature Keyboard, and Surface Slim Pen bundle in the months following the launch of the Surface Pro X to understand customer sentiment and product strengths and weaknesses in order to improve the next generation of products.
Method
3-part study with 13 participants—8 ”buyers” (who bought the bundle on their own) and 5 “seeded” (who fit the buyer profile and were provided the bundle):
User testing sessions focused on unboxing, initial setup and use, and first impressions
Online diary study over 4 weeks
In-depth 90-minute interview at conclusion of study
My Role
Wrote the protocols and materials (e.g. session guides, scripts, questions)
Adapted the study to run safely during shelter-in-place protocols due to the COVID-19 pandemic
Conducted/moderated sessions
Took notes and video
Analyzed and organized the data
Created reports, including items such as key takeaways, details, metrics, video clips, and recommendations
Presented results to hundreds of people across multiple teams and stakeholder audiences, tailoring each presentation appropriately
I collaborated with finance, project management, a UX research manager, another researcher who handled logistics (e.g. obtaining materials) and provided feedback on my work, an external recruiter for participants, an external researcher who moderated some of the interview sessions, and an external testing facility.
Impact
The findings provided foundational knowledge for user perceptions of the three products. My presentations to various teams reached wide audiences, educating stakeholders and strengthening buy-in for research efforts to inform future versions of these products.
Approach
Recruitment
The study had 2 participant groups:
8 “Buyers” - People who had already purchased the Surface Pro X bundle independently with their own money right before we recruited them
5 “Seeded” - People who were looking for a new computer; were not employed in technology, hardware engineering, media, or journalism; and fit the marketing/target customer profile for the Surface Pro X
For the Buyers group, I worked with a UX research manager who then coordinated with the marketing department to secure participants who recently bought the products. For the Seeded group, I worked with an external research and recruitment vendor.
Out-of-Box Experience (OOBE)
User testing sessions (Seeded group):
Why? - To learn what the first touchpoints are like, from the moment of seeing the packaging to the first time using the products, so that I could understand users’ thoughts, feelings, and behaviors during the first impressions window.
The original plan was to have sessions fully in person, but due to the COVID-19 pandemic, I pivoted to having participants sit alone in a clean lab room with the devices and interact with me via video instead.
With the bundle of the three products provided, I instructed the participants to imagine they had just received the products as a gift and asked them to open and physically set up the devices while “thinking aloud.” I gathered feedback from observations, spontaneous comments, probing for comments, and Likert scale ratings to learn about the details of the unboxing experience, asking about things like ease of opening and device removal, thoughts on the look and feel, intuitiveness of the packaging, and overall satisfaction.
After physical device setup, I had users go through the digital setup and first use on the device and gathered feedback on those experiences as well.
Online diary study (Buyers group):
Why? - I wanted to learn the same things as with the Seeded group, but Buyers participants had already unboxed and set up their devices before we contacted them.
Since I could not run a timely dedicated usability session with Buyers, I instead had them complete a diary study entry where they answered survey questions and recorded themselves talking about what they remembered from the setup process.
(Limitations: This method relied on participants’ memory, which may not always be accurate. It was also less detailed and unmoderated with no opportunity to spontaneously probe on any comments provided.)
Weekly Check-Ins
Online diary study (both groups):
Why? - To learn the highlights and lowlights of using the Surface Pro X as told and shown first-hand by the users and to track satisfaction over time while staying safely within COVID-19 protocols.
Both groups went on to use the device bundle for at least a month and used dscout to keep an online diary where they completed check-ins every week for 4 weeks. Each check-in included survey responses, photos, and videos.
Deep Dive at the 1-Month Mark
User interviews (both groups):
Why? - To gain a deeper understanding of how the device fits into users’ lives “in the wild,” and to get rich feedback on specific features, likes, dislikes, uses, and overall thoughts.
Originally, I planned to conduct home visits and have participants show me how they use the Surface Pro X at home in different rooms, contexts, postures, and activities and then have in-depth discussions to reflect on their experience over the past month. However, I once again had to modify the in-person approach due to COVID-19 pandemic guidelines and adapted the protocol to be remote instead.
In this final phase, I moderated 90-minute sessions via video chat that still included a home tour along with everything I initially planned.
(Limitations: Adapting the home visit to be online meant my view was limited to whatever could fit in frame on the participant’s device, whatever views the participant was able to physically set up, and at the video resolution their device could provide.)
While I was the primary researcher and moderator, I partnered with another researcher to moderate a few of the sessions in this final phase due to workload constraints.
Timeline
March 2020: OOBE (unboxing, setup, first use)
March-April 2020: Weekly Check-Ins (online diary study)
May 2020: Deep Dive (home tour and in-depth interview)