IBM Watson Assistant: Managing Custom Extensions
The ask: Improve the experience for our technical developer persona (Deb) to build a custom extension from an OpenAPI document for the non-technical assistant builder (Tanya) to add to her assistant.
The team: Design lead (me), Junior Designer, Product Manager, Engineering Lead, 2 Engineers
The timeline: 2 months
The process: Stakeholder Management, Design Thinking Facilitation, Evaluative Research (Task Analysis), Competitive Research, Iterative Prototyping.
The outcome: Released, impact not measured before I left IBM.
I led the research effort to evaluate the current experience for Deb, a technical developer to build custom extensions for an assistant. I planned the research, screened, and recruited users that matched our technical developer persona (Deb). I conducted the task analysis sessions with 10 participants.
In parallel with the task analysis sessions, our team evaluated other applications that had a way to build custom extensions (Slack and Intercom).
Using this research, I led the our team to design a minimum-viable product for how our users will be able to build and manage custom extensions.
Current Experience
The current experience for building a custom extension was a wizard flow with many steps. We started off with an assumption that we needed to reduce the amount of steps to reduce the amount of time and complexity of the experience.
Evaluative Testing Plan
I advocated for evaluative research to test this assumption and to give us more points of data on how to best improve the experience. Below is the task analysis test I created and conducted with 10 participants. I worked closely with my team to determine what we wanted to learn and tailored the test accordingly.
What did we learn about our current experience?
Our users find managing their extensions difficult and tedious. The register and add flows are redundant and inefficient, with very little guidance on how to set up extensions correctly in the product.
Our users find it difficult to differentiate their extensions, requiring them to spend more time and energy to determine what each of their extensions do.
Our users find it difficult to add and remove extensions from their draft and live environments because it’s not obvious what environment you are initially setting up. Both environments are identical and the CTA to add to live assistant is hard to find.
Competitive Experience Research
Our team looked at the experience that Slack and Intercom offered for managing custom extensions. Although they are not our direct competitors, we found this to be a valuable exercise to inform how we improve our experience.
Our solution
We came up with 3 principles / goals to achieve as a team for the new experience we wanted to build. We went through a few iterations of prototyping (sketches, lo-fi, mid-fi to align as a team on what we think we should build. The goal for this prototype was to test again with users and share the results with leadership/stakeholders to get buy-in to build it.
