User Research Notes

a user research study by shir goldberg

Interview #1

Name: Jason Age: 22 YOE: 6

Background:

  • DevOps engineer, makes sure code is production ready
  • Was a programmer for 3 years prior to this
  • Primarily a Java + PHP developer, currently works heavily with Python
  • Familiar with Twitch, occasionally watches streams

Chatbot task:

  • Tried to find APIs
  • "None of their menus have dropdowns"
  • Thought he was looking for a Twitch extension
  • Gravitated towards "API" language but discarded it as unhelpful
  • Felt "Extensions" was closest to what he wanted
  • Ignored video content
  • Went to marketing site for extensions
  • Opened up documentation and design guidelines in a new tab
  • Hit a log-in screen, hit another disabled screen (down for maintanence)
  • Clicked through authorization screen
  • Found "Getting Started with Extensions" documentation page
  • Said he was overwhelmed by acronyms and information
  • "I don't even know if I'm looking for an extension anymore—none of these are chatbots"
  • Ctrl+f for "chatbot", found it in TOC
  • "I guess Extensions are totally useless"
  • Closed all tabs
  • Found Glitch but didn't know what it was

Extension task:

  • "Going back to extensions", clicked first link and got to Extensions layout
  • Normally would be looking for APIs—doesn't care about high level concepts, only interested in APIs
  • Confused by Developer Rig–"this isn't an SDK, it's like Xcode or something"
  • Compared it to Chromecast apps—specifying an endpoint and then embedding content into their UI
  • Was able to quickly orient after drawing that comparison
  • Went back to Dev Console to start actually building an extension
  • Found API documentation. "This is actually useful!"

Ratings:

On a scale of 1-5, how easy was it to find the information you were looking for? 2.

On a scale of 1-5, how helpful did you think the content you found was? As a first time user, a 3. If I was someone who was new to the idea of making extensions for existing platforms, I think this would have been a lot more useful. But as someone who knows how to make extensions for applications, the information immediately presented to me was not useful as it started too basic. I would much rather jump to the API docs immediately.

On a scale of 1-5, how visually appealing did you find this website to be? 3. Needs way more diagrams.

Any other feedback or thoughts? Nope.

Interview #2

Name: Bish Age: 24 YOE: 3

Background:

  • Software engineer
  • Works primarily in systems-level languages (C, etc.)
  • Familiar with Twitch, enjoys watches speedruns and tournaments

Chatbot task:

  • Looked for Twitch API
  • "Extension doesn't sound like what I'm looking for..maybe API?"
  • Felt the "Streams" API category was the most fitting, but still not perfect
  • Clicked on it and got to Twitch API reference
  • "This doesn't seem right…let's go back"
  • Returned to homepage, reread Extensions
  • Decided to click on Extensions
  • Landed on Extensions marketing page
  • "It asks a question (What types of extensions can I build?) and doesn't answer it!"
  • Clicked on "Create an extension" and landed on the Twitch login page, which lead to the developer consoler
  • Was about to enable 2FA when I stopped the task

Extension task:

  • Clicked on first Extension link
  • Read content on the page
  • Found Hello World section
  • Assumed that there was a wizard available
  • Downloaded Developer Rig
  • Clicked on "Designing Extensions" docs
  • "There's a lot of reference stuff and design guidelines but I'm at the point where I'm worried about the high-level mechanics. Like how do I even get this thing going?"

Ratings:

On a scale of 1-5, how easy was it to find the information you were looking for? 4, because it wasn't a 5, but it also wasn't that bad. I don't know if I actually found what I wanted though.

On a scale of 1-5, how helpful did you think the content you found was? I felt like the stuff on the [developer documentation] was a 4. I felt that the content on the splash page you originally land on [main Twitch Dev homepage] is like a 2.

On a scale of 1-5, how visually appealing did you find this website to be? 3.

Any other feedback or thoughts? That first page I landed on had a lot of links but I wasn't sure where any of them would take me. It wasn't clear whether it would be documentation or examples or something else. But once I was in the actual documentation, it's way easier to navigate—maybe because I've had more experience with developer documentation pages like this and can understand how to navigate. I didn't understand how that other page was supposed to help me get here.

Interview #3

Name: Dare Age: 24 YOE: 3

Background:

  • Full-stack web developer
  • Works primarily in Python, lately moving to Kotlin and JS/HTML/CSS
  • Been a Twitch streamer, viewer, and stream engineer
  • As a stream engineer, hosted a channel using Xsplit with preset functions, hooked things up to StreamLabs
  • Never needed to use Twitch APIs, everything was done through StreamLabs which takes care of most of the API and behind-the-scenes work

Chatbot task:

  • Scrolled to bottom of page, skimming for anything important
  • Clicked "View Sample Code" to see it on Glitch
  • Opened docs in a new tab
  • Moved over to docs while sample code loaded
  • Found chatbot info on docs landing page
  • Jumped past all introductory text to get to the "meat" of the page
  • Knew what Glitch was but didn't want to run a sample in the cloud
  • Looked at env variables first to figure out what was needed to build this project
  • Read the code for a Hello World chatbot
  • "I would just go copy this sample code to my clipboard, I wish there was a copy button"

Extension task:

  • Noticed Extensions callout at top of page with image, "Extensions" header in TOC, and "Extensions" link further down the page
  • "I'll click the big image"
  • Jumped to what types of extensions he could build
  • Decided he didn't care about most of the content on the page—"this is important, but it's not my goal"
  • Gravitated towards "Hello World" example
  • "Since it's a download I won't do it immediately. If it was some bash commands I would do it right away. Downloads arebigger, I care about them more"
  • Read the "Hello World" example in depth without following it locally
  • "HTML, JS, and CSS…makes sense, it's probably running on Chromium or something"
  • "So it's just like React with a bunch of events…and the backend is maybe Node?"
  • "First things first is to learn what an Extension is. I want to know about all the extensions I can build, I'm looking for information gathering to generate ideas"
  • "So I would get the Hello World example together, write some CSS to test it out, and then set up a backend"

Ratings:

On a scale of 1-5, how easy was it to find the information you were looking for? 5, docs go where I think they'll go, and I didn't have to use any kind of search to find a starter example. I just want the docs to quickly get me to a Hello World so I can move on from there. Other information is great but that's the goal.

On a scale of 1-5, how helpful did you think the content you found was? 4. Well written, solid pieces of information. Part of it is my own skimming, but I'd like more pictures and diagrams. It's a wall of text, and being able to absorb information in multiple formats is best. Explain things twice, have different methods to convey, to understand better.

On a scale of 1-5, how visually appealing did you find this website to be? 5. I like the color scheme, it's easy to read, and the fonts are nice.

Any other feedback or thoughts? There's a lot of blank space on the right side of the page. Maybe you could put a floating action button there or something.

Return to top ↵