mark[sweep]

mark-sweep: in computer science, a simple algorithm for garbage collection.
Hi I'm Dave, I work in product at Meetup.

Clicky

“To invent your own life’s meaning is not easy… but it’s still allowed… and I think you’ll be happier for the trouble.”

– Bill Watterson

What do you Expect to See - Capturing Mental Models with Keynote

image

One of the great things about working at Meetup is that we have a top notch usability lab. Meetup brings in multiple people almost every day of the week and the lucky product teams gets to throw anything into usability with very short notice.

My teammate John Cockrell recently introduced me to something cool he was doing with usability mockups using Apple’s Keynote software. Keynote is a great tool for prototyping mobile flows and experiences. In fact, there are whole sites dedicated to templates for just such a purpose.

What was interesting about the mockups John was testing wasn’t how elaborate the presentation was, it was how simple it was. John would mock up a few flows that he was interested in but then as testers were clicking through the wireframes he would present them with this screen.

image

The beauty of this approach is that it captures the tester right in the moment where he or she is building a mental model.

We all build mental models about how things work - human beings are wired to try to make sense of things. The same is true of our products; users are trying to figure out how things are expected to work. By providing a blank screen in the flow of what the user was already trying to do, John could investigate what kind of mental model was being formed.

The important caveat with this approach is that you have to give the tester enough structure to form the model. You can’t just ask someone out of the blue “What do you expect this app to do?” - You will get many aspirational ideas that have no basis in reality

Instead, before the tester taps over to the next screen they have ALREADY pictured in their mind what this screen will do for them. That’s what you’re trying to capture, and that’s the beauty of this simple technique.

Other things I’ve written:

Why Homer Simpson doesn’t run product (Why traditional product interviews stink)

Homer Simpson once made a car called, aptly, the Homer. His long lost half-brother, owner of “Powell Motors”, give him carte blanche to create his dream car - a car for the “average” American.

For days, he slaved away drawing, designing, telling engineers what to build and how to build it. In the end, he came up with this design

image

The Homer:

  • Had two bubble domes, so you could throw the kids in the back and not have to hear their whining
  • Played “La Cucaracha” on it’s horns
  • Sported a bowler as a hood ornament
  • Came with tail fins
  • And shag carpeting

In the end, this monstrosity of a car cost too much to build and caused his brother’s company to fall into bankruptcy.

The reason Homer’s design failed is in the first paragraph: “carte blanche”.

When you do traditional product interviews, you are giving your user carte blanche to dream up any possible feature they want on your product. Their answers are all aspirational - they are dreaming up the Homer.

How many times have you asked someone, “Do you think this is something you would use?”

Dreaming…

As Ron Williams, CEO of the startup Knod.es, like to say, “I’ve fallen many times for the trap where someone says ‘If only you’ll build this, I’ll buy it’.”

The problem is that once you build what they said, you find out that they actually don’t buy it. They may have told you with all the best of intentions but what they did was tell you a dream that required no constraints.

In the real world, we deal in tradeoffs. We are willing to accept things of lesser quality or with “good enough” features because we trade off features that are less important for ones that are more important. 

In the “real” world, Homer drives a card like this:

image

And the reason he drives this pink sedan instead of the bubble domed Homer car is because he has had to made tradeoffs. Perhaps he wanted a green car, but when it came to a question of having a green car vs. saving money, he chose to save money. It wasn’t aspirational, it was actual.

So Nir Eyal, creator of the Desire Engine, likes to put it this way - when you do an interview don’t ask “What do you think of this?” ask them, “When was the last time you did that?”

Chances are, you won’t end up with the Homer….

More resources:

Why Jobs to be Done interviews don’t “Lead the Witness”

image

One of the great things about doing Jobs to be Done interviews is that you’re much less worried about “leading” the interviewee with your questions.

In classic product interviews, you have to be very careful when asking opinion questions about a product. If you place something in front of someone and ask them “Some people think green is a cool color. Do you think this is cool?”; you’ve already biased the interviewee to consider that this item is either cool or uncool. Their initial reaction might have been something totally different, but you’ve put them into a frame of mind with your question. This is a subtle form of influence called biased questioning.

We’ve been doing Jobs to be Done interviews at Meetup for a few months now and one of the great things about them is that you don’t need to worry about moderator bias as much. The reason is that instead of dealing with aspirational emotions about a future product, you’re uncovering history.

Because you’re working with past behavior, it’s something that has already happened and already has attributes, thoughts and emotions associated with it.

In fact, one of the techniques I’ve been using in my interviews is something I’m calling “correct me”. When I’m trying to uncover the emotion of a decision, I’ll sometimes paint an exaggerated scene and have them correct me.

image

I’ll say something like, “So you’re sitting on your couch, it’s a Monday morning and you have the laptop in your lap. The kids are occupied with cartoons and then you find Meetup… So what was the scene like? Did you throw up your hands and say ‘Awesome, this is exactly what I’ve been looking for?’”

This usually elicits a big grin from the interviewee as they jump in to correct me. Instead of introducing bias, they suddenly have the need to set the record straight.

"No no, it wasn’t so dramatic, actually what I felt was a bit of nervousness…"

And BAM, you’ve uncovered anxiety force and you can follow that thread with more questions. “Anxiety? What do you mean….”

The one caveat would be where the memory of the event is not strong. If you find your interviewee following the “maybe, probably, yeah” chain (maybe it was like that, probably that’s what I did, yeah I did that) then you have to be wary about the responses.

So the next interview you do, try having your interviewee correct you, you may be surprised where history takes you…

More…

Love to hear about your JTBD experiences - hit me up on twitter @marksweep

“A life spent making mistakes is not only more honorable, but more useful than a life spent doing nothing.”

– George Bernard Shaw

How to Interview Candidates - Jobs to be Done style

image

I recently had to interview a technical candidate who I was completely unqualified to assess. We were interviewing a potential iOS developer and I was thrown into the interview mix simply because at a hack-a-thon I had hacked together an iPhone app… in Ruby. Suffice it to say that there were better people than me for technical interviews.

Instead, I decided to try using techniques learned from the Jobs to Be Done framework we’ve been experimenting with,  to interview the candidate and see where it led me. Sometimes, when you have a hammer, everything looks like a nail. 

The key thing I wanted to find out was where was the emotion and push/pull that brought the candidate to work every day. What were the forces that made this candidate get up in the morning and want to program. What was the story behind how they got into iOS development. I ended up building a JTBD timeline of a different sort, one which told the story of “first thought” of wanting to program for the iPhone, to the “consumption” of building an app and getting it onto the app store.

Along the way, I had the opportunity to unpack the forces that motivated them and what impassioned them about coding. 

Google recently talked to the NY Times about their hiring practices and what they learned after using “Big Data” to sift through hiring practices. They found that “brain teaser” questions didn’t work - it only served to make the interviewer feel smart. One of the things that did work was “behavioral interviewing.”

"[Behavioral interviewing] — where you’re not giving someone a hypothetical, but you’re starting with a question like, “Give me an example of a time when you solved an analytically difficult problem.” The interesting thing about the behavioral interview is that when you ask somebody to speak to their own experience, and you drill into that, you get two kinds of information. One is you get to see how they actually interacted in a real-world situation, and the valuable “meta” information you get about the candidate is a sense of what they consider to be difficult."

This is exactly what JTBD interviews do. They uncover past “buying” experience instead of aspirational answers. You’re uncovering what they did, what they had emotion about and the tradeoffs they actually made along the way.

Another reason JTBD interviews work is because they get so detailed that it’s almost impossible to bluff your way through them. When you’re asking someone where they were sitting, who else was in the room, was it day or night, were you on a laptop or desktop… you’re really making a bluffer sweat keeping up all the made up details. Pretty soon they will collapse under the pressure.

All in all it was a great experiment and I’m hoping to try more in the future.


Resources:

Unpacking the Jobs To Be Done Timeline: A Purse Interview

image

My wife has a purse that she bought in December of last year. I noticed that one of the straps was fraying and pointed it out to her. What started as a simple public service announcement soon turned into a Jobs To Be Done interview about when and where she bought it.

At the Switch Workshop NYC, one of the tips given was to start at the point of purchase, so I followed this advice and began at the “end” of the JTBD timeline. In Jobs To Be Done, one of the tools used is the concept of a timeline. The idea is that you want to place the consumer back in time and relive their emotions and thoughts going “in” to the purchase. Post purchase, if you ask them how they feel about things, they are prone to rationalize and you’ll get false information - if you recreate their thinking going “in” to the purchase you get a better understanding of the forces at work when they hired the purchase to do a job.

image

She kept claiming that the purse was an “impulse buy” but we shall see that that’s not quite true. She explained how she was shopping at DSW on her lunch break, it was December.

"Why were you shopping?" I asked.

She explained that the impetus for shopping was that her niece’s birthday was coming up and she was looking for a gift for her.

"So you were looking for shoes for her?"

"Well, no, actually, I really needed a pair of new boots."

It turns out that her black boots had holes in them and she needed new boots. She had holes in her boots since last year but, as she explained, it wasn’t until it got to the Fall and was cold, rainy and snowy that the holes became a problem. She would still wear the boots when it didn’t rain or snow, but it had rained recently and since it was cold she had to wear pumps and got her feet icky and wet!

"Okay," I said, "So you waiting until December to buy new boots and picked up a purse as well? Why did you wait so long?"

She paused to think about this, “There must be some reason… I know! I have another pair of brown boots and they finally got a hole in them too! That’s why I could wait on the black boots until December.”

So here I realized that there was actually a parallel timeline involving boots that I had uncovered that was related to her purse purchase.

In the Jobs To Be Done timeline, you try to identify key points or events and capture the emotional energy around those. For my wife, she noticed 1 year ago her old black boots had developed holes in them. In the timeline, this would be called the “First Thought”. However, she didn’t actively act on this idea for a year. She had her other pair of boots to fall back on and with the warm weather in Spring and Summer, the boots weren’t on her mind. It wasn’t until her second pair of boots developed holes that her thoughts were triggered back to “I should really look for a new boot”. In the timeline, this event marks the change to “passive looking" - and the 2nd boots hole discovered is called Event #1. She still wasn’t shopping yet, but the level of emotion was rising.

Later, she really developed urgency and switched to “active looking" when she got caught on a rainy day and had to wear pumps. Her feet got wet and gross (New York City streets) and then the emotion really ramped up. Event #2 was the trigger to start shopping.

image

I wanted to get back to the purse, so we switched gears .

"So back to the purse purchase, before this did you not have a purse to use? Why get a new one?"

She pointed to her old purse, “Oh I had that one. But you can see it’s cute but too big. I can’t use it on the weekend.”

"Weekend? What do you mean?"

She then went on to describe how she wanted a big, but not too big purse (a long monologue ensued about women and huge purses). The old purse was good because it could fit her book, which she needed for her long subway ride to work, but on the weekends, she didn’t have a book to read, and then the purse was a little too big. She did own a cute small purse that was better for weekends.

So every weekend before we headed out, she would unpack the contents of her weekday purse onto the bed, and them repack all the items into the weekend purse - minus the book and her work pass. Then on Sunday night, she would unpack the contents of her weekend purse and put all that back into the weekday purse - plus her book and work pass. This went on for a year.

I kinda stared at her in disbelief.

"Crazy huh?"

I asked her when she bought her old purse - it was also about a year ago when a friend was visiting from California. They went shopping together and her friend found the purse and suggested she buy it. She did indeed buy it but soon after that she had the first thought: “This purse is too stiff and bulky for the weekend.” From there, she starts to passively look the first time she has to empty her purse and the only actively look as she enters the DSW and sees the purse rack on the way to purchasing her boots.

This highlights another point in the JTBD timeline - the experience of the purchase and the satisfaction or the dissatisfaction with the purchase. In this case, the dissatisfaction with the original purse became the first thought to getting another purse. The timeline for her old purse purchase flows right into the timeline for her new purse purchase.

image

So in this single interview, there are actually three purchase decisions we could unpack. The boots, the old purse and the new purse. I also asked her about how satisfied she felt after using her purse.

"I like it. It can fit my book for the subway and is small and cute enough for the weekend. Plus it has a lot of pockets to organize stuff."

Of course, now that I’ve pointed out the strap was frayed on her new purse…

Resources:

Why “Jobs To Be Done” is the next “Lean Startup”

image

There’s a way of thinking about startups and products that sat at the feet of W. Edwards Deming, godfather of Lean Manufacturing and the famed Toyota Production System. It is a simple but powerful concept that can cut down on product wastage and dismantle misconceptions about what people actually want.

Lean Startup?

Nope - it’s the Jobs To Be Done Framework.

Jobs To Be Done is a way of uncovering the forces behind a product purchase. The idea is that customers don’t buy your product, they “hire” them to perform a job. Clay Christensen, author of the Innovator’s Dilemma,  talked about the framework in the “Milkshake" talk.

I remember seeing the milkshake video online and being captured by the power of the idea. It was such a simple concept - stop thinking about features and satisfaction, start thinking about jobs your product is hired to do. But then I was stuck. There wasn’t any meat to the idea, nothing to grab a hold of.

And so Jobs To Be Done was relegated to the dustbin of my brain where cool but impractical ideas live. This went on for a while until…

Fast forward to the Mattress Interview. Here finally was something I could sink my teeth into. The beginnings of a digestible framework. Bob Moesta, Chris Spiek and a community of practitioners at Jobstobedoneradio.org have started to disseminate practical tools and examples of how to tease out the job to be done. If you haven’t listened to it yet, stop now and queue it up.

Remember when you first heard of The Lean Startup? It was that same revelation of a powerful, simple and logical idea. Since 2008, Lean Startup has grown to be a worldwide movement and phenomenon. It’s been debated and practiced and evangelized throughout the startup community. I remember wishing I heard learned about it earlier and trying absorb as much as possible in as short a time as possible.

Jobs To Be Done is just as powerful an idea - in fact, it’s the perfect complement to Lean Startup. It’s on the same trajectory so don’t miss the boat, this one is going to be big.

Resources:

“People don’t want to buy a quarter-inch drill, they want a quarter-inch hole.”

– Theodore Levitt