Posted on by bobw

From getting started to getting finished

As a technical writer who occasionally does some development on the side, I appreciate a fast Time to First Hello World (TTFHW)—the time it would take to create and run a simple program using a new (to me) API. The time to get started, in other words. The Nest API documentation I recently reviewed  reminded me of a much less frequently mentioned notion, the time to get finished.

If you’re not familiar with TTFHW, it is basically, how long it takes a new user to:

  1. Find your product
  2. Resonate with its value proposition
  3. Test it out to verify it provides them with a viable solution

The metric for this is operationalized by how long it takes someone to write a simple program to demonstrate step 3. From a measurement standpoint, this is a reasonable goal in that:

  1. It’s tangible and relatively unambiguous in its meaning (the reader was successful in accomplishing the goal).
  2. It can be measured accurately if your help documentation provides a way to link the documentation to the use of the API, through a user-specific key or interactive development environment.
  3. It can show a tangible engagement metric to show that your product is attracting and engaging potential customers.

While it’s good to use that as a metric for engagement and potential adoption, it’s important to not treat it as the only metric that counts because it doesn’t necessarily represent retention or other aspects where the money is made or lost.

History of TTFHW

The first web mention of TTFHW I was able to find on this was as a link to a note in a now unpublished blog in 2010. The buzz about TTFHW grew louder in 2012 with some posts from on The ProgrammableWeb and others:

These references describe TTFHW is a good measure of the API’s usability and ease of onboarding which are clearly important elements when talking about attracting and acquiring new customers. Developers can’t really fall in love with your API if they can’t figure it out and actually using it is a good indicator they have it figured out to some extent. However, Willmott mentions other possible measures when he quotes Lawson’s presentation,

Finally, in addition to your TTFHW, you might also want to measure TTFCA or TTFPA – Time to First Cool App or Time To First Profitable App. If you can get these two down low, then the engagement on your API will jump up even more quickly!

I think TTFCA or TTFPA are great, but I haven’t seen them mentioned since. Now might be a good time talk about them more, but, for now, let’s continue with TTFHW.

TTFHW has appeared occasionally since 2012, and even in an academic journal (Weisman & LIbris, 2014).

The more recent articles seem to consist mostly of elements from earlier articles, so it would seem that all that can be said about this has been said. That means it’s time to focus on the other elements of the customer journey.

The lure of TTFHW

The time to first Hello World metric is attractive from a content as well as a product success perspective. Unfortunately, it’s really the equivalent of downloads for apps and page views for web content in that they provide a number but don’t tell the story that really matters.

TTFHW relates very closely to the API’s content experience. Granted the API’s functionality and usability are more significant factors than the documentation to a good TTFHW, so it’s important to consider those elements before congratulating (or blaming) the documentation for the TTFHW metric. However, the connection between TTFHW and the content experience is easy to trace and, even better, it’s easy to trace using online metrics.

It’s a quick measure to take, in that a good TTFHW should be less than a day, ideally MUCH less. So, it doesn’t require much waiting around to see it. The sooner you can show “success,” the better, for many reasons.

It touches all the important parts of the content and product strategy. The good news is that to get a low TTFHW metric, the organization must have a consistent product strategy and execution. Marketing, product development, technical documentation, and support all have to be coordinated to get the lowest TTFHW. However, as with any aggregate metric, you need to be able to identify how each component contributes to the aggregate, which takes me to the next part.

TTFHW is only one metric

TTFHW is one metric, but it should not be the only one and it really isn’t the one for which you should optimize. If your organization gets paid when people use it, instead of just downloading or subscribing to it, tracking downloads and TTFHW are just the beginning. Even if you do make money on initial downloads or access, you’ll probably want to make more money by keeping them coming back for more. Granted, no one can use it if they don’t download or subscribe to it, but it’s important to keep TTFHW in context.

In the more global context, you’ll see how other metrics track the larger success goals of the product and the company.

While time to first cool app or time to first profitable app will be important metrics to your customers, they might be difficult for you to measure. The real measure of the success of your product’s onboarding will be observed in the context of the entire customer journey, such as:

  1. How well you’re attracting new customers to your site (site traffic)
  2. The time to first Hello World (TTFHW)
  3. The time to their first customer use of the API/product
  4. The ongoing product metrics that drive your business, such as those listed in 10 Essential Mobile App KPIs and Metrics (and How To Use Them).

Where content fits in

This is where it gets tricky, but, if you plan ahead, you can collect metrics on how well your content is contributing to each of the stages of the customer’s journey.

  1. Site traffic and how customers go through your onboarding funnel are content metrics that are easily tracked through conventional web metrics (e.g. Google Analytics). Your content should have a clear path that guides new users to where you want them to go. Basic metrics can track how well your content is getting them through the funnel.
  2. The time to first Hello World (TTFHW) might need some technical support to relate new users to their first successful use of the product. If you know that’s what you want to track, you can work that into the help site’s design to make sure those metrics are collected or can be associated. For example, relating the API key issued to a specific user session, so you can connect the session information with the Hello World experience.
  3. Tracking the time to their first customer use of the API/product should be easy enough to do through the API key (e.g. time of use –time of issue) but you’ll want to make sure to keep track of that information!
  4. Ongoing product metrics that drive your business should be built into the API’s design, so as long as you can tie that back to user’s (the developer’s, in this case) experience, you can use this to track back to possible content issues. Make sure that those metrics are available to merge into your content metrics (or vice versa) so you can make sure your content is helping your customers be successful.

The real challenge to this process, and what reviewing the Nest API brought this to mind, is that unlike the onboarding funnel, the path from Hello World to productive app (step 3) is not as linear as the path from home page to Hello World. Hello World and Tutorials are great (and necessary) leaning tools that might get you 80% of the way to the finish line, but it’s the reference topics (and StackOverflow) that get you across the finish line. Because the path that covers the last 20% of the road to the finish line is so different from the first 80% of the path, the metrics and instrumentation for that path will also be different. The Nest API docs seemed to accomplish the attract and onboard experiences well, but the low-quality reference topics seemed to ignore the last lap of the customer’s race to the finish line.

To track that last lap, the instrumentation of the path from Hello World to finished app can be built into the API’s logging and such instrumentation and logging, can provide some visibility into that path. For example, if:

  • Some methods are returning errors more often than others. (Maybe they are not as clearly documented as they need to be?)
  • Some errors are more common with new users. (Maybe the onboarding process needs review or maybe you need to reevaluate your audience?)

Watching the customer’s journey

Web APIs can provide the ability to track almost the entire customer’s journey with an API and that information can be used to improve the customer’s experience throughout.  John Musser’s slide share, Ten Reasons Developers Hate Your API, talks about this. While content plays a role in a successful customer journey, all the moving parts (marketing, product development, support, and the content) have to work together to get your  customers to where you and they want to be: finished!

Leave a Reply