Measuring your technical content – Part 1

This started out as a single post, but grew, and grew, and… Well, here’s the first installment.

After the last few posts, it would be easy to get the impression that I don’t like Google Analytics.

Not true.

I just don’t like when it’s treated like it’s the only tool that can collect data about a web site—especially a non-funnel (informational) web site.

In this collection of posts, I’ll look at my favorite topic, API documentation, and how you might analyze it in the context of the customers’ (readers’) journeys. This analysis doesn’t have to apply only to API documentation, because it’s based on customers’ goals, which are more universal and, if you look carefully, you might see a customer goal that matches some of your content.

So let’s start with the basic questions…

Basic questions

For most measurement matters, I like to start with the questions for which I seek answers and then find the data that will provide those answers in as valid of a fashion as possible.  So, before you start instrumenting and measuring anything, including content, ask yourself these questions:

• Why do you want to measure?
  What to you hope to gain from the effort?
• What will you do with the information?
  If you won’t use the information, why bother to collect it? Storing and sifting through it are not free. Neither is trying to make sense of it after the fact.
• How will those reasons help the goals of the organization?
  Again, if they aren’t helping advance the goals of the organization, why do you care?

The last is probably the most important question, but you really can’t answer it until you’ve answered the first two. Feel free to cycle back to the top of the list and repeat the process, until everyone is satisfied (or satisfied enough to continue).

You’ll have to come up with your own answers, but some examples might be:

• Why do you want to measure?
  • To track how our customers are interacting with our content (because you’re spending a lot of money on the content, it’s nice to know if it’s doing anything)
  • To be able to detect problems and opportunities.
  • To see how our responses handle the aforementioned.
• What will you do with the information?
  • Show stakeholders the impact of their investment/engagement/participation in the content effort
  • Respond to problems and opportunities in a timely fashion
  • Monitor our responses and response processes to make/keep them effective
• How will those reasons help the business?
  • Demonstrate that the investment in content is having the desired impact for the business (bang for the buck, return on investment, and similar ideas)
  • Keep on top of or ahead of small customer problems to prevent them from becoming large (i.e. expensive) ones
  • Demonstrate that our content is supporting the company’s goals and objectives for the product(s)

Yours will be different, but you should be able to see the pattern.

If you can’t come up with compelling reasons for these questions, then STOP. You don’t need to measure anything about your content, yet. Instead, you might need to figure out why you’re writing it in the first place. (Not a comfortable question when your job is to write it, but maybe a good topic for another post).

If you have a good set of responses to those main bullet questions, you can continue, but only with the understanding that you might need to return and refine them in case you run into an instrumentation problem. If you can’t measure towards one of your goals, you’ll need to work on the goal and the measurement until they fit.

Types of measurement instruments

You have your motivations for data collection. Now I’m going to divide the measurement instruments into two classes.

Alarm: An indication that something needs immediate attention because a value exceeded a limit. The Check Engine light on your dash that says pull over NOW! (just before you see the steam emerge from under the car’s hood.

Control: Something that you watch regularly to maintain a desired condition in an environment that can change. Usually a part of a control loop. For example, your car’s speedometer. You constantly watch the indicated speed so you can control it by adjusting the gas or brakes as you go up and down hills, for example.

Generally, you watch the monitoring goals to prevent alarms, but you want both, in case you can’t see everything at once. Note that you can have varying levels of alarms, such as warning and danger alarms. An alarm simply indicates a threshold was crossed. What that means and how you respond is up to you and how you design them.

Keep these in mind as we continue on to instrumentation.

Content instrumentation

With the preceding types of measurement instruments in mind, consider the following types of content that are typically found with API documentation. Other types of technical writing also share some or all of these types of content.

• Introduction
• Getting Started
• Tutorial/How-to
• Conceptual
• Reference

As you consider each of these categories, also keep in mind how often the needle moves (or should move) and how quickly you should react to it. These will vary depending on the product, its place in the market, and the audience, so they are hard to characterize in this summary.

Most important, remember that each of these types of content has a different goal and a different interaction experience from the customer’s perspective. To the writer, it’s all content, so it’s easy to try one-size-fits-all instrumentation solution—solutions which are inevitably, one-size-fits-nothing solutions.

For each content type and interaction, I’ll consider the following:

• The success factors for the customer and the business–how do they know they are having a successful experience? What does that look like to them?
• The success indicators for the customer and the business–what happens when they are successful? What do they do? How can we see they were successful?
• The measures used to capture those indicators.

Finally, in this analysis, I’ll be trying to avoid cases where I have to “interpret” the results (i.e. guess what the user was thinking).

Let’s start with an easy one, Introduction topics.

Introduction topics

Introduction topics are what people read when they first come to your site to learn more about your product but might not be ready to commit to it. They’re window-shopping.

This is different from a welcome or home page, whose purpose is to direct you someplace else. But, in some cases you might have introduction content on the home page. It really depends on the size of your product and its documentation.

The key to this content is that it must have a clear call-to-action or goal. This content isn’t just to introduce, it’s to educate and persuade. The goal is to tell the reader this product will help you and that you want it. In doing so, it will also identify and filter out, early in the process, the readers for whom it won’t help.

• Success factors
  • Customer are successful when:
    • What they read/learn appears to solve a problem or a need that they have (whether or not they realized it before reading the introduction).
  • The business is successful when:
    • The customer learns that your product will solve a need they have.
• Success indications
  • When a customer is successful, they will:
    • Read more about the topic
    • Buy and/or download the product (Depends on the product)
  • The business is successful when:
    • Customers buy or download the product.
• Measures
  • Customer success is measured by:
    • Search hits (search terms that land on introduction content)
      This could be summarized by top hits
    • Session duration (longer might be better)
      This would be monitored as a control value. You might want to watch it and adjust your content until it provides the desire result.
    • Session depth (deeper might be better)
      Same ideas as the duration.
    • Satisfaction feedback (more satisfied is almost always better)
      This could easily be a control chart with some alarm limits if it drops below or exceeds a specific value.
    • Downloads/Purchases (more is also probably better).
      This would also be a control value (downloads/day, for example).
  • Metrics for business:
    • Satisfaction feedback (more satisfied = better)
      This value and the others would probably be control values.
    • Downloads/Purchases (more = better)
    • API accesses (for online software)

Unlike a lot of technical content, this content can have a funnel shape (content that leads to a specific action), so take advantage of it with the tools that support it, and be sure to include feedback. Don’t just rely on Google Analytics.

With Google Analytics, for example, you could define a goal page so that Google Analytics knows what you’re hoping the user will do. This type of page could have conversion events, so Google Analytics can help you out. Be sure to author your content so that it will produce the events (clicks or page views) that indicate your customer has accomplished the page’s goal.

Remember, you want to know what success looks like in a way that you (or your instrumentation) can actually capture–and you might need to modify your content, to accommodate that.

That’s a good place to stop, for now.

I’ll look at some more content types in the next installment. Don’t worry, it’s about to get more interesting.