API reference topic study – summary results

During November and December, 2014, I ran a study to test how varying the design and content of an API reference topic influenced participants’ time to decide if the topic was relevant to a scenario.

Summary

  • I collected data from 698 individual task scenarios were from 201 participants.
  • The shorter API reference topics were assessed 20% more quickly than the longer ones, but were less credible and were judged to have a less professional appearance than the longer ones.
  • The API reference topics with more design elements were not assessed any more quickly than those with only a few design elements, but the topics with more design elements were more credible and judged to have a more professional appearance.
  • Testing API documentation isn’t that difficult (now that I know how to do it, anyway).

The most unexpected result, based on the literature, was how the variations of visual design did not significantly influence the decision time. Another surprise was how long the average decision time was–almost 44 seconds, overall. That’s more than long enough to read the entire topic. Did they scan or read? Unfortunately, I couldn’t tell from my study.

Details

The experiment measured how quickly participants assessed the relevance of an API reference topic to a task-based programming scenario. Each participant was presented with four task scenarios:  There were two scenarios for each task: one to which the topic applied and another to which the topic was not relevant and each participant saw two of each. There were four variations of each API reference topic; however, each participant only saw one–they had no way to compare one variation to another.

The four variations of API reference topics resulted from two levels of visual design and two levels of the amount of information presented in the topic.

Low visual design High visual design Findings:
Information variations
High
information
copy_ld_hi copy_hd_hi
  • Higher credibility
  • More professional appearance
Low
information
copy_ld_lo copy_hd_lo
  • Lower credibility
  • Less professional appearance
Findings:
Design variations
  • Faster decision time
  • Lower credibility
  • Less professional appearance
  • Slower decision time
  • Higher credibility
  • More professional appearance

Variations in the visual design elements of API reference topics did not significantly affect the time required to assess the topics’ relevance to the task scenario. The variations in visual design, however, did significantly influence participants’ assessments of the topics’ credibility and professional appearance. The topics with more design elements were assessed as being more credible and having a more professional appearance than those with fewer.

Variations in the number of information concepts presented in the API reference topics, on the other hand, significantly influenced the time participants took to evaluate the topics’ relevance-shorter versions of the topics were assessed about 20% more quickly than the longer versions.

The participants’ assessments of the topics’ credibility and professional appearance also varied significantly between the shorter versions of the topics and the longer versions. The longer ones were assessed as being more credible and having a more professional appearance than the shorter ones.

One Reply to “API reference topic study – summary results”

Leave a Reply