I’ve just collected some more articles for my bibliography of API documentation-related articles and the trend I saw earlier this year hasn’t changed much. In all fairness, eight months is probably not enough time to see a change given the pace of academic publishing. I now have 114 articles in my list of API documentation-related topics.
Searching for API DOCUMENTATION produces a lot of hits on actual API documentation (good news: there’s a lot of API Documentation out there!) Searching for Writing API Documentation produces more articles relevant to what I’m looking for. I’ve also merged my academic and non-academic API documentation bibliographic data, so I can compare and contrast them together.
The merged lists have these characteristics:
114 articles! and I know I haven’t found them all, yet.
71% (81/114) of the articles are from CS-oriented sources
29% (33/114) are from TC-oriented sources
81% of the CS-oriented articles are from edited publications (books, journals)
27% of the TC-oriented articles are from edited pubs.
27% (31/114) of the API documentation articles were published in 2017
These are some the new articles I found while browsing Google Scholar on the subject of API documentation since my last update on the topic. I found about 20-some new articles since January, which is exciting! Here’s a review of just the ones I’ve had a chance to skim and write up, so far.
This recent trove of documents makes me happy and sad as did the papers I reviewed in an earlier post. These articles contain lots of great research into API documentation, who uses it, and how it’s used. All of these are available for download and I would encourage you to do that if you are interested in API documentation.
At the same time, I’m still disappointed that there has been absolutely zero research published on the topic from the technical communication community this year (was I really the only one who was doing that lately?!) and little to no reference to anything tech-comm related in these papers. If you know of some API research published in a tech-comm venue (and that’s not already in my list), please let me know. At this point, all this is just a warning that some of my disappointment might seep through into my article reviews.
Many of these articles (3/4) had both academic and industry authors, which suggests that industry-academy partnerships aren’t that unusual in the research of API Documentation—if you’re in computer science. In tech comm, as Tom Johnson’s recent survey laments, not so much.
Some of these articles cite earlier research in developer user studies and also contribute to that body of work. None of them, however, cite what I would call writing or reading research. That wouldn’t be so troubling, if it were just these articles, but I can’t think of an article on the topic of API documentation published in a computer science venue that talks about actually reading and using the content. At the same time, there now seems to be a recognizable genre in API documentation literature that Head et al. describes as, “finding anti-patterns in documentation” (see reviews below).
So, here’s a short review of four of the articles I read this morning.
My list of API documentation articles -non academic has grown from 18 to 30 articles. Most new ones are from the past year, but I found a few earlier articles that I seemed to have missed in my last update.
Plotting them over time shows some recent growth in frequency as with the academic articles on API documentation, however there are 66% more academic articles on the topic in my [ever expanding] collection, so it might still be too soon to draw and comparisons.
I’m still intrigued by my discovery from yesterday that I describe in API documentation research-Where’s Tech Comm? The idea seems like it might have some legs for some tenure-worthy academic paper(s), so I thought I’d dig into it a bit more to see what I might be signing myself up for.
And the plot thickened. Which, for an academic, is a good thing in that it means it has potential for papers, talks, and maybe even a book? (Yippee!). But, I’m getting ahead of myself. (No book tours, yet.).
Disclaimer: what follows is a peek into my notebook and represents a work-in-progress. Any claims I make (or appear to make) are subject to change as the research progresses.
I started to take the study a little more seriously and methodically–i.e., I started collecting data. For now, my working research questions evolved from, “Why are only computer science researchers studying API documentation and why don’t they refer to technical communication research much (or at all)?” to “Have computer science researchers ever heard of technical communication research?!” and vice versa.
Google Scholar is for Computer Science research
In searching for “API Documentation” on Google Scholar, I collected something like 50 articles and papers having to do with API documentation over the years (The earliest paper I’ve found, so far, is from 1996). The following figure shows the distribution of papers by year published.
It’s good to see the topic being researched (finally!). I started studying them around 2008-2009 when, as you can see in the chart, there weren’t many to use as a reference. As academic fields of study go, this is definitely a niche topic. And, yes the bars add up to more than 50-something because I’m still in the process of cleaning my data set. The numbers are going to be rough in the meantime, but they seem representative if they aren’t precise, yet.
The more I looked at the academic papers I was collecting, the trend of them being from computer science(y) journals and conferences was even more skewed as this chart shows.
From the look of that graph, you could think that TC is hardly talking about API documentation. It’s even more stark when you take into account that I wrote 5 of the 8 articles in the TC slice of the pie.
Yet, I hear lots of API documentation talk on TC social media channels and Linked in has several groups dedicated to the topic so something isn’t passing the sniff test.
Is TC research on API documentation only published in blogs
As a reality check, I went to regular Google and search on API documentation and found a whole bunch of relevant topics. I haven’t formally collected them into my list, so I can’t say how many, yet, but it was clearly being talked about on the web. From a quick skim, the articles from the general web seemed to be more TC oriented than CS oriented (but, don’t quote me, on that, yet).
What slowed my progress down, momentarily, was I wanted to study the scholarly articles they were using as their references. I think that’s where some tenure-worthy research opportunities will be found. I’m (slowly and tediously) categorizing the references cited by each of the scholarly articles to get an idea of what they’re basing their research on. While that’s going to be tedious (i.e. incredibly tedious: 50+ articles each with 30+ references = 1,500+ citations to clean up and organize–if only they just attached a metadata file).
Unfortunately, the blog posts and informal references from the TC community in “regular” Google don’t tend to list as many citations as the academic articles, so looking for their foundations is going to take some more detective work. But, one thing at a time.
What it’s looking like, however, is that the academic CS scholars don’t refer to much in the way of TC research. I suppose that if they don’t provide any citations, you could say that the TC research doesn’t either. Maybe I’ll need to turn to some tcmyths for more info?
In any case, regardless of where this goes, I’ll end up with a killer bibliography of API documentation!
Google Scholar was reading my mind this morning and provided me a list of recommended papers. This was both interesting and troubling at the same time. It was interesting that they knew what I’d find interesting (they have been listening!) and troubling that I was only familiar with a couple from the list (i.e. I need to read more, apparently). Most disturbing was that in the articles on API documentation, most all references that those articles cited were from computer science literature and not from documentation or technical writing. One article cited my dissertation, so I can’t say there were no references to documentation research.
It could be there’s nothing to find. Searching Google Scholar for articles on “API documentation” yields no articles from non computer-science journals until one of mine appears on the fourth page of search results. So, it’s a reasonable question to ask if there are other sources to find.
I published my first API around 1988 for a peripheral to the IBM PC in which the API consisted of software interrupts to MS-DOS. (A software interrupt is similar in function to a procedure call, but used for operating system and device driver functions. I didn’t write the documentation (at least not the published version), but a couple of co-workers and I wrote the interface.
Later, I want to say in 1997-ish, I wrote the API provided by the Performance Data Helper (PDH) dynamic link library (DLL) that shipped in Microsoft Windows NT 4.0 (IIRC). I didn’t write the documentation for this API either as I was still developing software at the time. A super technical writer did the heavy lifting of making sense out of the functions the PDH.DLL provided.
While this isn’t a particularly impressive resume of API development, it shows that APIs and I go way back. They are a very useful tool for solving many problems. Yet, over the past 30+ years that I’ve been using, developing, and documenting APIs, there hasn’t really been much written about their documentation–until recently.
In my informal count (i.e. what I could find on Google while having my morning coffee a few days ago). I came across 17 articles and blog posts on the topic in just the past seven years (full list at the end of the post).
Last month, I published a summary of my dissertation study and I wanted to summarize some of the thoughts that the study results provoked. My first thought was that my experiment was broken. I had four distinctly different versions of each topic yet saw no significant difference between them in the time participants took to determine the relevance of the topic to the task scenario. Based on all the literature about how people read on the web and the importance of headings and in-page navigation cues in web documents, I expected to see at least some difference. But, no.
The other finding that surprised me was the average length of time that participants spent evaluating the topics. Whether the topic was relevant or not, participants reviewed a topic for an average of about 44 seconds before they decided its relevance. This was interesting for several reasons.
The average time spend reading a reference topic to determine its relevance in my study was the same whether the topic was relevant to the scenario or not. I would have expected them to be different–the non-relevant topics taking longer than the relevant ones on the assumption that readers would spend more time looking for an answer. But no. They seemed to take about 44 seconds to decide whether the topic would apply or not in both cases.
While, these findings are interesting, and bear further investigation, they point out the importance of readers’ contexts and tasks when considering page content and design. In this case, changing one aspect of a document’s design can improve one metric (e.g. information details and decision speed) at the cost of degrading others (credibility and appearance).
The challenges then become:
Finding ways to understand the audience and their tasks better to know what’s important to them
Finding ways to measure the success of the content in helping accomplishing those tasks
During November and December, 2014, I ran a study to test how varying the design and content of an API reference topic influenced participants’ time to decide if the topic was relevant to a scenario.
I collected data from 698 individual task scenarios were from 201 participants.
The shorter API reference topics were assessed 20% more quickly than the longer ones, but were less credible and were judged to have a less professional appearance than the longer ones.
The API reference topics with more design elements were not assessed any more quickly than those with only a few design elements, but the topics with more design elements were more credible and judged to have a more professional appearance.
Testing API documentation isn’t that difficult (now that I know how to do it, anyway).
The most unexpected result, based on the literature, was how the variations of visual design did not significantly influence the decision time. Another surprise was how long the average decision time was–almost 44 seconds, overall. That’s more than long enough to read the entire topic. Did they scan or read? Unfortunately, I couldn’t tell from my study.
The experiment measured how quickly participants assessed the relevance of an API reference topic to a task-based programming scenario. Each participant was presented with four task scenarios: There were two scenarios for each task: one to which the topic applied and another to which the topic was not relevant and each participant saw two of each. There were four variations of each API reference topic; however, each participant only saw one–they had no way to compare one variation to another.
The four variations of API reference topics resulted from two levels of visual design and two levels of the amount of information presented in the topic.