If we could only test docs like we can test code

Postman logo
Postman logo

As I continue to catch up on delinquent and neglected tasks during the inter-semester break, I’ve started porting the software from last year’s piClinic Console to make it ready for prime time. I don’t want to have any prototype code in the software that I’ll be leaving in the clinics this coming summer!

So, module by module, I’m reviewing the code and tightening all the loose screws. To help me along the way, I’m developing automated tests, which is something I haven’t done for quite a while.

The good news about automated tests is they find bugs. The bad news is they find bugs (a lot of bugs, especially as I get things off the ground). The code, however, is noticeably more solid as a result of all this automated testing and I no longer have to wonder if the code can handle this case or that, because I’ll have a test for that!

With testing, I’m getting to know the joy that comes with making a change to the code and not breaking any of the previous tests and the excitement of having the new features work the first time!

 I’m also learning to live with the pain of troubleshooting a failed test. Anywhere during the test and development cycle a test could fail because:

  1.  The test is broken, which happens when I didn’t update a test to accommodate a change in the code.
  2. The code is broken, which happens (occasionally).
  3. The environment is broken. Some tests work only in a specific context like with a certain user account or after a previous test has passes.
  4. Cosmic rays. Sometimes they just fail.

The challenge in troubleshooting these test failures is picking the right option from the start to make sure you don’t break something that actually was working (but whatever was really broken is hiding that fact).

But, this is nothing new for developers (or testers). It is, however, completely foreign to a writer.

Here are some of the differences.

Unit tests or sub tests

In coding, you can (or should be able to) isolate elements of the code to test them at a very focused to identify problems that could affect or be affected by other aspects of the program. While writing, I’ve sent snippets and paragraphs out for SME review, so that might be close, but the SME is still more of a structural test (is it correct?) not an effectiveness test (does it work?).

As any developer (or tester) can tell you compiling (being structurally sound) does not guarantee it will do what you want it to.

Testing

While writers have tools that can check for structural defects, there aren’t too many tools around to tell if a document works, at least not while you’re writing it. You might have to wait weeks or months for that type of feedback, if you get any feedback at all. As I write the code, I can test, what would amount to a paragraph if it were prose. Because of this, I can never really get more than a module (paragraph) off course before realizing I need to fix something.

With writing, I’ve sent out entire document sets, to CUSTOMERS!, without any testing! No developer in their right mind would think of doing that.

And, what does it even mean for documentation to “work?”

Specifications

To write a test means you have to know what you want the code to do at a very detailed and specific level. In general a test definition looks something like this.

When given these specific inputs, the code under test will return these particular outputs.

Writing a test, therefore, consists of configuring a test to provide those specific inputs and test for these particular outputs. It works or it’s broken (or the environment’s out of whack, or a preceding test didn’t complete…) But, that’s about as cut and dried as you can get.

With writing? Not so much. For example, a documentation test might look like: If the reader reads this document, they will be able to complete this task in the specified time period. Not every document lends itself to that type of test, but some do. Unfortunately, even for those that do, being able to run that test on demand is a considerable challenge (to put it mildly).

Testing is a lot of work!

But recovering from errors is more work. So, it’s time well spent. How much time? Right now, writing tests is taking about 2-3x the time it takes to write code. Part of that is due to the fact that, writing the code is quick as I’m porting it over from v1.0, I’m getting the test environment set up from scratch, and I’ve never used the tool (Postman) before, but it still adds a lot of work. What I can’t measure now, but I can see in the code and the test results, is that the end result is a lot more solid.

So, the effort is paying off. As I add the more complicated features, I’ll be reaping the dividends on this initial investment.

Now, if there was just some way to apply this to writing…

Leave a Reply