Usability studies of online documentation (index-versus-search)

Topic status automatically displays here - do not remove.

Bookmark this topic  Print this topic

By Colin Ramsden, May 2008.

 

In the technical communication industry, there has been much talk over the years about the usability of indexes-versus-search for online documentation (Help and PDF), and it's been several years since I last researched the subject, so I thought it was time to update my statistics on the matter. This topic discusses my online research into the all important question of index-versus-search for online documentation, and presents a summary of the latest online documentation usability study results and conclusions.

The Google search I performed on 8 May 2008 produced over 925 thousand hits for the following list of keywords "usability study index search online print" (without the quotes), so there's no shortage of web resources making use of these words in the same resource. Too many for just one person to examine.

When I subsequently grouped these words using quotes around subject phrasing, the search results were more subject focussed. I grouped "usability study" with the other four left ungrouped and returned 2970 hits. Adding the word "documentation" and grouping it with "online" returned a much more manageable result of 156 hits. These all fall within the broad category of "Usability studies of online documentation".

Picking through the results of the first few pages, I found 3 recurring themes emerge which were of particular interest to me: studies on improving the usability of indexing; studies on improving web site usability design; and studies on improving the usability of online documentation. I've grouped some of each for reference:

Studies on improving the usability of indexing

1. Index versus full-text search- a usability study of user ... Describes some of the usability issues resulting from the conversion of paper documents to Adobe PDF format and reports that users preferred the full text search tools but obtained more accurate results with an electronic back-of-the-book index. This article reports on the results of testing two versions of an information product, Usability testing and research: one version, an Adobe Acrobat Reader e-book with an index with the locators hyperlinked to the page reference for each entry; the other version, the same e-book without an index, but with the full-text search capabilities provided by Acrobat Reader. We first summarize the current literature regarding human indexing and information retrieval by machine (search engines). We then describe the methodology for testing, the testing results, our conclusions, and implications for future research.
2. Using a Task-Based Approach in Evaluating the Usability of BoBIs in an E-book Environment This paper reports on a usability evaluation of BoBIs (Back-of-the-book Indexes) as searching and browsing tools in an e-book environment. This study employed a task-based approach and within-subject design. The retrieval performance of a BoBI was compared with a ToC and Full-Text Search tool in terms of their respective effectiveness and efficiency for finding information in e-books. The results demonstrated that a BoBI was significantly more efficient (faster) and useful compared to a ToC or Full-Text Search tool for finding information in an e-book environment.
3. A usability assessment of online indexing structures in the ... American Society for Information Science Journal article for the US Bureau of Labor Statistics which investigated the index structure of the FedStats Web site at (http://www.fedstats.gov). The findings indicate that a hypertext index with multiple access points for each concept, all linked to the same resource, led to greater effectiveness and efficiency of retrieval on almost all measures. However, satisfaction measures were more variable. 

Studies on improving web site usability design

1. Research-Based Web Design and Usability Guidelines US Department of Health and Human Services publication of 209 individually rated design guidelines to create better and more usable websites, and to highlight areas for further research-based web-design studies.
2. Writing for the Web Sun Microsystems recommendations for web authors.
3. Museums and the Web 2000 Impact of Subjective Issues on ... University of Limerick (Ireland) study examining the role of cultural issues on the usability of the Louvre Museum (France) Official Web site. The primary research involved a usability study of different culture groups, in four languages. Results show that labels and captions on diagrams were not localized. The report goes on to discuss the requirements of multi-lingual and multi-cultural sites and the usability procedure and results of that study. 

Studies on improving the usability of online documentation

1. Access ThinkPad- The right information at the right time and place IBM moved from a 500 page printed manual to a 50 topic online help system, and this report documents their compromises and measures their performance based upon extensive usability testing before and after the migration and delivery processes. 
2. Combining Usability Testing and Documentation Projects Report IEEE TC97. This paper describes two cases in which usability testing and documentation projects were performed in conjunction with one other. It describes how usability testing affected the design and content of the documentation and how follow-on usability studies added significant new data not revealed in the initial tests.
3. Creating Easy-To-Use Documentation for Paper, Online and Multimedia addressing some of the differences between paper and online documentation that impact the development of easy-to-use online documentation, and outline some of the high-level, emerging issues to be aware of in the development of multimedia documentation. 

Conclusion

More and more people are connected to the internet every year, and peoples' research and reading behaviour has changed and adapted accordingly. The search facility for web browsers has introduced search technology to many people, who increasingly expect and apply search principles to their other online activities, such as online documentation. An emerging trend has been observed in several usability studies showing that many people prefer to use the search functionality, if available, in preference to using menu navigation or lists of links. In some cases, it was observed and reported that this preference for using the search functionality persisted in users even after several previous failed search attempts, and also after being prompted to try other methods such as the index.

Problem

[Insert explanation of search producing exact word matching results, and not subject matter matching.]

 

Answer

This presents a relatively new challenge to online documentation authors, in light of this emerging usage tendency for searching, to create appropriate contextual keywords within the document text itself, to improve the chances of a successful search using search criteria which might otherwise have been omitted from the document text, and placed within the index.

Many websites are already providing visible keyword listing in the topics they refer to (belong with?)

[COL: Provide example snippet of MS UA team blog site]

 

 

[COL: Move the following extracts to a separate topic]

 

Writing for the Web by Jakob Nielsen, distinguished engineer; PJ Schemenaur, technical editor; and Jonathan Fox, editor-in-chief, www.sun.com You can double the usability of your web site by following these guidelines: for two sample sites studied in Sun's Science Office, we improved measured usability by 159% and 124% by rewriting the content according to the guidelines. Writing for the Web is very different from writing for print: 79% of users scan the page instead of reading word-for-word Reading from computer screens is 25% slower than from paper Web content should have 50% of the word count of its paper equivalent Table of Contents Difference Between Paper and Online Presentation: Limit scrolling; use simple sentence structure; avoid cute headlines; update facts frequently. Working With a Designer: Combine page templates with professional artwork. Scannability: Highlight keywords; use bulleted lists; start with the conclusion. Navigation: Move detailed info to secondary pages. Writing to Be Read: Headlines and subheads; lists, captions, and hyperlinks. Writing to Be Found: Half of the users will navigate through search engines. Terms to Avoid: Don't call attention to web artifacts. Editorial Review of Web Pages: Fresh eyes and skilled editing improve your work. Web Facts: Significant improvements in all metrics.

 

In print, your document forms a whole and the user is focused on the entire set of information. On the Web, you need to split each document into multiple hyperlinked pages since users are not willing to read long pages.

Users can enter a site at any page and move between pages as they chose, so make every page independent and explain its topic without assumptions about the previous page seen by the user.

Link to background or explanatory information to help users who do not have the necessary knowledge to understand or use the page.


Make the word count for the online version of a given topic about half the word count used when writing for print: Users find it painful to read too much text on screens, and they read about 25 percent more slowly from screens than from paper.


Users don't like to scroll through masses of text, so put the most important information at the top.


Web users are impatient and critical: They have not chosen your site because you are great but because they have something they need to do. Write in the "news you can use" style to allow users to quickly find the information they want.

Credibility is important on the Web where users connect to unknown servers at remote locations. You have to work to earn the user's trust, which is rapidly lost if you use exaggerated claims or overly boastful language; avoid "marketese" in favor of a more objective style.

A few hyperlinks to other sites with supporting information increase the credibility of your pages. If at all possible, link quotes from magazine reviews and other articles to the source.


The Web is an informal and immediate medium, compared to print, so users appreciate a somewhat informal writing style and small amounts of humor.

Do not use clever or cute headings since users rely on scanning to pick up the meaning of the text.



Limit the use of metaphors, particularly in headings: Users might take you literally.



Use simple sentence structures: Convoluted writing and complex words are even harder to understand online.



Puns do not work for international users; find some other way to be humorous.



Add bylines and other ways of communicating some of your personality. (This also increases credibility.)


The Web is a fluid medium: Update pages as time goes by to reflect all changes. Statistics, numbers, and examples all need to be recent or credibility suffers.

For example: Before a conference, the page about the event might point to a registration form; afterward, point to slides or presentation transcripts instead.

Scannability
Seventy-nine percent of Web users scan pages; they do not read word-by-word. Design your web document to be scannable:

To make keywords stand out, use highlighting liberally: Highlight about three times as many words as you would when writing for print.

Use the <STRONG> tag rather than the <EM> tag for keyword highlighting. Since STRONG is usually rendered as boldface, this is typically the best way to highlight words.

Colored text or colored backgrounds can also be used for highlighting, but don't use blue for words. That color is reserved for hyperlinks.

The hyperlinks also stand out by virtue of being colored, so they should be written to do double duty as highlighted keywords.

Highlight only key information-carrying words. Avoid highlighting entire sentences or long phrases since a scanning eye can only pick up two (or at most three) words at a time.

Highlight words that differentiate your page from other pages and words that symbolize what a given paragraph is about (for example, do not highlight the word "Sun" when writing for the Sun Web site since all the pages are about Sun.)

The <EM> tag is usually rendered as italics and can be used to make figure captions or emphasized sentences or phrases stand out. Do not use it for large blocks of text, since italic typefaces are slower to read online.

Bulleted and numbered lists slow down the scanning eye and can draw attention to important points.

Each paragraph should contain one main idea; use a second paragraph for a second idea, since users tend to skip any second point as they scan over the paragraph.

Start the page with the conclusion as well as a short summary of the remaining contents ("inverted pyramid" style).

Navigation

Navigating a web document differs from navigating the Web. A web document fits within one or more web pages and covers a focused topic. The web page is the unit displayed to the user and can contain one or more web documents (as well as other web elements).

NAVIGATING DOCUMENTS

When writing a document for the Web, use links to guide the reader through the document. Think of "linking" as the quickest means to get the user to the most relevant information. Whenever possible, state conclusions and link to supporting details; enumerate categories and link to lists; summarize and link to full-length treatments. This allows the user to scan the contents of a page and select relevant and useful information.

Links embedded in a document are the primary links that you want a reader to see; since readers use links as guideposts in scanning, you want to use them correctly and write in a way that takes best advantage of them. Only the most pertinent should be "part" of the document. Don't let links become a distraction. Position less relevant, but meaningful links of additional information in the web page's margin or at the end of the document under a "See Also" label.

NAVIGATING THE WEB

If a link takes the user "outside" the document, then its purpose is to navigate the Web site (or direct the reader to a third-party web site). Whenever possible, links such as these should guide the user to additional information that is directly connected--not only to the topic of the document, but to the topic of the paragraph or section being read.

Writing to be read:

Part of web page design includes the consistent use of textual elements. These guidelines will improve readability:

Heads:	Make the topmost head on the page an H1, worded so that the user knows why the page is important. Make sure that heads clearly indicate the content of the sections. Avoid in-line character formatting to heads--the results are unpredictable, varying from browser to browser. Organize your text so that the hierarchy is no deeper than four levels. Lower-level heads are hard to distinguish and disorienting to online readers.
Lists:	You can include a greater number of lists on a web page than on a printed paper page. Use numbered lists when the order of entries is important. Use unnumbered lists whenever the sequence of the entries is not important. Limit the number of items in a single list to no more than nine. Generally, limit lists to no more than two levels: primary and secondary.
Captions:	Make sure that the caption uniquely identifies the illustration or table. For example, do not give the same name to the caption as you have given to a head on the same page or another page. Caption illustrations except when the context is so clear that captions would be redundant. Don't number illustrations sequentially by chapter, section, or the like. If a screen capture has more than one illustration to which you must refer, use a simple numbering scheme (Figure 1, Figure 2). If you follow the "one topic per screen" guideline, however, figure numbers usually won't be necessary. Don't include figure captions unless you need them or have a lot of conceptual or reference material.
Hyperlinks:	Don't use a hypertext link if the information can be succinctly presented on the current page. Don't mention that you are providing links at all. Use a description of the information to be found in the link, or perhaps the link address. Use hyperlinks to provide supplemental information like definitions of terms and abbreviations, reference information, and background reading. Cluster cross-references under a "See also" (or similar) heading where appropriate. Generally, such lists of cross-references are easiest to read if they include only headings or titles with a few words of explanation. NOTE: The left navigation bar on www.sun.com correctly lists cross-references with no explanatory text.

Writing to be found:

More than half of web users rely on search engines to navigate pages.

When users link to a page from a search engine, they should know immediately how the page relates to their query. Highlight keywords, start the page with a summary, and follow the guidelines listed under "Scannability".

Include in each page all possible query terms that can be used to search for the topic of the page. List the most important terms in a keywords meta-tag together with all common synonyms (even ones not included in the body text). Include generic terms used by customers or competing companies to describe the topic of the page; do not include competitors' trademarks in the meta-tag.

Syntax: <META name="keywords" content="Solaris 2.6, upgrade requirements, operating system versions">

Use a controlled vocabulary to add keywords to the meta-tags for your pages: Create a list of common terms for your subject area and make sure that each of these terms is added to the keywords meta-tag for those related pages.

Do not add a keyword if the page is only peripherally related to the term. Only use keywords that describe the main topic of a page.

Each page must have a <TITLE> tag.

Create title text of a single line, typically no more than 60 characters. Make sure that the first 40 characters of the title describe the topic of the page: titles are often truncated in navigation menus and by search engines.


Make the first word of the title the most important descriptor of the page: Users often scan down long lists of titles to choose pages.


Do not begin a title with a generic term ("Welcome to") or an article ("The").


The title should make sense when viewed completely out of context, as part of a long list of other page titles.


Give different titles to different pages: Pages about the same topic can start with the same words but should end with words that explain the difference between them.


Titles can contain no markup or highlighting.


Titles must be written in mixed case.


In rare cases, a single keyword might be emphasized by use of UPPERCASE (do not use for the first word in the title: being first is sufficient emphasis).


Each page should have a short summary in a description meta-tag. This summary is shown below the page title by some search engines.


Descriptions must be 150 characters or less.


Descriptions must make sense when read completely out of context of the site (though you can assume that they will be seen together with the page title).


Descriptions should tell users what the page is about and allow them to judge whether it is of relevance to their current problem.


Do not fill descriptions with hyperbole or promotional language; concentrate on the facts.


Syntax: <META name="description" content="How to upgrade from Solaris 2.5 to Solaris 2.6: system requirements, where to buy, link to online download.">

Terms to avoid:

Writing well for the Web means taking advantage of the options the Web offers, but at the same time, not calling attention to the Web. "Click here," "follow this link," and "this Web site" are just a few self-referential terms to avoid.

Generally, if the words or phrases are specific to Web use, then they are probably words to avoid. A good test of web-term overuse is to print the page out, read it, and ask yourself if it makes as much sense on paper as it does on screen.

You can't eliminate all references to the Web, especially when giving browser-related instructions. However, a common error to beware of is assuming that everyone reading the page uses the same browser. For instance, instructions on how to download a file are different from browser to browser. Make sure that your instructions are detailed enough to be understood without being specific to browser version or brand of browser.

Editorial review:

An editor can help you polish the content of your web pages before you release them to the rest of the world by improving the grammar, punctuation, and consistency, and by making content suggestions.

The editor can also serve as your usability tester, so be sure to create a list of any aspects of your web page design or content for which you particularly need feedback. (You can provide this information in the appropriate area of the editing request form, listed in "Editing References.")

To schedule editing, submit a hard-copy version of your web pages for the editorial review along with the completed editing request form. Or, provide the URL and the completed electronic editing request. A hard-copy edit decreases the likelihood that questionable corrections will be made to the electronic file. The paper version also gives you a handwritten record of the changes.

USAGE REFERENCES

For more information on writing style conventions, see the following:

The Chicago Manual of Style. 14th ed. Chicago: University of Chicago Press, 1993.

Merriam-Webster's Collegiate Dictionary. 10th ed. Springfield, Mass.: Merriam-Webster, 1994.

The Random House Dictionary of the English Language. 2d ed. New York: Random House, 1987.

Web Facts
79% of users always scan; only 16% read word-by-word

Reading from computer screens is 25% slower than from paper

Web content should be 50% the size of its paper equivalent

White Papers re-written according to these guidelines have shown significant improvements in all metrics:

Task Time: User Error: Memory: Subjective satisfaction: Overall usability:

180% faster 809% fewer 100% more 37% higher 159% better

Research Article

A usability assessment of online indexing structures in the networked environment

Carol A. Hert 1 *, Elin K. Jacob 1, Patrick Dawson 1

1School of Information Studies, Syracuse University, Syracuse, NY 13244-4100 2School of Library and Information Science, Indiana University, Bloomington, IN 47405 3Citadel Investment Group, Chicago, IL

email: Carol A. Hert (cahert@syr.edu) Elin K. Jacob (ejacob@indiana.edu) Patrick Dawson (pdawson@wfg.com)

^*Correspondence to Carol A. Hert, School of Information Studies, Syracuse University, Syracuse, NY 13244-4100

Index Terms

user satisfaction • usability • network computers • telecommunications networks • information retrieval indexes

Abstract

Usability of Web sites has become an increasingly important area of research as Web sites proliferate and problems with use are noted. Generally, aspects of Web sites that have been investigated focus on such areas as overall design and navigation. The exploratory study reported on here investigates one specific component of a Web site - the index structure. By employing index usability metrics developed by Liddy and Jörgensen ([1993]; Jörgensen & Liddy, [1996]) and modified to accommodate a hypertext environment, the study compared the effectiveness and efficiency of 20 subjects who used one existing index (the A-Z index on the FedStats Web site at http://www.fedstats.gov) and three experimental variants to complete five researcher-generated tasks. User satisfaction with the indexes was also evaluated. The findings indicate that a hypertext index with multiple access points for each concept, all linked to the same resource, led to greater effectiveness and efficiency of retrieval on almost all measures. Satisfaction measures were more variable. The study offers insight into potential improvements in the design of Web-based indexes and provides preliminary assessment of the validity of the measures employed.

 

 

 

 

 

Who am I? > find out more