Day 14 Readings

Today's readings are a continuation in a series studying usability practices in Open Source projects. The papers looked at include:

• Perceptions and practices of usability in the free/open source software (FoSS) community
• Usability in Open Source Software Development: Opinions and Practice
• How power users help and hinder open bug reporting

The first paper, Perceptions and practices of usability in the FoSS community, interviewed members of the FOSS community at a FOSS conference, at a corporation that produces FOSS software and remotely via Skype. There were a total of 12 developers, 5 user experience (UX) professionals, 5 non-code contributing project members (documentation,writers,localization contributers) and 5 core users of the software. In terms of the core users studied, they looked at users with a close social tie to project members, with domain expertise and extensive experience using the software. They also looked at users who regularly use nightly builds of the software. The projects studied included a wide gamut of OSS projects from a 3D animation package, to a web browser to a desktop operating system.

The study then posed four questions and answered them in turn.

Firstly, how do open source developers define and conceptualize the notion of usability. The study found that the definitions the participants provided span the range of definitions commonly found in HCI textbooks, which implies that the community as a whole posses a fairly sophisticated notion of the concept. The following are the top five definitions of usability provided (although there were more than a dozen definitions provided by the participants):

• Learnability/discoverability - features can be learned and discovered
• Logical - interface and interaction follow a certain logic that users can follow
• Efficiency - users can perform tasks efficiently
• Simplicity - Interface design and presentation is simple
• Transparency - the tool doesn't get in the way of the user

The second question, "What motivations do FOSS developers have for creating software that is usable by people other than themselves?" The study finds that social relationships, social rewards and subtle social pressures are some of the most important forces to keep people involved in a project. They provide quotes from participants which support each of these motivations. This is in contrast with conventional wisdom in OSS projects which is that a growing user base acted as a intrinsic motivator for contributors to OSS usability. However, the study also finds that a growing user base acts as a disincentive as it increases the number of bug reports, feature requests and complaints.

"What are current usability practices in the FOSS community?," was another important research question considered. This was the heart of the study, and so this summary will be a bit longer. Some novel usability methods practiced in FOSS communities include Graduated Testing and Open Content Projects. Graduated testing is the practice of new designs being incrementally evaluted by increasingly larger groups of users. At the beginning only the bleeding or cutting edge users use the software and provide feedback. Then a larger group of users use the software and provide further feedback. Finally, after all of this preliminary feedback, the new design is integrated into the "stable" release, touching all users and again waiting for users to provide feedback. In this way, critical design changes can be carefully evaluated, refined and released to the appropriate audience. Open Content Projects are community supported and funded projects where content creators use OSS to create content (for instance a short film in Blender, a open source movie 3D animation package). In these projects developers and end users work together to improve the software in the process.

The following are the top 5 methods for discovering usability issues provided by subjects in the study:

• Paying attention to what is asked, discussed or requested in primary communication channels (i.e. IRC, mailing lists, and forums)
• Through bug reports (i.e. Bugzilla)
• By discovering what doesn't work for them as they develop the software
• Via "reference users," or "bleeding edge," users who try out the latest nightly builds, as well as professional users who take full advantage of the capabilities offered by the software
• Through informal observations of friends, family and other colleagues

The following are the top 5 methods for addressing usability issues:

• Discussions via primary communication channels (i.e. IRC, mailing lists)
• Seeking feedback from mock-ups, prototypes and custom builds from others.
• Drawing up specifications, setting milestones, or articulating visions
• Via dedicated usability people
• Use of Human Interface Guidelines (HIG)

FOSS usability practices differ from traditional usability practices in a number of respects. The motivations for getting usability right in a OSS project are much different compared to getting usability right in a traditional closed source model. In the closed source model, getting usability right means growing the user base which in turn means more revenue and profit. This same motivation, however, isn't seen in many OSS projects, as improving usability will increase the user base, increase the number of feature request, problem reports and tasks that the contributors (many of whom are volunteers) don't enjoy as a task. In addition, certain practices in traditional usability are too costly for many open source projects, such as formal usability testing in a lab.

Usability in Open Source Software Development : Opinions and Practice looks into what contributor's opinions and knowledge of usability are. It then evaluates how open source usability is being practiced. They find in the study that OSS developers are interested in usability, as they can understand why getting usability right will allow them to win the war against commercial closed source applications, some of which have exceptional application usability practices. On the other hand, in practice usability is not a top priority compared to other technical issues that are more quickly to be addressed.

The study begins by performing an online questionnaire survey which asked contributors of a variety of OSS projects questions which explored the topics of: "About your current project,", "Communication," and "Usability." The questionnaire survey had a combination of quantitative and qualitative questions to get both easily analyzable result data as well as rich descriptive data that only qualitative data can provide. They then used the results of the questionnaire survey to provide guidance in planning a personal face to face interview with a sample of those who completed the questionnaire. Finally, they interviewed usability evaluators from a usability firm who works with open source projects on a regular basis.

The presented several surprising results for the researchers around the themes of OSS Contributors, Opinions about Usability, The OSS development process and finally Usability Evaluation methods in Practice.

Firstly, they found that the motivations participating in a OSS project were for a variety of reasons, however the five most popular reasons provided include:

• To strengthen free software
• OSS development is intellectually stimulating
• Improve skills of contributors
• Community reputation
• Professional Status

The opinions about usability from the open source community can be viewed in several different lights. We can look at how the open source community defines usability and compare it to how the field is defined. The interviewers asked open source contributors to define usability and have found that most of them (about 85%), use definitions which have to do with the effectiveness, efficiency and satisfaction aspects of usability as defined by ISO-9241. Therefore, it would seem fairly to conclude that open source contributors have an understanding of what usability is. Another way to view this theme is to consider how open source contributors rate usability as a priority. The survey found that a large majority of open source contributors see usability as a high (17%), very high (37%), or extremely high (29%) priority, with only 13% seeing usability as a moderate priority, 4% of contributors seeing usability as a slight priority and no participants seeing usability as having no priority. The conclusion that we can draw from this theme is that open source contributors feel that usability matters, as they both have an understanding of what usability is, and would rank usability as a priority.

On the other hand, their views on usability do not reflect how usability is practiced in many OSS projects. Usability experts are treated as advisors to a project likely because the community is afraid that they may dictate how things should be done and developers tend not to like to be told what to do even if it may be advice from a usability expert. In addition, while many developers felt that usability should be practiced from the beginning, there was a nearly equal proportion of developers who felt that usability could be performed iteratively, at the end of a development cycle, or during the testing/quality assurance part of the development cycle. However, it is well known that usability is not something that can be bolted on at the end of a project, but needs to be considered at the beginning as that will help dictate what functionality and features are needed to help support the way end users work. Moving usability further down the development cycle will only make costly redesigns more likely. Finally, in terms of evaluating the usability of their projects, the Open Source community appears to be far behind the industry in terms of practice. In the questionnaire completed by members of the open source community, the study found that 79% of respondents answered that they followed common usability conventions (i.e. "common sense). Less than half of all respondents answered that they used expert inspections, but were rarely performed by usability professionals. Furthermore, 21% of respondents have used remote usability in the past, which can be useful, but not as useful as performing usability evaluations in a usability laboratory, which only 8% of all respondents answered they performed.

The paper goes on to make many recommendations and discusses some notions about the open source world, and whether the data supports these results. One of the most surprising recommendations that it makes is to separate usability issue tracking from bug reporting. Currently, technical bugs (i.e. feature does not perform as specified), are clumped together with usability issues (i.e. the label of this menu item is unclear) in a bug tracking system (i.e. Bugzilla). However, as discussed in the previous days readings, these systems tend not to be designed for the complicated and nuanced nature of usability issue reporting. In addition, if developers had a choice between solving a technical problem or a usability problem in their bug tracking database, chances are they will fix the technical problem, as the usability issue is seen as being unexciting and less important in their eyes. By separating these two systems, developers will be forced to think about these two problems in different contexts. On the other hand, by not integrating bug tracking and usability issue tracking together, developers may not even be aware of, or track usability issues on a day to day basis. Perhaps one could develop a usability issue tracking system that integrates seamlessly into a bug tracking system, but that separates these two different concerns (technical bug reporting vs usability issue tracking).

Finally, the paper How Power Users Help and Hinder Open Source Bug Reporting, explores Mozilla Firefox bug reporting data since the Netscape days (all the way back to around 1994). The report categorizes people who use bug reporting systems into four types of people:

1. Core - who represent module owners, super reviewers and release drivers or managers
2. Active - who are people who report bugs and have been assigned to at least one bug
   
3. Reports - a contributor who reports at least one bug
4. Users - people who comment on bug reports and attach relevant files relating to an already reported bug

It took a look at how bugs were resolved. In most bug reporting systems, the following are common ways in which bugs can be resolved:

• Duplicate reports - the bug has already been reported in some other bug report
• Incomplete - the bug does not contain enough information to look into the issue
• WONTFIX - issues the community has decided not to fix
• Invalid reports - identify a problem that the project is not responsible for
  
• WorksforMe reports - did not involve a problem
• Fixed reports - leads to a change in the software (patch)

The paper seemed to be very data intensive which makes it difficult to summarize the findings precisely. Rather than discuss individual data points, I will move on and discuss what the results of the data collected means for the community. The study found that the "most effective," bug reports (if we define effective as the highest proportion of bug reports that resulted in fixes) was found in the active and core group. As you may recall from a previous reading, core and active users tend to be significantly smaller than the larger user population. The study found that 60% of all bug reports submitted by core developers resulted in fixes, compared to 13% of all reporters. In addition it also found that reporters mostly reported duplicate reports, sometimes even on bugs currently in the process of being fixed or that have recently been fixed. The study also found that bug reporting systems were used for more than just their obvious (stated) purpose. Some reporters for instance would use bug reporting systems to express their frustration over a particular bug, or to support a fellow reporter's argument of the existence of a bug (i.e. reporting that this happens to me too). It is also frequently used as technical support for users who may not be aware of a particular feature, or who may have difficulty in using a particular feature. At first one may be afraid that a good chunk of all bug reports may not be acted upon, and that the raw quantity of reports may have some impact on the effectiveness of resolving and discussing bugs and the bandwidth. However, the study found that if you split up all the bugs reported over the time period of interest (11 years), then you only have an average of 1 reporter report per developer every 50 days. Therefore, while the number of bug reports may seem to be large, and while it may seem necessary to take measures to stem the flow of bugs reported to avoid information overload, this is not the case seeing as there are 3,500 core+active developers total plus this is over the period of 11 years. Therefore suggestions such as making bug reporting mechanisms more approachable to everyday users (for instance removing the registration requirements and tedious tasks that "average," users have to go through to begin reporting bugs), may be a valuable suggestion, especially if it can be shown that the community bandwidth is not being wasted by excessive reporting.