Software Development
November 2003

Dealing with Defects

Tracking bugs isn’t too tricky—but streamlining the process on a shoestring presents a challenge, and Seapine Software’s TestTrack Pro rises to it.

By Allan McNaughton

With IT budgets under ever-increasing scrutiny, it’s important to make the most of every dollar spent, but when development tool vendors dangle costly baubles in your face, it can be a serious challenge not to empty your wallet. From enterprise-class Java application servers to high-end load-testing tools, developers regularly face tough decisions. Luckily, if you’re in the market for a bug-tracking system, you can get a quality product without going broke: Seapine Software’s TestTrack Pro 5.1, a recent Jolt Award winner in the Project Management Tools category, offers a robust feature set at a price that won’t make your CFO get the vapors.

Filling In the Cracks

Tracking bugs isn’t rocket science, but the value of a good tracking tool is inestimable, as all too many projects have foundered due to critical bugs slipping through the cracks. Aside from price, what really differentiates bug-tracking systems is the richness of the functionality that manipulates those tables. Developers should look for a tool that meets the following criteria:

TestTrack Pro was clearly designed with many of these criteria in mind. The product also extends these capabilities to those who don’t have TestTrack Pro accounts—useful for bringing customers into the loop.

With its straightforward client/server architecture, TestTrack Pro requires limited administration and can be deployed with minimal effort. This simplicity comes in part from its reliance on a proprietary database implementation. Defects can be managed through a Windows client or an equally capable Web implementation. SCC integration is supported to ensure proper tracking of code changes. TestTrack Pro also provides a flexible reporting engine based on industry-standard XSL (Extensible Style Language) technology. Integration with existing business applications is made possible with support for a SOAP API.

TestTrack Pro is a snap to get up and running. To use the TestTrack Pro Web and SOAP interfaces, you must have a Web server configured on your target machine prior to installation. After installation, you have the option of starting the server process as background service. Setting up the Windows client is also smooth sailing.

With the system up and running, next, you’ll populate the defect database. Unless you’re starting with a clean-slate project, you must import existing data into TestTrack Pro, which probably involves some script writing, as you can’t directly import data from other bug-tracking systems. Fortunately, TestTrack Pro does include an import feature that can handle XML and CSV files.

Seapine recommends using the XML format and provides an easily understandable DTD. If you’re not up to converting a text dump of your existing database to XML, you can use the text file import feature—although you’ll have to map the columns manually to the TestTrack Pro schema.

Creating a new defect database with the server administration tool is a breeze, as the tool distills this task to its bare essentials. If you’re already using TestTrack Pro and want to create another database, the server administration tool lets you use an existing database as a template, preserving critical information about your users, user groups, customers, query filters and test machine configurations.

A handy wizard governs the process of importing XML data. Formatting the data correctly will probably be your biggest problem. Before starting the import process, you should have TestTrack Pro validate the XML—resolving data problems before importing is easier than cleaning them up afterward. You’re also given the option of transferring defect numbers or starting with a new sequence. User information in the defect record is transferred in its simplest form, so all you get is the user name and a newly created entry in the user table.

The newly created users from my XML import still required manual editing before they could be used, as an account name must be assigned to the record so the user can log in to manage his bugs. Additionally, you must assign the user to a user group, a collection of users who share responsibilities and perform similar tasks. Access to TestTrack Pro functions, such as adding a defect or using SCC integration, is controlled by group-level security. In addition to setting the commands that a group can execute and the fields that can be edited, the security model can also restrict the set of defects that a group can see.

Customizing the security model for a newly created bug database can be tedious—a more efficient approach would get it right once and then create a new database using the existing one as a template. Of course, if you already have 10 bug databases and need to change the security model, you must edit each one manually, thus exposing a shortcoming in TestTrack Pro’s administration capabilities. Each defect database is effectively an island, so basic information that is generally shared among databases must be modified for each instance. Administrators of large-scale projects will find this problem further compounded by TestTrack Pro’s lack of LDAP support, as each user’s account must be entered by hand.

Managing Defects

Not only is it easy to submit a defect using TestTrack Pro’s SoloSubmit feature, but those who use your product outside of your organization are supported, as well, providing external customers with a straightforward Web-based bug-submission interface in which they can enter their contact information and a problem description, and attach supporting files. It also captures product, version and bug-severity information, using the terminology you defined when setting up the database. Enabling SoloSubmit is as simple as clicking a check box for each defect database you wish to expose. However, you’ll still have to open a hole in your firewall for outside users to talk with the TestTrack Pro server—something not every operations team may be comfortable with.

SoloBug, another useful bug-submission feature, is a 400KB stand-alone app that enables users to create a bug report offline. Each report creates a single file that you must e-mail to a preassigned address. TestTrack Pro reads e-mail from that address, automatically adding those defects to the database. By bundling the freely distributed SoloBug with your software, you’ve effectively integrated the customer into your QA organization.

Although TestTrack Pro offers a variety of ways to submit bugs, users will most likely use the richly functional Windows and Web client interfaces. The screen for adding a bug is logically divided into a tabbed interface in which you can enter the bug description, describe how to reproduce the issue, attach supporting files and enter details about the user’s test configuration. These steps are easy to follow, as the UI is organized in a clear, logical manner. I especially appreciated the ability to create a unique list of users (per bug) to be notified by e-mail whenever a defect changes.

SCC integration has become a checklist item for bug-tracking systems. TestTrack Pro includes support for CS-RCS, CVS, ClearCase, PVCS, Perforce, SourceOffSite Classic, StarTeam, Surround SCM and Visual SourceSafe. The usage model chosen by Seapine and most other vendors requires that the bug-tracking tool drive the actions of the SCC system: Check-ins and other file-related operations must be initiated using the TestTrack Pro Windows client. Prior to working with a file, you must manually associate it with a specific defect for SCC actions to be recorded. This approach is cumbersome, as you must always remember to follow these steps—not an easy task when you’re debugging a problem that could end up touching a dozen source files. A more streamlined solution employs tools that use an activity-based process for managing change sets—such as the Unified Change Management model of Rational ClearQuest working in conjunction with ClearCase (although this comes with a far heftier price tag).


[click for larger image]

Report Cards
TestTrack Pro’s reporting engine takes a filter’s results, sorts the data as needed and creates either aggregate or detailed reports. TestTrack Pro can also create reports on lists of individual bugs in either summary or detailed form.

Filtering and Reporting

Over time, a defect database can grow to include thousands of issues. TestTrack Pro attempts to trim this bloat through the use of filters, both simple and complex, to list only those defects that meet your selected criteria. Filters are created using a visual query builder tool—I was able to create a very complex nested Boolean query (four levels deep) without difficulty. Filters can be kept private or shared with varying levels of access. New filters are automatically added to a drop-down menu on the main defect list window, so they can be executed with a single click. TestTrack Pro also uses filters to constrain report size. The reporting engine takes a filter’s results, sorts the data as needed and creates either aggregate or detailed reports. You can also create reports on lists of individual bugs in either summary or detailed form. With customized setup screens for each report type, creating new variants is a snap.

Modifying report presentation is straightforward, as TestTrack Pro keeps all report templates in XSL format. I easily modified the look of an existing report to match my corporate site using XML Spy. Reports are rendered using HTML, so they can be posted to your intranet or sent out in e-mails.

With the addition of a SOAP API, TestTrack Pro has addressed many concerns raised by its use of a proprietary repository. The API supports basic user-level TestTrack Pro client operations such as executing filters and adding, deleting or modifying bugs, enabling the integration of defect data with existing business processes. Say you want to add some logic that escalates a defect priority after 30 days of inactivity. Using the SOAP API, you can write a script to execute a TestTrack Pro filter, iterate through the results and update each record accordingly. Although the task of creating such an application is best left to a competent developer, it can be simplified if you use Java or a Microsoft .NET language, as they come with tools to generate callable stub classes from the WSDL (Web Services Description Language) file.

TestTrack Pro also integrates with Microsoft Visual Basic 6.0, Visual C++ 6.0 and Visual Studio .NET. Its add-in dockable toolbar appears when the IDE is started, offering access to the TestTrack Pro Windows client. Integration with Visual Studio is rudimentary, as all defect-tracking windows are owned by TestTrack Pro, not the IDE. Requesting a defect list from the toolbar causes a TestTrack Pro model window to appear separately from the IDE. This presents a usability problem, as Visual Studio loses focus until the window is closed. Seapine should integrate TestTrack Pro directly into the Visual Studio shell.

Departmental Focus

TestTrack Pro is clearly intended for deployment at the departmental level—or, best case, across small enterprises. Although the user-level feature set is highly capable, the tool falls short in its support for large-scale geographically dispersed projects. Teams with a follow-the-sun approach to development will find its lack of database replication problematic, as they must rely on a single database instance halfway around the globe. The use of a proprietary database technology may also be a concern for shops that have standardized operational procedures on a specific database product. The addition of support for third-party databases would bring TestTrack Pro neck and neck with its competitors—such as Merant’s PVCS Tracker and DevTrack from TechExcel. On the plus side, its tightly integrated design makes server administration simple.

Thinking Small

Although TestTrack Pro may not be the absolute cheapest solution around, its reasonable price and rich feature set make it a good value. Thanks to its ease of use, combined with powerful query capabilities, flexible reporting and capable Web interface, TestTrack Pro is a winner.

Allan McNaughton is the principal analyst at Technical Insight LLC, a technology research firm that focuses on software development tools and practices. He can be reached at allen@technical-insight.com.

TestTrack Pro 5.0 Rating

Contact Information
Seapine Software Inc.
5390 Courseview Dr., Ste. 115
Mason, OH 45040
Tel: (513) 754-1655
Fax: (513) 754-1660
E-mail: sales@seapine.com
http://seapine.com/ttpro.html

Technical Requirements:

Windows client: Pentium-compatible PC running Windows 95 or newer. Web client: Any computer with a Web browser supporting HTML 3.0 or later.
Server: Pentium-compatible PC running Windows 95 or Linux 2.2 or newer; UltraSparc running Solaris (2.6 or later); any Apple running Mac OS X (10.1 or later).

Pricing Scheme:

Windows client: $295 per named machine or $795 for a floating license. Web client: $395 per concurrent session. Server: Included with client access. SOAP SDK: $995 for three concurrent connections.
SoloSubmit: $395 for unlimited users. Add mandatory 15 percent to all purchases for one year of support and maintenance.

Rating: 4 stars [Rate Card]
Pros:
  1. The Web interface is excellent.
  2. The reporting engine is flexible.
  3. The program combines a reasonable price with minimal hardware requirements.
Cons:

  1. TestTrack Pro isn't suitable for large-scale geographically dispersed projects.
  2. IDE support is poorly integrated.
  3. Administration of multiple defect databases is less than perfect.