Numbers May Not Lie, But The AccessData Report Is Far From The Truth

Ken Mizota Ken Mizota, Product Manager, Forensic Solutions

A little over a year ago, back in March 2012, in a previous EnCase Forensic blog post, “A Development Perspective,” we discussed the improvements that we had made to EnCase, including evidence processing speeds and the comprehensiveness of the indexed results. Now, AccessData, after waiting over a year, has conducted testing at its facilities on its equipment (nominally conducted by an “independent” third party, Opus One), and has issued a report (the “AccessData Report”) which I’ll address in detail, below. The AccessData public relations campaign over the last few weeks calls to mind the famous quote from Mark Twain:


There are three kinds of lies: lies, damned lies, and statistics.

As an initial matter, we regularly conduct benchmark testing of EnCase versus competitive products, and in all our testing have never seen test results anywhere near what AccessData claims. On the contrary, EnCase consistently processes data faster and is more comprehensive in its processing. We have also heard from third parties who have conducted their own testing and confirm our results. So how could AccessData make such outrageous claims? Five things stand out:

1. First, the AccessData Report doesn’t address fundamental questions that affect performance, such as: was 32-bit or 64-bit EnCase used? (they will have dramatically different performance.) Were extra “remote processing nodes” used with FTK, or was it just the examiner? Were there non-standard registry configurations made by AccessData technical staff? Indeed, an even more fundamental question might be: did it take them 13 months to find a few data sets which favored them? For instance, FTK does not support Unix Login or Syslog for Unix or Linux data sets, and, conveniently, those features were not required for the data sets used.
2. Second, AccessData refused to use EnCase-recommended settings. For instance, the indexing maximum word length was set to a non-EnCase recommended setting. Even though the default word length setting in EnCase is 64, the AccessData testing deliberately decided to use the AccessData preferred setting. Similarly, the AccessData Report did not provide any settings for Win Event Logs or Win Artifact Parser.
3. Third, AccessData’s hardware choices were unrealistic for most computer forensics investigators. We conduct our testing using hardware that meets the minimum recommended configuration for both EnCase and FTK, on the premise that forensic investigators typically don’t have thousands of extra dollars available for server-class, high-end hardware. Further, independent, third party testing by a leading forensic systems integrator confirms the responsiveness of EnCase Forensic in reasonably priced computer systems in a recently released report. The AccessData Report, however, details that their testing used hardware well beyond the recommended levels. Not only is the configuration unrealistic, but more than likely, unattainable for the majority of digital investigators.¹ What’s worse, the AccessData Report assumes that a forensic investigator has an extra high-end machine available to dedicate solely to processing data – with all of the cores occupied by AccessData’s processing, a forensic investigator can do nothing else (for instance, work on a report) on that machine until processing completes. EnCase Forensic, on the other hand, is designed so that processing can be accomplished quickly, while the machine can also be used at the same time for other forensic work.²
4. Fourth, the testing was of old versions of both products – version 4.2 of FTK (they are now on Version 5) and Version 7.05 (actually, the report is ambiguous on this point, but it seems to be referring to 7.05) of EnCase (the current version is 7.07). We continuously make improvements to the processing engine that we have developed and control; in contrast, AccessData licenses its capabilities from a third party, so it does not have the ability to make improvements to it.
5. Finally, and perhaps most importantly, the AccessData Report ignores the most important topic of all: the comprehensiveness of the processing and indexing of data. Our testing showed that EnCase indexed more items than FTK which means that a search of an FTK index could miss evidence that may be crucial to your case. For instance, EnCase provides full indexing of all data, including the outputs of any Evidence Processor module (e.g., Yahoo IM artifacts, Firefox artifacts, etc.); this is a clear difference between the two products. In addition, EnCase handles East Asian words appropriately, using language-specific word-breakers. Looking at file carving, EnCase provides support for 314 file types, whereas FTK provides support for 42. And, as previously mentioned, EnCase supports Unix Login and Syslog. Bottom line: the AccessData Report ignores the quality of the results, which is the most important factor to a professional investigator.

As we have stated many times, we welcome testing, because we believe it ultimately makes our product better, makes the industry better, and serves the user community. But in order for testing to serve those goals, it must be conducted fairly and transparently, not as a PR stunt or marketing charade. One of the great things about our industry is that it has always had a rigorous, almost scientific focus on repeatability, on transparency, and on quality. We look forward to continuing that tradition in our software development, our product testing, and our communications with the forensics community.

I hope to see many of you at CEIC in a few weeks, and would love to discuss the topic of testing with you there. In the meantime, if you have test results you’d be willing to share, please send them our way.

---

¹ In addition, a simple comparison of the configuration specified for the “16 Core Configuration” in the AccessData Report shows that they grossly understate the street price of the hardware specified: they provide an estimated price of $11,000, but a comparison on a forensic hardware vendor’s website indicates a price closer to $15,000

² For large labs or evidence processing “factories,” we offer other products that distribute processing with the expectation that the high-end hardware used there will be solely dedicated to processing data.