In the 1970’s, the United States suffered an oil shortage. There were long lines at the gas pump, and people began stealing gas from one car to another. One outcome of the crisis was the federal government created a metric (MPG) to measure vehicle fuel efficiency. Today, all cars have an MPG ratings, and more importantly, consumers frequently consider vehicle MPG in their purchase decisions. Since the measurement has entered into the purchase decision, vehicle fuel efficiency has improved considerably. Today we have electric and hybrid cars. This was made possible thanks to the MPG measurement.
Today, cyber security faces a similar issue. Ransomware infections are common, threatening our economy and way of life. There is no standard measurement for detection rates of security software. More importantly, consumers, businesses, and government agencies, frequently purchase security software without considering detection rates.
There is a small yet dedicated security software detection rate industry. The three largest players are AV Test from Magdeburg, Germany; AV Comparative from Innsbruck, Austria; and Virus Bulletin from Oxford, England. The chart below analyzes the public test history of the major security products in the American market place over the last ten years.
Malwarebytes has not participated in a public test in the last 10 years. It has been 4 years since Webroot has participated in a public test. Both of these companies are significant brands, and confirms the notion that people do not purchase security products based on detection rates.
PC Pitstop partnered with AV Comparatives to create a comprehensive and superior detection rate test that hopefully will focus purchase decisions on detection rates. The test overcomes many of the flaws of prior public tests.
The test is involuntary, that is no participant was aware when the test was being performed. Products were downloaded from each of the web sites, and run in default mode. The current model between the testers and the testees is financial. The testees pay the testers and submit their products.
Since it is involuntary, the test can be comprehensive permitting the public test results for both Malwarebytes and Webroot.
A criticism of AV testing are the small sample sizes. AV Test and AV Comparatives typically employ samples sizes measured in the hundreds which is a contrast to the Virus Bulletin, which has sample sizes in the tens of thousands. This test bridged the gap with a sample size of 5000. None of the samples were known to Virus Total at the time of the test. No vendor, including PC Pitstop, provided samples for the test.
The samples were categorized into ransomware and non-ransomware samples. In addition to the first comprehensive test, this is one of the first tests to analyze ransomware detection rates.
The tests were run “On Execute” with the internet connection enabled. Each sample was run through a script and executed, and then analyzed to see if the AV product in question was able to properly detect the sample.
Conclusions
This is the first of many tests that we plan to develop to further the discussion of detection rates in the security industry. The MPG rating is certainly not perfect, but the world is better off for having it. In fact, another automotive acronym is YMMV (Your Mileage May Vary). Our new test is not perfect, but it is ground breaking in many ways, and we believe that it will force the industry to improve detection rates to thwart the threat of ransomware. We’re planning to do more involuntary tests with AV Comparatives, if you’d like to stay up to date when new results are released click here to sign up for our newsletter.
Don’t see your antivirus vendor in our results? Let us know in the comments and we’ll do our best to include them in our next test.
Please note: In the chart above Norton is listed under their company name Symantec.
UPDATE:
PC Matic commissioned another test with AV Comparatives. The number of security solutions tested increased to 28, compared to the 18 listed above. Also, additional testing measures were implemented to include polymorphic ransomware and false positives. You may view the latest AV Comparatives malware detection results here.
