Contact us for your penetration testing needs 1-866-PLYNT-24    |   Contact Us   Plynt UK Website  
Click to get Security Testing Quote

Learn about tools

  1. What vulnerabilities do automated scanners find?
  2. Is there published research on the effectiveness of scanners?
  3. Tell me 5 things a scanner can’t find.
  4. How is scanning different from an application pen test?
  5. Vendor A says his tool discovers SQL Injection. Is that true?
  6. Which are the scanners available in the market?
  7. How much does an application security scanner cost?
  8. Are there any free scanners available?
  9. How do free scanners compare with commercial scanners?
What vulnerabilities do automated scanners find?

An automated scanner is good at finding these vulnerabilities:

  • Known holes in the web server (IIS, Apache, etc.)

  • Known holes in the sample applications that come installed with a web server

  • Error messages thrown out by your application

  • Any comments lying around in your HTML code

  • Old source code lying around on your web server

  • Misconfigurations in your web server

Is there published research on the effectiveness of scanners?

Dr. Holger Peine of Fraunhofer IESE published a research report on the effectiveness of scanners in May 2006. The report is available for download here, but please note it’s in German:) His conclusion was that the best scanners found about 1 in 5 holes, that false positives were rampant. This is similar to anecdotal evidence we’ve been hearing for years. The report was discussed in the Webappsec mailing list.

At the OWASP Conference in 2005 at Washington DC, Arian Evans a researcher with FishNet security gave a presentation, comparing the different tools available. It’s available for download here.

Jeremiah Grossman of Whitehat Security gave a presentation at Black Hat Security Conference in 2004 on the challenges of automated scanning. He should know, considering he wrote a scanner himself. The presentation is available here.

Robert Auger of CGISecurity wrote a good piece on Challenges faced by automated web application security assessment tools.

Tell me 5 things a scanner can’t find.

The threats a scanner miss are usually related to your business. For an eCommerce site, these include:

  • An adversary stealing credit cards from your application

  • An adversary manipulating the price of an order

  • An adversary cancelling an order after it has been shipped

  • An adversary faking an order on behalf of another user

  • An adversary denying access to all other shoppers

For an online banking site, the threats a scanner ignores include:

  • An adversary siphoning off funds from other users

  • An adversary stealing account statements of other users

  • An adversary generating fake account statements

  • An adversary reversing older transactions

  • An adversary escalating his privileges to an administrator

In the case of an Online Reservations site, a scanner will not find these threats:

  • An adversary stealing credit cards of patrons from the application

  • An adversary cancelling reservations of other users

  • An adversary faking reservations on behalf of other users

  • An adversary stealing the customer database

  • An adversary closing the reservations at his will

How is scanning different from an application pen test?

They discover different kinds of holes.

A scanner is good to find web server holes, ugly error messages and old source code lying around on your web server. At times, it can even find SQL Injection and Cross Site Scripting holes. A scanner has a large database of known vulnerabilities in web servers, and it checks if your site is safe against those.

An application pen tester dons the hat of an attacker. He discovers logical holes real adversaries are interested in that allow them to steal credit cards, manipulate prices, siphon off funds etc. That requires a study of the application - its business context, the motives of the adversary, the workflows, the privilege levels. He then crafts exploits and tests the application. The threats discovered by a pen tester have much greater impact than those discovered by a scanner.

There are other smaller differences too:

  • A scanner takes less time to run after it is setup – usually a day or two.

  • An application pen test can take between a week to three weeks, depending on the size of the app and experience of the tester.

  • Scan reports tend to have many false positives, and even more false negatives. [False positives are false alarms. A false negative is when a hole is missed.] Pen test reports are free from false positives. The number of false negatives depends on the skill of the tester.

  • Most pen testers use a scanner to discover web server holes, error messages, comments in code etc. during the pen test.

Vendor A says his tool discovers SQL Injection. Is that true?

He is partly true. Most tools today can discover the simple form of SQL Injection. Unfortunately, most of them miss out every other form.

Let’s see what is going on.

In a SQL Injection attack, the adversary induces the database to run SQL commands that he provides. The simplest way to discover if an input is vulnerable to SQL Injection is to manipulate the input, and see the response. [There are over 30 manipulations, we’ll come back to that in a jiffy.] If the server throws back a database error message, usually an ODBC error message, the adversary knows that the input he gave went all the way to the database – that’s why the error message came back from the database.

So here’s how we discover the simplest form of SQL Injection: if the response includes a database error message, then the app is vulnerable to SQL Injection. Every scanning tool we have seen uses this logic. Some extend it to other error messages too.

But an app can be vulnerable to SQL Injection and not throw any error messages to the user. When the input reaches the database, it gives an error to the application. But the application, as part of common courtesy, usually hides the error message from the user. Thus, an application can be vulnerable to SQL Injection and yet hide all error messages from the user. In such cases, a scanner wrongly concludes that the app is safe.

The right way to find SQL Injection in such cases, (and this is by far the most common case today), is to infer from the app’s behavior that it ran the manipulated input we gave it. Most pen testers have learnt how to make these deductions and refine the exploit step-by-step to expose the vulnerability.

A variation of the simple SQL Injection attack (called Blind SQL Injection) is designed to test for SQL Injection attacks without looking for the error message. Some tools, notably Spidynamics WebInspect, use Blind SQL Injection also in their repertoire.

Now, more about those 30+ manipulations to test for SQL Injection. They are sometimes called test cases, sometimes Injection strings, at times fault injectors. They are refinements over the basic SQL Injection test case which is a simple single quote: ‘ That single quote has the power to throw an error message. Most of the 30-odd refinements improve our ability to exploit SQL Injection. If you just want to know whether the app is vulnerable, the single quote suffices. Many scanners and testers boast how many different “test cases” they use for SQL Injection. The number is really unimportant. Whether you use 15 or 30 or 50 test cases, discovering SQL Injection is about making the right inferences from a few simple inputs. All the other test cases are only for exploitation, after you are able to discover a vulnerable variable. Thus, all these boasts about the number of test cases is misplaced. It doesn’t matter if you have a 100 exploit refinements if you can’t spot the right variable that’s vulnerable to it.

Our favorite piece on SQL Injection is the Advanced SQL Injection paper by Chris Anley. It explains the sequence of inferences a tester has to make to refine his test cases.

Which are the scanners available in the market?

Here’s a list of application scanning tools, randomly ordered:

Arian Evans gave a presentation at OWASP on some of these tools.

If you'd like us to add your favourite scanner, please drop us a mail and we'll be glad to do so.

How much does an application security scanner cost?

Scanner prices vary widely based on the product, the licensing option you chose and your negotiating power.

The licensing options you will encounter are:

  • Per-application license: limits the number of apps you can scan

  • Per-seat license: one scanner, any number of apps to scan

  • Consulting license: any number of apps, including those outside your company

We have seen prices quoted from $5000 for a per-app license to $35,000 for a consulting license. Please ask your vendors for their latest quote.

Depending on your budget, you might want to check the cost of a pen test too.

Are there any free scanners available?

Here’re some of the free application vulnerability scanners available:

How do free scanners compare with commercial scanners?

The free scanners we have seen lag behind commercial scanners. There’s no published report comparing the two categories (if there’s one we have missed, please let us know), but here’s what we have made out over the last few years.

Most importantly, the open source scanners do not let you log in to an application and test it. That’s a major shortcoming that we hope will get fixed soon.

Scanners have traditionally been good at detecting web server vulnerabilities. The free scanners support less number of web servers, and also discover fewer vulnerabilities in those they support.

The chances of a free scanner eliciting errors is lesser as the number of fault injection test cases is lesser in free scanners.

The number of test cases for attacks like SQL Injection/XSS is also less in free scanners. This is not as big an issue as it’s sometimes made out to be. We explain why the number of test cases is not so important here.

Request a proposal

Our quote contains the best price, the time estimate, and our methodology; and we'll mail you the quote in 24 hrs.

Movable Type Appliance - Powered by TurnKey Linux