Is there published research on the effectiveness of scanners?
Dr. Holger Peine of Fraunhofer IESE published a research report on the effectiveness of scanners in May 2006. The report is available for download here, but please note it’s in German:) His conclusion was that the best scanners found about 1 in 5 holes, that false positives were rampant. This is similar to anecdotal evidence we’ve been hearing for years. The report was discussed in the Webappsec mailing list.
Jeremiah Grossman of Whitehat Security gave a presentation at Black Hat Security Conference in 2004 on the challenges of automated scanning. He should know, considering he wrote a scanner himself. The presentation is available here.
Robert Auger of CGISecurity wrote a good piece on Challenges faced by automated web application security assessment tools.