My friend, Ben Rothke, asked me to post my comments to his recent piece "Sue the Auditor and Shut Down the Firm" over on CSO Online. The topic is one I've thought about a lot over the years; namely, how do you control quality and performance for 3rd party auditors. After all, quality is the core problem being targeted in the Savvis law suits, and the basis of the aforementioned article.
Overall, I think the point of the article is fine. I think there's a fundamental problem that needs to be addressed with auditing, though, and incompetent auditors is merely a symptom. The way IT audits often work is that you sign multi-year contracts and then, as the auditor, you find ways to cut your costs (aka "time spent auditing") every year in order to maximize your realization. Such an incentive inevitably means that auditors will try to staff gigs as cheaply as possible, which means you'll get inexperienced or incompetent auditors. It's a bad cycle, and one that will only be changed at expense to all involved. Certification, licensure, and strict liability might help, but it will also result in increasing the cost of audits.
There's also a question about the utility of having much stronger auditors. Does the benefit outweigh the cost? Do we really need auditors who are much stronger? I guess it depends on how high the stakes are for the audit. In the case of PCI DSS, where QSAs are publishing attestations of compliance upon which people can hang their hats, there certainly is a concern about quality. However, at the same time, there has to be a realization of limited liability. An audit is only a snapshot in time of a limited number of systems across a limited number of requirements. Especially when you factor in sampling, it should be painfully obvious that no audit result should be considered a gold standard.
What, then, is the solution? It's unclear to me if there are any good solutions today. A combination of metrics, security testing, audits, and overall due diligence documentation seem like a good starting point. Demonstrated and measurable organizational maturity may also be worthwhile. However, again, all of these may rely to more or less a degree on point-in-time attestation, which could expire before the auditor walks out of the building. It seems unlikely that this particular problem can or will be "solved" per se. It's not clear that it's "solvable" - or even if it's defined well enough to be solvable.
It's an interesting place we've come to in this industry. Clearly something is broken. Many things, most likely. A new paradigm sure would be nice.
