Estimating the security of software is something that most systems administrators do instinctively, whether consciously or subconsciously. I have heard people say things like "Don't use WordPress it's always getting hacked", "Linux is more secure than Windows"1 or "Macs don't get viruses"2. All of these are estimates about the relative security of software.
The problem with looking at the security of software is that it's only possible to prove the negative (i.e. this software is not secure) but it's not possible to prove the positive (i.e. this software is secure) and once a piece of software become sufficiently complex I don't believe it is practically possible to write software with no bugs at all.
Let's take for example two piece's of software Apache OpenOffice and LibreOffice both were affected by CVE-2015-1774 and at the time of this writing it hasn't been patched in OpenOffice so I could prove the negative and say "OpenOffice is not secure, if a users opens a malicious .odt file3 it could result in remote code execution.". While with LibreOffice it has been patched and there are (again, at the time of this writing) no public CVEs so I could say "There are no public vulnerabilities that have not been patched." but that's not the same as saying it's secure.
There are probably bugs in LibreOffice that have been discovered but haven't been disclosed, found by folks like Vupen and other shady groups selling 0-days on the black market. And there are undoubtedly bugs that simply haven't been found by anyone yet.
So in the case of software where there are no known vulnerabilities it can't be proven to be secure, or proven to be insecure we are left in a situation where we have to estimate.
I don't think we as an industry have a good way of estimating the security health of software but I think one would be useful. There are some good indicators, a great example is the Linux Foundation's Core Infrastructure Initiative, which recently tried to score open source projects, although they were not just assessing security but also impact (e.g. is the program widely used, dose it handle network traffic, etc...).
One interesting metric is CVEs but as mentioned in the Core Infrastructure white paper, it can be misleading because a low number could indicate a lack of review (i.e. no one is looking for bugs) or it could be that there are very few bugs to find. Conversely a high number of CVEs could mean that the software is full of bugs or that it's getting a lot of attention and bugs are found and fixed quickly.
Other good metrics include the choice of programming language and framework. I recall reading an argument on line4 where one user said something like "There are no bad programming languages, only bad developers" to which the response was "Then it's reasonable to develop in Brainfuck?". I think that sums it up perfectly, some languages are simply a better choice and make it easier to write secure software.
Our opinions of software security can also be biased by the popularity of software. I think both Windows and WordPress suffer from this to a degree, for example according to W3Techs Surveys 58.7% of sites where they could detect the CMS it was WordPress. But even if we look at the lower number 24.5% of all sites run WordPress. If you think that almost 1 in every 4 sites on the net runs WordPress then it's not surprising that a lot of the horrible hacked sites that are being used to drop malware are running out of date WordPress installs.
Similarly if you think how many times you have seen a nasty Windows desktop a bunch of spyware and junk installed. Systems where there is no option but to format and reinstall, but then think how many times have you seen a non Windows desktops, sure there are a few mac floating about but not that many. It's not surprising that most desktop malware targets Windows. Just like most SSH password brute forcing worms target Linux, because there are more Linux servers with SSH exposed on the net.
I think in most situations picking the most secure software ends up being a gut feeling thing or at best an informed guess, but there are some objective measures out there if you look for them.
-
While Debian GNU Linux is my preferred operating system I don't necessarily think it's more secure. This is just an example of things I've heard people say. ↩
-
Of course all operating systems more complex that an calculator can get viruses. Macs have bash, you could run
:(){ :|:& };:that's self replicating code although it won't survive a reboot ↩ -
The vulnerability is with the HWP format, but you can give a hwp file a .odt extension and OpenOffice will still parse the file and trigger the vulnerability. ↩
-
I think it was about Python vs Java on Security Stack Exchange but can't seem to find the link. ↩
