When scores were based solely on safety features and code hygiene, Windows was in the lead, with Linux trailing. Linux did so much better in terms of code complexity, however, that even though that's just 20% of the overall score, the total scores are now fairly close. You can browse the different evaluation categories below to determine which elements are more important to you.
We are including a desktop operating system here for comparison purposes, as both smart tvs were running Linux variants. Rather than comparing to off the shelf Linux, we are comparing to a version where a modest hardening effort ($500 paid to a consultant) has been made, as we would expect a tv vendor to spend at least the cost of one tv on hardening their firmware. While neither tv was up to the safety standards of the hardened Linux, Visio had a clear security and safety advantage over LG, which did surprisingly poorly.
Here we consider two versions of Microsoft Office for OS X. How does the newer release compare to the old?
Of the three operating systems reviewed, Windows 10 has the most consistent use of safety features, and is the only one to broadly deploy CFI. Linux is well behind MacOS and Windows when it comes to its use of ASLR. MacOS takes a hit because of its diminishing use of the heap protection flag, and because it lags behind Linux in the use of source fortification.
Directly comparing scores across different operating systems is tricky in this category, because each OS has different safety capabilities and liabilities. Some safety features are only available or applicable in one operating system, which is why some OSes are absent entirely from some sets. Safe SEH and RELRO, for example, are addressing security concerns specific to their operating systems, so they are only applicable for one OS each. Others, like CFI, are newer features that are available in some OSes and not others.
Use of DEP and ASLR was consistent across the board, but Edge far outstripped the other browsers in terms of CFI, with Opera in second place. Firefox's use of stack guards lagged a bit behind the others, and Chrome didn't have as many 64 bit binaries as expected.
Chrome is the only one of the browsers to use the heap protection flag, but it loses that advantage due to the slightly lower values for use of 64 bit binaries, DEP, and ASLR. Firefox does the best in terms of fortification, and Opera does the best for stack guard usage. Safari is the worst in both of these categories, getting it the worst safety feature score overall.
Firefox leads the group, showing strong usage of build safety features.
Both of the smart tvs were entirely 32 bit, but Visio did much better than LG in terms of safety features. Where Visio's scores were comparable to hardened Linux for all categories (aside from being 32 bit), LG had no ASLR, and almost no use of source fortification or stack guards.
In terms of safety features, the 2016 release showed significant improvement over Office 2011.
*The SafeSEH only applies to 32-bit binaries that opt into the SEH compatible flag, thus the number of SafeSEH binaries is a subset of the total binaries in the dataset.
Some functions are historically difficult to use without introducing vulnerabilities. We have broken these into categories, based on how much of a risk they pose. In many cases, newer, safer versions of these functions have been introduced to replace them - these are the "good" category of functions. Windows got the highest code hygiene score because they showed the most frequent use of good functions, and least frequent use of the very risky category. MacOS came in second, and Linux came in third. Instances of extremely risky function use were very rare.
Firefox was the only browser to show significant use of good functions, but looses that advantage and then some by having the most "very risky" functions as well, by a significant margin. Chrome and Opera end up being nearly identical in their ratios, and Safari ends up being the best of the pack.
When looking at code hygiene, all three browsers performed similarly.
Neither tv used many good functions, and both used more extremely risky functions than we generally expect to see in commercial code. LG had the largest ratio of very risky functions.
In terms of code hygiene, Office 2016 saw an increased ratio of "good" functions, resulting in a slightly improved score.
Ratio: Good-Risky Functions
The circular bar charts to the left show how many good functions were present in comparison to the number of risky functions of varying degrees. If more of the circle is brightly colored, then that indicates that a greater part of all of the categorized functions were from the "good" category. Darker shades of grey indicate higher levels of risk.
- Very Risky
- Extremely Risky
Linux links to fewer libraries than either of its competitor OSes, and has a much smaller code base, so it scores much better than either in terms of complexity. Windows has fewer linked libraries than MacOS, but a larger overall code base by far, so it gets the worst complexity score.
On the whole, all three browsers weigh in with similar code sizes.
We expect the desktop operating system to be much more complex than the one on a smart tv, and when it comes to code size, that turns out to be true. When it comes to linked libraries, though, LG uses almost as many libraries as a full deployment of Linux. Visio had the smallest code size and used the fewest libraries.
With the upgrade to Office 2016, one notable change is the increase in code size.
# of Libraries
Some product features aren't currently captured in our scores (usually because we can't automatedly measure them), but could be relevant to decision-making. If a tool or operating system is open source, then it's possible to fix some of its security flaws yourself, and create a customized version for your personal or organizational use (eg, recompile with safer settings than the off the shelf version). If an application engages in sandboxing, then while parts of the software might be vulnerable, attacks on those parts could be contained within a designated "sandbox".
|Is Open Source?|
* A yellow check mark represents partial support. This usually means the software is using an open source library, but has custom additions that are not open source.
I Use Low-Scoring Software
What Should I Do?
If You're An Average User
Switch to a secure choice
See whether a better-scoring option has the features you need.
If You're A Technically Advanced User
Re-Compile the Software
If it's open source, recompile with better safety features. If it's not, contact the vendor and ask them to enable the missing safety features.