Windows Phone is far harder to use than competing smartphone operating systems, with a sometimes "oppressive" and "forbidding" interface, says a recent usability shoot-out done by a consulting firm. Is the report accurate, or are the testers completely off base?
The Pfeiffer Report's Smartphone User Experience Shootout compared iOS 7, iOS 6, Android, BlackBerry 10, and Windows Phone 8. As fellow Computerworld blogger Jonny Evans reports on his blog, iOS 7 came out on top. (For the full report, get this PDF.)
Windows Phone came out on the bottom, well below the competition. Here are the results --- higher means better:
The shootout compared the OSes using four main benchmarks: cognitive load (how quickly you can get used to the operating system when you first try it); efficiency and integration (how easily you can use all of the OS's main features, such as notifications, multitasking, and using the camera); customization; and user experience friction (how annoying or frustrating the OS is to use).
Windows Phone 8 did well in only a single category: Cognitive load. It was tied with iOS 7 for second place, lagging iOS 6, and well ahead of both BlackBerry and Android. The report notes:
"Windows Phone 8 has a streamlined user interface...but at the cost of eliminating some core user experience and efficiency features."
However, even though it did well, the report also said that in this category, Windows Phone 8 is "more serious looking than Android or iOS, but can also be perceived as more forbidding."
Windows Phone 8 was at the bottom when it comes to efficiency and integration. The report says:
"Windows Phone 8 offers only basic efficiency integration: notification management is limited; multitasking control is very basic, and does not provide any way of selectively quitting running apps, and there is no quick access to key settings."
When it comes to customization, once again Windows Phone was at the bottom, well below the competition with only a 2 out of 10 rating, compared to Android at a 7, iOS7 at a 6, iOS 6 at a 5, and BlackBerry at a 4.
Windows Phone 8 was also well behind the other OSes in user experience friction (UXF) or how annoying or frustrating the OS can be. While the report lauds an operating system it calls "completely different from the iPhone," it concludes:
"Windows Phone 8 also a significant amount of UXF: customization of the user environment is minimal, for instance, making the stylish interface oppressive after a while."
The report concludes that Windows Phone 8:
"Is not very competitive in terms of overall user experience...Microsoft deserves credit for designing a highly original mobile operating system that does not imitate the market leaders and boldly goes its own way. That would be great --- if the OS actually delivered all the user experience aspects that users around the world have come to expect. The truth is that under the slick veneer of swiveling tiles is an operating system that is challenging to use. One can feel trapped in the rigid user interface that leaves no room for customization, and user interface in general is not conceived to deal efficiently with the dozens and dozens of apps smartphone users want. Add to that the lack of some core user interface features and you get an idea of Microsoft's challenges to make Windows Phone truly competitive."
In a word: Ouch!
I don't buy many of the report's conclusions. It's right to laud Microsoft for designing a smartphone OS different than the market leaders. And it's also right that it's extremely easy to get up and running quickly with Windows Phone. But is Windows Phone 8 really oppressive and forbidding? And does it really matter that it's more difficult to customize than the competition?
The answer to all those questions is no. The problem with the report is that it assumes that the old style method of using smartphones is the only proper one: Display screen after screen of apps so that people can find an app that does the work they want to get done and then run that app.
Windows Phone breaks with that convention. Instead, it displays information via its tiles, so that you often don't need to run an app --- just glance at the phone and you get what you need. When you want more details, tap the tile.
Windows Phone certainly has its problems, and I agree that its lack of customization can be frustrating. But this report gets it wrong. If it measured by using real-world tests on how quickly you can get information, it would find Windows Phone a solid performer.