Also in the last ACM interaction issue, Donald Norman is conscientiously shifting from his past stance ("study first, design second") to a pragmatic take: "for many projects the order is design, then study". And this, for different reasons:

Once a project is announced, it is too late to study what it should be—that's what the announcement was about. If you want to do creative study, you have to do it before the project's launch. (...) Most projects are enhancements of preexisting projects. Why do we have to start studying the users all over again? Haven't we already learned a lot about them? (...)

Our continual plea for up-front user studies, field observations, and the discovery of true user needs is a step backwards: This is a linear, inflexible process inserted prior to the design and coding stages. We are advocating a waterfall method for us, even as we deny it for others. Yes, folks. By saying we need time to do field studies, observations, rapid paper prototypes and the like, we are contradicting the very methods that we claim to be promoting.

So what's the point?

Field studies, user observations, contextual analyses, and all procedures that aim to determine true human needs are still just as important as ever—but they should all be done outside of the product process. This is the information needed to determine what product to build, which projects to fund.

Usability testing is like Beta testing of software. It should never be used to determine "what users need." It is for catching bugs, and so this kind of usability testing still fits the new, iterative programming models, just as Beta testing for software bugs fits the models. (...) UI and Beta testing are meant simply to find bugs, not to redesign.

So let's separate the field and observational studies, the conceptual design work, and the needs analyses from the actual product project. We need to discover what users need before the project starts; for once started, the direction has already been determined. We need to embrace rapid, iterative methods.

Why do I blog this? This is of interest to me because I faced similar issues when working with game designers: articulating the field studies / usability tests with game design process is often tough and what Norman is describing should be taken into account. Besides, to certain people there is often confusion between usability test/field studies and it's pertinent to see how Norman clarifies that.