Rapid Feedback

In his talk Inventing on Principle, Bret Victor sets the scene by explaining his own guiding principle, that "creators need an immediate connection with what they create." The interactive demonstrations in his talk are very impressive. You look at them and think "Why isn't my interaction with the computer like that? Why am I still struggling, still working by remote control?" The difference between his demos and the way we usually work is as dramatic as the difference between batch-processing and time-sharing.

"When you make a decision," says Victor, "you need to see the effect of that immediately — there can't be any delay, can't be anything hidden." The demos are lovely, and it's certainly nice to get rapid feedback, but "need"? Do we really need it? We can find a pretty clear answer to that question if we turn around and consult the paper Conditions for Intuitive Expertise: A Failure to Disagree by psychologists Daniel Kahneman and Gary Klein (American Psychologist, Vol 64(6), Sep 2009, pp515-526).

For many years, Kahneman and Klein have been the central figures in two opposed camps of psychologists studying human expertise. Kahneman, in the "heuristics and biases" camp, enjoyed demonstrating that human intuition is often flawed and that experts can often be overconfident about their decisions. Klein, in the "naturalistic decision making" camp, enjoyed celebrating the uncanny ability of experts to make good decisions in complex situations and under time pressure.

For example, Klein found from many interviews with firefighters that rather than thinking of a number of options (or even just two) and deciding between them, they usually only thought of one option — the right one — and then quickly checked that it was appropriate for the current situation. Their intuition really works. On the other hand, "experts" in other fields don't seem so capable. Unfortunately, their intuitions are still compelling and feel to them just as just as correct as the firefighters' — but this feeling is deceptive. For example, in Moneyball, Michael Lewis describes how baseball scouts fail time and again to choose the most effective players because they fall for the "representativeness" heuristic: they pick good lookers, rather than good performers.

Was it possible to reconcile these two conflicting points of view? Kahneman and Klein decided to try settle their differences by writing a paper together. Both camps were clearly right some of the time — experts are of course sometimes accurate and sometimes wide of the mark. But was there a pattern? "What are the activities," they asked, "in which skilled intuitive judgement develops with experience? And what are the activities in which experience is more likely to produce overconfidence than genuine skill?"

In the end, the two psychologists demonstrated "a failure to disagree". Reconciling the evidence from each camp, they describe in their paper the precise conditions for skilled intuition to develop. Firstly, there must be a "high-validity" environment, or in other words there must be cues in the observable environment which indicate the true nature of the current situation. For example, firefighters and paediatric nurses inhabit high-validity environments, because they can perceive early indications that a building will collapse or that a baby will suffer from an infection. They will often be unaware of the precise cues which guide their intuitive decisions, but the cues are there all the same.

Secondly, people must have an opportunity to learn the cues and to get feedback when they are right and when they are wrong. But humans are pretty good at this, provided they are in a high-validity environment. As Kahneman and Klein note: "Where simple and valid cues exist, humans will find them if they are given sufficient experience and enough rapid feedback to do so". But the key for us here is the feedback: to build skilled intuition this must be "both rapid and unequivocal".

So, let's now come back to Brett Victor and his principle of rapid feedback. Do we need it? Computer programming certainly forms a high-validity environment. We should be able to become intuitive experts, but as you can now see, that depends critically on whether the feedback we get is "both rapid and unequivocal". In most programming environments nowadays the feedback is neither.

(I leave it as an exercise for the reader to work out whether a kitchen is a high-validity environment. The point Victor was making about rapid feedback seems to have been missed by so many people that he felt it necessary to construct a web-page explaining it all over again. That account is certainly worth reading too, and you can find it here: Learnable Programming.)