Making child's play of power tools

Historically, knowledge and skills start in the hands of an elite few and gradually disperse outward through the population at large. There was a time when the process took centuries or millennia to unfold; nowadays it usually occurs in only years. Multiplication and division, for long stretches of relatively recent history, were university graduate skills and the computation of square roots only for the cognoscenti. The rate of progress is technology dependent and socially directed. Writing, printing, tables of logarithms, mass education in industrialised societies, the slide rule, the electronic pocket calculator, have moved the calculation of a square root from a difficult accomplishment through to a trivial black box action by anyone who can press a key. The spreadsheet, at the cost of a few additional key presses and a little extra understanding, applied to that same black box to potentially infinite data ranges.

What has all this to do with an overview of data analysis in 2007? Bear with me a while.

That example of a square root is not chosen lightly: the conceptual acceptance of a square root as a world view component is a marker in the specific strand of cognitive development upon which a statistical view of data depends. This is true whether one can compute it or not, but internalising acceptance of something which one cannot do or use is difficult.

The movement is not only downward. Methods that allow black box handling of complex ideas and techniques also allow those at the leading developmental edge to collapse time-consuming tasks into automated tools that either directly facilitate the growth of higher level methods – or at least clear the decks and free up resources for their pursuit.

Sophisticated data analysis has, for some time, been moving away from statisticians into wider domains: first to become a routine tool of all scientists, then to widespread use in business, and finally a black box tool for general use. Although the foundations lie in agriculture and operational research predating electronics, the technology which has driven this and decided its pace is of course the spread of increasing computing power at falling cost. [read more...]