“Data without a theory are like a baby without a parent: their life expectancy is low.”

– G. Gigerenzer, in Surrogates for theories (1998, p.202)

Nicely summarizes my approach to science, without a theory (a formal one!), it is hard to make sense of data. Gerd goes further and argues that even amazing experimental work will be forgotten if it is not explained with a strong theory.

“The [MAB] problem was formulated during the war, and efforts to solve it so sapped the energies and minds of Allied scientists that the suggestion was made that the problem be dropped over Germany, as the ultimate instrument of intellectual sabotage.”

– P. Whittle (1980)

Bandit problem, looking deceivingly simple – my favourite puzzle and an entry point to the world of exploitation and exploration. These days I see them everywhere.

“What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.”

– Herbert Simon (according to Hal Varian, Scientific American, 1995)

Scarcity of attention. So obvious in today’s world of abundance. Just look at the number of books, shows or music you can choose from. I thank Pantelis for pointing it out.

“Human reasoners are not interested in validity for its own sake. Their goal is to reach conclusions that are true and useful.”

– Philip Johnson-Laird (1993)

I don’t always agree with this, but I find it a nice guide for choosing the topic to work on. Scientists are by necessarily curious creatures and can get interested in any problem, regardless of its usefulness. Though, some caution needs to be exercised, it is also difficult to judge how useful a solution to some obscure problem can be in the long run.

“What is sometimes required is not more data or more refined data but a different conception of the problem.”

– Roger N. Shepard

People who are able to do that make the largest impact in science. Shepard was definitely one of them.

“When we’re learning to see, nobody’s telling us what the right answers are—we just look,” Hinton says. “Every so often, your mother says ‘that’s a dog,’ but that’s very little information. You’d be lucky if you got a few bits of information—even one bit per second—that way. The brain’s visual system requires 1014 [neural] connections. And you only live for 109 seconds. So it’s no use learning one bit per second. You need more like 105 bits per second. And there’s only one place you can get that much information—from the input itself.”

– Geoff Hinton (from Gorder 2006)

Most research about concept formation in cognitive psychology focuses on supervised learning. Comparatively, there is very little on how people learn in unsupervised fashion. I like Geoff Hinton’s argument why unsupervised learning is necessary for explaining human cognition.

“Nature creates curved lines while humans create straight lines.”

– Hideki Yukawa

Sometimes I think what Hideki meant is that we are always wrong in our reasoning, while at other times I think it refers to the nature of our reasoning, trying to find the patterns and simplest explanations.

“I believe there is no philosophical high-road in science, with epistemological signposts. No, we are in a jungle and find our way by trial and error, building our road behind us as we proceed. We do not find signposts at crossroads, but our own scouts erect them, to help the rest.”

– Max Born, Experiment and Theory in Physics (1943), p. 44

So true. Science from the outside looks like a beautiful process, things falling into place neatly. It is rarely that way; most of the time it’s a messy affair.

“If your experiment needs statistics, you ought to have done a better experiment.”

– Ernest Rutherford (?)

Indeed, good experimental design can save you a lot of trouble once you start analysing the data. Zvi Grilliches famously said that a lot of issues in economics could be dealt with by improving the quality of the data. However, what would then econometricians (or statisticians in this case) do?

“Models are like toothbrushes – everyone should have one, but you would never dream of using someone else’s”

– Michael Watkins, Models as toothbrushes (BBS, 1984)

Researchers with affinity towards modeling really do often suffer of this affliction, at least in my research domain. We should invest more energy into evaluating and discarding existing models.