“Respect your own attempts to think.”

– Rich Sutton (Deep learning and reinforcement learning summer school 2019)

From a very insightful career advice talk. This statement resonated with me. Scientists tend to be very critical of themselves, and I am not any different. We should take our thinking seriously, write it down regularly and give it time.

“We focus too much on methods and solutions, rather than on problems.”

– Sebastian Thrun

In an interview on Lex Fridman’s podcast Thrun that working on end-to-end problems might bring more progress in practice. Science kind of forces you to decompose a problem on subcomponents and focus on these, and we are rewarded for such work, but from time to time I would think that this makes it harder to see the big picture and how everything can fit together. Hence, I would agree, we should pick out important problems and work out solutions holistically.

“We wanted flying cars, instead we got 140 characters.”

– Peter Thiel

This speaks to the fact that innovation rate actually slowed down in recent decades, even though to an average consumer it doesn’t seem that way. Most innovations these days are very small iterative improvements – just think about new smartphones that come out every year! However, more serious AI might change this situation.

“Computers are useless. They provide only answers.”

– Pablo Picasso

I picked this up in Garry Kasparov’s recent book “Deep Thinking”. Well, Pablo’s statement is actually quite deep, how can computer assess what questions are important to ask?

“Data without a theory are like a baby without a parent: their life expectancy is low.”

– G. Gigerenzer, in Surrogates for theories (1998, p.202)

Nicely summarizes my approach to science, without a theory (a formal one!), it is hard to make sense of data. Gerd goes further and argues that even amazing experimental work will be forgotten if it is not explained with a strong theory.

“The [MAB] problem was formulated during the war, and efforts to solve it so sapped the energies and minds of Allied scientists that the suggestion was made that the problem be dropped over Germany, as the ultimate instrument of intellectual sabotage.”

– P. Whittle (1980)

Bandit problem, looking deceivingly simple – my favourite puzzle and an entry point to the world of exploitation and exploration. These days I see them everywhere.

“The difficulty lies, not in the new ideas, but in escaping from the old ones, which ramify, for those brought up as most of us have been, into every corner of our minds.”

– John Maynard Keynes (1935), The General Theory of Employment, Interest and Money

Aaaand, it’s a trap. I wonder if I will be able to unsee bandit problems when needed!

“What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.”

– Herbert Simon (according to Hal Varian, Scientific American, 1995)

Scarcity of attention. So obvious in today’s world of abundance. Just look at the number of books, shows or music you can choose from. I thank Pantelis for pointing it out.

“Human reasoners are not interested in validity for its own sake. Their goal is to reach conclusions that are true and useful.”

– Philip Johnson-Laird (1993)

I don’t always agree with this, but I find it a nice guide for choosing the topic to work on. Scientists are by necessarily curious creatures and can get interested in any problem, regardless of its usefulness. Though, some caution needs to be exercised, it is also difficult to judge how useful a solution to some obscure problem can be in the long run.

“What is sometimes required is not more data or more refined data but a different conception of the problem.”

– Roger N. Shepard

People who are able to do that make the largest impact in science. Shepard was definitely one of them.

“When we’re learning to see, nobody’s telling us what the right answers are—we just look,” Hinton says. “Every so often, your mother says ‘that’s a dog,’ but that’s very little information. You’d be lucky if you got a few bits of information—even one bit per second—that way. The brain’s visual system requires 1014 [neural] connections. And you only live for 109 seconds. So it’s no use learning one bit per second. You need more like 105 bits per second. And there’s only one place you can get that much information—from the input itself.”

– Geoff Hinton (from Gorder 2006)

Most research about concept formation in cognitive psychology focuses on supervised learning. Comparatively, there is very little on how people learn in unsupervised fashion. I like Geoff Hinton’s argument why unsupervised learning is necessary for explaining human cognition.

“Nature creates curved lines while humans create straight lines.”

– Hideki Yukawa

Sometimes I think what Hideki meant is that we are always wrong in our reasoning, while at other times I think it refers to the nature of our reasoning, trying to find the patterns and simplest explanations.

“I believe there is no philosophical high-road in science, with epistemological signposts. No, we are in a jungle and find our way by trial and error, building our road behind us as we proceed. We do not find signposts at crossroads, but our own scouts erect them, to help the rest.”

– Max Born, Experiment and Theory in Physics (1943), p. 44

So true. Science from the outside looks like a beautiful process, things falling into place neatly. It is rarely that way; most of the time it’s a messy affair.

“If your experiment needs statistics, you ought to have done a better experiment.”

– Ernest Rutherford (?)

Indeed, good experimental design can save you a lot of trouble once you start analysing the data. Zvi Grilliches famously said that a lot of issues in economics could be dealt with by improving the quality of the data. However, what would then econometricians (or statisticians in this case) do?

“Models are like toothbrushes – everyone should have one, but you would never dream of using someone else’s”

– Michael Watkins, Models as toothbrushes (BBS, 1984)

Researchers with affinity towards modeling really do often suffer of this affliction, at least in my research domain. We should invest more energy into evaluating and discarding existing models.