Similarly, physical theories differ in the basic assumptions they make about observation and reality. If we are not careful to spell them out, confusion can and will occur when we try to compare descriptions of the world that come out of different theories.
In this book we shall be concerned with two very basic ways in which theories may differ. The first is in the answer they give to the question of what space and time are. Newton’s theory was based on one answer to this question, general relativity on quite another. We shall see shortly what these were, but the important fact is that Einstein altered forever our understanding of space and time.
Another way in which theories may differ is in how observers are believed to be related to the system they observe. There must be some relationship, otherwise the observers would not even be aware of the existence of the system. But different theories can and do differ strongly in the assumptions they make about the relationship between observer and observed. In particular, quantum theory makes radically different assumptions from those made by Newton about this question.
The problem is that while quantum theory changed radically the assumptions about the relationship between the observer and the observed, it accepted without alteration Newton’s old answer to the question of what space and time are. Just the opposite happened with Einstein’s general relativity theory, in which the concept of space and time was radically changed, while Newton’s view of the relationship between observer and observed was retained. Each theory seems to be at least partly true, yet each retains assumptions from the old physics that the other contradicts.
Relativity and quantum theory were therefore just the first steps in a revolution that now, a century later, remains unfinished. To complete the revolution, we must find a single theory that brings together the insights gained from relativity and quantum theory. This new theory must somehow merge the new conception of space and time Einstein introduced with the new conception of the relationship between the observer and the observed which the quantum theory teaches
us. If that does not prove possible, it must reject both and find new answers to the questions of what space and time are and what the relationship between observer and observed is.
The new theory is not yet complete, but it already has a name: it is called the quantum theory of gravity. This is because a key part of it involves extending the quantum theory, which is the basis of our understanding of atoms and the elementary particles, to a theory of gravity. Gravity is presently understood in the context of general relativity, which teaches us that gravity is actually a manifestation of the structure of space and time. This was Einstein’s most surprising and most beautiful insight, and we shall have a great deal to say about it as we go along. The problem we now face is (in the jargon of fundamental physics) to unify Einstein’s theory of general relativity with the quantum theory. The product of this unification will be a quantum theory of gravity.
When we have it, the quantum theory of gravity will provide new answers to the questions of what space and time are. But that is not all. The quantum theory of gravity will also have to be a theory of matter. It will have to contain all the insights gained over the last century into the elementary particles and the forces that govern them. It must also be a theory of cosmology. It will, when we have it, answer what now seem very mysterious questions about the origin of the universe, such as whether the big bang was the first moment of time or only a transition from a different world that existed previously. It may even help us to answer the question of whether the universe was fated to contain life, or whether our own existence is merely the consequence of a lucky accident.
As we enter the twenty-first century, there is no more challenging problem in science than the completion of this theory. You may wonder, as many have, whether it is too hard - whether it will remain always unsolved, in the class of impossible problems like certain mathematical problems or the nature of consciousness. It would not be surprising if, once you see the scope of the problem, you were to take this view. Many good physicists have. Twenty-five years ago, when I began to work on the quantum theory of gravity in
college, several of my teachers told me that only fools worked on this problem. At that time very few people worked seriously on quantum gravity. I don’t know if they ever all got together for a dinner party, but they might have.
My advisor in graduate school, Sidney Coleman, tried to talk me into doing something else. When I persisted he told me he would give me a year to get started and that if, as he expected, I made no progress, he would assign me a more doable project in elementary particle physics. Then he did me a great favour: he asked one of the pioneers of the subject, Stanley Deser, to look after me and share my supervision. Deser had recently been one of the inventors of a new theory of gravity called supergravity, which for a few years seemed to solve many of the problems that had resisted all earlier attempts to solve them. I was also lucky during my first year at graduate school to hear lectures by someone else who had made an important contribution to the search for quantum gravity: Gerard ’t Hooft. If I have not always followed either of their directions, I learned a crucial lesson from the example of their work - that it is possible to make progress on a seemingly impossible problem if one just ignores the sceptics and gets on with it. After all, atoms do fall, so the relationship between gravity and the quantum is not a problem for nature. If it is a problem for us it must be because somewhere in our thinking there is at least one, and possibly several, wrong assumptions. At the very least, these assumptions involve our concept of space and time and the connection between the observer and the observed.
It was obvious to me then that before we could find the quantum theory of gravity we first had to isolate these wrong assumptions. This made it possible to push ahead for there is an obvious strategy for rooting out false assumptions: try to construct the theory, and see where it fails. Since all the avenues that had been followed up to that time had, sooner or later, led to a dead end, there was ample work to do. It may not have inspired many people, but it was necessary work and, for a time, it was enough.
The situation now is very different. We are still not quite there, but few who work in the field doubt that we have come
a long way towards our goal. The reason is that, beginning in the mid-1980s, we began to find ways of combining quantum theory and relativity that did not fail, as all previous attempts had. As a result, it is possible to say that in the last few years large parts of the puzzle have been solved.
One consequence of our having made progress is that all of a sudden our pursuit has become fashionable. The small number of pioneers who were working on the subject a few decades ago have now grown into a large community of hundreds of people who work full time on some aspect of the problem of quantum gravity. There are, indeed, so many of us that, like the jealous primates we are, we have splintered into different communities pursuing different approaches. These go under different names, such as strings, loops, twistors, non-commutative geometry and topi. This over-specialization has had unfortunate effects. In each community there are people who are sure that their approach is the only key to the problem. Sadly, most of them do not understand in any detail the main results that excite the people working on the other approaches. There are even cases in which someone taking one approach does not seem to realize that a problem they find hard has been completely solved by someone taking another approach. One consequence of this is that many people who work on some aspect of quantum gravity do not have a view of the field that is wide enough to take in all the progress that has recently been made towards its solution.
This is perhaps not so surprising - it seems not very different from the present state of cancer research or evolutionary theory. Because the problem is hard, it might be expected that, like climbers confronting a virgin peak, different people would attempt different approaches. Of course, some of these approaches will turn out to be total failures. But, at least in the case of quantum gravity, several approaches seem recently to have led to genuine discoveries about the nature of space and time.
The most compelling developments, taking place as I write, have to do with bringing together the different lessons that have been learned by following the different approaches, so
that they can be incorporated into a single theory - the quantum theory of gravity. Although we do not yet have this single theory in its final form, we do know a lot about it, and this is the basis of what I shall be describing in the chapters to come.
I should warn the reader that I am by temperament a very optimistic person. My own view is that we are only a few years away from having the complete quantum theory of gravity, but I do have friends and colleagues who are more cautious. So I want to emphasize that what follows is a personal view, one that not every scientist or mathematician working on the problem of quantum gravity will endorse. I should also add that there are a few mysteries that have yet to be solved. The final stone that finishes the arch has yet to be found.
Furthermore, I must emphasize that so far it has not been possible to test any of our new theories of quantum gravity experimentally. Until very recently it was even believed that the quantum theory of gravity could not be tested with existing technology, and that it would therefore be many years into the future before the theory could be confronted with data from experimental science. However, it now appears that this pessimism may have been short-sighted. Philosophers of science such as Paul Feyerabend have stressed that new theories often suggest new kinds of experiment which may be used to test them. This is very definitely happening in quantum gravity. Very recently, new experiments have been proposed which it appears will make it possible to test at least some of the theory’s predictions in the very near future. These new experiments will employ existing technology, but used in surprising ways, to study phenomena that would not have been thought, on the basis of the old theories, to have anything to do with quantum gravity. This is indeed a sign of real progress. However, we must never forget that until the experiments are performed it will always be possible that, as beautiful and compelling as the new theories may seem, they are simply wrong.
During the past few years there has been a growing sense of excitement and confidence among many of the people working
on quantum gravity. It is hard to avoid the feeling that we are indeed closing in on the beast. We may not have it in our net, but it feels as if we have it cornered and we have seen, with our flashlights, a few glimpses of it.
Among the many different paths to quantum gravity, most recent traffic, and most progress, has been along three broad roads. Given that quantum gravity is supposed to arise from a unification of two theories - relativity and quantum theory - two of these paths are perhaps not unexpected. There is the route from quantum theory, in which most of the ideas and methods used were developed first in other parts of quantum theory. Then there is the road from relativity, along which one starts with the essential principles of Einstein’s theory of general relativity and seeks to modify them to include quantum phenomena. These two roads have each led to a well worked-out and partly successful theory of quantum gravity. The first road gave birth to string theory, while the second led to a seemingly different theory (although with a similar name) called loop quantum gravity.
Both loop quantum gravity and string theory agree on some of the basics. They agree that there is a physical scale on which the nature of space and time is very different from that which we observe. This scale is extremely small, far out of the reach of experiments done with even the largest particle accelerators. It may in fact be very much smaller than we have so far probed. It is usually thought to be as much as 20 orders of magnitude (i.e. a factor of 10
20
) smaller than an atomic nucleus. However, we are not really sure at which point it is reached, and recently there have been some very imaginative suggestions that, if they bear fruit, will bring quantum gravity effects within the range of present-day experimental capabilities.
The scale where quantum gravity is necessary to describe space and time is called the Planck scale. Both string theory and loop quantum gravity are theories about what space and time are like on this tiny scale. One of the stories I shall be telling is how the pictures that each theory gives us are converging. Not everyone yet agrees, but there is more and
more evidence that these different approaches are different windows into the same very tiny world.
Having said this, I should confess my own situation and bias. I was one of the first people to work on loop quantum gravity. The most exhilarating days of my life (apart from the purely personal) were those when, all of a sudden, after months of hard work, we suddenly understood one of our theory’s basic lessons. The friends I did that with are friends for life, and I feel equal affection and hope for the discoveries we made. But before then I worked on string theory and, for the past four years, most of my work has been in the very fertile domain that lies between the two theories. I believe that the essential results of both string theory and loop quantum gravity are true, and the picture of the world I shall be presenting here is one that comes from taking both seriously.
Apart from string theory and loop quantum gravity, there has always been a third road. This has been taken by people who discarded both relativity and quantum theory as being too flawed and incomplete to be proper starting points. Instead, these people wrestle with the fundamental principles and attempt to fashion the new theory directly from them. While they make reference to the older theories, these people are not afraid to invent whole new conceptual worlds and mathematical formalisms. Thus, unlike the other two paths, which are trodden by communities of people each large enough to exhibit the full spectrum of human group behaviour, this third path is followed by just a few individuals, each pursuing his or her own vision, each either a prophet or a fool, who prefers that essential uncertainty to the comfort of travelling with a crowd of like-minded seekers.