Open any physics textbook and you will learn that gravity is a force of attraction between two objects that depends on their individual masses and the distance between them. The description will be clear (if it's a good textbook) and precise. It will be in terms of the gravitational field and field lines. The math will be pretty simple, directed through the Newtonian formula, F_gravity = GMm∕r². The tone will be confident, as is the style of textbooks. We know what gravity is, and what it isn't. There will be no talk of an object's proper place in the universe or its goal of getting there. There will be no references to crystalline spheres in the astronomy section, since we now know it's simply this force of gravity that holds a planet in orbit.

There will be no worries in the textbooks about, as Ernst Mach put it, the “uncommon unintelligibility” (Mach, Reference Mach1911, p. 56) of an invisible field that can exert a force instantaneously, at great distance, between any two objects. Whether or not it has become intelligible, this idea has become common knowledge. The Newtonian theory is the truth about gravity taught to physics students throughout their undergraduate education. It's not until graduate school that the more modern theory, the general theory of relativity, is introduced as the proper replacement for Newton. And even then, a class in general relativity is usually optional, not a requirement for the graduate degree. You can get a degree in physics, surely a bachelor's degree and maybe even a doctorate, with a Newtonian understanding of gravity.

Often the textbooks will hint that there are troubles for the Newtonian theory, and that, strictly speaking, it's not the last word on gravity. But it's surely good enough. It's good enough for most work in physics. It's good enough for all work by engineers. And it's good enough for rocket science.

Good enough means that if you use the Newtonian theory of gravity to figure out how to launch and aim a rocket to get to Mars, the rocket will get to Mars. You can even land it safely on Mars, again by using the Newtonian theory to design and deploy the steering and braking mechanisms. This sort of successful application makes it seem as if the theory is true. Even if it is not spot-on true, it is at least nearly true. It's true enough.

A word of caution is in order. It is certainly possible for a scientific theory to facilitate accurate applications even when the theory is dramatically false. Consider the caloric theory of heat. If you had opened a general physics textbook, or a more specific thermodynamics textbook, in the eighteenth and early nineteenth centuries, you would have learned that heat is an invisible substance, a fluid that flows from hot objects to cold. This accurately predicted that most things will expand as they get hotter, to make room for the extra stuff, the caloric fluid. It was also the theoretical basics for the industrial revolution, the invention and development of steam engines and other practical ways of using heat to get work done. But the theory was, and is, false. Really false. It's based on a fundamental fiction. There is no such thing as a caloric fluid.

We worked with the difference between a theory being true and its being useful in Chapter 4. That was about the Ptolemaic model of planetary orbits, the one with epicycles and void points. The suggestion was that practical applications and success with the evidence are all we should really care about in a scientific theory. The theory being true or not is irrelevant. This attitude toward theories is called instrumentalism, a sort of shut up and calculate response to the question about truth.

Instrumentalism may get you through the work day as an engineer or a scientist, but it conflicts with some inherent human curiosity, the one that wants to know what's really going on in nature. Besides, the textbooks don't say it's as if there is a force field; they say there is a force field. Instrumentalism is dissatisfying because it denies the natural inference that a theory works because it's true. That is, the success with evidence and applications indicates that the theory is basically right. How could it work if it were false? Germ theory is fantastically successful in keeping us healthy, and that's because there are, in fact, germs.

So, what, if anything, is the connection between the utility and the truth of a theory? On the one hand, theories can work really well even if they are false. On the other hand, it seems like a theory couldn't work as well as Newtonian gravity without being true, or nearly true. To sort this out, we need to look carefully at the reasons one might think the theory is in fact true. There are two kinds of things to look at, empirical testing (obviously), and the internal, conceptual workings of the theory itself. This second aspect of evaluating a theory asks about things like the simplicity and theoretical coherence of the ideas. Maybe the indication of truth is in the combination of these things. A true theory is one that works with the evidence and is as simple and coherent as possible. It has to both make sense of the observations and make sense itself.

Does the Newtonian theory of gravity make sense? Is it intrinsically plausible in how it describes the mechanism of gravity? Despite our comfort with the idea of a gravitational field, the initial reception of Newtonian gravity faced considerable conceptual challenge. The problem was the action at a distance. The force between two objects requires no contact or substantive medium. And it happens instantaneously. We live with this by letting the field do the work. This may be just hiding one mystery behind another. The gravitational field is a dynamic player in the gravitational interaction. There is nothing else there between the two. The field is the thing itself that gets the job done.

This action without contact used to be regarded as not only mysterious but mystical. When Kepler suggested that the Moon was the cause of the ocean tides on the Earth, Galileo dismissed this no-contact interaction as occult and argued for his own, local cause. But then Newton allowed the distant force suggested by Kepler, and made it universal gravitation. There was still some objection at the time. For example,

This is, remarkably, from a letter that Newton himself wrote to a colleague, Richard Bentley. The mechanism of universal gravitation is an “absurdity.” But it works. And we have come to overlook the absurdity. That's how Mach saw it in Reference Mach1911.

Mach's point is that the theory still doesn't make sense, but we have gotten used to it. Complacency is no indication of truth. If we are to regard the theory as true, it will have to be despite its not making sense. This will require some pretty robust evidence. The theory will have to pass some strict tests.

In the case of a theory of gravity, and specifically the Newtonian theory of gravity, the testing will have to be indirect. As we realized in Chapter 1, neither the force of gravity nor the field is something we can directly observe. So testing the theory will have to be done by observing something else, something that is logically related to the theoretical ideas about forces and fields. We should further clarify this logical relation, something we started in Chapter 6, before considering the evidence, the observations.

It is often said that a scientific theory is tested by its observable predictions. The evidence is in seeing whether the predictions turn out to be true. There is also favorable evidence if a theory can explain what has already been observed. The logic is the same. In both cases, prediction and explanation, it's an if–then relation between theory and evidence. It's always: If the theory is true, then a specific observation is expected. To illustrate with a simple medical example, if malaria is a mosquito-borne parasite, then we predict that preventing mosquito bites will prevent the disease. And if malaria is a mosquito-borne parasite, then that explains why the disease is most common in warm, wet climates. Explanation or prediction, theory and evidence always meet in an if–then relation. Theory is the if; evidence is the then.

Consider the phenomena that the Newtonian theory of gravity explains. One of Newton's great mathematical achievements was to prove that if the gravitational attraction between two bodies depends on the inverse-square of the distance between them, the 1∕r² in the formula, then Kepler's three laws of planetary motion followed. The theory explains Kepler's laws, and the laws themselves are summaries of observations, the regularities in the orbits. The Newtonian theory also explains the ocean tides, again by mathematical derivation of the form, if the theory is true, then the tides as we observe them follow. And the theory explains what is observed in the phenomenon of free-fall, specifically the time-square law of Galileo and the fact that heavy and light objects fall at the same rate.

In all cases of explanation, the argument has two premises.

1. (1) If the theory is true, then the phenomenon will be observed.

2. (2) The phenomenon has been observed.

Here's where we have to be careful, because to draw the conclusion that

1. (3) the theory is true,

would be to commit a logical fallacy.

Logicians call this the fallacy of affirming the consequent. The mistake is obvious when you apply the same form of reasoning to diagnosing disease. Think of a case in which the patient has a fever. That's all we know at this point, the fever. A doctor, a bad doctor, may reason that malaria would explain the fever, so the truth of the matter is the patient has malaria. It's true that malaria explains the fever. If the patient has malaria, then they will have a fever. And it's true the patient has a fever. But those two premises don't prove that it's malaria. There are other diseases that present with a fever, flu, for example, or Ebola. Any of these explains the symptoms, and any of these fits the premise, if this theory is true, then we will observe these symptoms. Fever underdetermines malaria, and in general a symptom underdetermines a diagnosis. To conclude that the malaria theory is true, on the basis of its successfully explaining the fever, is to commit the fallacy of affirming the consequent. It would be the same fallacy if we conclude that a theory of gravity is true, on the basis of its explaining the tides. There are plenty of alternative explanations out there.

So, the doctor, this time a good doctor, continues. What are other symptoms of malaria? If it's malaria, expect recurring episodes of shaking chills and profuse sweating. That's the prediction. And now the doctor is testing the malaria theory, not by shaping the theory to accommodate the evidence in hand, the fever, but by seeing if the theory matches evidence yet to be gathered.

Suppose the prediction turns out to be true? The patient is subsequently observed to have the episodic chills and sweating. What conclusion follows? Strictly speaking, no conclusion follows. To conclude that the malaria theory is true would be to commit exactly the same logical fallacy, affirming the consequent. The logic is the same as in the case of explanation.

1. (1) If the theory is true, then the phenomenon will be observed.

2. (2) The phenomenon is observed.

3. (3) Therefore, the theory is true?

No. That's the same fallacy.

But suppose the prediction turns out to be false? Is it possible to disprove the theory? This is logically more promising, since the argument in this case is not a fallacy. It's a valid argument. The form is called modus tollens.

1. (1) If the theory is true, then the phenomenon will be observed.

2. (2) The phenomenon is not observed.

3. (3) Therefore, the theory is false.

If the patient does not have the predicted symptoms, at least we can rule out malaria. And this is what doctors do, they zero in on what the disease is by ruling out what it isn't.

Testing a theory of gravity will have the same logic as diagnosing a disease, so we should look at the predictions of the Newtonian theory and see what conclusions can be drawn.

The history of science, as usual, offers some cautionary examples against over-confidence. This time it's about over-confidence in the decisive logic of disproof. Tycho, recall, reasoned that if the Earth moves, orbits the Sun, then the angular positions of stars will change over the course of a year. No such stellar parallax was observed. He concluded that the moving-Earth theory was false. But the Earth does orbit the Sun. Similarly, Aristotle reasoned that if the Earth rotates on its axis, a dropped object would fall not straight down but some distance to the west, left behind by the moving Earth. Dropped objects are observed to fall straight down, so Aristotle concluded that the Earth is stationary. But the theory, the one that says the Earth rotates, is true, despite the false prediction.

So, the logic can't be as tidy as the malarial modus tollens. To understand what's going on in scientific testing, and to see how Newton's theory was tested, we'll look at two important cases, the discovery of the planet Neptune, and the non-discovery of the planet Vulcan. One was a great success for Newtonian theory, the other a failure.

Start with the success story, Neptune. Neptune is the eighth planet, both in order of discovery and in distance from the Sun. It was discovered in 1846, and the story is interesting on its own, for its quirky characters, missed opportunities, and a dramatic climax. But we are most interested in the details of the case to clarify important aspects of scientific testing. In particular, we will add to the minimal logic developed with the medical example, and point out a mutual influence between theory and observation.

The discovery of Neptune will also help clarify the distinction between the truth of a scientific claim and its justification. The limitations of science are often summarized by saying that we can never arrive at the truth (or the absolute truth, or the Truth), because we can never be certain that what we say now won't be disproved later. But this confuses the truth of a claim with our ability to demonstrate its truth. Each of our theories is, in fact, true or false, independent of our abilities to demonstrate which it is. The truth of a theory neither comes in degrees nor changes over time (unless nature itself changes). Justification, the reasons to believe that a theory is true, does come in degrees, and it changes as more evidence is available. It's not that we can never arrive at the truth, but that we can never arrive with certainty. This distinction will be clear in the case of Neptune.

No theory is tested on its own, one-on-one with the evidence. There is always a broader theoretical context to connect what you plan to test, the hypothesis, with observable expectations and results. In the case of Newton's theory of gravity, the companion laws of mechanics, the F = ma$F=ma$, link the dynamics of gravity to the observable kinematics of planetary motion. The entire theoretical package can be tested in a ready-made laboratory, the Solar system.

Together, Newtonian mechanics and Newtonian gravity are beautifully coherent. They were also remarkably successful in tests against observations of the planets. Six planets were known at the time Newton published his theory. Predictions of where each planet would move under the gravitational influence of the Sun and the other planets was not a trivial calculation. It still isn't. The equations can be solved exactly only if there are only two objects present. If it were just the Sun and the Earth, the equations of motion could be solved with no trouble. But with three or more objects, there is no exact solution. The procedure is to solve the two-body problem, and then add in the perturbing effects of the other bodies. It's tedious, but ultimately precise and reliable.

Things went very well for Newton's theories, making mathematical predictions of where the planets would be, predictions that matched subsequent observations. This, of course, doesn't prove the theory is true, but it seems some good reason to think so.

The careful observations of planets paid off in other ways. In 1772, for example, the German astronomer Johann Bode revealed a mathematical regularity in the planetary distances from the Sun. Using the Earth–Sun distance as one unit, an Astronomical Unit (AU), the observed positions of all the planets known at the time are shown on Table 8.1.

Bode borrowed an idea from Johann Titus and demonstrated a pattern in these numbers. Start with the sequence of numbers 0, 3, 6, 12, 24, 48, 96,…Except for the 0 and the 3, each number is double its predecessor. This is the regularity. Now add 4 to each number to get 4, 7, 10, 16, 28, 52, 100, 196, and so on. Divide each number by 10. The result is 0.4, 0.7, 1.0, 1.6, 2.8, 5.2, 10.0, 19.6. Except for the missing correspondent at 2.8, these numbers of Bode's law pretty well match the radial positions of the planets. The comparison is in Table 8.2.

Bode's law does not in any way explain why the planets are at these locations, but the law does make predictions. There is the gap at position 2.8. In 1801, the first of the so-called minor planets was discovered, and over time the asteroid belt filled in the space between 2.3 and 3.3 AU. Furthermore, Bode's law predicts that if there is a planet beyond Saturn, it will be roughly 19.6 AU from the Sun. In 1781, even before the gap at 2.8 was filled in, William Herschel discovered the seventh planet, Uranus, at radius 19.2 AU.

It is noteworthy that a survey of astronomical charts made prior to the discovery of Uranus showed that the planet had been observed and recorded several times, at least as early as the first Astronomer Royal, John Flamsteed, in 1690. It had been observed, but never observed as a planet. Seeing something as a planet rather than a star requires either seeing it extended as a disk rather than a point of light, or having credible measurements of its movement relative to the fixed stars. Observation in astronomy, like observation in all of science, is more than mere physical sensation. It all depends on what you make of it.

By 1821, the orbital positions of Uranus were tabulated with sufficient precision to show an undeniable disagreement between the actual orbit and that predicted by the Newtonian laws. By 1840, the disagreement had grown to an enormous 2ʹ of arc, making it impossible to blame the tools or methods of observation. Various explanations were proposed, from the incidental – a comet had struck the planet and knocked it out of its Newtonian orbit – to the profound – that Newton's law of gravitation was flawed. But the judgment of the scientific establishment, for which there is no better authority than Sir George Airy, the Astronomer Royal from 1835 to 1881, was that everyone was “fully impressed with the universality of [Newton's] law of gravitation.” (Quoted in Moore, Reference Moore1996, p. 94.) This suggests that Newton's theory was no longer being tested; it was being used to discover new things about the Solar system. And the most likely explanation to the aberrant motion of Uranus was the effect of an exterior planet that was as yet unseen.

Sir George was not particularly anxious to find the hypothesized eighth planet. In fact, he discounted the chances of ever seeing it, saying, of the perturbation in Uranus’ orbit, “If it be the effect of any unseen body, it will be nearly impossible ever to find out its place.” (Quoted in Moore, Reference Moore1996, p. 97.) This attitude was characteristic of Airy, a man obsessed with order and reluctant to have his tidy understanding of things, either the law of gravitation or the population of the Solar system, changed. His way of dealing with anomalies in the scheme of things was to ignore them. But then this is a man who is said to have spent the better part of a day in the basement of his observatory labeling empty boxes with the word “empty” so there would be no mistaking them.

Airy's disregard for the challenge of locating the unseen body was not shared throughout the community of astronomers. The first precise prediction of where to find the planet was in 1843 by a young English mathematician, John Couch Adams. Adams used Bode's law to estimate the planet's radial position. He zeroed in on its angular motion by doing a kind of reverse-perturbations analysis on the motion of Uranus. He sent his calculations and his results to Airy, hoping the Royal Astronomical Society would undertake to look for the planet where he told them it would be.

Airy received the calculations with dismissive skepticism. Added to his reluctance to disturb the model of the Solar system as he knew it was a general disrespect for youth. Adams, just one year beyond his graduation from Cambridge, was unlikely to have discovered a planet by pen and paper alone. Nonetheless, Airy responded by sending Adams a kind of test question, perhaps intended to gauge the young man's credibility. The question involved the deviation of Uranus from its Bode's law position of 19.6 AU, and the explanation of that deviation in terms of the hypothesized eighth planet. Adams apparently thought the question to be silly, its answer already implicit in the calculations he had sent to Airy. He never responded.

Meanwhile, others were at work on the mathematical challenge of predicting where to find the new planet, among them a French chemist-turned-mathematician, Urbain Jean-Joseph Le Verrier. Using a mathematical approach somewhat different than Adams’, he arrived at a similar result in 1846. Le Verrier sent his work, including the specifications on where and when the new planet could be found, to Airy. In return, he received the same question regarding Uranus’ aberration from Bode's law. Le Verrier responded. Airy now began to take the predictions of an eighth planet seriously enough to direct James Challis to have a look, using the Cambridge Observatory's 30 cm telescope.

The English astronomers had two distinct disadvantages in their search for the hypothetical planet. For one, their star charts were not the most complete, thus making it difficult to identify a novelty. For two, they were generally skeptical of the theoretical work that predicted the location. Indeed, British scientists and philosophers had a tradition of empiricism, from John Locke to Newton to David Hume. Observation is supposed to tell us what to think, not the other way around. And certainly astronomy had always been an empirical science in which observation preceded theory. So, with little respect for the precision of Adams’ and Le Verrier's prediction, Challis adopted a strategy of sweeping his telescope back and forth to systematically scan the general area rather than zero in immediately on the specified spot. Weeks of searching in this patterned way failed to find the planet.

Le Verrier, meanwhile, grew impatient. On September 18, 1846, he wrote to Johann Galle of the Berlin Observatory, asking him to use their 23 cm telescope to look for the planet. Galle received the letter on September 23 and directed his observing that night to the very spot indicated by Le Verrier. Expecting to see the disk of a planet, Galle was initially disappointed. It was Galle's young assistant, Heinrich D'Arrest, who suggested comparing what they were seeing to the detailed star charts on hand, and soon they found what they were looking for. “That star is not on the map!” D'Arrest is reported to have said, and by the following night that star had moved. That star was the planet Neptune. And knowing that it was a planet they were looking at, the disk was visible.

Galle wrote to Le Verrier,

Thus was Neptune discovered, right where both Adams and Le Verrier had predicted it to be.

It was embarrassing to the English astronomers, especially when they looked back at the records of their search for the planet and saw that they had in fact observed the same object. They had seen it as a star rather than a planet. It had even been recorded as worthy of rechecking, but for some reason they never did. A story goes that it was Mrs. Challis who offered her husband and a dinner guest another cup of tea before the astronomer went back to work. They accepted, and by the time they were finished, the sky had clouded over. But with more confidence in the calculations and the hypothesis predicting the planet's location, chances are he would have been less easily distracted.

The Cambridge astronomers were not the only ones to have observed and recorded Neptune without realizing that it was a planet. Galileo had seen it and called it a star. And in 1793, the French astronomer Joseph Lelande had seen it and even noted its movement, which he blamed on observational error. Neither of these people had a telescope as powerful as the 23 cm refractor of the Berlin Observatory, let alone the 30 cm refractor at Cambridge. But even with these, seeing Neptune as a planet required knowing that it was a planet. As Johann Encke, the director of the Berlin Observatory put it, “the disc can be recognized only when one knows that it exists.” (Quoted in Standage, Reference Standage2000, p. 122.) It's not just instrumentation that advances and refines scientific observation; theory plays both a motivational and interpretive role.

Who was the first to observe Neptune? Galileo saw it in 1612, but he didn't see it as a planet. Challis saw it before Galle, and even though the Englishman was looking for a planet, he didn't see it as a planet. Galle, with a smaller telescope but greater respect for the theory, was the first to not just see Neptune but see it for what it is, a planet. This is a clear example of theory influencing scientific observation. Adams' and Le Verrier's calculations told the astronomers where to look and what to look for. And when they looked in the right place, there was a theory in place to tell them how to interpret the images. If it's a planet it must move with respect to the fixed stars and it must, with proper magnification, be seen as an extended disk. In the case of Neptune, it was respect for the theories that made Galle look hard enough to recognize the disk.

Before there was notable difference between the predicted and observed orbits of Uranus, there was little reason to believe there was an eighth planet. Prior to Bode's law, there was really no good reason, no justification at all. By 1840 though, the irregularities in the orbit of Uranus provided some reason for believing in the existence of an eighth planet. And in 1846, when Neptune was observed, the claim was as close to being fully justified as you can get. In other words, the justification, the good reasons to believe the claim is true, increased with the accumulating evidence. But the claim was true all along. It is no truer now than it was in Galileo's or even Aristotle's day.

This makes a humble but important point. Uncertain does not mean untrue. Science does aim to deliver the truth, the absolute truth, and sometimes it succeeds. There is no hubris in saying this. What science can't deliver is certainty, that is, absolute justification. That's no reason not to aim for certainty, since even unachievable goals can keep you heading in the right direction. The responsible thing to do is to acknowledge the limitation, to keep track of the degrees of justification, and to proportion our beliefs accordingly.

The original work on comparing the orbit of Uranus to the predictions of Newtonian mechanics is another demonstration of why one failed prediction does not falsify a theory. Falsification is indecisive because it is not the theory alone that makes the prediction. When testing Newton's theory of gravity by predicting locations of planets, the reasoning involves the laws of mechanics and claims about the make-up of the Solar system as well. If the theory of gravity is true, and if the laws of mechanics are true, and if the current model of the Solar system is accurate, then this planet will be in that spot at that time. If the planet is not at the predicted spot, any one (or more) of the pieces used to make the prediction could be at fault. The hypothesis in this case is Newton's law of gravity. It made a prediction about Uranus that turned out to be false, but that did not automatically mean the hypothesis was false.

Perseverance through the challenge of falsely predicting the motion of Uranus led to a novel prediction for Newton's theory of gravity. It predicted the existence of an eighth planet, and the prediction turned out to be true. The discovery of Neptune was surely a triumph for the theory. The prediction was precise and the observation of the planet was unassailable. This does not prove the theory is true. That would be the fallacy of affirming the consequent. But it is surely some measure of success to the theory, some reason to think it's true.

If the discovery of Neptune was a success for the theory, the non-discovery of Vulcan was as much a failure. Confidence was high in both the theory of gravity and the analytic method of finding the unseen cause of an observed phenomenon. This is another inverse problem. The results are known; the challenge is to figure out what brought them about. The unusual motion of Uranus was known; Adams’ and Le Verrier's accomplishment was to precisely identify the cause. It turned out to be another planet. Order was restored, and the harmony between theory and observation was renewed.

But Uranus wasn't the only planet giving Newton's theory some trouble. The orbit of Mercury, the planet closest to the Sun, was also a bit off from the theoretical predictions.

From Kepler and Newton, and our own Chapter 7, we know that planetary orbits are elliptical. Kepler had the Sun at one focus of the ellipse, but Newton moved it a bit, putting the center of mass of the Sun–planet pair at the focus. The point on the ellipse at which the planet is closest to the Sun is called the perihelion. If it were just these two bodies, the Sun and the one planet, Newtonian theory predicts that the orientation of the ellipse would not move against the background to the stars. The planet would complete a full orbit and return to the same spot and start over, retracing its previous orbit. But that's not what happens, because it's not just these two bodies. There are other planets in the Solar system, tugging on each other the way Neptune tugs on Uranus. This causes the ellipse to precess, so that the planet over-shoots its starting point when it completes an orbit. The orientation of the ellipse slowly rotates with each orbit. Astronomers measure this in terms of the precession of the perihelion.

Newton's theory of gravity has all these details covered, and it fully predicts the precession of perihelions. It's not at all an easy calculation, since it involves multiple objects, but it is, as the physicists say, doable. And it was done for Mercury. It was done by Le Verrier himself in 1859. His result was a predicted precession of 527 arc-seconds per century (527″/century) for the perihelion of Mercury. This isn't very much. An arc-second is 1/3600 of an angular degree. And it's 527 arc-seconds per century, not per orbit. It's a tiny amount, but well within observational precision in the mid-nineteenth century. The measured value was, and is, 565″. With Mercury, as with Uranus, the Newtonian theory of gravity made a false prediction. It was off by 38″ per century.

Letting a theory get away with a failed prediction seems to compromise the scientific standard of falsifiability. If the evidence doesn't force us to abandon the theory, what will? Le Verrier, flush from his success with Neptune, was not about to question the Newtonian theory of gravity. He knew what to do to accommodate the anomalous data, and it wasn't to change the theory. “We will certainly not be tempted into charging the law of universal gravitation with inadequacy.” (Quoted in Roseveare, Reference Roseveare1982, p. 20.) So in 1859, Le Verrier hypothesized the existence of a hidden planet, between Mercury and the Sun, responsible for the extra 38″ of precession.

Confidence in the reality of this interior planet was not nearly as high as confidence in the planet beyond Uranus, even before the observation of Neptune. With Neptune, William Herschel declared victory in a way that made actually looking for the planet almost superfluous. “We see [the planet] as Columbus saw America from the far shores of Spain. Its movements have been felt, trembling along the far-reaching line of our analysis with a certainty hardly inferior to ocular demonstrations.” (Quoted in Baum and Sheehan, Reference Baum and Sheehan1997, p. 102.) They knew Neptune was out there, even before seeing it. But this new planet, the one so close to the Sun, was dubious from the start.

For one thing, Le Verrier did not provide a precise location for the planet, as he and Adams had for Neptune. It's in there somewhere. If it is, and this is another reason for doubt, it should be visible, certainly during a solar eclipse. But it had never been seen. Perhaps it is always behind the Sun, its orbit exactly in time with our own but on the opposite side of the Sun. This is impossible by Newton's own laws. If the planet has a different orbital radius from the Earth, being much closer to the Sun, it must have a different orbital period. That's basic Kepler. Despite these obvious problems, a search for the new planet was begun.

At the same time, because of these obvious problems, alternative explanations for the anomalous 38″ were proposed. Some were in the form of additional matter affecting Mercury, but not matter concentrated as a planet. Diffuse matter, dust, if there was enough of it in the space between Mercury and the Sun, would exert a strong enough gravitational pull on the planet to perturb its precession. And this dust would also explain the so-called zodiacal light, a subtle glow in the night sky that seems to emanate from the Sun and extend along the ecliptic, growing fainter with distance from the Sun. But the brightness of the light allows for an estimate of the amount of dust, and it was much too small to provide the mass needed to explain the extra 38″ in Mercurial precession.

A compromise between the matter concentrated as a planet and diffuse as dust is a collection of asteroids, tiny not-quite-planets, in orbit between Mercury and the Sun. Again, the density of such objects sufficient to account for the observed motion of Mercury would result in the asteroids being visible.

So it was exciting when on March 26, 1859, an amateur astronomer, Edmond Modeste Lescarbault, a physician and a Frenchman at that, reported to Le Verrier that he had observed a previously unseen object transit across a portion of the Sun. This could be the predicted planet. On a visit to Lescarbault's observatory, Le Verrier was sufficiently convinced of the doctor's credibility, and the new planet was declared discovered. Le Verrier named it Vulcan, after the Roman god of fire and metal-work. It was, after all, right next to the hearth, as Kepler called the focus of the orbit.

This was not a triumph of Neptune proportions, however. The estimated mass of Lescarbault's planet was enough to provide only 1/17 of the pull required to cause the extra 38″ of precession in Mercury. And it was never seen again. Add to this an embarrassing follow-up in which Lescarbault reported finding another new celestial object, this time a star, that turned out to be Saturn. His credibility, and the alleged sighting of Vulcan, suffered.

None of the unseen-matter hypotheses worked to explain the anomalous precession of Mercury. There was simply not enough mass, based on indirect, or dubious, observations. There were also conceptual concerns. The diffuse matter would have to orbit the Sun, as would asteroids or a planet, but by Newtonian calculations the orbit would have to be tipped, not in the same plane as the orbit of Mercury, in order to affect the precession. Such an orientation, again using Newtonian theory to understanding the situation, would be unstable. Like an unbalanced washing machine, it wouldn't last.

With no new matter to blame for the failed prediction of the orbit of Mercury, question turned to the theory of gravity itself. There was no call to toss Newtonian gravity wholesale, only to tinker. Small changes were suggested, changes that would show up for planets close to the Sun, like Mercury, but not for planets far from the Sun, like Uranus and Neptune. The tinkering, in other words, would be focused on the distance parameter, the r in F = GMm/r²$F=GMm/r^2$.

Asaph Hall, in 1895, suggested a slightly revised formula for universal gravitation, F = GMm/r^{2 + δ}$F=GMm/r^{2+\delta }$, where δ is an adjustable parameter that takes on whatever value is needed to get the formula to match the evidence. It's like the parameters in the Ptolemaic model of planetary orbits, the radius and period of the epicycle. You put in whatever value it takes to match the data, to save the phenomena. In this case it took a very small number, a very small adjustment:

$\delta =0.0000001612$

The charitable description of this revision is in terms of theory being responsive to evidence. That's how we think science should work. But the disparaging description of the same event is of ad hoc tinkering with the theory, putting in that δ fudge-factor in the exponent, to accommodate the falsifying data. A theory so flexible is not falsifiable, and that's not how we think science should work. The Newtonian theory of gravity has adjustable parameters. It's not, to borrow a term from Steven Weinberg, theoretically “rigid” (Weinberg, Reference Weinberg1992, p. 105). The general theory of relativity, we will find out, is significantly less flexible. The 1∕r² dependence in the relativistic formula will be a matter of principle, not a derivation from the collected evidence. It will not accommodate tinkering.

Whether in positive terms of being responsive to evidence, or negative in terms of unseemly patching theoretical leaks, the Hall theory works well under an instrumentalist attitude about science. Is the gravitational force really dependent on such an unlikely geometric factor of 1∕r^2+δ? Instrumentalism makes this moot. Gravity is as if the dependence is 1∕r^2+δ, and calculating in this way gets the predictive job done. It offers no explanation, but with this attitude, no explanation is called for. This is science, where theories are slaves to evidence, nothing more than formulaic summaries of data, and a mere summary can't, and shouldn't pretend to, explain.

Summarizing our own historical data on the attempts to reconcile theory with the anomalous precession of Mercury, no solution was perfect. Hall's theory was not only inelegant and artificial, it was also inaccurate in its predictions of the motion of the Moon. Hypotheses about unseen matter, whether diffuse or planetary, were easier to form, not having to conform to any conceptual constraints like elegance, simplicity, or coherence with other physical theories. But none of the matter hypotheses could be made to provide enough mass while remaining otherwise undetected. That much stuff would make a visible mark. In this case the absence of evidence is evidence of absence. It was clear by the start of the twentieth century that something pretty radical had to be done.

But it wasn't Mercury, or the non-discovery of Vulcan, that leveraged the radical change in the theoretical understanding of gravity. It was a more fundamental, principled flaw in Newton's theory. The change in the theory was nothing like the fix-up proposed by Hall. It was a wholesale discarding of the Newtonian theory, to be replaced by a very different way of describing gravity. The new account, relativity, will be based on principles. It's a top-down way of arriving at a theory, and it will be less adaptable to evidence and less accommodating of tinkering.

The guiding principles for the new theory of gravity are familiar, the Principle of Relativity and the Principle of Equivalence. It will be physics, not astronomy, that forces the issue. It will be the action at a distance in Newton's theory that is fundamentally at odds with the principles.

Einstein did not put together the new theory of gravity, the general theory of relativity, to accommodate the extra 38″ of precession in the orbit of Mercury. He put it together to abide by the Principle of Relativity and the Principle of Equivalence. But once assembled, the new theory in fact predicted the precession spot-on. It included the 38″. It may be more accurate to call this a post-diction rather than a prediction, since the observations had already been made. And getting a post-diction right is arguably a better test than getting a prediction right. The evidence is on the books before the theory is in the books, so there is no opportunity to interpret or select data with an eye towards hitting the theoretical target. And if, as Harold Jeffreys described the general theory of relativity, the theory “contains no arbitrary constituent capable of adjustment to suit empirical facts” (Jeffreys, Reference Jeffreys1919, p. 138), either the theory and evidence match, or they don't. This is the basis of falsifiability.

Einstein was proud that the general theory got Mercury right. Not only did the theory match the evidence, but it achieved the match in the right way, from first principles. This was unlike previous attempts invoking an unseen planet or the δ revision. These he described as “the assumption of hypotheses which have little probability, and were devised solely for this purpose.” (Einstein, Reference Einstein1920, p. 123.) The “little probability” must stem from the obvious problem that if there was a planet we would see it. That's a problem with the content of the hypotheses. But the “devised solely for this purpose” is a problem of method. Evidence, according to Einstein, should not force the hand of theory. Theoretical change should be a matter of principle.

Back to the textbooks, then, the ones that describe gravity in Newtonian terms. The description is good enough for non-experts. It's good enough for using the phenomenon to get things done or predict how things move. But it's not good enough if you want to not only use gravity, but also want to understand gravity. It's not good enough because it's false.