For a way to understand how business and the military treat scientific ethics, the best, clearest case is the Manhattan Project.  When discussing the project, it is important to remember that the program wasn’t to “develop an atomic bomb.” But to “develop an atomic bomb before the Nazi project produced one.” The potentially civilization-ending powers of the weapon were known. And, obviously, in a world where wooden bullets and poison gas were forbidden on the battlefield, something as horrific as an atomic bomb must be, right?

Many Manhattan Project scientists made this exact point before the bombs were dropped on Hiroshima and Nagasaki, including Leo Szilard, the person who first hypothesized an uncontrollable chain reaction could be weaponized. He said, “Let me say only this much to the moral issue involved: Suppose Germany had developed two bombs before we had any bombs. And suppose Germany had dropped one bomb, say, on Rochester and the other on Buffalo, and then having run out of bombs she would have lost the war. Can anyone doubt that we would then have defined the dropping of atomic bombs on cities as a war crime, and that we would have sentenced the Germans who were guilty of this crime to death at Nuremberg and hanged them?”

Another Manhattan Project alum, James Franck, who led the metallurgic chemistry department at Los Alamos, said, “If the United States were to be the first to release this new means of indiscriminate destruction upon mankind, she would sacrifice public support throughout the world, precipitate the race for armaments, and prejudice the possibility of reaching an international agreement on the future control of such weapons.” Exactly, Professor Franck, exactly.

Even Dwight Eisenhower spoke out against using atomic weapons in Japan, telling the US Secretary of State on two occasions not to use the bombs. It was one thing to develop the weapons to counter the real threat of the Nazi program, but Japan’s atomic bomb program was laughable, and the weapons were not needed for any legitimate military reason. There were a lot of people in this camp, including many of the major figures involved in the war against Japan, including Douglas MacArthur (who led the ground forces in the Pacific campaign) and Chester Nimitz and William Halsey (the commanders of the sea forces in the Pacific campaign.) Even Curtis LeMay, the commander of the US air forces, acknowledged no military necessity for dropping the atomic bomb. Even LeMay!

And what some people wanted was much worse than what we got. Initially, the bomb was to be dropped on Kyoto because, in part, it was an enormous city on a flat plain that would kill the most people and demonstrate the fullest potential of the weapon and create the greatest psychological impact.  It was barely rejected on cultural grounds.  Too much nice stuff in Kyoto.  Not because of the human cost but the cultural cost.

But the bomb had never been designed to be used to do anything other than counter the Nazi program. All of the arguments against its use – including those made by the commanders of the forces that would have invaded Japan who said in no uncertain terms that they didn’t want to drop the bomb – were swept aside.

The bomb had gotten a life of its own, and like a monster from the most terrifying movie in history, it struck and killed hundreds of thousands of people. Men. Women. Children. They died when the buildings they were in were smashed to dust or vaporized in a blinding white light, or days or weeks later when their organs were reduced to a toxic slurry by radiation, or years later when cancer caught up to them. And like any true horror movie monster, it made a hole underground and swam in the deep waters of the oceans to await the day when the stars are right, and it can spread more death, and disease, and terror.

The ethical concerns – the correct, true, real ethical concerns – of people were ignored. The scientists in charge of the program turned on its use and were ignored. The military commanders in the field were ignored. We had the bomb, and, by God, we were going to use it.

Now, the specific function of atomic weapons is to vaporize large numbers of humans. It is the most terrible weapon yet discovered by humans. Ethical concerns did not prevent its construction and use. Ongoing ethical concerns have not forbidden it, either, even though various arms treaties have condemned the use of far less horrific weapons, like poison gas. To this day, countries like Pakistan, India, and North Korea are playing games of nuclear brinksmanship. Of course, it is a tale well told about the brush with nuclear war in the Cuban Missile Crisis. Richard Nixon used a kind of nuclear brinksmanship in the Vietnam War, the “madman strategy,” where he cultivated an image that he was a madman and might do anything, including using nuclear weapons. Russia is playing nuclear games right now, in Ukraine. “Remember,” Putin has said, “that we have nuclear weapons, too,” ensuring that the EU and the US will not intervene directly.

If ethicists can’t stop these uses of a murderous weapon, what are the odds that anyone will listen to them when developing artificial general intelligence? Roughly zero percent. Rounded up.