X-Message-Number: 15977 Date: Fri, 30 Mar 2001 13:47:49 -0500 From: "Eliezer S. Yudkowsky" <> Subject: Reliability of Friendly AI (was: Trust in All-Powerful Lords) References: <> From: Lee Corbin <> > > Eliezer S. Yudkowsky wrote > > > PEOPLE ARE NOT YOUR PROPERTY. They can NEVER be your > > property. Everyone owns himself, herself, or verself > > and can never be owned by anyone else. > > > Tell me, if I suddenly revealed that I'm an Observer > > construct and that you're living in a sim, would you > > suddenly decide that you have no rights? > > I really don't think that "rights" in the abstract mean > anything. Usually when someone forms a sentence "X has > the right to do Y", it really means nothing more than > "I approve of X being able to do Y". A common misconception, widespread because of its apparent objectivity. "X has the right to do Y" means (1) "I approve of X being able to do Y"; plus either (2a) "I believe that society at large also approves of X being able to do Y", or (2b) "I believe that society at large believes in moral principles whereby I should be able to convince them that they should approve of X being able to do Y, even if they do not presently believe as such". At least, that's the moral relativist's version. There is also a strong psychological difference in how humans process "rights" and approvals that are internally believed to be more observer-independent, but that's not really relevant to the discussion. It's what I meant to ask when I wanted to know whether you would "decide that you have no rights" - i.e., whether you would feel a sense of violated entitlement - but evidently you've left those intuitions behind, which is certainly not something I would object to. > So far as I can see, the infinite nosiness of the Could you *please* restrain your impulse to use terminology like this and just concentrate on debating whether or not the Sysop is, in fact, infinitely nosy? > Sysop that you promote arises from a deep utilitarian > belief or position, that makes it your business whether > or not I give someone life or not. We need hardly talk > of the future to expose the principle at work here. I > am opposed to legal rights for animals, and do not believe > that we actually need statutes on the books forbidding > cruelty to animals (if we ever did need them), even > though I base my beliefs on utilitarianism also. Forgive me, but aren't you now talking about rights? If this entire conversation is phrased in terms of simple cause and effect, then a Friendly AI comes into existence which respects your rights and also the rights of any citizens which you create. This may be a consequence of the fact that Eliezer, a Friendship programmer, respects both you and the citizens which you create. Or it may have nothing to do with Eliezer as such; it may simply be a consequence of the means whereby the Friendly AI recognizes "people" as such. In either case, your language seems to me to betray a strong sense of entitlement to the *right* to - as I would put it - dictate the lives of your children; to, as you would put it, do what you will with your computing power. My attempt to "intervene" does not betray a lack of respect for private property on my part; it betrays a definition of "citizen" which operates on pure patterns, irrespective of levels of abstraction. Thus, to stop respecting your children, I would have to stop respecting you. > The > reason is simply that if given freedom, so few people > would seriously mistreat animals that the laws on the > books cause more harm than good. Fine; then that is a sufficient condition to make the Sysop Scenario undesirable under Friendliness. I think you're wrong. > If in the near future, it became clear that only one AI > in the solar system would rule (for techical reasons), > and we had a chance to contribute to the principles by > which that government Government?! A "government" is a socially plausible structure composed of interacting evolved-normal humans, separated into discrete nontelepathic/nonagglomerative entities, each with observer-centered desirability metrics and observer-biased cognitive processes. Do you have any idea how much that constrains your solution space? > would rule, how many reading this > would agree with Eliezer that people should not be > allowed to run simulations (and not be allowed to torture > animals), ?? I never said anything about animals. > and how many would agree with me, that our > government should exist only to enforce freely arrived-at > contracts, and to protect private property? A rather prejudiced way of phrasing it, don't you think? Let's make it simpler: Regardless of who or what runs the future Universe, whether it's a society or an artificial cognitive structure, do you - personally - like or dislike the following scenario: Mr. Smith creates a simulation of he doesn't like; say, Mrs. Jones next door from Old Earth. This entity is fully conscious, fully volitional, human-equivalent in every way, et cetera; she may be a computer simulation, or she may be constructed out of human-type biology and real atoms. In either case, the atoms, or computers, are owned by Mr. Smith. Mrs. Jones is a normal human from our perspective; that is, her internal desires are such as to wish to explore the Universe, live forever, exist free from pain, upgrade to superintelligence, and so on. (That's "normal", right?) Mr. Smith then works up a recreation of a medieval torture chamber - either as a virtual reality, or using more owned atoms - and keeps Mrs. Jones there for a thousand years. Then he kills her. Some possible answers: 1) Eliezer's answer: "The lowest layer of my morality says this is a bad thing. While there's a principle against interfering with Mr. Smith, violating private property, et cetera, this principle is on the same level as the principle against involuntary torture. The moral principles urging me not to interfere with Mr. Smith don't have the power to override the moral principle urging me to free Mrs. Jones. Smith and Jones are equals, at war with each other, and I'll support whichever side I think is in the right. Thus, while I might not launch a physical war and so disrupt the solar system, I wouldn't hesitate to crack Mr. Smith's computer." 2) "No, I hate it, but there are deeper or higher moral principles which forbid me to interfere, such as my respect for private property. I'd beg Mr. Smith not to do it, but if he decided to, I would never intervene." 3) "No, I hate it, and I'd do something about it if I could get away with it, but I believe that the morally optimal structure for society is such that society would successfully prevent me from interfering with Mr. Smith in any way." 4) "Mr. Smith owns Mrs. Jones. I might personally dislike his actions, but I don't consider my disliking to rise to the level of a moral principle. It's like disliking his taste in music, just more intense - a strictly personal thing." 5) "Mr. Smith owns Mrs. Jones. I have no problem whatsoever with any of the things he chooses to do to her." 6) "Mr. Smith owns Mrs. Jones. However, I believe that, throughout the entire future, no owner will ever torture an owned entity; or that these cases will be so few and far between that the overhead of preventing them would simply not be worth it." -- -- -- -- -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=15977