X-Message-Number: 15932 Date: Sat, 24 Mar 2001 18:04:03 -0500 From: Sabine Atkins <> Subject: Re: Trust in All-Powerful Lords (As you brought up some very important points, and as I'm still a newbie regarding Friendly AI, I asked Eliezer to address them.) ------- Start of forwarded message ------- From: "Eliezer S. Yudkowsky" <> To: Sabine Atkins <> Cc: Brian Atkins <> Subject: reply Date: 3/24/01 11:57:41 AM Lee Corbin wrote: > > Evidently, I did not make myself clear. What _exactly_ > do you intend to do with peaceful yet very advanced > entities (such as many of us hope to become) who intend > to live outside the control of your AI? (Please be > specific.) Unless it can be demonstrated as effectively certain that you are, and will remain, peaceful yet very advanced - i.e. will not create a simulated civilization and torture it, nor take actions that lead to that possibility - then I think the right to life and freedom of a trillion potential beings will probably outweigh your (admittedly real) volition to step outside the Sysop. I ask again: Can you name something that a peaceful entity wants to do, and should be able to do, that can't be done from inside the Sysop region? I think you are confusing the potential for interference with actual interference, like the philosopher's god who was omnipotent on the condition that it never exercise its power. To a human, "freedom" means nobody having the theoretical capability to interfere with you. In our world, anyone who has the power to mess with you, usually does. Lord Acton's heuristic applies to all freely evolving imperfectly deceptive social organisms, and humans in particular. As a result, we attempt to maximize actual freedom - that is, minimize actual interference - by minimizing what our mind perceives as the "capability" for agents to interfere. Human motives are too changeable - are in fact biased by Acton's heuristic - for us to take motives into account as a constraint on possible interference. This is what failed revolutions don't understand; that freedom requires minimizing the total amount of power, not shifting power from "untrustworthy" to "trustworthy" groups. Beyond the human realms, I'll settle for that freedom which consists of nobody ever actually interfering with me - i.e., as long as the only entity(s) with the power to mess up my life are known Friendly ones. If Friendliness is not absolutely knowable, then I don't want there to be more than one entity, to minimize the risk. If there are zero entities who possess the *potential* to interfere with me - and the situation is symmetrical, so that there are many other entities whom nobody possesses the potential to interfere with - then entities can assemble, at will, the technological capability to interfere with me, and soon I'm staring down the barrel of a *lot* of unFriendly guns. Niven's Law: Anarchy is unstable. > >As far as SIAI is concerned, the super intelligent > >AI we are planning to build will be a protector and > >a facilitator. It is also planned to prevent us from > >doing harm to each other and ourselves. > > Evidently you cannot see why people are finding your > words alarming. Please: history is replete with the > efforts of the best-intentioned people to provide > "workers' paradises" and other benevolent dictatorships. > > Are you unaware of Lord Acton's principle? (Power > corrupts and absolute power corrupts absolutely.) That's right. Acton's heuristic applies to humans, who evolved in social power structures, not to AIs. That is; the heuristic (as a finite tendency, not an absolute rule) applies to humans such as SIAI's Board and Friendship programmers. Therefore, all else being equal, it is desirable to use structural definitions that present few (or zero) degrees of freedom in supergoal content, so that no one human (or group of humans) has the capability to mess things up without starting the project over again completely from scratch. Rest assured that I have no intention of building a gun that could be pointed at my own head. > Do you not think that Joseph Stalin, the young > revolutionary, was completely sincere in his desire > to help the Russian people? Or Mao Ze Dong? What is this, some kind of destructive twentieth-century cynicism? Haven't you ever heard of George Washington? There was a successful American revolution before there was a failed French revolution. Idealists don't always fail. Contrary to your misquote, Lord Acton said that power *tends to* corrupt. The tendency is finite. It can be, and has been, defeated. Of course, part of that discipline is learning to avoid the *necessity* for asking others to trust you with power. I choose to follow a strategy which minimizes anyone's ability, including my own, to screw things up. > My third question: by what miracle of computational > science can you be sure that a tiny rogue-element has > not been inserted by some programmer (or by some > external fiend) into the architecture of your AI? > I believe that any attempt to prove that your AI does > not contain such an element is NP complete, if not > much, much, harder. I don't think a young self-modifying AI can successfully bury a rogue-element in such fashion that the element is never spotted by a far more mature version of verself which believes verself to be Friendly and would be horrified to spot such an element. It's a question of relative abilities. There's also the idea of a secure flight recorder - which *would* allow for absolute verification of all inputs, code, and hardware - but SIAI probably won't be able to afford one of those until later in the game. Unless you're offering to pay for one? > Fourth: So, in short, are you asking us to just > "trust you, and everything will be all right"? No. I would sort of appreciate if you were to assume that we are not idiots, and that everything you can see, we can see. Failing that, you can read the online material on the subject and judge for yourself. Failing that, you can try to get a feel for Eliezer Yudkowsky and figure out whether he's the sort of person who probably follows protocols that avoid the need for trust. -- -- -- -- -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence -------- End of forwarded message -------- -- Sabine Atkins http://www.posthuman.com/ -- Singularity Institute for Artificial Intelligence http://singinst.org/ Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=15932