X-Message-Number: 15977
Date: Fri, 30 Mar 2001 13:47:49 -0500
From: "Eliezer S. Yudkowsky" <>
Subject: Reliability of Friendly AI (was: Trust in All-Powerful Lords)
References: <>

From: Lee Corbin <>
> 
> Eliezer S. Yudkowsky wrote
> 
> > PEOPLE ARE NOT YOUR PROPERTY.  They can NEVER be your
> > property.  Everyone owns himself, herself, or verself
> > and can never be owned by anyone else.
> 
> > Tell me, if I suddenly revealed that I'm an Observer
> > construct and that you're living in a sim, would you
> > suddenly decide that you have no rights?
> 
> I really don't think that "rights" in the abstract mean
> anything.  Usually when someone forms a sentence "X has
> the right to do Y", it really means nothing more than
> "I approve of X being able to do Y".

A common misconception, widespread because of its apparent objectivity. 
"X has the right to do Y" means (1) "I approve of X being able to do Y";
plus either (2a) "I believe that society at large also approves of X being
able to do Y", or (2b) "I believe that society at large believes in moral
principles whereby I should be able to convince them that they should
approve of X being able to do Y, even if they do not presently believe as
such".  At least, that's the moral relativist's version.

There is also a strong psychological difference in how humans process
"rights" and approvals that are internally believed to be more
observer-independent, but that's not really relevant to the discussion. 
It's what I meant to ask when I wanted to know whether you would "decide
that you have no rights" - i.e., whether you would feel a sense of
violated entitlement - but evidently you've left those intuitions behind,
which is certainly not something I would object to.

> So far as I can see, the infinite nosiness of the

Could you *please* restrain your impulse to use terminology like this and
just concentrate on debating whether or not the Sysop is, in fact,
infinitely nosy?

> Sysop that you promote arises from a deep utilitarian
> belief or position, that makes it your business whether
> or not I give someone life or not.  We need hardly talk
> of the future to expose the principle at work here.  I
> am opposed to legal rights for animals, and do not believe
> that we actually need statutes on the books forbidding
> cruelty to animals (if we ever did need them), even
> though I base my beliefs on utilitarianism also.

Forgive me, but aren't you now talking about rights?  If this entire
conversation is phrased in terms of simple cause and effect, then a
Friendly AI comes into existence which respects your rights and also the
rights of any citizens which you create.  This may be a consequence of the
fact that Eliezer, a Friendship programmer, respects both you and the
citizens which you create.  Or it may have nothing to do with Eliezer as
such; it may simply be a consequence of the means whereby the Friendly AI
recognizes "people" as such.

In either case, your language seems to me to betray a strong sense of
entitlement to the *right* to - as I would put it - dictate the lives of
your children; to, as you would put it, do what you will with your
computing power.  My attempt to "intervene" does not betray a lack of
respect for private property on my part; it betrays a definition of
"citizen" which operates on pure patterns, irrespective of levels of
abstraction.  Thus, to stop respecting your children, I would have to stop
respecting you.

> The
> reason is simply that if given freedom, so few people
> would seriously mistreat animals that the laws on the
> books cause more harm than good.

Fine; then that is a sufficient condition to make the Sysop Scenario
undesirable under Friendliness.  I think you're wrong.

> If in the near future, it became clear that only one AI
> in the solar system would rule (for techical reasons),
> and we had a chance to contribute to the principles by
> which that government

Government?!  A "government" is a socially plausible structure composed of
interacting evolved-normal humans, separated into discrete
nontelepathic/nonagglomerative entities, each with observer-centered
desirability metrics and observer-biased cognitive processes.  Do you have
any idea how much that constrains your solution space?

> would rule, how many reading this
> would agree with Eliezer that people should not be
> allowed to run simulations (and not be allowed to torture
> animals),

??  I never said anything about animals.

> and how many would agree with me, that our
> government should exist only to enforce freely arrived-at
> contracts, and to protect private property?

A rather prejudiced way of phrasing it, don't you think?

Let's make it simpler:

Regardless of who or what runs the future Universe, whether it's a society
or an artificial cognitive structure, do you - personally - like or
dislike the following scenario:

Mr. Smith creates a simulation of he doesn't like; say, Mrs. Jones next
door from Old Earth.  This entity is fully conscious, fully volitional,
human-equivalent in every way, et cetera; she may be a computer
simulation, or she may be constructed out of human-type biology and real
atoms.  In either case, the atoms, or computers, are owned by Mr. Smith. 
Mrs. Jones is a normal human from our perspective; that is, her internal
desires are such as to wish to explore the Universe, live forever, exist
free from pain, upgrade to superintelligence, and so on.  (That's
"normal", right?)  Mr. Smith then works up a recreation of a medieval
torture chamber - either as a virtual reality, or using more owned atoms -
and keeps Mrs. Jones there for a thousand years.  Then he kills her.

Some possible answers:

1)  Eliezer's answer:  "The lowest layer of my morality says this is a bad
thing.  While there's a principle against interfering with Mr. Smith,
violating private property, et cetera, this principle is on the same level
as the principle against involuntary torture.  The moral principles urging
me not to interfere with Mr. Smith don't have the power to override the
moral principle urging me to free Mrs. Jones.  Smith and Jones are equals,
at war with each other, and I'll support whichever side I think is in the
right.  Thus, while I might not launch a physical war and so disrupt the
solar system, I wouldn't hesitate to crack Mr. Smith's computer."

2)  "No, I hate it, but there are deeper or higher moral principles which
forbid me to interfere, such as my respect for private property.  I'd beg
Mr. Smith not to do it, but if he decided to, I would never intervene."

3) "No, I hate it, and I'd do something about it if I could get away with
it, but I believe that the morally optimal structure for society is such
that society would successfully prevent me from interfering with Mr. Smith
in any way."

4)  "Mr. Smith owns Mrs. Jones.  I might personally dislike his actions,
but I don't consider my disliking to rise to the level of a moral
principle.  It's like disliking his taste in music, just more intense - a
strictly personal thing."

5)  "Mr. Smith owns Mrs. Jones.  I have no problem whatsoever with any of
the things he chooses to do to her."

6)  "Mr. Smith owns Mrs. Jones.  However, I believe that, throughout the
entire future, no owner will ever torture an owned entity; or that these
cases will be so few and far between that the overhead of preventing them
would simply not be worth it."

--              --              --              --              -- 
Eliezer S. Yudkowsky                          http://singinst.org/ 
Research Fellow, Singularity Institute for Artificial Intelligence

Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=15977