I guess it depends if friendliness is to be judged by actions or intention. But as the saying goes, the road to hell is paved with good intention.

@James I think that’s a similar but related idea which I wouldn’t mind learning more about. The representation of other people’s consciousness is more of a two way thing and at a slightly more direct level than my impression of Jung’s ‘collective unconscious’ – perhaps we should call it our ‘collective consciousness’?

This sounds to me very much like the ‘collective unconscious’ that Jung talks about—the cultural paradigm that we use to filter our ideas and actions through in daily life. We need to empathise in order to be successful and to attain happiness in our lives, and we use this paradigm as a tool of interpretation. If we hold in our minds an accurate image of the collective unconscious, then we interact in more meaningful and beneficial ways with others, and thus become more successful and achieve more happiness.

Seems only fair that any AI entity should have the same goals, using empathy as a tool to that end.

With Friendly AI (FAI), you care about a certain relation between it and (say) a human: in each situation X (incl. all input, etc.), the FAI needs to prefer to do the same action as the human would prefer in the same situation X (on reflection, if they were both capable of understanding X fully, etc.). This is more complicated than same behavior of of two identical programs, since preferring the same thing doesn’t necessarily mean that they’ll actually do the same thing, or do the most preferred thing: when solving a difficult program, your goal might be to find the right solution, but you’ll only be able to find some approximation to a solution, or a candidate solution that you aren’t sure is the right solution. This holds both for us stupid apes, and for superintelligent AIs, given appropriately difficult problems.

This is to reply to two ideas you voiced in this post: first, you don’t need FAI to actually interact or observe humans, to “emphasize”, in order to have a knowable property of being Friendly, at least not during the actual operation (you of course need to confer the info about humans into the FAI at the start, but this is again the info about humans as systems, not magic knowledge of what they’ll actually do, or explicit info about what they will do in all possible situations). Second, uncomputability or chaotic behavior of environment are not game-stoppers, as you only need to have a certain abstract property of FAI as a system, that can then be let to freely develop for any unanticipated circumstance it happens to encounter. Friendliness need to be shown in general, as a property of a system, from which Friendly behavior in the specific situation (which is not knowable in advance) follows as a special case.

You might want to read the current sequence I’m writing on my blog, to get a better idea of where I’m coming from.

“If you have two identical complicated programs, of which you don’t and can’t know what they compute on what possible inputs, you can still be sure that they’ll compute exactly the same thing on all occasions.”

Unless you mean that they receive exactly the same inputs at the same times, then yes, I agree. But any intelligent agent doesn’t exist in a world with an invariant temporal chain of inputs.

So two identical programs could diverge depending on the relative ordering of inputs. Not necessarily diverge from friendliness, but in terms of eventual knowledge and belief.