It is my belief that empathy is a a prerequisite to consciousness.

I recently read Hofstadter’s I am a strange loop, whose central themes are around recursive representations of self leading to our perception of consciousness. For some, the idea that our consciousness is somewhat of an illusion might be hard to swallow – but then, quite likely, so are all the other qualia. They seem real to us, because our mind makes it real. To me, it’s not a huge hurdle to believe. I find the idea that our minds are infinitely representing themselves via self-reflection kind of beautiful in simplicity. You can get some very strange things happening when things start self-reflecting.

For example, Gödel’s incompleteness theorem originally broke Principia Mathematica and can do the same for any sufficiently expressive formal system when you force that formal system to reason about itself. One day I’ll commit to explaining this in a post, but people write entire books about the idea to make Godel’s theorem and it’s consequences easy to understand!

And as an example of self-reflection and recursion being beautiful, I merely have to point to fractals which exhibit self-similarity at arbitrary levels of recursion. Or perhaps the recursive and repeating hallucinations induced by psychedelics give us some clue about the recursive structures within the brain.

Hofstadter also later in the book delves into slightly murky mystical waters, which I find quite entertaining and not without merit. He says that, due to us modelling of the behaviour of others, we also start representing their consciousness too. The eventual conclusion, which is explained in much greater and philosophical detail in his book, is that our “consciousness” isn’t just the sum of what’s in our head but is a holistic total of ourselves and everyone’s representation of us in their heads.

I don’t think the Turing test will really be complete until a machine can model humans as individual and make insightful comments on their motivations. Ok, so that wouldn’t formally be the Turing test any more, but I think that as a judgement of conscious intelligence, the artificial agent needs to at least be able to reflect the motivations of others and understand the representation of itself within others. Lots of recursive representations!

The development of consciousness within AI via empathy is what, in my opinion, will allow us to create friendly AI. Formal proofs won’t work due to computational irreducibility of complex systems. In an admittedly strained analogy this is similar to trying to formally prove where a toy sailboat will end up after dropping it in a river upstream. Trying to prove that it won’t get caught in an eddy before it reaches the ocean of friendliness (or perhaps if you’re pessimistic and you view the eddy as the small space of possibilities for friendly AI). Sure computers and silicon act deterministically (for the most part), but any useful intelligence will interact with an uncertain universe. It will also have to model humans out of necessity as humans are one of the primary agents on the Earth that will need to interact with… perhaps not if it becomes all-powerful but certainly initially. By modelling humans, it’s effectively empathising with our motivations and causing parts of our consciousness to be represented inside it[1].

Given that machine could increase it’s computationally capacity exponentially via Moore’s law (not to mention via potentially large investment and subsequently rapid datacenter expansion) it could eventually model many more individuals than any one human does. So if the AI had a large number of simulated human minds, which would, if accurately modelled, probably bawk at killing the original, then any actions the AI performed would likely benefit the largest number of individuals.

Or perhaps the AI would become neurotic trying to satisfy the desires and wants of conflicting opinions.

In some ways this is similar to Eliezer’s Collected Extrapolated Volition (as I remember it at least… It was a long time ago that I read it. I should do so again to see how/if it fits with what I’ve said here).

[1] People might claim that this won’t be an issue because digital minds designed from scratch will be able to box up individual representations to prevent a bleed through of beliefs. Unfortunately, I don’t think this is a tractable design for AI, even if it was desirable. AI is about efficiency of computation and representation, so these concepts and beliefs will blend. Besides, conceptual blending is quite likely a strong source of new ideas and hypotheses in the human brain.

How is “Advanced Intelligence” different from general AI? The release says:

Typical features of Advanced Intelligence include: (1) Close interaction and coordination between Natural Intelligence and Artificial Intelligence, (2) Ideas and applications that push the frontiers of both Artificial Intelligence and Natural Intelligence, (3) Large-scale Distributed Intelligence and Web Intelligence

FWIW, I’m not sure I agree with the name, since “advanced” is a relative term, and the field of AGI (artificial general intelligence) is already fragmented enough without adding additional labels. Having said that, I’ve been invited to be an assistant co-chair, so I cordially invite you to submit interesting papers since then they’ll be a greater chance of me getting interesting papers to review

“The Singularity Institutes “main purpose” is meant to be to investigate whether a recursively improving intelligence can maintain “friendliness” towards human kind. “

Okay, but my standpoint is: Why does the recursively improving intelligence need to be non-human? It seems counter-intuitive to me to devolve this power to something outside of ourselves – and also a bit like we’re just trying vainly to become a kind of God, creating another type of being.

I think the main reason there is a focus on AI rather than improvement of human intelligence is because it’s so damn hard to do experiments on people’s brains. It’s ethically difficult to justify various experiments, and it only gets harder as things become more regulated (and rightfully so for the most case). I think they’ll definitely be continuing research into this stuff though. For myself, occasionally taking Modafinil enhances my productivity significantly (so long as I maintain focus on what I’m meant to be doing, it’s easy to get enthralled with something that interests me, but isn’t related to my work).

But there’s no exclusion of human intelligence amplification from the singularity concept. If we create smarter humans, then this begets even smarter humans. Again we can’t really predict what those enhanced “humans” would do, because they are a significant step smarter than us.

Human intelligence amplification has a whole raft of other ethical issues associated with it though too. When it becomes more mainstream/available it’s going to be a major political and social issue. What happens when not everyone can afford (or wants) to enhance themselves? Will we develop two classes? One of naturals and another of post-humans? Will employers require certain professions to use performance enhancement (say for example, for brain surgeons performing long surgeries)? It’s also going to raise the question about the ownership of our bodies. There are laws against taking recreational drugs, but for some, LSD helps with certain types of thought and could be seen as a form of an intelligence manipulator (or an amplifier of certain facets of intelligence).

At the moment at least, governments and enforcement agencies seem completely uninterested in actively stopping this, due to the prevalence of various performance enhancement drugs throughout academia and other cognitively demanding professions. Obviously it’s not necessary, but for some the edge or boost it gives them is sufficient to outweigh the risks of off-prescription drug use.

That was my main beef with Kurzweil: He assumed that the intelligence beyond the horizon would be non-human. This, of course, begs the deeper philosophical question: what *is* “human?” But I’m sure you’ve mulled that one over plenty. Indeed, it would be interesting to hear what you have to say on the subject.

It simply strikes me that, the human ego being what it is, we would naturally be trying to improve our own intelligence and not worrying too much about creating AI as a standalone entity. Am I wrong? Is the human ego instead more interested in giving birth to a new species?

There is one other reason there is a focus on AI. Which is related to human ego. Once humans have an advantage over others, such as a significant step up in intelligence, then there’s a good
chance some will use that power over others in a negative way. It’s the old adage, “power corrupts, and absolute power corrupts absolutely” – and it’s my opinion, that this could lead to the same power/intelligence imbalance between an intelligence-amplified human and humans, as that between humans and dogs.

Or more succinctly, humans have plenty of evolutionary baggage that we
don’t necessarily want to amplify!

Yeah I’ve thought about that a little as well. What I wonder (and this really is just pure speculation) is whether hyper-intelligent humans would have more faculty for reason and logic; and, if so, if they would see the benefits in spreading the gift and/or using their gifts in more compassionate or benevolent ways…? Because, surely, that’s evolution as well.

Again, it would seem somehow counter-intuitive to improve intelligence without, for example, attempting to improve our capacity for processing emotions — our ‘emotional intelligence’, as it were.

I tend to think that many of the world’s problems today come down to a focus on what is scientifically possible rather than what is philosophically ‘good’ or ‘true’.

Basic agreement, improving intelligence is really just a catch all for all sort of cognition improvements. Depending on where we focussed neuron regeneration, we could expect to see improvements in different aspects of cognition.

Science and technology will inevitably advance, and while we can regulate technology to some extent, we can’t stop it. Telling people they can’t work on AI would merely push it underground, and one of the reason I work on OpenCog is because it’s an open-source framework. It can be inspected by other experts for flaws to ensure there isn’t some hidden time-bomb sitting in the code, and allows an international approach which might otherwise be over-regulated within a university-specific project (this ignores the actual issue of friendly vs. unfriendly AI, since the latter doesn’t have to be intentionally designed – I may write more about that one day, but SIAI and it’s fellows have already written plenty on that).

To bring it back to AI, what place is there for emotions in an artifically constructed intelligence? Is there a capacity for compassion or fear, if they serve a function for processing input and generating responses? I am guessing that cognitive scientists have thought about this one…

That’s a BIG topic, so all I’ll say is that my opinion is that emotions are just a particular mind state. Sure this mind state is influenced by hormones and neurotransmitters, but so is the rest of the brain’s functioning.

My opinion is that emotions are just extremely strong aspects of a human’s mental world. If we gave an AI an extremely strong desire/goal to make humans happy, and the AI design had some kind of reward based system (such that the AI was trying to maximise these) and achieving these rewards caused other effects in the AI’s mind, such a propensity to use positive phrases to describe the world, then is there any reason to believe that the AI isn’t happy itself?

Disclaimer: I’m no longer in the employ of the SIAI, so my thoughts are not endorsed by them at all.