One of the key problems with the singularity proposition, of “greater than human intelligence” machines is the comparison to human intelligence. What do you mean by “more intelligent than human” or “as intelligent as human”?
If in the first place it was accurate in some meaningful sense to make this kind of comparison, is it not true that if one machine can be as intelligent as human, then by the power of numbers, machines can be much smarter than human. There can be virtually infinite number of machines, where there can be only a finite number of humans.
If we see “intelligence” as some kind of an objective currency of doing things, which it most certainly is not, then it should not matter if it’s one really smart machine, or trillions of machines that are “as smart as human”.
The most important thing to contemplate for this discussion is what “intelligence” means. We tend to look at intelligence on the scale of our perception of time. For example we don’t think that the intelligence of a rock is very impressive, because we can’t see anything happening. We don’t also consider water too intelligent, yet within it we can find the intelligence to create all life forms.
What AI promises to provide us, is support for humanity in those areas of intelligence we are weak in. Like JCR Licklider said it in the 60s, man-machine symbiosis. Not symbiosis in the sense of some kind of a physical/mechanical connection, but in the sense of the man being empowered by the (still largely hidden) power of the machine.
The conclusion we’ve had in our research spanning over the past ten years, is that moral values play a huge role in us developing strong AI. Contrary to the common belief, responsibly created, the machine has great potential for moral judgement. Unlike us, it is not burdened by millions of generations of ruthless battle for survival.