I have recently re-read Michael Frayn’s The Tin Men, a superb satire of AI. Among the many wonderful passages is this, on the semantic verification problem of agent communications:
“Ah,” said Rowe, “there’s a difference between a man and a machine when it comes to praying.” “Aye. The machine would do it better. It wouldn’t pray for things it oughtn’t pray for, and its thoughts wouldn’t wander.”
“Y-e-e-s. But the computer saying the words wouldn’t be the same . . .”
“Oh, I don’t know. If the words ‘O Lord, bless the Queen and her Ministers‘ are going to produce any tangible effects on the Government, it can’t matter who or what says them, can it?”
“Y-e-e-s, I see that. But if a man says the words he means them.”
“So does the computer. Or at any rate, it would take a damned complicated computer to say the words without meaning them. I mean, what do we mean by ‘mean’? If we want to know whether a man or a computer means ‘O Lord, bless the Queen and her Ministers,’ we look to see whether it’s grinning insincerely or ironically as it says the words. We try to find out whether it belongs to the Communist Party. We observe whether it simultaneously passes notes about lunch or fornication. If it passes all the tests of this sort, what other tests are there for telling if it means what it says? All the computers in my department, at any rate, would pray with great sincerity and single-mindedness. They’re devout wee things, computers.” (pages 109-110).
Michael Frayn [1995/1965]: The Tin Men (London, UK: Penguin, originally published by William Collins, 1965)