jnd.org

ORIGINALLY PUBLISHED IN SOFTWARE AGENTS,J. BRADSHAW, ED¹

Agents occupy a strange place in the realm of technology, leading to much fear, fiction, and extravagant claims. The reasons for this are not hard to find: the concept of an “agent,” especially when modified by the term “intelligent,” brings forth images of human-like automatons, working without supervision on tasks thought to be for our benefit, but not necessarily to our liking. Probably all the major software manufacturers are exploring the use of intelligent agents. Myths, promises, and reality are all colliding. But the main difficulties I foresee are social, not technical: How will intelligent agents interact with people and perhaps more important, how might people think about agents?

Automata are not new concepts. Intelligent machines have existed in fact or fiction for centuries. Perhaps the most relevant predecessors to today’s intelligent agents are servomechanisms and other control devices, including factory control and the automated takeoff, landing, and flight control of aircraft. The new crop of “intelligent agents” are different from the automated devices of earlier eras because of their computational power: they have Turing-machine powers, they take over human tasks, and they interact with people in human-like ways, perhaps with a form of natural language, perhaps with animated graphics or video. Some agents have the potential to form their own goals and intentions, to initiate actions on their own without explicit instruction or guidance, and to offer suggestions to people. Thus, agents might set up schedules, reserve hotel and meeting rooms, arrange transportation, and even outline meeting topics, all without human intervention. Other, more complex interventions in human activities are contemplated. These human-like activities and characteristics are what lead to the special concern over today’s agents. Moreover, today’s agents are simple in comparison to those that are being planned.

To ensure a smooth introduction of this technology, two major themes are relevant. One theme deals with the way people feel about agents, the other with comfort and acceptance of their automatic, autonomous actions.

• Ensuring that people feel in control of their computational systems;
• The nature of the human-agent interaction;
• Built-in safeguards to prevent runaway computation;
• Providing accurate expectations (and minimizing false hopes);
• Privacy concerns (a subset of the feeling of control problem);
• Hiding complexity while simultaneously revealing the underlying operations.

The Feeling of Control

One of the first problems to face is that of the person’s feeling of control. An important psychological aspect of people’s comfort with their activities — all of their activities, from social relations, to jobs, to their interaction with technology — is the feeling of control they have over these activities and their personal lives. It’s bad enough when people are intimidated by their home appliances: what will happen when automatic systems select the articles they should read, determine the importance and priority of their daily mail, and automatically answer mail, send messages, and schedule appointments? It is essential that people feel in control of their lives and surroundings, and that when automata do tasks for them, that they are comfortable with the actions, in part through a feeling of understanding, in part through confidence in the systems.

Confidence comes slowly, and the track record of existing automation does not lead to much optimism. Thus, the first introduction of automation into automobiles was met with resistance: the automatic spark advance, the automatic choke, the automatic transmission all took decades before they were accepted. The record of automation in the process control and transportation industries has not been good, with numerous failures. Although commercial airplanes can take off, navigate, and fly under automatic control, most passengers — including me — would not fly in a pilotless plane: indeed, we require two pilots partially because of workload, but partially as a spare: if one pilot becomes incapacitated, the other can take over. Pilots are not comfortable with all the automation: they feel “out of the loop” [6].

Mind you, the lack of confidence in automation is justified. We have a poor track record in developing large scale, complex systems. Systems do run amok (see, for example, [1] ). Agents pose especially complex technical questions because they are intended to be autonomous processes, sometimes capable of migrating across networks and processors in the most complex of asynchronous, autonomous, distributed processing environments. Moreover, each agent is apt to be created independently of the others, often in ignorance of the existence of others, so conflicts, contradictory actions, and synchronizing problems are bound to occur.

Two things are necessary to make all this technology acceptable: one is technical, the other social. The technical aspect is to devise a computational structure that guarantees that from the technical standpoint, all is under control. This is not an easy task. The social part of acceptability is to provide reassurance to the user that all is working according to plan. The best way to do this is through developing an appropriate conceptual model of the actions, one in which the actions of agents are understood in context so users feel comfortable in their ability to find out what actions have been taken in their behalf, that private matters remain private, that expensive or unwanted actions will not be taken without explicit permission, and that it is always possible to trace back the actual sequence of acts and undo any that are seen as unwarranted. This is a non-trivial task. We don’t yet know how to do this, and to a large extent, the amount of information and explicit control that has to be provided is a function of the state of the individual’s own comfort level: this will change over time, both for individuals and for society.

Probably, in the early days, agents will have to make their presence and actions known through a conceptual model of the underlying operations and then, through graphics, sound, and appropriately chosen verbal messages, provide a continual updating of the conceptual state. As reliability increases, so too will people’s comfort and acceptance. The user should therefore be able to change the amount and form of feedback, decreasing to some minimal level. I imagine that at first, people may want to know all the actions taken for them, but after they have come to trust the actions, they will be annoyed to have complete reporting. Nonetheless, I suspect there will always be the need to have the potential for such complete reports, even if they are seldom requested. Just as many people wish to be able to review their bank records each month, even though they seldom find an error, I suspect people will always want to be able to know about the actions of their agents.

Overblown Expectations

If the one aspect of people’s attitudes about agents is fear over their capabilities and actions, the other is over-exaggerated expectations, triggered to a large extent because much more has promised than can be delivered. Why? Part of this is the natural enthusiasm of the researcher who sees far into the future and imagines a world of perfect and complete actions. Part of this is in the nature of people’s tendency to false anthropomorphizing, seeing human attributes in any action that appears in the least intelligent. Speech recognition has this problem: develop a system that recognizes words of speech and people assume that the system has full language understanding, which is not at all the same thing. Have a system act as if it has its own goals and intelligence, and there is an expectation of full knowledge and understanding of human goals.

The problem is amplified by the natural tendency of researchers and manufacturers to show their agents in human form. You can imagine the advertisements: “Want to schedule a trip, the new MacroAgent System offers you Helena, your friendly agent, ready to do your bidding.” As soon as we put a human face into the model, perhaps with reasonably appropriate dynamic facial expressions, carefully tuned speech characteristics, and human-like language interactions, we build upon natural expectations for human-like intelligence, understanding, and actions.

There are some who believe that it is wrong — immoral even — to offer artificial systems in the guise of human appearance, for to do so makes false promises. Some believe that the more human-like the appearance and interaction style of the agent, the more deceptive and misleading it becomes: personification suggests promises of performance that cannot be met. I believe that as long as there is no deception, there is no moral problem. Be warned that this is a controversial area. As a result, it would not be wise to present an agent in human-like structures without also offering a choice to those who would rather not have them. People will be more accepting of intelligent agents if their expectations are consistent with reality. This is achieved by presenting an appropriate conceptual model — a “system image” [2] — that accurately depicts the capabilities and actions.

Safety

Safety plays a part in the feeling of control: making sure that the agent does not do things that would jeopardize the physical, mental, or monetary well-being of the owner. But how can this be guaranteed when intelligent agents might enter one’s system from outside? Sometimes one won’t even know, as when they arrive in the mail, or are parts of some new set of capabilities being added to the computational system. How does one guard against error, maliciousness (as in the spread of computer viruses), and deliberate intent to pry and probe within one’s personal records?

Privacy

Privacy could be considered a subset of the sense of control, but because the technical and social implications are considerably different, it deserves its ownspecial consideration. Privacy is a complex topic, one deeply rooted in human cultural and legal systems. The concerns for privacy within the United States are not necessarily mirrored in the rest of the world, nor for that matter, even in the prior history of the United States.

Privacy often pits the interests of one group against another: the right of citizens to know what their government is doing; the right of one family to know what its neighbors are doing; the right or necessity of a government or person to keep its activities private and confidential.

Law enforcement has a need to be able to detect illegal actions: citizens have a right to be free from unwanted surveillance. Citizens do not trust their fellow citizens, industry, police, or government to use information about their activities in legitimate, beneficial ways. Business feels it can be more efficient and helpful the more information it has about the desires and behavior of its customers.

Not all the need for privacy is to avoid the detection of wrong-doing. White lies and other deceptions are an essential, positive aspect of social interaction, allowing for smoother, friendlier social discourse. Sometimes we want to protect a self image. Sometimes we simply want to be removed from the hustle and bustle of modern communication — note the increasing prevalence of unlisted telephone numbers.

The issues are too complex to be given full treatment here. However, the idea that autonomous, intelligent agents could have access to personal records, correspondence, and financial activities is disturbing to many individuals, no matter how helpful the agents might be. Moreover, as the ability to imbed agents within electronic mail messages becomes more prevalent, who will be comfortable with the mail systems? Any mail message might release agents that search the recipient’s records and return confidential information to the sender. I have already seen one such system demonstrated, and although it was shown with benign intent, where the agent “requested” permission before searching the recipient’s address book and returning the information to the sender, it was easy to imagine other situations. Suppose the request was deceptive, with what was asked for differing from what was done.

Privacy and confidentiality of actions will be among the major issues confronting the use of intelligent agents in our future of a fully interconnected, fully communicating society. We must address those issues now, not just in the technical sense, but in the local, national, and global legal systems.

Human-Agent Interaction

What is the appropriate form of interaction between agent and person? The question has many different components, including how the person shall instruct and control the agent, the nature of the feedback from agent to person, the manner by which the person’s conceptual model of the agent’s method of operation and activities is presented, and the manner by which the agent offers advice and information to the person.

Take the problem of instruction: programming the agent. This is a complex issue. Various suggestions exist, from having the agent instruct itself by watching over people’s activities and deciding how it can offer help, to instruction “by doing”: “watch what I do,” says the person,” “and then do it for me.” Other suggestions include the development of simple programming languages (for example, [5] , this issue), some graphical, some declarative.

None of these seem satisfactory. The kinds of activities we assume will be performed by agents are quite complex. Scheduling events or ordering and making payments involve temporal relationships with other activities, distributed in space and time, not under control of the agent. Asynchronous coordination is not a simple task domain. The profound difficulty of programming complex tasks is well known by professional programmers.

Concluding Remarks

Agents are here to stay: once unleashed, technologies do not disappear. Agents may well have numerous positive contributions to our lives. They can simplify our use of computers, allowing us to move away from the complexity of command languages or the tedium of direct manipulation toward intelligent, agent-guided interaction. Agents offer the possibility of providing friendly assistance, so smoothly done that users need not even be aware, much as the modern automobile controls parts of the engine that used to require human intervention — e.g., the spark advance, choke, shifting — most of us are delighted to forget these things. Agents promise to hide complexity, to perform actions we could not or would rather not do ourselves. And agents could add to human intelligence, adding one more tool to the domain of cognitive artifacts that indeed do make people smarter [3, 4] .

But along with the promise comes potential danger. Agents are unlike other artifacts of society in that they have some level of intelligence, some form of self-initiated, self-determined goals. Along with their benefits and capabilities come the potential for social mischief, for systems that run amok, for a loss of privacy, and for further alienation of society form technology through increasing loss of the sense of control. None of these negative aspects of agents are inevitable. All can be eliminated or minimized, but only if we consider these aspects in the design of our intelligent systems.

Footnote

1. Published as: Norman, D. A. (1997). How might people interact with agents. In J. Bradshaw (Ed.), Software agentsReturn to text]

References

1. Neumann, P. (1995). Computer-related risks. Reading, MA: Addison-Wesley.

2. Norman, D. A. Cognitive engineering. In User centered system design, D. A. Norman and S. W. Draper Ed. Lawrence Erlbaum Associates, Hillsdale, NJ, 1986.

3. Norman, D. A. Cognitive artifacts. In Designing interaction: Psychology at the human-computer interface, J. M. Carroll Ed. Cambridge University Press, NewYork, 1991.

4. Norman, D. A. Things that make us smart. Addison-Wesley, Reading, MA, 1993.

5. Smith, D. C., Cypher, A. and Spohrer, J. KidSim: Programming agents without a programming language. Communications of the ACM . This issue.

6. Wiener, E. L. Cockpit automation. In Human factors in aviation, E. L. Wiener and D. C. Nagel Ed. Academic Press, Orlando, FL, 1988.

Acknowledgments

I thank Julie Norman, Tom Erickson, Harry Saddler and Pavel Curtis for critical readings and helpful suggestions.