Intelligent User Interfaces
Table of Contents
What are Intelligent Interfaces?
Introduction
Many computer programs exist to help the operator perform a task. These programs, of course, require an interface, a means for the user to interact with the program. The interface with which the operator interacts with the program should be as invisible and intuitive as possible -- working with and understanding the interface should not be a task. The operator should be able to concentrate on the task which he is to perform. Intelligent interfaces attempt to make the interface as intuitive and helpful as possible.
What is an intelligent interface? To answer this question, we must first know what an interface is, what intelligence is, and about what the interface can be intelligent. Once we know the answers to these questions, we can formulate a definition for the term "intelligent interface".
What is an interface?
An interface provides a means of communication between two or more entities. For example, the interface between a driver and a car would be the steering wheel, the speedometer, the gas gauge, and other controls and meters on the dashboard. This interface allows the driver to translate his desires into something that the car can understand. The driver gives input: touching the gas pedal, hitting the brakes, or turning the steering wheel. The car responds with output: the current speed, the amount of gas left in the vehicle, or a change in direction. The driver uses the car's output, plus his own motivation, to decide what input to send into the car.
Similar interfaces exist between a user and a program. Specifically, a human- computer interface is the part of a computer system with which a person interects to accomplish some task. It includes both the methods for allowing the user to perform actions, as well as the means by which the computer reports information to the user [ref]. Terveen breaks interfaces into two types: an intermediary interface, in which the user enters information to the interface, which passes that information to the application (like the DOS and UNIX interfaces); and a model world interface, which presents the user with a visual representation of the system, which the user can manipulate directly. This type of interface is used in Windows and similar systems.
What is intelligence?
Intelligence is not an easy term to define. What makes a system intelligent? In intelligent interfaces, the intelligence might be in predicting what the user wants to do, and presenting information with this prediction in mind. Intelligent interfaces can also make doing a task more intuitive and helpful. Instead of trudging along a task in the mire of an inefficient and clumsy interface, the user might find a helpful and information-using interface to be more "intelligent." Thus, intelligence does not actually mean cognition in this context; instead, it means using information in an appropriate manner.
Intelligence in interfacing is a subjective term. One person may look at a system with context-sensitive help and say that the system seems smart; another person might look at the same system and see nothing special about it. In a sense, intelligence in interfaces might be defined as "the next best thing" -- once we have a system which one would say is intelligent, the novelty of the system wears off, and people are in search for more intelligent interfaces. Intelligence is that goal which is always one step ahead of us; once we conquer it, it is no longer intelligence.
What can an interface be intelligent about?
We have established what an interface is, and what intelligence is, but what can an interface be intelligent about?
The interface can be intelligent about a variety of things. One of these is system functionality. The interface might have some knowledge of how to get around the system, or tasks a user would want to do. With this information, the system can present its interface in an intelligent manner, making navigation and operation more intuitive to the user.
Interfaces can be intelligent about the user. Through the use of a user model, the system can tailor communication (both input and output) to the user. Examples of tailored communications include methods of communicating (voice? visual? tactile?) and way of presenting data (bar graph? pie chart? line graph?).
The interface can also be sensitive to the wants and needs of the user. This ties closely with the user model, but it deals more with interface adaptability than outright use of models. One scenario where interfaces might be able to detect the needs of the user would be in the case of a system detecting that the user needs help doing a particular task; the system might volunteer this help. Or, a system may notice that a user is having trouble with the interface; perhaps the system can alter its interface a bit to compensate (for example, maybe the user keeps selecting the wrong button; maybe an alteration to compensate would be to describe the button in more detail, or give more hints as to the correct button to press). This idea can also be brought to a physical level: maybe the system detects that the user keeps missing the selectable zones on a button; the system might compensate by expanding the "hot region" around this button. This would be especially helpful in a system which might be used by someone with poor eyesight or motor control.
The difference between intelligent interfaces and intelligent systems with an interface
There is a distinction between programs with intelligent interfaces, and programs which are intelligent and have interfaces. A program which has an intelligent interface uses intelligent techniques in working with the user. It might use user models, or it might be knowledgable about system functionality, or it might compensate or help its user. An intelligent program may produce information which is intelligent, but the interface itself might not contain any intelligence.
Uses of Intelligent Interfaces
Motivation behind intelligent interfacing
Why do we need intelligent interfaces? There are a couple of good reasons for wanting to create systems with intelligent interfaces:
- Applications are becoming increasingly complex; users may need some guidance on how to use a particular part of a program, especially if the parts in question are rarely used or are confusing.
- Applications are managing a lot of information. Often, there is too much information to display to the user. Techniques for determining what information is the most pertinent for a particular user is important so the user isn't overwhelmed with data.
- As computers become more widely used in the workplace, more non-experts are finding themselves in front of systems which they don't fully understand. The assistance offered by intelligent interfaces might alleviate some problems and misunderstandings between computers and users.
- Computers are being used in an increasing number of special or extreme situations, or are being used by special users. Examples of special or extreme situations include military programs, medical software, and programs in high-stress environments (such as distaster management). Intelligent interfaces would be needed in these situations because they can better provide for the user through the use of multimodal communication and knowledge of system functionality. Special users include primarily those with conditions that would otherwise prohibit them from effectively using a computer, such as people with visual or motor afflictions. Through multimodal communication and interface adaptability, intelligent interfaces can help make computers accessible to these users.
Existing examples of intelligent interfaces
There are quite a few programs which use intelligent interfaces. Some of the more significant of these (those systems which are in actual use, as opposed to those written for the sole purpose of demonstrating IntInt techniques) include the following:
- CUBRICON. This is a system used for Air Force command and control. It incorporates speech input and output, natural-language text, graphics, and pointing gestures by the user [ref]. The objective of using IntInt techniques in this system is to "simplify operator communication with sophisticated computer systems."
- SAGE. This program creates intelligent data-graphics. The intelligent interfacing is in determining the best way to display a set of data so that the data is readable and understandable. [ref]
- CHORIS. This system is designed to enable a wide range of users to interact effectively with varying types of complex systems [ref]. Its main strength lies in its knowledge of these complex systems.
- UCEgo, the intelligent component of UNIX Consultant. This is a natural language system which helps the user solve problems encountered while using UNIX [ref].
In each of these instances, intelligent interfacing techniques make the programs more intuitive, more helpful, and more usable.
Components of Intelligent Interfaces
About Components of Intelligent Interfaces
Intelligent interfaces possess one or more of the following components. These components are discussed, and examples are given to further illustrate the principles involved.
The User Model
The user model is one of the most important parts of an intelligent interface.
Without it, the interface would not have any information on which to base
the design and individuality of the interface. The user model is a
compilation of information which describes the user, and
which is used in determining how to present data, what type of help to
give, and how the user interacts with the interface. Therefore, user models
are beneficial in systems with the following characteristics: [ref]
- The system seeks to adapt its behavior to users
- The system assumes or shares responsibility for ensuring the success of user-system communication
- The class of potential users, or potential uses of the system, is diverse
- Getting input from the user
- Deciding what to say
- Deciding how to ask
- Interpeting responses
- Resolving ambiguity
- Interpreting referring expressions
- Providing help and advice
- Evaluating relevance
- Knowing when to volunteer information
- Handling misconceptions
- Recognizing misconceptions
- Correcting misconceptions
- Understanding the user's information-seeking behavior
- Recognizing user goals
- Recognizing user plans
- Providing output to the user
- Deciding what to say
- Providing prerequisite information
- Modeling relevance
- Deciding how to say it
- Constructing referring expressions
- Lexical choice
- Deciding what to say
The user model is composed of a variety of facts about the user. Depending on the system which is being used, the model may contain any type of applicable fact, such as these:
- Stereotype: what job the user holds
- Expertise
- Preferences
- Fatigue and stress
- Age
- Education
- Familiarity with computers
- Medical conditions
- Availablility of person's attention (can the user actually look at the screen, for example, or should the system speak output instead of displaying it?)
Multimodal Communication
The use of various methods of communication with an interface is referred to as multimodal communication. Examples of multimodal include natural language entered through type or speech, computer speech, and the ability to point to a spot on the screen and refer to it ("place an aircraft carrier here [point]").
In the CUBRICON system [ref], a system used for Air Force command and control, the user is presented with a multimodal interface which allows gesturing, tactile, and visual interaction to effectively display and accept information. The particular combination of multimodal techniques in CUBRICON makes it an effective system.
There are at least two purposes behind multimodal communication: it enables the user to use the system more intuitively by using gestures, and it gives users more freedom, so they are not restricted to sitting in front of a computer as they work on a computer system.
Multimodal communication is especially important in situations where the user cannot afford to use all of his senses towards a particular task. For example, if the user has to be looking closely at something, a computer system which produces visual output would not be a good idea. Instead, the computer system might be made to speak the output instead of displaying it. As a result, the user is able to examine something without paying visual attention to the computer.
In many cases, an effective multimodal interface requires a user model. In this case, the user model helps to determine the best communication techniques to use depending on the user's task: for example, whether output should be vocal or visual, whether input should be tactile or vocal or typed.
Plan Recognition
Plan recognition is used in an intelligent interface to deduce what the user plans to do. It takes system knowledge, the user model, and the user's actions into consideration. This helps the user by providing guidance, requiring less repetition (as the case may be if the system begins to recognize a frequently entered sequence of commands), and understanding what the user wants to do. For example, a user of a fire station dispatch program might send the same emergency vehicles to certain types of fires. The system might start to pick up on this. Eventually, the system might say the equivalent of, "I notice that for 2-alarm fires that you always send Engine-1 and Rescue-5. Would you like to automate this process?" Then, when the system sees the user dispatching for a 2-alarm fire, it would offer to send Engine-1 and Rescue-5 automatically.
Plan recognition can also be used to predict the user's course of action, and may provide guidance along the way. For example, a system to aid in computer repair might automatically suggest checking particular chips if the system recognizes a correlation between other circuits which have just been analyzed by the user.
Dynamic Presentation
Different people should be able to view data in different ways which would make that data more understandable to the individual users. One way in which the system decides to display data is determined by examining the user model. This allows for the display of data which is tailored to the user. The user should not be overwhelmed by too much data, nor underinformed by too little data. The presentation of data should be done in an effective and understandable way.
For an example of intelligent data display, consider a fictional probe which could be sent deep into a jungle. There are a lot of things to look at in a jungle, creating far too much data for one person to comprehend. The probe would send back data to a computer system, which would split up the data into parts that would be understandable to individual users (in this case, scientists). An entomologist would receive data about jungle insects; a botanist would receive data about plants. Even these can be broken down further: there might be an entymologist who is specifically researching the social behavior of ants, while another might be looking into the life cycle of butterflies. While each individual scientist is receiving information about his particular discipline, keep in mind that in the jungle, everything is very intertwined. The behavior of pollinating insects might be very important to the development of a type of plant, for example. The system could allow the user to delve into another scientist's field: perhaps the entomologist could call up data on jungle botany.
Even without crossing scientific fields, information would have to be presented in an intelligent manner. With so much data coming in, it would be important to have this data being displayed in the most comprehendable format possible. Thus, the system might know to show the changing population of ants in a line graph, while showing the percentage of ants engaged in different tasks (foraging for food, defending nest, building nest, caring for young) as a pie chart. The dynamic display of data should also be sensitive to the user model; for example, if the user finds that having two different charts is confusing or not informative enough, he may prefer to have the population and task charts combined into a single chart in which these data are combined:
Alternatively, dynamic presentation may be attained by presenting data in an intelligent manner -- that is, in a way which makes the facts of the graphics obvious and clear. This type of system was explored in the SAGE project [ref]. SAGE is capable of producing intelligent data graphics. The graphics it designs are not output from a system; they are drawn from data entered into the system for the purpose of obtaining the most clear and informative graph possible. See Minard's chart of Napoleon's 1812 Campaign.
Natural Language
Natural language is one of the best techniques for making a system more intuitve. Instead of trying to remember commands, the user enters what he wants to happen. Examples include, "rotate the crane arm 45 degrees," or, "what does the ls command do?" [ref]
A wide variety of programs which can use natural language, specifically those which would be cumbersome if the user was faced with a menu interface. Natural language commands allow for a high degree of freedom on the user's part. Commands do not have to be explicitly listed or categorized in a menu system, and it is easier to create commands which are composed of other commands (such as the non-natural "more filename | grep keyword", which could be stated in natural language as "show all instances of keyword in filename").
Intelligent Help
In complex systems, asking for help may bring up more information than is necessary, or information which is not specific to the user's neeeds. Intelligent help presents the user with help that the user would most likely need at a particular time, or in a particular situation. Suppose a user is using a system to aid the user in building a house. The user might be having trouble sawing through a support beam. If the system knows that the user is sawing through a beam, it should know what type of saw would be best for cutting through the beam, and make a suggestion to change saws. As another example, consider a program to help a gardener. The gardener may want help as to why his cucumbers aren't growing. The system might know that there has been little rainfall recently, and suggest that this may be the cause. These uses of intelligent help require knowledge about application functionality.
Intelligent help can also be used when the user calls up 'help' on a system. The system may recognize that the user is having trouble with a task, like printing a paper. The help system may suggest some things that the user can do: see if the printer is set up correctly, if it is plugged in, and so on.
Some plan recognition can also be used as intelligent help. In a help system for UNIX, the user might ask, "what does 'ls -v' do?" [ref]. The system could respond, "ls lists the files in your current directory," ignoring the fact that ls doesn't have a -v option. Or, the system could say, "There is no -v option for ls," which does not answer the user's question, but it does clear up the user's misconception.
Interface Adaptability
Users may desire a degree of preference in their interface. Also, the system may adapt itself to interact better with the user without the user telling it to do so. An example of this would be if a user keeps pressing the mouse button when the pointer is close to, but not on, a button, the system might expand the "hot" boundries of the button.
Adaptive interfaces can also determine what type of interface to present to the user depending on an analysis of the user model [ref]. For example, one type of user might feel more comfortable with a menu, while another user might like the flexibility of a command-line interface.
Adaptation is a step in response planning, which is the process in which the program determines what feedback and abilities should be given to the user depending on the user model, thereby producing interfaces which would be prefered by the user [ref].
Concluding Remarks
General Discussion of Intelligent Interfaces
Although there is a great variety of techniques which may be used in intelligent interfacing, a general diagram can be drawn for a system containing intelligent interfaces.
The following is a basic model of an IntInt system:
Locations of intelligent interfacing:
- User Input Interpreter. At this part of the system are functions to make inputting easier for the user. Examples of functions which would go here include tab completion of commands, spell checking, interpretation of abbreviations, detection of mouse clicks, and so forth.
- Application Input Interpreter. Here is where input is taken and made into an undestandable form by the program. For example, if the user has entered a command, this part of the system would interpet that command. Plan recognition can take place here as well
- Inside the program. This is where inner workings of the program take place: creation of data, doing actions specified by commands, preparing output, and so on.
- Application Output Interpreter. This part of the system would be responsible for making the data created by the program into a form which would be most effective when given to the user.
- User Output Interpreter. Once the system determines what data to display and how to display it, this part of the system displays the information.
Conclusion
Applications are becoming more complex, and more people are finding themselves using computers. The interface between the user and the program must be developed to compensate for these changes. The progress made in intelligent interfaces is promising. As interfaces continue to improve, users will be able to use complex systems more effectively.