 |
This part of the thesis concerns an agent-based human-robot interface via the Internet. A user client accessing the AGV and an embedded software within the AGV are both viewed as agents with limited computational and communication resources. To facilitate the communication between the real-time embedded agents and the user interface agents via the communication channel of an uncertain quality, a proxy agent is proposed with the role of a mediator. The functions assigned to the proxy agent target the reduction of inter-agent communication load and minimization of computational resources taken by the embedded agents and user interface agents for communication-related tasks. An XML-based language, RoboML, is designed to serve as a common language for robot programming, agent communication and knowledge representation. The human-robot interface software prototype is developed for the AGV and the proposed techniques are evaluated on the AGV platform. The interface techniques may be applied to other systems requiring access to robotic or similar resources via the Internet.
As applications of robots have been extended to such areas as service and entertainment leading to gradually increasing variations of robot designs and interfaces, the lack of open standards for robot hardware and software implies increasing costs of robot production and unsatisfactory end-user experience. Indeed, the variety and complexity of different hardware and software interfaces in modern robotics may account for some of the disappointments on the pace of adoption of robots in human environments.
An open architecture to provide a common interface at the application programming and hardware levels has been proposed (OpenR [17]). We recognize a need for similar initiatives in the area of human-robot interface (HRI), in particular for AGVs and mobile robots. The wide accessibility and relative simplicity of many Internet-based interface technologies may provide the power that will bridge the gap between humans and robots. This work is concerned with two issues of HRI via the Internet: (i) formulation and utilization of an agent-based architecture and (ii) development of a common markup language for robot programming, agent communication and knowledge representation.
Starting from the early works on Internet-based HRI there is a number of successful implementations of teleoperation interfaces via the Internet (e. g. [49,53]). Weak quality-of-service guarantees over the Internet however limit the applications of Internet-based telerobotics and suggest using more complex interface models, i. e. interface agents that control the robot at a higher level rather than through direct teleoperation, thus relaxing the time constraints imposed by the real-time robotic system. Despite the fact that there are agent applications on robots [29,41] and agent-based interfaces [36,44], few works combine agents embedded in robots and user interface agents in a single agent space [32,50].
In this study we propose an architecture that includes user interface agents (IAs) and embedded agents (EAs) communicating over the Internet. The proxy agent, with functions close to those of communication facilitators [19], is utilized to reduce inter-agent communication load and computational resources needed by the EAs for interface-related tasks. It is assumed that the proxy agent is has much more relaxed restrictions on its computational resources, as opposed to EAs and IAs. The proxy agent dynamically creates and removes links to IAs and EAs, builds and supports representations of the domain ontologies related to the respective IAs and EAs by means of XML Schemas [56], performs translation of inter-agent communication, and provides data journaling to facilitate asynchronous bidirectional communication between IAs and EAs.
A number of task-level programming languages has been developed for robots, including manipulators and mobile robots: TDL (Task Definition Language) [47]; the human-oriented robotic programming language CURL (Cambridge University Robot Language) [26]; reactive robot control languages RPL (Reactive Plan Language) [39], RAP (Reactive Action Packages) [13], ESL (Executive Support Language) [18]; the logic-based action language GOLOG [38]. The need for simplification of robot programming has led to a number of dedicated national projects, e. g. in Japan [2] and in Germany [33]. One way to simplify programming is through utilization of graphical interfaces, e. g. Roboglyph [26], Robot Programming Simplification Project [2], Onika [21]). An alternative (and complimentary) strategy is to simplify the underlying data representation so that its interpretation and editing with the help of relatively simple software or directly by a non-expert is possible. In this work we follow the latter approach.
There have appeared several proposed common languages for agent communication and knowledge representation. The agent communication languages include KQML (Knowledge Query Manipulation Language) [10], FIPA (Foundation for Intelligent Physical Agents) [11], AOP (Agent-Oriented Programming) [46] and Telescript [55]. The formal logic-based (first order predicate calculus) language for expressing the content of a knowledge-base is the KIF (Knowledge Interchange Format) [22].
While these languages are powerful and convenient for expressing the information related to their respective applications, a simultaneous utilization of a robot programming language, agent communication language and knowledge representation language (i. e. the separate languages) in a single application, as it would be necessary for an agent-based HRI, could create significant difficulty for a human operator.
Our goal is to propose a language for the purposes of Internet-based HRI applications that would unify the languages of the three categories considered above while being transparent for a human with minimal training. Namely, the language should:
The rest of the work is organized as follows. Section 1.2 describes the agent-based HRI architecture. RoboML is presented in section 1.3. The proposed HRI model and RoboML are evaluated on an HRI software prototype as shown in section 1.4. Section 1.5 gives our conclusion remarks.
The idea of using facilitator agents to mediate the communication between agents of different domains is a well known agent architecture, e. g. [19]. The applications of HRI, however, impose some specific constraints, i. e. the embedded agent runs in real time and has limited computational resources. To provide the true Internet-based HRI it is also useful to avoid extra assumptions about the computational resources available for the user interface agent. Thus not only the information should be preprocessed to reduce communication load, but as much of the communication-related processing as possible should be performed outside the EA/IA host computers. On the other hand, it is a popular setup of the Internet-based HRI that there is a dedicated (onboard or offboard) computer to host internet services (httpd et al.) with less severe limitations on computational resources compared to EA/IA hosts. In this case it can be advantageous to use this computer for hosting a facilitator agent which would provide some additional services to ease the computational and communication load of the other agents. In this work we call such an agent proxy.
The proxy agent in our model supports a knowledge-base to facilitate the asynchronous communication between IAs and EAs. The knowledge-base is supported by the proxy agent engine and its particular realization is not relevant to the topic of this research (it can be a database or a data repository of a more complex structure). To reduce the inter-agent communication load and to provide a set of data access levels the proxy supports data schemas announcing the data available for subscription and the currently subscribed data for input and output data streams of each IA/EA. These schemas can be viewed as representation of subsets of the domain ontologies [12] corresponding to each of the IAs/EAs. Thus four data schemas for each of the IAs/EAs are hosted by the proxy agent: available data schema (ADS) and subscribed data schema (SDS) for to-agent and from-agent data streams (see fig. 1.1).
The possible scenario of the communication using the proxy agent is as follows. The IAs and EAs send queries and data to the proxy agent in their own domain ontologies. The query can be expressed in the get or subscribe performatives of RoboML (section 1.3), and the generic data transmission is done under the set performative. Query of the type subscribe from agent A updates the to-agent SDS corresponding to agent A and from-agent SDS corresponding to agent B, recruited to provide the necessary information. Query of the type get from agent A may not cause permanent or temporary changes in its to-agent SDS but the request should be announced in the from-agent SDS of agent B, chosen to respond to the query. A message with the set performative from agent A may update its from-agent ADS and the content of the message can be stored in the proxy agent knowledge-base for future use. It may seem to be natural, at least for many applications, to have SDS as a subset of the corresponding ADS. However, we would not like to specify such restriction in the model, allowing a wider range of interaction scenarios. It may be useful, for example, for a user to subscribe to information in advance, thus modifying to-IA SDS even if the information requested is not yet available in the respective to-IA ADS.
Linking of ADS and SDS is represented by the sign of intersection of sets in the figure 1.1 for both to-agent and from-agent streams, assuming that in most cases the data stream is filtered to satisfy both SDS and ADS. However this may not always be true for the from-agent streams, as the proxy agent may choose not to ignore unsubscribed data but to store it for future reference. The mechanisms of advance subscription and of storage of unrequested data can effectively allow asynchronous interaction between IAs and EAs, accounting for the real-time constraints imposed on EAs (and, possibly, IAs).
The syntax of available data schemas and the subscribed data schemas is based upon XML Schemas specification [56] and RoboML (section 1.3). The sample schemas that are created by an experimental HRI application are described in section 1.4.
Our goal is to develop a single Internet-oriented language to account for the major needs of agent-based HRI. We have chosen to use XML as a base for the specification of RoboML, the markup language for robotics. In view of the requirements outlined in section 1.1.2 the choice to build the language using XML is based on the following considerations:
In the rest of this section we shall outline the main concepts of RoboML. For the sake of clarity we omit the language constructs that are not directly related to the vocabulary of RoboML (e. g. the XML document type declarations).
We utilize the KQML model [10] to define the semantics of agent communication elements of RoboML. The following performatives are introduced as elements of RoboML: set as analogous to tell, get as analogous to ask, subscribe as analogous to subscribe of KQML. Each of these elements can have optional attributes sender and receiver (corresponding to the communication layer of KQML model) and ontology (corresponding to the message layer of KQML model), e. g. a RoboML request can be wrapped into tags of the get element as:
<get sender="MyInterface" receiver="AGV1"
ontology="UserInterface">
...
</get>
To express the content of a RoboML message the following alternatives are considered: (a) the content is already expressed in a language which has an XML-based analog, or (b) there is no known XML-based representation of the content. In the former case the known XML-based language can be readily adopted to express the content of the message. To account for the latter case and for the case when the utilization of the vocabulary of another XML-based language may be undesirable, we propose to use a few basic elements for description of the information of robot-related domains.
As noted in [17], the robot's hardware usually can be represented as a semantic tree according to the physical and logical interconnection of the units. Software components and user interfaces can be locally represented by tree structures as well. This representation is particularly suitable for the nested markup structure of XML-based languages. This fact is exploited through utilization of the container elements [59].
RoboML uses container elements to represent basic compound objects within a domain. For example, the following container elements are defined for the Hardware ontology: robot, wheel, motor, sensor, controller, and so forth. These container elements can be further specified by the the name and the ontology attributes. The name is provided to uniquely identify the element within the respective layer of the nesting. Note that a container element may have its own ontology, which makes it possible to describe data of various domains in a single RoboML document. A container element can have a number of children elements, in accordance with the domain ontology.
The actual data should appear within the token elements, or, following the XML Schema terminology, elements that have simple types, i. e. those that cannot contain any subelements or attributes. The tokens specified for Hardware ontology of RoboML are: position, velocity, acceleration, the generic token value, and so forth. Other tokens may be adopted from the respective XML-based languages when other domain ontologies are utilized. A RoboML message from the AGV1 embedded agent to the MyInterface user interface agent specifying the positions of steering motors of wheels 1 and 2 may look like:
<set sender="AGV1" receiver="MyInterface"
ontology="Hardware">
<robot name="AGV1">
<wheel name="1">
<motor name="steering motor">
<position>2577</position>
</motor>
</wheel>
<wheel name="2">
<motor name="steering motor">
<position>-754</position>
</motor>
</wheel>
</robot>
</set>
Well developed formalism of programming languages, including those for robot programming, makes their translation into an XML-based markup language straightforward. Although this is unlikely to produce any advantage for direct manipulation of the code by a human, it allows reuse of parsing, and possibly of other components of the respective agent software by the language interpreter/editor. Simple programming constructs can then be expressed in RoboML between the <program> and </program> tags in the way analogous to using the progn element in KIF [22].
The proposed HRI architecture and RoboML have been evaluated with the software HRI prototype consisting of (a) the user interface agent IA, (b) the proxy agent (in the current version both hosted by an off-board Macintosh computer), and (c) the embedded agent EA running on Harmony RTOS [25] (hosted by a Motorola MVME controller that is embedded in the AGV). At the expense of some extra communication traffic between the EA and the proxy agent, the EA is relieved of any communication processing tasks (i. e. the EA ignores its to-agent ADS and from-agent SDS), except for periodical report/query generation and RoboML parsing.
The communication using the Hardware and UserInterface ontologies has been implemented. To demonstrate the use of XML Schemas for implementation of ADS and SDS we show a sample from-EA ADS generated during the operation of the HRI. The from-EA ADS represents the information available in the from-stream of the EA agent. In this case, it refers to the positions of the steering motors and velocities of the driving motors corresponding to the four wheels of the AGV1 robot (as before, we omit the XML document type declaration, the standard XML Schema wrapping and annotations):
<xsd:element name="set" type="SetPerformative"/>
<xsd:complexType name="SetPerformative">
<xsd:element name="robot" type="Robot"/>
<xsd:attribute name="sender" type="xsd:string"/>
<xsd:attribute name="receiver" type="xsd:string"/>
<xsd:attribute name="ontology" type="xsd:string"
fixed="Hardware"/>
</xsd:complexType>
<xsd:complexType name="Robot">
<xsd:element name="wheel" type="Wheel"
minOccurs="4" maxOccurs="4"/>
<xsd:attribute name="name" type="xsd:string"/>
</xsd:complexType>
<xsd:complexType name="Wheel">
<xsd:element name="motor" type="SteeringMotor"
minOccurs="1" maxOccurs="1"/>
<xsd:element name="motor" type="DrivingMotor"
minOccurs="1" maxOccurs="1"/>
<xsd:attribute name="name" type="xsd:string"/>
</xsd:complexType>
<xsd:complexType name="SteeringMotor">
<xsd:element name="position" type="xsd:decimal"
minOccurs="1" maxOccurs="1"/>
</xsd:complexType>
<xsd:complexType name="DrivingMotor">
<xsd:element name="velocity" type="xsd:decimal"
minOccurs="1" maxOccurs="1"/>
</xsd:complexType>
Assume that the IA requests the information related to the top view of the vehicle. The subscription could be expressed in the UserInterface domain ontology by the respective to-IA SDS. The from-EA SDS, corresponding to this request, would then be a translation of the to-IA SDS into Hardware domain ontology, e. g., it would subscribe for the steering motor positions and would ignore the driving motor data. In this case, the from-EA SDS would look essentially like the from-EA ADS above, except for the definition of the Wheel complex type lacking the declaration of the motor element of the DrivingMotor type (or, equivalently, having in this declaration minOccurs="0" maxOccurs="0").
The further details, including specifications and manual for RoboML and Zvyazok can be found in the project Web site [42].
In order to use Internet as a communication medium for the human-robot interface, it is required to address a number of specific problems: communication between real-time systems via the channel with its relatively low quality-of-service guarantees, limited or uncertain computational resources available for the interface client and the embedded software, compliance with the Internet standards, and compatibility with available software.
This work describes the approach to solving these problems by the utilization of the agent-based HRI architecture and the common XML-based language for HRI applications.
The proxy-mediated HRI model aims at reducing the communication and computational load of embedded and user interface agents and provides means for asynchronous data exchange, accounting for the real-time nature of the embedded agent and, possibly, of the user interface agent. Available data schema and subscribed data schema, expressed in XML Schema definition language, are supported by the proxy agent for each of the data streams to facilitate cross-agent translation and to reduce inter-agent traffic.
The common Internet-oriented language for agent communication, knowledge representation and robot programming is proposed. The XML-based markup language for robotic applications, RoboML, is intended to be used in conjunction with other Internet standards, including other XML-based languages for specific domains.
The proposed interface architecture and the language are implemented in the HRI for the autonomous guided vehicle prototype. The Hardware and UserInterface ontologies are developed for the experimental HRI configuration.
While the proposed methodologies are being successfully tested, we recognize that much more work needs to be done to refine the formal specifications of the proxy-mediated HRI architecture and RoboML. This work is closely related to the development of the parent Internet standards of XML-based languages and protocols [56], as well as to the progress in HRI, agent communication, knowledge representation, and robot programming. Possible further directions of work on RoboML include (a) specification of ontologies for robot control architectures, multimodal user interfaces, and robot programming, and (b) development of the RoboML application support via applets and browser plug-ins.
This document was generated using the LaTeX2HTML translator Version 2K.1beta (1.55)
Copyright © 1993, 1994, 1995, 1996,
Nikos Drakos,
Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999,
Ross Moore,
Mathematics Department, Macquarie University, Sydney.
The command line arguments were:
latex2html -split 0 -no_navigation -no_math -html_version 3.2,math part3_a.tex
The translation was initiated by Maxim on 2001-06-23