IVOA

CEA Interfaces: How to Invoke Applications in the Common Execution Architecture
Version 1.0

IVOA Working Draft 30 October, 2007

This version:
http://www.ivoa.net/Documents/WD/GWS/CEAInterface-XXXX.html
Latest version:
http://www.ivoa.net/Documents/latest/CEAInterface.html
Previous versions
Authors:
Paul Harrison, Editor
and the IVOA Grid and Web Services Working Group.

Abstract

This document describes the Common Execution Connector web service interface that may be used to invoke an application remotely that conforms to the Common Execution Architecture.

Status of this document

This is an IVOA Working Draft for review by IVOA members and other interested parties. It is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use IVOA Working Drafts as reference materials or to cite them as other than "work in progress."

Parts that the editor considers should be removed from the document are marked in red with a strike through line, and parts to be expanded are in green

Comments on this document are due 15 December 2006 for consideration in the next version of this document. They should be sent to registry@ivoa.net, a mailing list with a public archive or on the VOApplication twiki discussion page. General discussion of related technology is also welcome on the Rwp03 wiki site.

A list of current IVOA Recommendations and other technical documents can be found at http://www.ivoa.net/Documents/.

Acknowledgements

This document has been developed with support from the UK Particle Physics and Astronomy Research Council (PPARC), and from the European Commission's Sixth Framework Program via the VOTech project

Conformance-related definitions

The words "MUST", "SHALL", "SHOULD", "MAY", "RECOMMENDED", and "OPTIONAL" (in upper or lower case) used in this document are to be interpreted as described in IETF standard, RFC 2119 [RFC 2119].

The Virtual Observatory (VO) is general term for a collection of federated resources that can be used to conduct astronomical research, education, and outreach. The International Virtual Observatory Alliance (IVOA) is a global collaboration of separately funded projects to develop standards and infrastructure that enable VO applications.

Syntax Notation Using XML Schema

The eXtensible Markup Language, or XML, is document syntax for marking textual information with named tags and is defined by the World Wide Web Consortium (W3C) Recommendation, XML 1.0 [XML]. The set of XML tag names and the syntax rules for their use is referred to as the document schema. One way to formally define a schema for XML documents is using the W3C standard known as XML Schema [Schema].

This document defines the interfaces to CEA servers using Web Services Description Language (WSDL) [WSDL] and schema using XML Schema. The full WSDL and Schema documents are listed in the Appendices . The WSDL files additionally import schema that were defined in the CEAApplication Data Model standard [CEAAPP].

Note that as this document represents a draft the cea: and ceab: namespaces in attached schema and instance documents have the string "rc<n>" appended where "rc" stands for release candidate and <n> is an integer representing the iteration number of the release candidate draft of the associated namespace.

Contents

Introduction

The concept of a Common Execution Architecture (hereafter CEA) was introduced in [CEA]. This document represents a formal publication of the interfaces to a service that provides the execution capability of applications that can be described by the CEA application model that is now defined in a separate standard [CEAAPP]. This document defined the Common Execution Connector SOAP web service that was described initially in [CEA], as well as a new [REST] interface, both of which are conformant with the IVOA Universal Worker Service [UWS] pattern. The fact that many various kinds of application can be exposed with a single web service definition and application model is an important feature of CEA, as it allows universal server and client side software facades to be created that hide the details of the web service implementation, which in turn simplifies the adoption into existing frameworks such as workflows.

Design

The rationale behind creating a Common Execution Architecture have been expounded in [CEA] and [UWS] in detail, so here only the most important features will be highlighted.

Relationship to UWS

The Universal Worker Service [UWS] pattern was originally inspired by the early CEA implementation within the Astrogrid project, which provided a pattern for asynchronous interaction and data staging with remote services using a SOAP web service interface. The UWS pattern attempts to abstract these ideas slightly further so that they might be reapplied in other services, and as such exists only as a document, there are no concrete 'pure' UWS implementations. The UWS pattern has had a separate track of development, notably the addition of a REST profile, which in-turn is being fed-back into this CEA standard.

Interactions

The interaction between the CEA client and the CEA server follows a similar pattern for both the interfaces. There are some extra intermediate steps that may be called between these steps, but the steps listed below are the essential interactions that can be performed on either interface for the complete execution of a job and retrieval of the results.

  1. Client requests that a new "job" be created passing a <tool> element that describes the application to run and the parameters to pass to it. The Client receives a job identifier in return that it can use to identify this particular job in subsequent calls.
  2. Client requests that the job be started.
  3. Client repeatedly inquires about the status of the job until it is informed that the job is finished.
  4. Client retrieves the results from the job.

The job goes through several phases during this interaction. The first 4 phases that are listed are what occurs for normal successful execution.

PENDING
The initial phase when the client has just requested the job to be created, and the parameters are being validated.
QUEUED
The job has been validated and accepted, but no request to start the job has yet been received
EXECUTING
The job is running, after the signal to start has been given.
COMPLETED
The job has successfully completed.
ERROR
An error of some kind has occured
UNKNOWN
The job is in an unknown state - no clear error signal has been received, but nor has it completed within an expected time.
SUSPENDED
The job has been requested to be executed by the client, but the server has not yet started the job, perhaps because of lack of available resources. (cf QUEUED). The job will be automatically started when resources become available without client intervention.
HELD
The job has been executing, but the server has suspended the job. The job will not be restarted without client intervention.

Interfaces

There are two possible interfaces to a CEA server using either Simple Object Access Protocol [SOAP] or "Representational State Transfer" [REST] technologies. A CEA server must implement one of these interfaces and should ideally implement both so that a CEA client has a choice to use whichever is most convenient in the given situation. These two interfaces are described in more detail below.

Common Execution Connector SOAP

The SOAP interface that implements the UWS pattern in CEA is called the Common Execution Connector and is described by the WSDL contract in Appendix A. The CEA SOAP interface adds a "call-back" feature to the UWS pattern, that means that the client does not have to continually poll the server to obtain job status, but instead can register to be "called-back" when the job status has changed. In large grids this can be beneficial as it cuts down on network traffic and cpu cycles compared with polling. There are two such call backs that can be registered.

  1. A progress listener, which can be used to receive messages about the current status of the job. This is registered with the registerProgressListener() operation of the CommonExecutionConnector port. The operation expects the endpoint of a server that implements the CEAJobMonitor WSDL contract (see Appendix C) which implements a one.
  2. A results listener, which can receive the results when the job has completed. This is registered with the registerResultsListener() operation of the CommonExecutionConnector port. The operation expects the endpoint of a server that implements the CEAResultsListener WSDL contract (see Appendix D).

A typical interactaction with the CommonExecutionConnetor interface is illustrated in the following UML diagram.

CommonExecutionConnector UML sequence diagram

Each operation in the CommonExecutionConnector interface is described in more detail in the following sections.TBC

createJob

The job is specified by passing a tool document.

Note on output parameters. The output parameters need not usually be sent of course, as they are output parameters! However, to instruct the CEA server to automatically stage the output to somewhere the output parameter should be sent with the indirect attribute set to true and the value set to the URI of the location where it is desired that the output should be sent. If this is not done the CEA server will attempt to return results in-line in the getResults SOAP call.

registerProgressListener

This allows the CEA client to register the end-point of a service that progress log messages may be sent to. This does not stop a client also requesting similar messages by polling with a getPhase call. The frequency of messages that are sent is left as an implementation detail for the CEA server, but it would be useful to sent messages when for instance;

registerResultsListener

Allows the CEA client to register the end-point of a service that the results can be sent to when ready. The client may also request results explicitly by using the getResults call.

getResults

Request the results from an execution. These will be returned in a ceat:result-list-type structure, which may return results

delete

Request that the job be deleted. If the job is currently running then it will be aborted and the record of the job deleted from the CEA server. If the job has already completed then this call will delete any record of the job from the CEA server and free any resources associated with the job.

getPhase

Returns the execution phase of the job, along with some associated metadata about the current status of the job.

getExecutionSummary

Returns a detailed summary of the job including the input parameters and the results.

setTerminationTime

Sets the time at which the CEA server will automatically delete the record of the job and also reclaim any resources associated with the job. After this time it is not possible any longer to retrieve any metadata about or results from the job. Any results that have been staged to another storage location will of course not be deleted.

UWS-PA REST (preliminary)

A REST interface to CEA is described in the table below. The operation names are the same as for the corresponding operation in the SOAP interface. The namespace prefixes are

prefix namespace schema file
ceab http://www.ivoa.net/xml/CEA/base/v1.0rc2 located in appendix C of [CEAAPP]
ceat http://www.ivoa.net/xml/CEA/types/v1.0rc1 located in Appendix E of this document.

 

Operation URI HTTP Method Example Input Example Output
createJob /jobs POST

<ceat:tool id="ivo://dummy" interface="test">
<input>
<parameter id="p1">
<array>
<value>1</value>
<value>2</value>
</array>
</parameter>
<parameter id="p2">
<value>hello</value>
</parameter>
</input>
<output>
<parameter id="out" indirect="true">
<value>vos://org.test!vospace/mydata</value>
</parameter>
</output>
</ceat:tool>

<ceat:jobId>1234</ceat:jobId>
execute /jobs/{jobId}/phase POST
<ceat:phase>EXECUTING</ceat:phase>
<ceat:phase>QUEUED</ceat:phase>
getPhase /jobs/{jobId}/phase GET NONE
<tns:getPhaseResponse>
<ceat:phase>QUEUED</ceat:phase>
<ceat:content>Job parameters valid</ceat:content>
<ceat:level>info</ceat:level>
<ceat:source></ceat:source>
<ceat:timestamp>2007-05-01T12:00:00</ceat:timestamp>
</tns:getPhaseResponse>
getResults /jobs/(jobid)/results GET NONE
<tns:getResultsResponse>
<!-- a standard indirect result that the CEA server has paced in VOspace -->
<ceat:result id="output" indirect="true"><value>vos://org.test!vospace/mydata</value></ceat:result>
<!-- a direct result that is retuned in-line -->
<ceat:result id="mean"><value>45.32</value></ceat:result>
<!-- an indirect result that is returned as a sub uri of the main results uri. -->
<!-- would be nice to be able to use a xlink:href-->
<ceat:result id="image" indirect="true"><value>./image</value></ceat:result>
</tns:getResultsResponse>
getExecutionSummary /jobs/(jobId) GET NONE
<tns:getExecutionSummaryResponse>
<ceat:application-name>ivo://dummy</ceat:application-name>
<ceat:jobId>1234</ceat:jobId>
<ceat:input-list>
<ceat:parameter id="p1">
<array>
<value>1</value>
<value>2</value>
</array>
</ceat:parameter>
<ceat:parameter id="p2">
<value>hello</value>
</ceat:parameter>
</ceat:input-list>
<ceat:result-list>
<!-- a standard indirect result that the CEA server has paced in VOspace -->
<ceat:result id="output" indirect="true">
<value>vos://org.test!vospace/mydata</value>
</ceat:result>
<!-- a direct result that is retuned in-line -->
<ceat:result id="mean">
<value>45.32</value>
</ceat:result>
<!-- an indirect result that is returned as a sub uri of the main results uri. -->
<!-- would be nice to be able to use a xlink:href -->
<ceat:result id="image" indirect="true">
<value>./image</value>
</ceat:result>
</ceat:result-list>
<ceat:phase>COMPLETED</ceat:phase>
<ceat:startTime>2007-05-01T12:30:21</ceat:startTime>
<ceat:endTime>2007-05-01T13:17:11</ceat:endTime>
<ceat:termination>2007-07-01T00:00:00</ceat:termination>
</tns:getExecutionSummaryResponse>
delete /jobs/(jobId) DELETE/POST NONE NONE

 

Differences from UWS 0.3 Draft implementation [UWS]

The REST interface for CEA follows [UWS] except where noted below;

Chosing between the SOAP and REST interfaces.

There factors involved for a CEA client in making a choice between using either the SOAP or the REST versions of the interface are;

Registering CEA Services

The registration of CEA components is covered in detail in [CEAAPP], however it is worth noting some specific points here about service endpoints.

Security

CEA implementations generally follow the security recommendations of the IVOA GWS group [SSO]. Specifically for the SOAP version of the interface, message level security using WS-Security is implemented, and for the REST version transport layer TLS security is used. At a minimum CEA servers should check client authentication via these means. A CEA server should publish it's authorization policy, but as a minimum policy the server should allow read and write access only to the creator of a job.

CEA as a Meta-Standard

This standard together with the CEA Application Model standard [CEAAPP] provide a complete description of how to describe and invoke a "parameterized application" utilizing agreed IVOA patterns [UWS]. It is true that many other IVOA standards e.g. SIAP, SSAP, TAP, SNAP also fall into this general category, and as such it would be possible to re-cast such standards in terms of CEA, and as such CEA would take on the role of a meta-standard. The advantages would be

Example of SIA defined as CEA application

Possible Extensions to CEA Interfaces

There are some extensions to the CEA interface definitions that could be made, that would detract from the "purity" of the original design, but might make the interfaces more easily usable in many practical deployment scenarios.

Appendix A: The complete Common Execution Controller WSDL 1.1

 

Appendix B: Preliminary REST WSDL 2.0

This appendix contains an attempt to represent the REST interface with WSDL. This has been done using [WSDL2] as this new version of the standard has better support for the representation of REST services in a standard fashion within the WSDL document.

Appendix C: The complete CEAJobMonitor WSDL 1.1

Appendix D: The complete CEAResultsListener WSDL 1.1

Appendix E: CEATypes Schema

Appendix E: Change History

This is the first version that has been made public - some content is based on [CEA].

References

[RFC 2119]
Bradner, S. 1997. Key words for use in RFCs to Indicate Requirement Levels, IETF RFC 2119, http://www.ietf.org/rfc/rfc2119.txt
[RM]
Hanisch, Robert (ed.) 2004. Resource Metadata for the Virtual Observatory, Version 1.01, IVOA Recommendation, http://www.ivoa.net/Documents/REC/ResMetadata/RM-20040426.htm
[VR]
Plante, Ray (ed.) 2006. VOResource: an XML Encoding Schema for Resource Metadata, IVOA Working Draft, http://www.ivoa.net/Documents/latest/VOResource.html
[VSTD]
Harrison, Paul. (ed.) 2006. VOStandard: an XML Encoding Schema for IVOA Standards, IVOA Working Draft, http://www.ivoa.net/Documents/latest/VOResource.html
[CEA]
Harrison, Paul. 2005. A Proposal for a Common Execution Architecture , IVOA Working Draft, http://www.ivoa.net/Documents/latest/CEA.html
[UWS]
Rixon, Guy. 2007. Universal Worker Service, V0.3, IVOA Working Draft, http://www.ivoa.net/internal/IVOA/IvoaGridAndWebServices/UWS-0.3.pdf
 
 
[CEAAPP]
Harrison, Paul. 2007. CEA Application Model: A model and XML Encoding Schema for Applications in the Common Execution Architecture, IVOA Working Draft, http://www.ivoa.net/Documents/latest/CEAApplicationDM.html
 
[xml]
Bray, Tim, Paoli, Jean, Sperberg-McQueen, C. M., Maler, Eve, Yergeau, Francois, Cowan, John (editors) 2006, Extensible Markup Language (XML) 1.1 (Second Edition),,http://www.w3.org/TR/xml11/
[schema]
Fallside, David C., Walmsley, Priscilla (editors) 2004, XML Schema Part 0: Primer Second Edition, W3C Recommendation 28 October 2004, http://www.w3.org/TR/xmlschema-0/
[ISO8601]
Wolf, Misha and Wicksteed, Charles 1997, Date and Time Format, http://www.w3.org/TR/NOTE-datetime.
 
[ID]
Plante, R., Linde, T., Williams, R., Noddle, K. 2005, IVOA Identifiers v1.1, http://www.ivoa.net/Documents/REC/Identifiers/Identifiers-200505XX.html.
[WSDL]
Christensen, E., Curbera, F., Meredith, G., Weerawarana, S. Web Services Description Language (WSDL) 1.1, W3C Note 15 March 2001, http://www.w3.org/TR/wsdl.
[WSDL2]
Chinnici, R., Moreau, J-J., Ryman, A., Weerawarana, S. Web Services Description Language (WSDL) Version 2.0 Part 1: Core Language, W3C Working Draft 26 March 2007, http://www.w3.org/TR/wsdl20/.
 
[SOAP]
Box, D., Ehnebuske, D., Kakivaya, G., Layman, A., Mendolsohn, N., Neilsen, H.F., Thatte, S., Winer, D. Simple Object Access Protocol (SOAP) 1.1, W3C Note 08 May 2000, http://www.w3.org/TR/2000/NOTE-SOAP-20000508/.
[REST]
Roy T. Fielding Architectural Styles and the Design of Network-based Software Architectures, http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm.

Last modified: 30-Oct-2007 11:44