TuringArena Web Evaluation API
A Web API based on SEWI is described, meant to perform submission evaluations based on TuringArena via Web.
The API supports the following methods.
- evaluate, used to start a new evaluation, using a given evaluator and a given submission,
- evaluation events, used to get a page of evaluation events of an ongoing evaluation.
base_url denote the base URL where the API is deployed.
A call to evaluate is SEWI request with:
- URL template:
The request includes the provided submission, with base name
See Submissions in Web APIs
The other parameters are:
repository[url]: the URL of the repo to clone.
commit_oid, the OID of the commit to consider.
directory, the directory of the problems, optional.
evaluator_cmd, the command to run as evaluator.
The commit OID (SHA-1 hash) is required in the requests for repeatability and caching.
The response body of evaluate is a JSON object, with the following fields.
evaluation_id: a string identifying the newly created evaluation.
A call to evaluation events is an HTTP request as follows.
- URL template:
- Template parameters
evaluation_id, required, containing the ID of the evaluation,
after, optional, a cursor indicating the beginning of the desired page (the cursor is implicitly
nullif this parameter is absent).
In normal conditions, the response body is a page of evaluation events, represented as JSON,
starting at the cursor provided in the
(i.e., the first page if
after is absent).
The page can be empty.
A request should be repeatable in case of errors,
until the client actually reads after the returned page.
More precisely, a call to evaluation events (request 1) should succeed as long as
no other requests to evaluation events have been performed (request 2),
such that the
after parameter of request 2 occurs strictly later than the
after parameter of request 1.
This assumption makes the protocol robust under HTTP errors,
but still allows an implementation to delete the events when it is sure that the client has received those.
Every time an implementation receives a call to evaluation events,
it can delete all the events before the
In order to be able to safely delete also the last page of events, an implementation may do the following.
- Define a dummy cursor
- When the last event is sent, set the
endcursor of the page to
- When a request is received with
DUMMY_END, return an empty page with
null, and remove any remaining data stored for the current evaluation.