Submission Evaluation Web Interface

The Submission Evaluation Web Interface (SEWI) is a set of conventions, useful to pass submissions and receive evaluations via Web APIs.

Form HTTP request

We work always with HTTP request which satisfy the following assumptions.

  • The request method is POST.
  • The request body has the multipart/form-data MIME type, and the Content-Type header is set accordingly.
  • The request body is such that it could have been generated by sending an HTML form (called the form in the following) using the FormData Javascript API.

We call such requests form-like requests.

Evaluations

Evaluation event pages

SEWI specifies a way to represent evaluations as JSON objects, meant mainly to be returned in HTTP responses.

In general, a JSON object is used to represent a sub-sequence of events occurring consecutively in an evaluation, called a page of events.

A page is identified using cursors. A cursor can be:

  • null, representing either the beginning or the end of the evaluation stream, or
  • an opaque JSON string, representing a location in the middle of an evaluation stream.

A cursor never refers to a specific events, but always to the location between two consecutive events. (More formally, the disjoint union of events and string cursors form a totally ordered set.)

A page is represented as a JSON object with the following fields.

  • begin: the cursor at the beginning of this page.
  • end: the cursor at the end of this page.
  • data: a JSON array containing the events in the page.

If the begin (resp. end) cursor is null, then the page is at the beginning (resp. end) of the evaluation. The whole evaluation can be represented a a page when the begin and end cursors are both null.

Evaluations via WebSockets

An evaluation can be transferred though a WebSockets connection as follows.

  • Data is sent only in one direction.
  • For every event, its JSON representation is sent in an distinct UTF-8 message.
  • No other messages are sent.
  • The connection is closed by the sender after the last event.

If an application wants more flexibility using WebSockets, then it should send event pages, represented as JSON objects as described above, instead of single events.

Submissions

A SEWI HTTP request can send one or more submissions. Each submission sent is identified by a submission base name. Different submission must have different base names.

The request is a form-like request with the following fields.

For every submission and for every of its fields, the form contains exactly one field, defined by:

  • name: the submission base name followed by the field name surrouded by brackets ([ and ])
  • value: the content of the submission file.

If the content is sent as a value instead of a file, i.e., the filename parameter is missing in the Content-Disposition header, then the filename defaults to the field name followed by .txt.

The form may have other fields, but it must not contain any other field whose name starts with a submission base name.

Example

Consider two submissions:

  • Submission 1.
    • Base name: new_submission.
    • Fields:
      • name: source, type: file, stored in /tmp/file1.cpp,
      • name: source_language, file content: c++.
  • Submission 2.
    • Base name: previous_submission.
    • Fields:
      • name: source, type: file, stored in /tmp/file2.py
      • name: source_language, file content: python,
      • name: source_python_version, file content: 3.

The following CURL command generates a valid SEWI request, sending the above two submissions (along with other form fields):


curl https://httpbin.org/anything \
    -F action=create_submission \
    -F user=john_smith \
    -F new_submission[source]=@/tmp/file1.cpp \
    -F new_submission[source_language]=c++ \
    -F previous_submission[source]=@/tmp/file2.py \
    -F previous_submission[source_language]=python \
    -F previous_submission[source_python_version]=3 \
    -F format=json