Submission evaluation CLI
Submission Evaluation CLI is a set of conventions, useful to specify submissions and receive evaluations using a Command Line Interface.
A submission field can be specified on a command line as a string (either a positional argument or the argument of an option) with the following syntax:
[ submission_name '/' ] field_name ( ':' path | '=' value )
submission_nameis used to identify a submission (say, if multiple submissions are provided).
field_nameis the name of the field.
pathis the path of the submission file
valueis the content of the submission file (if provided, a temporary file is created with the given content).
source:file.py source_language=c++ a/source:file.py b/source:file2.py
: is used for paths, since
bash seems to auto-complete paths
: is encountered.
A list of submission files can be also specified directly as positional arguments,
as long as their paths do not contain
: nor `='.
Submission files given as positional arguments are mapped to field names
taken from a list of default field names.
The list of default field names can depend on the program.
By default, it contains only one name:
A command line tool can output an evaluation
by printing its event stream to
in JSON Lines (http://jsonlines.org/) format.