Benchmarks

Benchmark

class comic.eyra.models.Benchmark(*args, **kwargs)[source]

A Benchmark defines a specific challenge.

To be a valid Benchmark, for which submissions can be created which can be evaluated, a couple of things need to be specified:

data_set field:

Sets the DataSet to be used in this benchmark. Determines which files are used as test data and ground truth in the evaluation pipeline.

evaluation_image field:

The evaluator field sets which docker image is used to evaluate the output of a user Submission.

Parameters

  • id (UUIDField) – UUID (primary key)

  • created (DateTimeField) – Moment of creation

  • modified (DateTimeField) – Moment of last modification

  • creator_id (ForeignKey) – Creator of the benchmark

  • name (CharField) – The name of the benchmark

  • short_description (TextField) – Short description in markdown

  • description (TextField) – Description in markdown

  • about (TextField) – About this benchmark in markdown

  • data_description (TextField) – Description of the data used in this benchmark in markdown

  • truth_description (TextField) – Description of the truth data in markdown

  • metrics_description (TextField) – Description of the metrics in markdown

  • submission_instruction (TextField) – submission instruction for user in markdown

  • banner_image (ImageField) – Banner image (wide image that shows in header of benchmark details page)

  • card_image (ImageField) – Image to show in small card of this benchmark

  • evaluation_image (CharField) – Docker image to use for evaluation.

  • data_set_id (ForeignKey) – Data set used in this benchmark

  • admin_group_id (OneToOneField) – Admin group for this benchmark

  • should_evaluate (BooleanField) – Should evaluate in Eyra

Submission

class comic.eyra.models.Submission(*args, **kwargs)[source]

A Submission is an algorithm (as a docker image) tested for a Benchmark. Whenever a Submission is created, two Jobs are created: the algorithm_job (which runs first) and the evaluation_job (which runs second, using the output of the first Job.

When the evaluation Job succeeds, the field metrics in its JSON output will be copied to this models metrics.

Parameters

  • id (UUIDField) – UUID (primary key)

  • creator_id (ForeignKey) – Creator

  • created (DateTimeField) – Created

  • modified (DateTimeField) – Modified

  • name (CharField) – Name

  • benchmark_id (ForeignKey) – Benchmark

  • algorithm_job_id (ForeignKey) – Algorithm job

  • evaluation_job_id (ForeignKey) – Evaluation job

  • metrics (JSONField) – Metrics

  • is_private (BooleanField) – Submission for private leaderboard

  • visualization_url (URLField) – Visualization URL

  • image (CharField) – Docker image (e.g. eyra/frb-eval:3)

  • command (CharField) – If specified, overrides default command as defined in Dockerfile

  • algorithm_id (ForeignKey) – Implemented algorithm

  • version (CharField) – Version