MLcomp version alpha Statistics: 2044 users, 665 datasets, 389 programs, 18920 runs (0 queued, 4 running), 0 workers

Anyone who has worked with machine learning knows that it's a zoo. There are a dazzling array of methods published at conferences each year, leaving the practitioner, who just wants to choose the best method for his/her task, baffled.

• You upload programs and/or datasets.
• Programs are run on datasets.
• Various performance metrics are reported on these runs.

MLcomp aims to improve research in the following ways:

• Transparency: all metrics (error, running time, etc.) of all programs and datasets are shown side-by-side for comparison.
• Reproducibility: one can download any program and dataset, which are stored with their exact settings/formats, to verify results.
• Collaboration: anyone can upload programs and/or datasets which can be run on and compared with existing programs and datasets.

Yes.

Intuitively, good programs have low error rates and good datasets distinguish programs. Specifically, the rating of a dataset is related to the standard deviation of the error rates of programs run on that dataset. For each dataset, a Gaussian is fit on the error rates of the programs, resulting a percentile (0 to 100) for each program. The rating of a program is the average percentile across all the datasets on which it is run. This is essentially a first cut at rating programs and we intend to improve it. Suggestions are welcome.

MLcomp serves two communities:

• Machine learning researchers: objectively evaluate and compare methods.
• Practitioners: quickly try out many different methods.

For the machine learning researcher: you develop a fancy new algorithm. Upload your program to MLcomp and run it on an array of different datasets in the system that others have already uploaded. You can then compare the results to those of programs that others have uploaded.

For the practitioner: suppose you're working in computational biology and want to know out which machine learning method works best on your task. Upload your dataset and run it on the many available programs that have been developed by machine learning researchers, and download the program that performs best.

If it is, it's a living and breathing repository. The great thing about MLcomp is that all programs and datasets are in standard formats, and hence can be run on each other, immediately producing results for all to see.

Follow our QuickStart to start uploading programs and datasets in minutes.

A domain defines a particular machine learning task, for example, binary classification or collaborative filtering. Each domain is equipped with the following:

• a particular dataset format,
• a particular program interface, and
• standard evaluation metrics.

For example, in the BinaryClassification domain, datasets are in SVMlight format, and program interface must read these datasets and output binary labels, and the evaluation metric is misclassification error rate.

See this page for a detailed description of all domains.

We encourage you to contact the MLcomp team to create a new domain. MLcomp is built modularly, so it should be relatively easy to put in the desired dataset formats, program interfaces, and evaluation scripts.

Currently, none. In the future, a complex domain might be defined in terms of multiple task types and dataset formats.

Concretely, a program is an executable (called run) that conforms to a particular program interface determined by the domain. Of course, the program can also include other files besides run. Think of a program as a Java class implemented through the shell.

Basically any language. Your program will be run on a standard Linux installation with various software packages already installed. Let us know if you need a particular library or package which doesn't exist.

Either, as long as your run executable satisfies the program interface for the domain. That being said, we recommend that you include source code. MLcomp is designed to be collaborative, and this only works if your code is available to be downloaded by others.

Include the C++ source files with your program package that you upload to MLcomp. Write a run script (e.g., in Bash) which compiles the source code and invokes it. Specifically, when we call ./run construct, the C++ source files should be compiled. See more information on the program interface.

You have two options:

• Implement your own hyperparameter tuning inside ./run learn and MLcomp will treat it as a black box.
• Implement ./run setHyperparameter x to store the hyperparameter x, which will be called before ./run learn. If hyperparameter tuning is chosen when a run is created, MLcomp will use the training set to perform cross-validation on x ∈ { 0.01, 0.1, 1, 10, 100 } and use the best value.

Yes, this is done transparently through a one-versus-all reduction.

We strongly encourage users to make their code available, but we provide the option to make it private. You are given the option of choosing whether to make it publicly downloadable or not at the time of upload. You can change this setting by clicking 'Edit Program' on the program's page.

Currently, programs (when zipped) can be no larger than 200MB.

You can use the CDE tool, which can automatically package all the libraries you need to run your program. You can then upload the entire self-contained package onto MLcomp. See the demo program for more information.

Each dataset is identified with a domain-dependent dataset format. In general, a dataset consists of multiple shards; for example, for some tasks, the training, development, and test sets are standardized; these would constitute three different shards. For supervised learning, typically two shards are used: train and test. One can also upload a dataset with just one shard called raw, which will be automatically split into training and test.

When a dataset is uploaded, it is processed by a dataset processor, which validates the dataset according to the expected format, computes statistics on the dataset, and splits the data if necessary. Runs can only be executed on a dataset only after it has been processed.

You have two options:

• Create two shards, called train and test.
• Create one shard, called raw. Upon upload, a dataset processor will automatically split this shard into a train shard and a test shard containing 70% and 30% of the examples, respectively.

By default, your dataset can be downloaded by anyone. However, if there are licensing restrictions, you can check the restricted access box for a dataset to prohibit downloads. However, note that others can use your data to test their algorithms.

Currently, datasets (when zipped) can be no larger than 200MB.

A run is the execution of a program on a dataset. For example, for the binary classification domain, a run includes both training, testing, and evaluation. Each run produces a log file and a set of statistics (time, memory, error rate, etc.), which can be viewed on the appropriate page of the run.

There are two ways:

1. Go to the page for a program. At the bottom of the screen, choose the dataset to run on.
2. Go to the page for a dataset. At the bottom of the screen, choose the program to run it on.

Go to the page for the run by clicking on a run ID. This run page includes a log file of the run which will automatically updates while the run is in progress. Results of the run are displayed once it finishes.

Any user with a (free) MLcomp account can start a run from any program to any compatible dataset. All run outputs are viewable by everyone.

When you generate a run, you can set a time limit for the run (no more than 24 hours). After that point, we will terminate the program. Your program can use 1.5GB of memory. More information here.

To see your runs, go to the 'My Stuff' page. Click the ID of a run to see its page.

All runs are executed on Amazon's EC2 cloud-computing infrastructure on a standard Linux distribution. More information here.

Currently, MLcomp is supported by generous donations of computational credits from Michael Jordan at UC Berkeley.

Go to the page for the run and look at the log file for signs of the responsible error. You can also download the run and run it locally on your machine (a README file should be included in the download which provides more information).

We said that a run was simply a program/dataset pair, but that's not the full story. A run actually includes other helper programs such as the evaluation program and various programs for reductions (e.g., one-versus-all, hyperparameter tuning).

More formally, a run is a given by a run specification, which can be found on the page for any run. A run specification is a tree where each internal node represents a program and its children represents the arguments to be passed into its constructor. For example, the one-versus-all program takes your binary classification program as a constructor argument and behaves like a multiclass classification program.