Website

User registration

A SyntaxGym account is required to create new test suites or add new language models. No account is needed to browse existing suites, models, or results on the site.

To sign up for an account, please click here.

Exports

SyntaxGym allows users to download test suites, raw surprisals, and prediction results in a variety of human-readable formats.

Exporting test suites

Note

We recommend that you familiarize yourself with our test suite format before reading further.

Users can download any test suite on SyntaxGym in .json or .csv format. The .json format is documented in full here.

When you download a test suite as a .csv file, the columns are ["item_number", "condition_name", "region_number", "region_name", "content"]. These refer to the basic building blocks of our test suite format, as described here. Please note that the .csv file represents the test suite data in long format.

Here is an example test suite item in .csv format:

item_number,condition_name,region_number,region_name,content
1,number_match,1,subject,The boy
1,number_match,2,verb,swims
1,number_match,3,post_verb,today.
1,number_mismatch,1,subject,The boy
1,number_mismatch,2,verb,swim
1,number_mismatch,3,post_verb,today.

Note

The .csv format only represents the items of a test suite, and does not represent the predictions. To make use of the prediction formulae, we recommend that you download the test suite in .json format.

Exporting raw surprisals

Users can download the raw surprisal results across all models for any test suite by visiting the test suite page and clicking on the Download all results dropdown menu at the bottom of the page.

The raw results are formatted as a .csv file with the following columns:

suite

Name of the test suite. (Note: this is the short name, not the long name displayed on the website for readability.)

model_name

Name of the language model.

item_number, condition_name, region_number, region_name, content

See the test suite documentation.

surprisal

The aggregate surprisal value at the relevant region, in bits.

oovs

Any out-of-vocabulary items in the relevant region.

Exporting prediction results

Users can download the prediction results across all models for any test suite by visiting the test suite page and clicking on the Download all results dropdown menu at the bottom of the page.