.. SyntaxGym documentation master file, created by
   sphinx-quickstart on Tue Aug 13 11:59:49 2019.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

.. figure:: ../img/logo_name_wide.png
   :width: 80%
   :align: center
   :target: http://www.syntaxgym.org

`SyntaxGym <http://www.syntaxgym.org>`_ is a unified platform where language
and NLP researchers can evaluate the performance of language models on 
targeted syntactic tests. Our goal is to make psycholinguistic
assessment of language models more **standardized**, **reproducible**, and
**accessible** to a wide variety of researchers.
The project is run out of the
`MIT Computational Psycholinguistics Laboratory <http://cpl.mit.edu/>`_.

.. .. figure:: ../img/diagram.png
..    :width: 60%
..    :align: center

Overview
=============

There are two primary components of SyntaxGym: test suites and language models.

Test suites
************

SyntaxGym represents targeted syntactic evaluation experiments as test suites. 
Test suites evaluate language models’ knowledge of some particular grammatical phenomenon. 
Our standardized format for test suites is described at the
`SyntaxGym core documentation <https://cpllab.github.io/syntaxgym-core/architecture.html>`_.

Language models
****************

Language models are fit to a standard API and then evaluated on our 
test suites. To browse the existing list of language models available on SyntaxGym, 
please visit the `LM Zoo registry <https://cpllab.github.io/lm-zoo/models.html>`_. 

Users can also add their own language models, which involves the following two steps. 
First, you will need to
`build an API-compliant Docker image <https://cpllab.github.io/lm-zoo/contributing.html>`_
for your model. Second, you will need to `register the model <http://syntaxgym.org/lang_model/create>`_
on the SyntaxGym site. A SyntaxGym account is required to add language models.

.. note::

   For more information about the (independent) tools ``syntaxgym`` and ``lm-zoo``,
   please read about our :ref:`tools`.

What can I do with SyntaxGym?
******************************

For an example of analysis on the test suites and language models, please
take a look at our `ACL 2020 long paper <https://www.aclweb.org/anthology/2020.acl-main.158/>`_.

If you use SyntaxGym in your own research, please `get in touch <mailto:contact@syntaxgym.org>`_ 
so it can be featured here!

Documentation
==================

.. toctree::
    about
    website
    tools
    
.. * :ref:`genindex`
.. * :ref:`modindex`
.. * :ref:`search`