Open-source tools

In addition to the clearinghouse website, the SyntaxGym ecosystem provides two open-source tools: syntaxgym and lm-zoo.

syntaxgym is a Python package which provides easy, standardized, reproducible access to targeted syntactic evaluations of language models. It replicates the core behavior of the SyntaxGym website. syntaxgym is documented in full here.

The Language Model Zoo (LM Zoo) is an open-source repository of state-of-the-art language models, designed to support black-box access to model predictions and representations. It provides the command line tool lm-zoo, a standard interface for interacting with language models. lm-zoo is documented in full here.

Note

To gain a better understanding of how these three tools work together (syntaxgym.org, syntaxgym, and lm-zoo), please take a look at our ACL 2020 System Demonstration paper.