Bigram Relatedness Dataset (BiRD)
March 2019
Authors: Shima Asaadi, Saif M. Mohammad, and Svetlana Kiritchenko


Contacts: Saif Mohammad (saif.mohammad@nrc-cnrc.gc.ca), Shima Asaadi (shima.asaadi@tu-dresden.de)
	



*****************************************
Terms of Use: 
*****************************************

1. If you use this dataset, cite the paper below:

Shima Asaadi, Saif M. Mohammad, and Svetlana Kiritchenko (2019) Big BiRD: A Large, Fine-Grained, Bigram Relatedness Dataset for Examining Semantic Composition. Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), Minneapolis, USA, June 2019.

@inproceedings{bird-naacl2019,
  
	title={Big BiRD: A Large, Fine-Grained, Bigram Relatedness Dataset for Examining Semantic Composition},
  
	author={Asaadi, Shima and Mohammad, Saif M. and Kiritchenko, Svetlana},
	
	booktitle={Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)}, 
	
	year={2019},
	
	address={Minneapolis, USA}    

}

2. Do not redistribute the data. Direct interested parties to this page: http://saifmohammad.com/WebPages/BiRD.html

3. National Research Council Canada (NRC) disclaims any responsibility for the use of the dataset and does not provide technical support. However, the contacts listed above will be happy to respond to queries and clarifications.



*****************************************
Files included in this distribution:
*****************************************

1. BiRD.txt is the BiRD dataset with detailed information. The file format is described below.




2. BiRD-annotations-raw.csv contains the annotators' responses to the questions in the annotation task on the crowdsourcing platform. The file format is described below.
3. Questionnaire-for-BiRD.pdf shows the questionnaire used to annotate the data along with the detailed annotation instructions and examples of appropriate and inappropriate responses provided to annotators in the crowdsourcing platform.



*****************************************
BiRD-annotations-raw.csv File Format
*****************************************


Each line in this file has the following format:
<unit_id>,<worker_id>,<term_pair1>,<term_pair2>,<term_pair3>,<term_pair4>,<q1_which_pair_is_most_close_in_meaning_or_most_related>,<q2_which_pair_is_least_close_in_meaning_or_least_related>.



<unit_id> is the unique ID for each 4-tuple;

<worker_id> is the ID of the annotator;

<term_pair1>,<term_pair2>,<term_pair3>,<term_pair4> are the four term pairs (4-tuple) to annotate using the BWS;

<q1_which_pair_is_most_close_in_meaning_or_most_related> is the annotation for the most related pair from the given four term pairs. <q2_which_pair_is_least_close_in_meaning_or_least_related> is the annotation for the least related pair from the given four term pairs.



To preserve privacy, we replaced the CrowdFlower worker IDs with sequential IDs, but kept the same ID for all annotations performed by a particular worker.



*****************************************
BiRD.txt File Format
*****************************************

Each line in BiRD.txt file has the following format:
<pair><tab><term1><tab><term2><tab><term2:unigram or bigram><tab><source><tab><relation><tab><relatedness score><tab><POS>

<pair> is a distinct term pair in the following format: (<term1>, <term2>);
<term1> is a bigram;
<term2> can be a unigram or a bigram;
<term2:unigram or bigram> indicates if <term2> is a unigram or a bigram;
<source> indicates the source of the <pair>. It can be one of the following sources: 		
	Wikipedia_transposed, 
	WordNet_synonym, 
	WordNet_is-a, 
	WordNet_part-whole, 
	PhraseTable_co-aligned.
<relation> indicates the relation type between <term1> and <term2> if the <source> is WordNet_is-a or WordNet_part-whole. If the source is WordNet_is-a, the relation is hypernym or hyponym. If the source is WordNet_part-whole, the relation is holonym or meronym. For other resources <relation> is None;
<relatedness score> is a real-valued number between 0 (lowest semantic relatedness) and 1 (highest semantic relatedness) indicating the degree of relatedness between <term1> and <term2>;
<POS> indicates the part-of-speech tag for <term1>. It can be a-n (adjective-noun) or n-n (noun-noun).

There are 3,345 distinct term pairs. The file is sorted in alphabetical order of <term1>. Term pairs associated with <term1> are sorted in decreasing order of <relatedness score>. 
 


*****************************************
More Information
*****************************************

For more details on the process of creating the dataset, please refer to the following paper:

Shima Asaadi, Saif M. Mohammad, and Svetlana Kiritchenko (2019) Big BiRD: Big BiRD: A Large, Fine-Grained, Bigram Relatedness Dataset for Examining Semantic Composition. Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), Minneapolis, USA, June 2019.
