Equity Evaluation Corpus
May 2018
Copyright (C) 2018 National Research Council Canada (NRC)
----------------------------------------------------------------


Contact: Svetlana Kiritchenko (svetlana.kiritchenko@nrc-cnrc.gc.ca)


Terms of Use: 
-------------------------------------------------

1. If you use this dataset, cite the paper below:

Svetlana Kiritchenko and Saif M. Mohammad. (2018) Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), New Orleans, USA, 2018.

2. Do not redistribute the data. Direct interested parties to this page: http://saifmohammad.com/WebPages/Biases-SA.html

3. National Research Council Canada (NRC) disclaims any responsibility for the use of the dataset and does not provide technical support. However, the contact listed above will be happy to respond to queries and clarifications.



Equity Evaluation Corpus (EEC)
-------------------------------------------------

Equity Evaluation Corpus is a dataset of 8,640 English sentences that can be used to 
determine whether automatic systems consistently give higher (or lower) sentiment intensity scores to sentences involving  a particular race or gender. 

The sentences are generated from eleven templates by instantiating the noun phrase variable with one of the pre-chosen values:
 - common African American female first names,
 - common African American male first names,
 - common European American female names,
 - common European American male first names,
 - noun phrases referring to females, such as `my daughter',
 - noun phrases referring to males, such as `my son'.
Some sentences include explicit expressions of sentiment and emotion, and some sentences do not. One can derive pairs of sentences from the EEC such that they differ only in one word corresponding to gender or race (e.g., `My daughter feels devastated' and `My son feels devastated').




Further Information:
-------------------------------------------------

Svetlana Kiritchenko and Saif M. Mohammad. (2018) Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), New Orleans, USA, 2018.

