# Fluency Locker word frequency lists

These are the adapted word lists behind the review cards in the Fluency Locker app for iPhone. There is one file for each course, and each file lists the words in the order the app teaches them.

They are shared under the Creative Commons Attribution-ShareAlike 4.0 International license (CC BY-SA 4.0), the same license as the lists they come from: https://creativecommons.org/licenses/by-sa/4.0/

## Files

| File | Language | Words | Source |
|---|---|---|---|
| es.tsv | Spanish | 2,512 | FrequencyWords, 2018 Spanish list |
| fr.tsv | French | 2,870 | FrequencyWords, 2018 French list |
| it.tsv | Italian | 2,760 | FrequencyWords, 2018 Italian list |
| de.tsv | German | 2,697 | FrequencyWords, 2018 German list |
| ja.tsv | Japanese | 2,591 | wordfreq 3, Japanese |
| ko.tsv | Korean | 2,419 | wordfreq 3, Korean |
| zh.tsv | Mandarin Chinese (simplified characters) | 2,893 | wordfreq 3, Chinese |
| en.tsv | American English | 2,520 | wordfreq 3, English |

Each file is UTF-8 text with tab-separated columns and one header row:

- `order`: the position of the word in the app's teaching order.
- `lemma`: the dictionary form of the word, written in the language's own script.
- `pos`: the part of speech (noun, verb, adj, adv, pron, prep, conj, det, num, interj, phrase, particle, counter or aux).
- `source_rank`: the word's position in the source list. When the source counts several forms of the same word separately (for example tengo and tiene for tener), this is the position of the most frequent of those forms.
- `source`: the list the word comes from.

## How we adapted the lists

- We joined the forms of each word into its dictionary form. In Japanese and Korean, the source lists count pieces of words, and we joined them into whole words.
- We removed names of people and places, brands, swear words, abbreviations, internet slang, English words in the lists of the other languages, and sounds that are frequent only in subtitles, such as oh, eh and hey.
- Some words are frequent in the source lists mostly because the lists come from movie subtitles, such as words about crime and violence. We moved those words to the end of the order or removed them.
- In the English list, we replaced words that are frequent mostly in news and politics with everyday words that are also in the source list, with their real position.
- We added the part of speech and put the words in the order the app teaches them, so `order` and `source_rank` are often different.

The translations, example sentences, scenes, grammar notes and audio in the app are not part of these lists and are not shared under this license.

## Credits

**FrequencyWords** by Hermit Dave (https://github.com/hermitdave/FrequencyWords), used for Spanish, French, Italian and German. The content of FrequencyWords is shared under CC BY-SA 4.0. Its 2018 lists are made from the OpenSubtitles 2018 collection of OPUS (http://opus.nlpl.eu/OpenSubtitles.php), whose data comes from OpenSubtitles.org (http://www.opensubtitles.org/).

**wordfreq** by Robyn Speer (https://github.com/rspeer/wordfreq), used for Japanese, Korean, Chinese and English. The data files of wordfreq are shared under CC BY-SA 4.0. As its documentation asks, we credit the sources it is built on:

- Google Books Ngrams (http://books.google.com/ngrams) and Google Books Syntactic Ngrams.
- The Leeds Internet Corpus, from the University of Leeds Centre for Translation Studies (http://corpus.leeds.ac.uk/list.html).
- Wikipedia (http://www.wikipedia.org).
- ParaCrawl (https://paracrawl.eu).
- OPUS OpenSubtitles 2018 (http://opus.nlpl.eu/OpenSubtitles.php), with data from OpenSubtitles.org (http://www.opensubtitles.org/).
- The SUBTLEX word lists by Marc Brysbaert and colleagues, which are freely available at http://crr.ugent.be/programs-data/subtitle-frequencies. The English list uses SUBTLEX-US and SUBTLEX-UK, and the Chinese list uses SUBTLEX-CH.
- Word statistics collected from the public Twitter streaming API, as described in the wordfreq documentation.

### Citations

- Robyn Speer. (2022). rspeer/wordfreq: v3.0 (v3.0.2). Zenodo. https://doi.org/10.5281/zenodo.7199437
- Brysbaert, M. & New, B. (2009). Moving beyond Kucera and Francis: A Critical Evaluation of Current Word Frequency Norms and the Introduction of a New and Improved Word Frequency Measure for American English. Behavior Research Methods, 41 (4), 977-990.
- van Heuven, W. J., Mandera, P., Keuleers, E., & Brysbaert, M. (2014). SUBTLEX-UK: A new and improved word frequency database for British English. The Quarterly Journal of Experimental Psychology, 67(6), 1176-1190.
- Cai, Q., & Brysbaert, M. (2010). SUBTLEX-CH: Chinese word and character frequencies based on film subtitles. PLoS One, 5(6), e10729.

## License

You may copy, share and adapt these lists, including for commercial use, as long as you give credit (to the sources above and to Fluency Locker) and share what you make from them under the same license. Read the full license at https://creativecommons.org/licenses/by-sa/4.0/legalcode

The lists come with no warranty.