Checking this repo against the Quote-500K Dataset

[TomLucidor]

Currently there is a dataset of quotes that are useful for language models and that I need to exclusively pick out the English ones to prevent data being unclean. There are a lot of quotes that are foreign (Spanish, French, Italian, Hindi, Chinese etc.) but I also realized that some of them are buggy (e.g. bad characters, numerals, formatting) and I would like to adventure into cleaning them out. Here are the ones being flagged as non-English (some are however English)
expected_foreign.txt

Questions:

1. Why is Lingua so sensitive to punctuation?
2. Are there ways to handle bilingual sentences and detect them explicitly (e.g. French with English translation)

Note: I will drop the "false negative" case once the data is handled carefully, as the original is 98KB

[pemistahl]

1. What do you mean by sensitive to punctuation? Lingua actually ignores punctuation during language identification.
2. There is the method LanguageDetector.detect_multiple_languages_of() which can identify multiple languages in a single text. But it's experimental as of now.

[TomLucidor]

In some of the text where it is clearly English, the punctuation might cause the rest of the text to look "off" even when punctuation is ignored, and I am not sure why some got misclassified (length issue?)

[pemistahl]

Punctuation is not at all involved in the language identification process. The most likely reason for English phrases being identified as non-English is that the sum of the ngram probabilities for English will be smaller than the sum of the ngram probabilities for a non-English language. Certain ngrams, i.e. letter sequences which happen to occur in English are more likely to occur in other languages, i.e. they occur more often in non-English languages. The word broken is identified as Dutch even though it's an English word because its ngrams occur more often in Dutch than in English.