Add Pāli Stemmer

[khemarato]

This is currently a draft PR for starting a discussion.

The test cases are defined in snowballstem/snowball-data#26

The current, super naive, implementation of just removing common suffixes achieves an admirable accuracy of 97.7% on the test set. See this gist for the failing cases.

Any feedback at all is appreciated.

[ojwb]

While this code is correct, I'd suggest removing the delete here and instead adding (delete) after line 17 below. Then it's clearer to a reader less familiar with the Snowball language that the deletion only happens when one of these suffixes matches. Framing it as an explicit action also more easily allows for different actions for different suffixes if that is useful.

[ojwb]

This will produce an empty stem if the input is exactly one of these suffixes, for example:

$ echo anta|./stemwords -l pali

$

Both anta and esi are in your test vocabulary and suffer from this, but it would be undesirable even if it only affected strings which weren't real words in the language as such strings can occur as proper nouns, words from other languages, typos, OCR errors, etc.

It can also produce very short stems (even a single character) - there are 86 cases of single character stems in your test vocabulary. I don't really know anything about Pali so maybe some or even all of these are actually appropriate, but typically it is a sign of overstemming since there are a very limited number of single character stems.

Most of the existing stemmers define R1 and R2 regions based on vowels and non-vowels, and limit suffix removal to one of these regions which has proven to be a good approach in general - see https://snowballstem.org/texts/r1r2.html for more details.

[ojwb]

I'm going to close this - I'm certainly open to adding a Pāli algorithm, but the current state here needs much more work and progress seems to have stalled.

Also Pāli is rather unusual case where the language is essentially fixed and maybe algorithmic stemming isn't the best approach. There's more discussion in snowballstem/snowball-data#26, especially snowballstem/snowball-data#26 (comment) lays down pros and cons for a snowball Pāli algorithm.