Skip to content

Latest commit

 

History

History
134 lines (100 loc) · 4.86 KB

README.md

File metadata and controls

134 lines (100 loc) · 4.86 KB
Raw

Indic Tagger (Indian Language Tagger)

In this project, we build part-of-speech (POS) taggers and chunkers for Indian Languages.

Languages supported: Telugu (te), Hindi (hi), Tamil (ta), Marathi (mr), Punjabi (pa), Kannada (kn), Malayalam (ml), Urdu (ur), Bengali (bn)

If you reuse this software, please use the following citation:

@inproceedings{PVS:SPSAL2007,
  editor    = {P.V.S., Avinesh and Gali, Karthik},
  title     = {Part of Speech Tagging and Chunking using Conditional Random Fields and Transformation Based Learning}
  booktitle = {Proceedings of the  Shallow Parsing for South Asian Languages (SPSAL) Workshop, held at IJCAI-07, Hyderabad, India},
  series    = {{SPSAL} Workshop Proceedings},
  month     = {January},
  year      = {2007},
  pages     = {21--24},
}

Training Data Statistics and System Performances (F1 macro)

Languages # Words # Sents CRF POS CRF Chunk BI-LSTM-CRF POS BI-LSTM CRF Chunk
te 347k 30k 93% 96% 92% 92%
hi 350k 16.3k 93% 97% 94% 93%
bn 298.3k 14.6k 84% 95% 85% 88%
pa 152.5k 5.6k 92% 98% 94% 96%
mr 207.9k 8.5k 89% 95% 88% 90%
ur 158.9k 7.6k 90% 96% 92% 89%
ta 337k 14.2k 88% 92% 87% 85%
ml 192k 11.4k 96% 95% 98% 98%
kn 294.3k 16.5k 90% 98% 88% 87%

Training Data Statistics and System Performances (F1 macro) for NER

Languages # Words # Sents CRF NER BI-LSTM-CRF NER
te 347k 30k 69% 65%
hi 503k 19k 62% 63%
bn 120k 6k 54% 48%
ur 35k 1.5k 65% 56%
or 93k 1.8k 68% 43%

Install using Anaconda

    # INSTALL python environment
    conda create -n tagger3.6 anaconda python=3.6
    source activate tagger3.6
    
    # Install the tokenizer
    cd polyglot-tokenizer
    python setup.py install
    
    # Install requirements
    pip install -r requirements.txt

Run

    python pipeline.py -p predict -l te -t pos -m crf -f txt -e utf -i input_file -o output_file

    -l, --languages       select language (2 letter ISO-639 code) 
                          {hi, be, ml, pu, te, ta, ka, mr, ur}
    -t, --tag_type      	pos, chunk, parse, ner
    -m, --model_type    	crf, hmm, lstm
    -f, --data_format   	ssf, txt, conll
    -e, --encoding      	utf8, wx   (default: utf8)
    -i, --input_file      <input-file>
    -o, --output_file     <output-file>
    -s, --sent_split      True/False (default: True)
	
    python pipeline.py --help

Train the POS tagger:

    # CRF model
    python pipeline.py -p train -o outputs -l te -t pos -m crf -e utf -f ssf
    
    # BI-LSTM-CRF model
    python pipeline.py -p train -t pos -f conll -m lstm -e utf -l te

Predict on text:

    # CRF models 
    python pipeline.py -p predict -l te -t pos -m crf -f txt -e utf -i data/test/te/test.utf.txt
    
    # BI-LSTM-CRF models
    python pipeline.py -p predict -l te -t pos -m lstm -f txt -e utf -i data/test/te/test.utf.txt
    
    # SpaCy models
    python spacy_tagger_test.py -l te -t pos

Train the NER tagger:

    # CRF model
    python pipeline.py -p train -o outputs -l te -t ner -m crf -e utf -f conll
    
    # BI-LSTM-CRF model
    python pipeline.py -p train -t ner -f conll -m lstm -e utf -l te

Predict NER on text:

    # CRF model
    python pipeline.py -p predict -l hi -t ner -m crf -f txt -e utf -i data/test/hi/test.utf.txt
    
    # BI-LSTM-CRF model
    python pipeline.py -p predict -l hi -t ner -m lstm -f txt -e utf -i data/test/hi/test.utf.txt

ToDo List

  • Telugu, Hindi trained CRF models
  • Bengali, Punjabi, Marathi, Urdu, Tamil trained CRF models
  • Bug: Utf-8 error Malayalam, Kannada trained CRF models
  • Deep learning (BI-LSTM-CRF)
  • Analysis Comparision w.r.t other ML algorithms
  • Bug: Punjabi & Urdu training file doesn't have "|" (or) end of sentence marker.
  • NER for Indian Languages
  • Feature addition to BI-LSTM-CRF models
  • Active Learning based sampling strategies