samples/word_alignment.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Alignment Tutorial\n",
    "\n",
    "In this notebook, we will demonstrate how to use machine to train statistical word alignment models and then use them to predict alignments between sentences. Machine uses the [Thot](https://github.com/sillsdev/thot) library to implement word alignment models. The implementations are contained in the [SIL.Machine.Translation.Thot](https://www.nuget.org/packages/SIL.Machine.Translation.Thot/) package.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>SIL.Scripture, 12.0.1</span></li><li><span>Thot, 3.4.1</span></li></ul></div></div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#r \"nuget:SIL.Scripture,12.0.1\"\n",
    "#r \"nuget:Thot\"\n",
    "#r \"../src/SIL.Machine.Translation.Thot/bin/Debug/netstandard2.0/SIL.Machine.dll\"\n",
    "#r \"../src/SIL.Machine.Translation.Thot/bin/Debug/netstandard2.0/SIL.Machine.Translation.Thot.dll\"\n",
    "\n",
    "void WriteLine(string text = \"\")\n",
    "{\n",
    "    Console.Write(text + \"\\n\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training models\n",
    "\n",
    "The first step in training a statistical word alignment model is setting up a parallel text corpus. Word alignment models are unsupervised, so they only require a parallel text corpus to train. Manually created alignments are not necessary. So let's create a parallel corpus from the source and target monolingual corpora.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "using SIL.Machine.Corpora;\n",
    "using SIL.Machine.Tokenization;\n",
    "\n",
    "var sourceCorpus = new ParatextTextCorpus(\"data/VBL-PT\");\n",
    "var targetCorpus = new ParatextTextCorpus(\"data/WEB-PT\");\n",
    "var parallelCorpus = sourceCorpus.AlignRows(targetCorpus).Tokenize<LatinWordTokenizer>();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine has implementations of all common statistical models, including the famous IBM models (1-4), HMM, and FastAlign. All alignment models implement the `IWordAlignmentModel` interface. This makes it easier to swap out different models in your code.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we are going to start by training an IBM-1 model. There are two possible ways to train a model. First, we will demonstrate training a model from a class that implements `IWordAlignmentModel`. We use the `CreateTrainer` method to create a trainer object that is used to train the model. If we do not specify a file path when creating the model object, then the model will only exist in memory. When we call the `SaveAsync` method, the model instance is updated with the trained model parameters, but the model is not written to disk. We are going to use the `Lowercase` token processor to pre-process that data, since that generally gives better results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training IBM-1 model: 0.00%\n",
      "Training IBM-1 model: 16.67%\n",
      "Training IBM-1 model: 33.33%\n",
      "Training IBM-1 model: 50.00%\n",
      "Training IBM-1 model: 66.67%\n",
      "Training IBM-1 model: 83.33%\n",
      "Training IBM-1 model: 100.00%\n"
     ]
    }
   ],
   "source": [
    "using SIL.Machine.Translation.Thot;\n",
    "using SIL.Machine.Utils;\n",
    "\n",
    "{\n",
    "    using var model = new ThotIbm1WordAlignmentModel();\n",
    "    using var trainer = model.CreateTrainer(parallelCorpus.Lowercase());\n",
    "    await trainer.TrainAsync(new DelegateProgress(status => WriteLine($\"Training IBM-1 model: {status.PercentCompleted:P}\")));\n",
    "    await trainer.SaveAsync();\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other option for training a model is to construct a trainer object directly. This method is useful for when you are only interested in training the model and saving it to disk for later use. We need to specify where the model will be saved after it is trained and we call the `SaveAsync` method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training IBM-1 model: 0.00%\n",
      "Training IBM-1 model: 16.67%\n",
      "Training IBM-1 model: 33.33%\n",
      "Training IBM-1 model: 50.00%\n",
      "Training IBM-1 model: 66.67%\n",
      "Training IBM-1 model: 83.33%\n",
      "Training IBM-1 model: 100.00%\n",
      "IBM-1 model saved\n"
     ]
    }
   ],
   "source": [
    "using System.IO;\n",
    "\n",
    "Directory.CreateDirectory(\"out/VBL-WEB-IBM1\");\n",
    "{\n",
    "    using var trainer = new ThotWordAlignmentModelTrainer(ThotWordAlignmentModelType.Ibm1, parallelCorpus.Lowercase(),\n",
    "        \"out/VBL-WEB-IBM1/src_trg\");\n",
    "    await trainer.TrainAsync(new DelegateProgress(status => WriteLine($\"Training IBM-1 model: {status.PercentCompleted:P}\")));\n",
    "    await trainer.SaveAsync();\n",
    "    WriteLine(\"IBM-1 model saved\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning parallel sentences\n",
    "\n",
    "Now that we have a trained alignment model, we can find the best alignment for a parallel sentence. We call `Align` method to find the best alignment. The results are returned as a `WordAlignmentMatrix` object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1\n",
      "Source: Esta carta trata sobre la Palabra de vida que existía desde el principio , que hemos escuchado , que hemos visto con nuestros propios ojos y le hemos contemplado , y que hemos tocado con nuestras manos .\n",
      "Target: That which was from the beginning , that which we have heard , that which we have seen with our eyes , that which we saw , and our hands touched , concerning the Word of life\n",
      "Alignment: 2-2 2-20 2-25 2-29 2-30 3-32 4-4 4-33 5-34 6-35 7-36 12-3 12-5 13-6 13-12 13-21 13-26 13-31 15-0 15-7 15-10 15-13 15-16 15-22 16-1 16-8 16-11 16-14 16-23 20-17 21-18 22-19 22-28 25-27 35-9 35-15 35-24\n",
      "1JN 1:2\n",
      "Source: Esta Vida nos fue revelada . La vimos y damos testimonio de ella . Estamos hablándoles de Aquél que es la Vida Eterna , que estaba con el Padre , y que nos fue revelado .\n",
      "Target: ( and the life was revealed , and we have seen , and testify , and declare to you the life , the eternal life , which was with the Father , and was revealed to us ) ;\n",
      "Alignment: 0-17 0-35 1-3 1-20 1-24 2-36 3-4 3-27 3-33 4-0 4-10 4-13 4-16 4-37 8-1 8-7 8-12 8-15 8-32 10-26 12-18 14-8 22-23 23-6 23-11 23-14 23-21 23-25 23-31 26-28 28-2 28-9 28-19 28-22 28-29 28-30 34-5 34-34 34-38\n",
      "1JN 1:3\n",
      "Source: Los que hemos visto y oído eso mismo les contamos , para que también puedan participar de esta amistad junto a nosotros . Esta amistad con el Padre y su Hijo Jesucristo .\n",
      "Target: that which we have seen and heard we declare to you , that you also may have fellowship with us . Yes , and our fellowship is with the Father and with his Son , Jesus Christ .\n",
      "Alignment: 2-0 2-1 2-2 2-3 2-7 2-12 2-16 3-4 4-5 4-23 4-30 5-21 6-9 8-10 8-13 10-11 10-22 10-34 13-14 14-15 18-6 18-8 18-17 18-24 18-25 21-19 22-20 22-37 25-18 25-27 25-31 26-26 27-28 27-29 29-32 30-33 31-35 31-36\n",
      "1JN 1:4\n",
      "Source: Escribimos para decirles esto , a fin de que nuestra felicidad sea completa .\n",
      "Target: And we write these things to you , that our joy may be fulfilled .\n",
      "Alignment: 0-9 0-10 0-13 1-8 2-3 2-4 3-5 3-6 4-7 6-0 6-2 9-12 12-1 12-11 13-14\n",
      "1JN 1:5\n",
      "Source: Este es el mensaje que recibimos de él y que nosotros les declaramos a ustedes : Dios es luz , y no hay ningún vestigio de oscuridad en él .\n",
      "Target: This is the message which we have heard from him and announce to you , that God is light , and in him is no darkness at all .\n",
      "Alignment: 0-8 1-1 1-17 1-23 2-2 3-3 3-4 3-7 4-15 5-11 5-26 5-27 7-9 7-22 8-10 8-20 10-5 14-6 14-13 15-0 16-16 18-18 19-14 19-19 22-12 22-24 26-25 27-21 29-28\n"
     ]
    }
   ],
   "source": [
    "{\n",
    "    using var model = new ThotIbm1WordAlignmentModel(\"out/VBL-WEB-IBM1/src_trg\");\n",
    "    foreach (var row in parallelCorpus.Take(5))\n",
    "    {\n",
    "        var alignment = model.Align(row.SourceSegment.Lowercase(), row.TargetSegment.Lowercase());\n",
    "\n",
    "        WriteLine($\"{row.Ref}\");\n",
    "        WriteLine($\"Source: {row.SourceText}\");\n",
    "        WriteLine($\"Target: {row.TargetText}\");\n",
    "        WriteLine($\"Alignment: {alignment}\");\n",
    "    }\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting model probabilities\n",
    "\n",
    "A statistical word alignment model consists of one or more conditional probability distributions that are estimated from the training data. For example, most models estimate a word translation probability distribution that can be queried to obtain the probability that a source word is a translation of a target word. Each model class has methods to obtain these probabilities. Let's try getting some translation probabilities from the IBM-1 model that we trained by calling the `GetTranslationProbability` method. In order to get the probability that a word does not translate to anything, you can pass `null` instead of the word string.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "es -> is: 0.2720\n",
      "NULL -> that: 0.0516\n"
     ]
    }
   ],
   "source": [
    "{\n",
    "    using var model = new ThotIbm1WordAlignmentModel(\"out/VBL-WEB-IBM1/src_trg\");\n",
    "    double prob = model.GetTranslationProbability(\"es\", \"is\");\n",
    "    WriteLine($\"es -> is: {prob:0.0000}\");\n",
    "    prob = model.GetTranslationProbability(null, \"that\");\n",
    "    WriteLine($\"NULL -> that: {prob:0.0000}\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Symmetrized alignment models\n",
    "\n",
    "Most statistical word alignment models are directional and asymmetric. This means that it can only model one-to-one and one-to-many alignments in one direction. They are not capable of modeling many-to-many alignments, which can occur in some language pairs. One way to get around this limitation is to train models in both directions (source-to-target and target-to-source), and then merge the resulting alignments from the two models into a single alignment. This is called symmetrization and is a common practice when using statistical word alignment models. In addition, researchers have found that symmetrized alignments are better quality.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine provides a special word alignment model class to support symmetrization called `SymmetrizedWordAlignmentModel`. Let's demonstrate how to use this class. First, we will train the symmetrized model using the `SymmetrizedWordAlignmentModelTrainer` class.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training direct alignment model: 0.00%\n",
      "Training direct alignment model: 0.00%\n",
      "Training direct alignment model: 8.33%\n",
      "Training direct alignment model: 16.67%\n",
      "Training direct alignment model: 25.00%\n",
      "Training direct alignment model: 33.33%\n",
      "Training direct alignment model: 41.67%\n",
      "Training direct alignment model: 50.00%\n",
      "Training inverse alignment model: 50.00%\n",
      "Training inverse alignment model: 50.00%\n",
      "Training inverse alignment model: 58.33%\n",
      "Training inverse alignment model: 66.67%\n",
      "Training inverse alignment model: 75.00%\n",
      "Training inverse alignment model: 83.33%\n",
      "Training inverse alignment model: 91.67%\n",
      "Training inverse alignment model: 100.00%\n",
      "Symmetrized IBM-1 model saved\n"
     ]
    }
   ],
   "source": [
    "using SIL.Machine.Translation;\n",
    "\n",
    "{\n",
    "    using var srcTrgTrainer = new ThotWordAlignmentModelTrainer(ThotWordAlignmentModelType.Ibm1,\n",
    "        parallelCorpus.Lowercase(), \"out/VBL-WEB-IBM1/src_trg\");\n",
    "    using var trgSrcTrainer = new ThotWordAlignmentModelTrainer(ThotWordAlignmentModelType.Ibm1,\n",
    "        parallelCorpus.Invert().Lowercase(), \"out/VBL-WEB-IBM1/trg_src\");\n",
    "    using var symmetrizedTrainer = new SymmetrizedWordAlignmentModelTrainer(srcTrgTrainer, trgSrcTrainer);\n",
    "    await symmetrizedTrainer.TrainAsync(new DelegateProgress(status =>\n",
    "        WriteLine($\"{status.Message}: {status.PercentCompleted:P}\")));\n",
    "    await symmetrizedTrainer.SaveAsync();\n",
    "    WriteLine(\"Symmetrized IBM-1 model saved\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can also be trained using the `CreateTrainer` method on `SymmetrizedWordAlignmentModel`. Now that we've trained the symmetrized model, let's obtain some alignments. Machine supports many different symmetrization heuristics. The symmetrization heuristic to use when merging alignments can be specified using the `Heuristic` property. In this case, we will use the `GrowDiagFinalAnd` heuristic.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1\n",
      "Source: Esta carta trata sobre la Palabra de vida que existía desde el principio , que hemos escuchado , que hemos visto con nuestros propios ojos y le hemos contemplado , y que hemos tocado con nuestras manos .\n",
      "Target: That which was from the beginning , that which we have heard , that which we have seen with our eyes , that which we saw , and our hands touched , concerning the Word of life\n",
      "Alignment: 1-25 2-25 3-32 4-4 4-33 5-34 6-35 7-36 8-0 12-5 13-6 15-10 16-11 19-20 20-17 21-18 22-19 25-27 35-9\n",
      "1JN 1:2\n",
      "Source: Esta Vida nos fue revelada . La vimos y damos testimonio de ella . Estamos hablándoles de Aquél que es la Vida Eterna , que estaba con el Padre , y que nos fue revelado .\n",
      "Target: ( and the life was revealed , and we have seen , and testify , and declare to you the life , the eternal life , which was with the Father , and was revealed to us ) ;\n",
      "Alignment: 0-17 1-3 2-36 3-4 4-0 6-38 7-0 8-1 9-0 10-0 11-13 12-18 14-8 18-9 22-23 23-6 26-28 27-2 28-29 28-30 33-4 34-5\n",
      "1JN 1:3\n",
      "Source: Los que hemos visto y oído eso mismo les contamos , para que también puedan participar de esta amistad junto a nosotros . Esta amistad con el Padre y su Hijo Jesucristo .\n",
      "Target: that which we have seen and heard we declare to you , that you also may have fellowship with us . Yes , and our fellowship is with the Father and with his Son , Jesus Christ .\n",
      "Alignment: 1-0 2-1 2-2 2-3 3-4 4-5 5-21 6-9 8-10 10-11 13-14 14-15 17-8 18-17 19-17 21-19 22-20 24-17 25-18 26-26 27-28 27-29 29-32 30-33 31-35 31-36\n",
      "1JN 1:4\n",
      "Source: Escribimos para decirles esto , a fin de que nuestra felicidad sea completa .\n",
      "Target: And we write these things to you , that our joy may be fulfilled .\n",
      "Alignment: 0-13 1-8 2-4 3-3 3-5 3-6 4-7 6-0 9-12 10-13 11-13 12-13 13-14\n",
      "1JN 1:5\n",
      "Source: Este es el mensaje que recibimos de él y que nosotros les declaramos a ustedes : Dios es luz , y no hay ningún vestigio de oscuridad en él .\n",
      "Target: This is the message which we have heard from him and announce to you , that God is light , and in him is no darkness at all .\n",
      "Alignment: 0-8 1-0 1-1 2-2 3-3 3-4 4-15 5-11 6-27 7-9 8-10 10-5 12-11 13-12 14-13 16-16 18-18 19-14 19-19 21-24 22-24 26-25 27-21 29-28\n"
     ]
    }
   ],
   "source": [
    "{\n",
    "    using var srcTrgModel = new ThotIbm1WordAlignmentModel(\"out/VBL-WEB-IBM1/src_trg\");\n",
    "    using var trgSrcModel = new ThotIbm1WordAlignmentModel(\"out/VBL-WEB-IBM1/trg_src\");\n",
    "    using var symmetrizedModel = new SymmetrizedWordAlignmentModel(srcTrgModel, trgSrcModel)\n",
    "    {\n",
    "        Heuristic = SymmetrizationHeuristic.GrowDiagFinalAnd\n",
    "    };\n",
    "    foreach (var row in parallelCorpus.Take(5))\n",
    "    {\n",
    "        var alignment = symmetrizedModel.Align(row.SourceSegment.Lowercase(), row.TargetSegment.Lowercase());\n",
    "\n",
    "        WriteLine($\"{row.Ref}\");\n",
    "        WriteLine($\"Source: {row.SourceText}\");\n",
    "        WriteLine($\"Target: {row.TargetText}\");\n",
    "        WriteLine($\"Alignment: {alignment}\");\n",
    "    }\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "C#"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}