Why do I see serialVersionUID or MLReadable or MLWritable errors

[maziyarpanahi]

Hi,

---

Question

Why do I see an exception/error that has serialVersionUID inside the trace:

Problem:
You are using a model in the wrong Apache Spark version. There are 6 annotators which you can either train and use in Apache Spark 2.3.x/2.4.x or Apache Spark 3.x, but you cannot train in one and use it in the other one. (This is due to the change of Scala 2.11 in Spark 2.3.x/2.4.x to 2.12 in Spark 3.x)

• TokenizerModel
• PerceptronApproach (POS Tagger)
• WordSegmenter
• DependencyParser
• TypedDependencyParser
• NerCrfModel

Solution:

• If you are using .pretrained("NAME_OF_THE_MODEL") and this happens, please report it. We probably forgot to update that model/pipeline for the newer Spark 3.x.
• If you are using an offline version of the models belonging to the 6 mentioned annotators, please download the new one that is compatible with Apache Spark 3.x either by using .pretrained() which takes care of everything for you, or directly from Models Hub
• If you have trained and saved any of those annotators, or in a pipeline, you have no other choice than to do it again this time in Apache Spark 3.x if you wish to move to Apache Spark/PySpark 3.x.

NOTE: Every other model or pipeline provided by Spark NLP or previously saved on your disk by you is compatible with Apache Spark/PySpark 3.x over Scala 2.12, only the annotators mentioned above cannot be used in both Scala 2.11 and 2.12 at the same time.

---

Question

Why do I see an exception/error that has MLReadable or MLWritable inside the trace:

Problem:
Spark NLP provides multiple artifacts for each major Apache Spark version. For instance, if you have installed or using Apache Spark 2.3.x, you must use the Maven/Jar made for that specific version. This error means your Spark NLP from Maven or a Fat JAR is not matching your Apache Spark major version.

Solution:

• Please read the name of the packages carefully in the README or use the correct Fat JAR provided in each Release Notes
• Make sure if you installed pyspark==3.1.x or pyspark 3.0.x you simply use spark=sparknlp.start() without any parameter
• Make sure if you installed pyspark==2.4.x you simply use spark=sparknlp.start(spark24=True) to use the right package
• Make sure if you installed pyspark==2.3.x you simply use spark=sparknlp.start(spark23=True) to use the right package

One tip is to be sure you don't have any leftover in your SPARK_HOME especially on Windows assuming your pip install pyspark==3.1.1 should work while you forgot to point your SPARK_HOME to Apache Spark 3.1.1.