ERROR: This Java ML type cannot be loaded into Python currently

[gourav-sg]

Hi,

I am using EMR 6.2.0 and the versions are mentioned below:
Spark NLP version 3.0.3
Spark NLP display version 1.7
Apache Spark version: 3.0.1-amzn-0

Since I am in VPC, I have downloaded from https://github.com/JohnSnowLabs/spark-nlp-models the Universal Sentence Encoder (tfhub_use_en_2.4.0_2.4_1587136330099.zip)

And then I am trying to load this model back using :

bert_embeddings = BertEmbeddings.load("s3://gourav-bucket/sparknlpmodel/<<path to unzipped file>>/")

And then the program quits with the error mentioned in the subject. I will be grateful for anyones help.

Regards,
Gourav

[gourav-sg]

Hi,
solved this issue which was happening because I was not using sparknlp.start() but just regular SparkSession.builder...... option.

So I compared the configuration variables between two different sessions, one started using sparknlp.start() and the other using SparkSession.start() and after adding the following configuration to the SparkSession things worked fine:

spark = SparkSession.builder.master("local[*]") \
                            .config("spark.driver.memory","16G") \
                            .config("spark.repl.local.jars","file:///root/.ivy2/jars/com.johnsnowlabs.nlp_spark-nlp_2.12-3.0.3.jar") \
                            .config("spark.app.initial.jar.urls","spark://9b8813e6a187:40533/jars/com.google.protobuf_protobuf-java-3.0.0-beta-3.jar") \
                            .config("spark.kryoserializer.buffer.max","2000M") \
                            .config("spark.files","file:///root/.ivy2/jars/com.johnsnowlabs.nlp_spark-nlp_2.12-3.0.3.jar") \
                            .config("spark.serializer","org.apache.spark.serializer.KryoSerializer") \
                            .config("spark.jars","file:///root/.ivy2/jars/com.johnsnowlabs.nlp_spark-nlp_2.12-3.0.3.jar") \
                            .config("spark.jars.packages","com.johnsnowlabs.nlp:spark-nlp_2.12:3.0.3") \
                            .config("spark.app.initial.file.urls","file:///root/.ivy2/jars/org.slf4j_slf4j-api-1.7.21.jar") \
                            .config("spark.driver.maxResultSize","0") \
                            .getOrCreate()

The best option is the use sparknlp.start() in case you are in secured platform, and you have to dynamically load the libraries then perhaps the above will be useful.

Regards,
Gourav

[maziyarpanahi]

• We have full instructions for EMR: https://github.com/JohnSnowLabs/spark-nlp#emr-cluster
• Full instructions for offline use of Spark NLP: https://github.com/JohnSnowLabs/spark-nlp#offline

PS: sparknlp.start() or SparkSession.builder won't work in any cluster especially .master("local[*]"! The SparkSession is created behind the scene and all you can do is configure it before it is started. Like Databricks, EMR, SageMaker, Cloudera, etc. ?(you can discard this if you have a working environment by now)