how to load model inside jar

[sunbuhui]

I make a jar and package explain_document_dl_en model inside jar. The jar structure is below.

So how could I load this model from this jar, my code is below.
val pipeline = PretrainedPipeline("explain_document_dl", lang = "en", diskLocation = Option("model_path_in_jar"))
I really want to know what is its right path.

[maziyarpanahi]

• it is not recommended to bundle such a large file inside your JAR.
• If you wish you do so anyway, you need to load the pipeline offline.

val explainedPipeline = PipelineModel.load("/tmp/explain_document_dl_en_2.0.2_2.4_1556530585689/")

You need to use a relative path to where this directory is once the JAR is packaged.

[sunbuhui]

spark-submit \
      --master local[4]  \
      --conf spark.yarn.priority=2 --conf spark.yarn.maxAppAttempts=1 \
      --conf spark.driver.memory=1g \
      --conf spark.kryoserializer.buffer.max=1g \
      --conf spark.jars.packages=com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:2.7.3 \
      --name nlp_sunhongwei \
      --class com.bytedance.user_profile.TestNLP \
      /Users/admin/Desktop/user_profile_script/output/run/user_profile_script-assembly-0.1.jar 

I will submit to my spark cluster using the above script with this jar, and I don't know where the path you mentioned-"You need to use a relative path to where this directory is once the JAR is packaged. I test val pipeline = PretrainedPipeline("explain_document_dl", lang = "en", diskLocation = Option("/tmp/explain_document_dl_en")), it's wrong.
Q1: Does it mean that I can't load model from packaged jar which bundles my model?
Q2: If I am wrong, can you show me how to load it?
I don't have direct w/r rights to my HDFS cluster, and the cluster net is cutted down from outside so I can't make my cluster
download your offline model. So I believe it's the only way... Sorry to bother you, thank you!

[maziyarpanahi]

If you are in the cluster you need to have access to HDFS. Those permissions are required to access the hdfs:///hadoop/ or your home directory on HDFS.
The example I gave you (/tmp/) was just an example, you need to find a relative path in your jar by yourself. That’s being said, I am not sure bundling the model inside the JAR would work in the cluster but you can try. (Easiest way is to ask for access to the HDFS that is co figured with your cluster)

[sunbuhui]

Thanks, I upload the model to HDFS and make it run, but I am stilling wondering about loading it from jar. When running jar, will it unzip the jar and put it in a tmp directory?

21/02/19 14:01:19 INFO SessionState: Created HDFS directory: /tmp/hive-scratch-dir/SCRATCH_DATE/hive-yarn/sunhongwei/8ce982e4-1c7c-47bf-92c1-1a3dfe83dec1
21/02/19 14:01:19 INFO SessionState: Created local directory: /tmp/yarn/8ce982e4-1c7c-47bf-92c1-1a3dfe83dec1
21/02/19 14:01:19 INFO SessionState: Created HDFS directory: /tmp/hive-scratch-dir/SCRATCH_DATE/hive-yarn/sunhongwei/8ce982e4-1c7c-47bf-92c1-1a3dfe83dec1/_tmp_space.db

I found it in my Spark running time syslog, is it related with my thought? Maybe I can load the model from one of these path.
Also, my workmate used a Chinese NLP package called 'HanLP', it supports load model from jar.(hankcs/HanLP#1579) I thought Spark-NLP also supports it but it doesn't.
Anyway, thanks for your dedication, you really help me a lot!