How does Data provider access CLDR data?

[riajain0412]

I was exploring about data providers in ICU4X but I'm little bit confused regarding the process of Data Providers. According to my understanding, Data providers can read JSON file and then provide responses accordingly but where is that JSON file present? Do we have to create blob or what?

[sffc]

You need to convert CLDR data into a format ICU4X accepts at runtime. You have several choices: JSON, Postcard, and Databake. You might choose a different format depending on your needs.

icu4x-datagen is the tool that does this conversion.

You can read more in the data management tutorial here:

https://github.com/unicode-org/icu4x/blob/main/docs/tutorials/data_management.md

[sffc]

cargo install --features experimental,bin icu_datagen

[riajain0412]

Okay, This finally worked. Thank you so much @sffc @Manishearth .
I've one last query that I was now trying WordBreakSegmenter but this time I only included "segmenter/word@1" and "segmenter/dictionary@1" but this gave incorrect results. I have not mentioned feature lstm anywhere. I want to use dictionary based segmentation.

[sffc]

With icu_segmenter version 0.7, if the lstm feature is off and your data file contains "segmenter/dictionary@1" data, yhou should get dictionary segmentation.

Note that the lstm feature could get implicitly enabled if you have a dependency on something that enables it, such as icu_datagen with the experimental feature. You don't need icu_datagen in your Cargo.toml file unless you run it in your build.rs. This is one of the issues we want to clean up before stabilizing icu_segmenter; see #2829

[riajain0412]

Okay, yes, removing icu_datagen from Cargo.toml worked. Thank you so much for your help.

[riajain0412]

Have you guys done any benchmark testing for segmentation wrt ICU4C? I want to know that how ICU4X is better than ICU4C in terms of segmentation.