You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The DBNL pipeline starts a single Frog process with many threads, this does not work as expected; Frog is way too slow (LanguageMachines/frog#45) and its parallellisation does not work as expected and in fact slows things down.
The simple alternative would be to run a Frog instance for every document, but this will bring initialisation times back into the equation so would not be ideal.
Instead: Redesign the dbnl pipeline to cut the input batch into N batches and start N frogs in parallel on each of these batches. This is not the most ideal form of parallellisation (the batches won't finish at the same time), but probably the best and most realistic choice now.
The text was updated successfully, but these errors were encountered:
The DBNL pipeline starts a single Frog process with many threads, this does not work as expected; Frog is way too slow (LanguageMachines/frog#45) and its parallellisation does not work as expected and in fact slows things down.
The simple alternative would be to run a Frog instance for every document, but this will bring initialisation times back into the equation so would not be ideal.
Instead: Redesign the dbnl pipeline to cut the input batch into N batches and start N frogs in parallel on each of these batches. This is not the most ideal form of parallellisation (the batches won't finish at the same time), but probably the best and most realistic choice now.
The text was updated successfully, but these errors were encountered: