-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Way how to query the tools with Gerbil #22
Comments
Hi Julien, currently, GERBIL is programmed in a way, that it will send a single sentence. After that it will wait for the response of the annotator before the next sentence will be sent. Cheers, |
Thanks Michael. |
@MichaelRoeder, we are finalizing the API of our system for the OKE challenge, and we are still a bit confused regarding how each system will receive the data to process so that GERBIL can compute our performance. It would tremendously help if you could provide an example! Hence, the OKE organizers will most likely have a single NIF file containing the test dataset. What will happen next? Gerbil will parse this NIF dataset, split it into sentences, and call our API with some parameters. We need to tell you how to call the system and you need to let us know what precisely you will send us so we need to know what to parse (and return back!). @jplu, can you please re-open this issue? |
Hi all, @rtroncy, you will receive ASAP an email about how configure the REST API in order to make you system accessible via GERBIL. I am going to report here the main points that @MichaelRoeder described about the integration of the annotators in GERBIL. @MichaelRoeder correct me, if I am wrong. Basically, we are configuring two experiments in GERBIL for evaluating the annotators. The following is a cURL example of a request that GERBIL performs to a single annotator. curl -H "Content-Type:application/x-turtle" -H "Accept:application/x-turtle" -d "Here Goes The NIF-compliant turtle" URI_OF_THE _ANNOTATOR |
Thanks @anuzzolese, this helps. You wrote that GERBIL will send a NIF-compliant turtle containing one ore more sentences while @MichaelRoeder wrote in this comment that it will be a single sentence. I assume the later is correct. |
Hi all, Andrea already described the main points. I just would like to add an example and the URL of a GEBRIL instance that you can use for testing. If you have an annotator for task 1, you will receive POST requests containing a single NIF document as it is defined in the dataset. As the most (or nearly all ?) documents contain only one single sentence, a request will contain something like this:
(I removed the prefix definitions to keep the example easy) GERBIL expects, that the response of the annotator contains a single NIF document and the annotations:
I set up a GERBIL instance on one of our servers, which you can use to test the communication between GERBIL and your annotator: http://139.18.2.164:1235/gerbil Note that this GERBIL instance does not contain the evaluation datasets, even if you can choose them in the front end ;-) Please let me know if you have problems or further questions. Cheers, |
@MichaelRoeder can you clarify what @rtroncy pointed out, i.e.,
This is very important for the description of the evaluation I am going to circulate. Thanks in advance. |
Hi all, GERBIL sends exactly one single document per request. Unfortunately, NIF does not define a class "document". Thus, we decided that for GERBIL a document is every Resource that is an instance of In practice, that means that the example dataset for task 1 contains 2 documents - Does this answer the question or did I missunderstood it? |
OK, this means that if I have 20 sentences then GERBIL sends 20 requests to each system to evaluate. |
What I tried to say is that it simply depends on the strucure of your RDF data. If you are defining every sentence as its own |
Perfectly clear. |
Yes, indeed, clear ... for you @anuzzolese :-) Since you're the only who has seen the test dataset and how you have packaged the rdf data, can you confirm us that it will be shaped as the training dataset and that consequently, we will receive as many requests from GERBIL as there are sentences to process? |
:-) Is this an issue for you @rtroncy ? |
No problem then, we just had to know what to expect. Now, this is clear (hopefully for all) |
Hi,
We were wondering how the participating tools through Gerbil will be queried. Does Gerbil will send one query with a bunch of sentences and waiting one result or it will send one query per sentence and wait one result per sentence?
Thanks.
The text was updated successfully, but these errors were encountered: