sru cursor model

Background

The swissbib SRU interface is based on a SearchEngine (SOLR). Especially for search engines running in distributed mode allowing deep paging is nearly prohibited. (For a more detailed description compare Pagination of results Because of this reason the usual paging used with SRU (using the queryParameters startRecord and maximumRecords (overview of SRU query parameters) is limited by our implementation to a fix number of documents (at the moment 50000 documents). If clients wants to fetch more documents the server response is an exception.

To provide clients the possibility to fetch a huge amount of documents (bulk request) we implemented the SOLR cursor model although this is not compatible with the SRU standard.

How to use it

We added an additional query parameter "cursor=" which indicates and starts the use of cursor mode on the server side

http://sru.swissbib.ch/sru/search/defaultdb?query= dc.anywhere = hello world&operation=searchRetrieve&recordSchema=info:srw/schema/1/marcxml-v1.1-light&maximumRecords=10&startRecord=0&recordPacking=XML&availableDBs=defaultdb&cursor=

Because of the cursor parameter the sru service adds an additional element into the result which is called nextCursormark and contains a random value. Example query with cursor

Now the client has to use this value and send it back to the sru server with the next request.

http://sru.swissbib.ch/sru/search/defaultdb?query=+dc.anywhere+%3D+hello+world&operation=searchRetrieve&recordSchema=info%3Asrw%2Fschema%2F1%2Fmarcxml-v1.1-light&maximumRecords=10&startRecord=0&recordPacking=XML&availableDBs=defaultdb&cursor=AoEpMDIzNTMzMjBY

The iteration over a complete resultset (related to the query) is going to be finished if the returned nextCursormark value is equal to the value sent back with the previous request.

first test results

The cursor mechanism seems to be very efficient. Fetching a result set with more than 1 million documents the average query time is more or less between 50 and 150 ms