Solr support in AtoM #1817

Set up docker-file which starts up AtoM on port 63001, solr with 3 instances on ports 8981-8983, and a simple php page on port 9001.

WIP Skeleton for CLI task and plugin added. CLI task creates new solr collection, but doesn't populate the index yet.

WIP

Converted elastic search information object model and all dependent classes to solr equivalent. WIP: arSolrPluginUtil still has a couple of Elastica references for search specific functions that need to be adapted/converted for solr. QubitSearch has not been converted, so no actual documents are added to Solr index yet. Changes will need to be made to either search itself or all of the update/populate methods inside models to add the documents to the solr index.

Added solr specific getInstance, enable, and disable methods. WIP: No documents are added to the solr index by the addDocument method in arSolrPlugin.

WIP: Data is being indexed using solr's REST API, but it is not searchable since copy fields have not been declared, and the schema is dynamic

fields to be created: - nested (fields: actorRelations, dates) - object (fields: alternativeIdentifiers, digitalObject, findingAid, i18n)

Fixed the multiValued parameter in the api request to convert binary to string accurately. Added code to skip adding fields to copy based on the include_in_all mapping value. Also added a function parameter for the stored api request parameter since the copy field does not need to be stored.

Adding a WIP solr CLI search tool to search solr. Deleted the experimental solr html search page since it is no longer needed.

Added a helper method for all the Solr HTTP requests and cleaned up the plugin code making the requests.

Updated solr CLI search task to use API requests instead of using SolrClient. Using edismax queries instead of regular solr queries to be able to apply boost to fields. Accepting command line field inputs is WIP.

Fixed CS-Fixer warnings

Added a class to handles Solr Queries and its parameters

arSolrSearchTask now accepts fields as a command line parameter. Also fixed minor bugs with the search task and arSolrQuery, and removed the 10 row default from the solr search handler.

Moved solr query related classes to its own folder, added solr query classes WIP: All solr query classes are just placeholder skeletons extending from arSolrAbstractQuery. Also, arSolrPluginQuery needs to adapt addAggs to set up all the appropriate aggregation parameters

Added Range Query support for Solr, and also moved methods for adding params to the Abstract Query class. Also cleaned up indentation. Note about TODO: Elastic Search uses a slightly different format than Solr for Range Queries so all calls to create a Range Query will have to be updated to use te Solr syntax instead of the ElasticSearch.

WIP Added skeletons for Match and MatchAll queries, and moved up common Query params up to arSolrAbstractQuery. Also cleaned up some of the mapping code and added logs for config params as well as the add-field query for debugging.

Updated indexing code to use a recursive function for adding mappings TODO: - Children of top level properties are duplicated. Figure out a way to index these while maintaining ES field names - The JSON request is massive, and may time out? Might have to make add field requests in batches.

Appended Qubit types to index mapping methods to succesfully create schema in solr. TODO: addDocument and search queries will need to account of Qubit types

Updated addDocument method to add the data under appropriate Qubit types

Added a setType method to arSolrQuery which appends the provided type to each of the search fields.

Temporarily setting all solr fields to multi valued since quite a few of the fields accept arrays, and the current Elastic Search mapping does not differentiate between atomic and multi valued fields. Also taking out config params since that doesn't seem to be required now that the schema is configured correctly.

Updated arSolrQuery to call the getAllFields method from arSolrPluginUtil to query the appropriate fields. Also changed the default query type to edismax.

Added classes for ResultSet and QubitSolrSearchPager. ResultSet needs to be adapted to be able to ingest response correctly, and Qubit Types need to be filtered out to be able to display information correctly.

Added a getDocument function to arSolrResultSet that formats the documents from the result set into a nested associative array to match the structure expected in the templates.

Add a new arSolrResult class for use with some templates that interface directly with the result instead of interfacing with documents returned by the result set.

Updated arSolrMatchAllQuery to skip fields entirely and use lucene parser to fetch results. Also updated arSolrSearchTask to add support for MatchAllQuery.

Update the docker config to remove references to the old solr folder. Also remove the solr php package as it isn't needed (and available) anymore.

Fix search results skipping documents due to offset field in the incorrect location.

Updated arSolrBoolQuery to function with arSolrQuery, added support for should, and added rudimentary support for it in arSolrSearchTask.

Add lucene based existence query for arSolrExistsQuery

Update arSolrRangeQuery to be compatible with Elastica like Range Queries

The addSubProperties function calls in arSolrPlugin were setting stored to false which was causing several fields to not be stored while indexing. TODO Fixed this change, but multivalued fields were still broken with the generated query setting all fields as single valued, and indexed results still missing some info. Hard coded all fields as multivalued for now to get working search results, but this will need to updated.

Updated makeHttpRequest to use curl for better error handling. Also updated autocomplete fields to be multivalued since for scenarios when they have both referenceCode and titles, and performed some code cleanups in arSolrPlugin.

Update solr to only use multivalue fields where they are needed. TODO: arSolrPlugin includes a large list of fields that are being used to test field names for being multivalued. This should be updated to work via yml. arSolrMapping also has a change that sets all partial foreign type id's to be multivalued, which needs to be updated to apply that to only the required id fields.

Adds arSolrTermQuery for Term Queries using Solr. Also updated arSolrMatchQuery to be an extension of arSolrTermQuery, and updated arSolrQuery to provide more fuzzy results.

Adds PHPUnit tests for arSolrExistsQuery class. Refactor arSolrExistsQuery to return exceptions when query generation preconditions are not met before query statement is generated.

Add tests for arSolrTermQuery. Also update the term query for handling situations with no terms or no fields.

Update ArSolrTermQueryTest messages to display 'term query' instead of 'exists query'

Update arSolrRangeQuery and ArSolrRangeQueryTest to account for types

Update arSolrBoolQuery to use the query params for each of the clauses instead of using edismax queries to extend support for query types that do not use edismax. Also change the _addQuery method to allow all queries that are instances of arSolrAbstractQuery instead of just arSolrQuery

Add support for sorting and aggregations to arSolrBoolQuery. TODO: - Add tests for arSolrBoolQuery

Add arSolrTermsQuery and associated tests. Also fix typo in a property name in arSolrTermQuery.

Add arSolrIdsQuery and associated tests

Add a method to arSolrBoolQuery that sets the types for its child queries. Also add a method for setting filters for bool queries.

Add a method that appends types to the aggregations before the query params are generated.

Add a metod to remove any term queries with a given field from the must clause in arSolrBoolQuery.

Change the generateQueryString method in arSolrPluginUtil to create a solr query from the input string and rename the method to generateQuery.

Set the version param from hit in arSolrResult

Change arSolrQuery to arSolrStringQuery to avoid confusion with Elastica's Query

Refactor code in arSolrPlugin that talked to solr into arSolrClient. Also renamed the query folder to client for clarity, and fixed a bug that was skipping over autocomplete fields in arSolrResult.

Update arSolrPluginQuery to use a single bool query directly instead of using a query container and a separate boolean query.

Add updateDocument and updateDocumentById to arSolrClient which enable updating existing documents in the solr index. Also add functions to arSolrPlugin to call these from AtoM.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Solr support in AtoM #1817

Solr support in AtoM #1817

Commits on Aug 29, 2024

Commits on Sep 4, 2024

Commits on Sep 10, 2024

Solr support in AtoM #1817

Are you sure you want to change the base?

Solr support in AtoM #1817

Commits on Aug 29, 2024

Commits on Sep 4, 2024

Commits on Sep 10, 2024