-
Notifications
You must be signed in to change notification settings - Fork 127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Solr support in AtoM #1817
base: qa/2.x
Are you sure you want to change the base?
Solr support in AtoM #1817
Commits on Aug 29, 2024
-
Experimental solr exploration branch init
Set up docker-file which starts up AtoM on port 63001, solr with 3 instances on ports 8981-8983, and a simple php page on port 9001.
Configuration menu - View commit details
-
Copy full SHA for 942ad59 - Browse repository at this point
Copy the full SHA 942ad59View commit details -
WIP Skeleton for CLI task and plugin added. CLI task creates new solr collection, but doesn't populate the index yet.
Configuration menu - View commit details
-
Copy full SHA for bb23cda - Browse repository at this point
Copy the full SHA bb23cdaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0662eaf - Browse repository at this point
Copy the full SHA 0662eafView commit details -
Add solr Information Object model
Converted elastic search information object model and all dependent classes to solr equivalent. WIP: arSolrPluginUtil still has a couple of Elastica references for search specific functions that need to be adapted/converted for solr. QubitSearch has not been converted, so no actual documents are added to Solr index yet. Changes will need to be made to either search itself or all of the update/populate methods inside models to add the documents to the solr index.
Configuration menu - View commit details
-
Copy full SHA for 90f8646 - Browse repository at this point
Copy the full SHA 90f8646View commit details -
Add solr specific methods to QubitSearch
Added solr specific getInstance, enable, and disable methods. WIP: No documents are added to the solr index by the addDocument method in arSolrPlugin.
Configuration menu - View commit details
-
Copy full SHA for dac8e84 - Browse repository at this point
Copy the full SHA dac8e84View commit details -
Add solr indexing using solr REST API
WIP: Data is being indexed using solr's REST API, but it is not searchable since copy fields have not been declared, and the schema is dynamic
Configuration menu - View commit details
-
Copy full SHA for 6df3cdb - Browse repository at this point
Copy the full SHA 6df3cdbView commit details -
fields to be created: - nested (fields: actorRelations, dates) - object (fields: alternativeIdentifiers, digitalObject, findingAid, i18n)
Configuration menu - View commit details
-
Copy full SHA for 9935d15 - Browse repository at this point
Copy the full SHA 9935d15View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2284b49 - Browse repository at this point
Copy the full SHA 2284b49View commit details -
Fix indexing issues due to copy field
Fixed the multiValued parameter in the api request to convert binary to string accurately. Added code to skip adding fields to copy based on the include_in_all mapping value. Also added a function parameter for the stored api request parameter since the copy field does not need to be stored.
Configuration menu - View commit details
-
Copy full SHA for a5d1ba3 - Browse repository at this point
Copy the full SHA a5d1ba3View commit details -
Adding a WIP solr CLI search tool to search solr. Deleted the experimental solr html search page since it is no longer needed.
Configuration menu - View commit details
-
Copy full SHA for 52cd82b - Browse repository at this point
Copy the full SHA 52cd82bView commit details -
Added a helper method for all the Solr HTTP requests and cleaned up the plugin code making the requests.
Configuration menu - View commit details
-
Copy full SHA for ae8630e - Browse repository at this point
Copy the full SHA ae8630eView commit details -
Change CLI solr task to query using solr API
Updated solr CLI search task to use API requests instead of using SolrClient. Using edismax queries instead of regular solr queries to be able to apply boost to fields. Accepting command line field inputs is WIP.
Configuration menu - View commit details
-
Copy full SHA for dd246a7 - Browse repository at this point
Copy the full SHA dd246a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7b18d2a - Browse repository at this point
Copy the full SHA 7b18d2aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 579a513 - Browse repository at this point
Copy the full SHA 579a513View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e9f2e5 - Browse repository at this point
Copy the full SHA 9e9f2e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for d2cccd9 - Browse repository at this point
Copy the full SHA d2cccd9View commit details -
Configuration menu - View commit details
-
Copy full SHA for ffd55d2 - Browse repository at this point
Copy the full SHA ffd55d2View commit details -
Added a class to handles Solr Queries and its parameters
Configuration menu - View commit details
-
Copy full SHA for 1332b8d - Browse repository at this point
Copy the full SHA 1332b8dView commit details -
Update arSolrSearchTask to accept fields param
arSolrSearchTask now accepts fields as a command line parameter. Also fixed minor bugs with the search task and arSolrQuery, and removed the 10 row default from the solr search handler.
Configuration menu - View commit details
-
Copy full SHA for 7a0c786 - Browse repository at this point
Copy the full SHA 7a0c786View commit details -
Configuration menu - View commit details
-
Copy full SHA for fd880cb - Browse repository at this point
Copy the full SHA fd880cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4e8c632 - Browse repository at this point
Copy the full SHA 4e8c632View commit details -
Configuration menu - View commit details
-
Copy full SHA for c50835d - Browse repository at this point
Copy the full SHA c50835dView commit details -
Configuration menu - View commit details
-
Copy full SHA for d5ff971 - Browse repository at this point
Copy the full SHA d5ff971View commit details -
Configuration menu - View commit details
-
Copy full SHA for 13dbbbc - Browse repository at this point
Copy the full SHA 13dbbbcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 95b44c8 - Browse repository at this point
Copy the full SHA 95b44c8View commit details -
Moved solr query related classes to its own folder, added solr query classes WIP: All solr query classes are just placeholder skeletons extending from arSolrAbstractQuery. Also, arSolrPluginQuery needs to adapt addAggs to set up all the appropriate aggregation parameters
Configuration menu - View commit details
-
Copy full SHA for 310440e - Browse repository at this point
Copy the full SHA 310440eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2c4d60a - Browse repository at this point
Copy the full SHA 2c4d60aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 23be945 - Browse repository at this point
Copy the full SHA 23be945View commit details -
Added Range Query support for Solr, and also moved methods for adding params to the Abstract Query class. Also cleaned up indentation. Note about TODO: Elastic Search uses a slightly different format than Solr for Range Queries so all calls to create a Range Query will have to be updated to use te Solr syntax instead of the ElasticSearch.
Configuration menu - View commit details
-
Copy full SHA for 3bfba78 - Browse repository at this point
Copy the full SHA 3bfba78View commit details -
Configuration menu - View commit details
-
Copy full SHA for 34fe21e - Browse repository at this point
Copy the full SHA 34fe21eView commit details -
WIP Added skeletons for Match and MatchAll queries, and moved up common Query params up to arSolrAbstractQuery. Also cleaned up some of the mapping code and added logs for config params as well as the add-field query for debugging.
Configuration menu - View commit details
-
Copy full SHA for 72577dc - Browse repository at this point
Copy the full SHA 72577dcView commit details -
Configuration menu - View commit details
-
Copy full SHA for d805efa - Browse repository at this point
Copy the full SHA d805efaView commit details -
Configuration menu - View commit details
-
Copy full SHA for f30dfe7 - Browse repository at this point
Copy the full SHA f30dfe7View commit details -
Configuration menu - View commit details
-
Copy full SHA for c238922 - Browse repository at this point
Copy the full SHA c238922View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0407bce - Browse repository at this point
Copy the full SHA 0407bceView commit details -
Update indexing mapping methods (WIP)
Updated indexing code to use a recursive function for adding mappings TODO: - Children of top level properties are duplicated. Figure out a way to index these while maintaining ES field names - The JSON request is massive, and may time out? Might have to make add field requests in batches.
Configuration menu - View commit details
-
Copy full SHA for 2754477 - Browse repository at this point
Copy the full SHA 2754477View commit details -
Added types to index mapping methods (WIP)
Appended Qubit types to index mapping methods to succesfully create schema in solr. TODO: addDocument and search queries will need to account of Qubit types
Configuration menu - View commit details
-
Copy full SHA for bc7565d - Browse repository at this point
Copy the full SHA bc7565dView commit details -
Update addDocument method to use types
Updated addDocument method to add the data under appropriate Qubit types
Configuration menu - View commit details
-
Copy full SHA for 56d41ac - Browse repository at this point
Copy the full SHA 56d41acView commit details -
Add setType method to arSolrQuery
Added a setType method to arSolrQuery which appends the provided type to each of the search fields.
Configuration menu - View commit details
-
Copy full SHA for 21be280 - Browse repository at this point
Copy the full SHA 21be280View commit details -
Configuration menu - View commit details
-
Copy full SHA for c9ccd24 - Browse repository at this point
Copy the full SHA c9ccd24View commit details -
Configuration menu - View commit details
-
Copy full SHA for 44a5af7 - Browse repository at this point
Copy the full SHA 44a5af7View commit details -
Set all solr fields to use multiValued (WIP)
Temporarily setting all solr fields to multi valued since quite a few of the fields accept arrays, and the current Elastic Search mapping does not differentiate between atomic and multi valued fields. Also taking out config params since that doesn't seem to be required now that the schema is configured correctly.
Configuration menu - View commit details
-
Copy full SHA for 0b05ce1 - Browse repository at this point
Copy the full SHA 0b05ce1View commit details -
Update arSolrQuery to use correct fields (WIP)
Updated arSolrQuery to call the getAllFields method from arSolrPluginUtil to query the appropriate fields. Also changed the default query type to edismax.
Configuration menu - View commit details
-
Copy full SHA for 5c9fcca - Browse repository at this point
Copy the full SHA 5c9fccaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 791508f - Browse repository at this point
Copy the full SHA 791508fView commit details -
Configuration menu - View commit details
-
Copy full SHA for c183342 - Browse repository at this point
Copy the full SHA c183342View commit details -
Add ResultSet and QubitSolrSearchPager (WIP)
Added classes for ResultSet and QubitSolrSearchPager. ResultSet needs to be adapted to be able to ingest response correctly, and Qubit Types need to be filtered out to be able to display information correctly.
Configuration menu - View commit details
-
Copy full SHA for d98ba68 - Browse repository at this point
Copy the full SHA d98ba68View commit details -
Restructure docs in arSolrResultSet
Added a getDocument function to arSolrResultSet that formats the documents from the result set into a nested associative array to match the structure expected in the templates.
Configuration menu - View commit details
-
Copy full SHA for f4bbf60 - Browse repository at this point
Copy the full SHA f4bbf60View commit details -
Configuration menu - View commit details
-
Copy full SHA for ffa2490 - Browse repository at this point
Copy the full SHA ffa2490View commit details -
Add a new arSolrResult class for use with some templates that interface directly with the result instead of interfacing with documents returned by the result set.
Configuration menu - View commit details
-
Copy full SHA for 00c9bc8 - Browse repository at this point
Copy the full SHA 00c9bc8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4494c16 - Browse repository at this point
Copy the full SHA 4494c16View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9f4d38c - Browse repository at this point
Copy the full SHA 9f4d38cView commit details -
Updated arSolrMatchAllQuery to skip fields entirely and use lucene parser to fetch results. Also updated arSolrSearchTask to add support for MatchAllQuery.
Configuration menu - View commit details
-
Copy full SHA for 45fa2ab - Browse repository at this point
Copy the full SHA 45fa2abView commit details -
Update the docker config to remove references to the old solr folder. Also remove the solr php package as it isn't needed (and available) anymore.
Configuration menu - View commit details
-
Copy full SHA for 8940f1f - Browse repository at this point
Copy the full SHA 8940f1fView commit details -
Fix search results skipping documents due to offset field in the incorrect location.
Configuration menu - View commit details
-
Copy full SHA for fed53e5 - Browse repository at this point
Copy the full SHA fed53e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 19f2dcc - Browse repository at this point
Copy the full SHA 19f2dccView commit details -
Updated arSolrBoolQuery to function with arSolrQuery, added support for should, and added rudimentary support for it in arSolrSearchTask.
Configuration menu - View commit details
-
Copy full SHA for 59d3b3a - Browse repository at this point
Copy the full SHA 59d3b3aView commit details -
Update arSolrExistsQuery and add functionality
Add lucene based existence query for arSolrExistsQuery
Configuration menu - View commit details
-
Copy full SHA for b9fb137 - Browse repository at this point
Copy the full SHA b9fb137View commit details -
Update arSolrRangeQuery to be compatible with Elastica like Range Queries
Configuration menu - View commit details
-
Copy full SHA for 4bd6ed0 - Browse repository at this point
Copy the full SHA 4bd6ed0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4c48ab0 - Browse repository at this point
Copy the full SHA 4c48ab0View commit details -
Configuration menu - View commit details
-
Copy full SHA for d420164 - Browse repository at this point
Copy the full SHA d420164View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7aeefdc - Browse repository at this point
Copy the full SHA 7aeefdcView commit details -
The addSubProperties function calls in arSolrPlugin were setting stored to false which was causing several fields to not be stored while indexing. TODO Fixed this change, but multivalued fields were still broken with the generated query setting all fields as single valued, and indexed results still missing some info. Hard coded all fields as multivalued for now to get working search results, but this will need to updated.
Configuration menu - View commit details
-
Copy full SHA for 687c14f - Browse repository at this point
Copy the full SHA 687c14fView commit details -
Update arSolrPlugin to use curl (WIP)
Updated makeHttpRequest to use curl for better error handling. Also updated autocomplete fields to be multivalued since for scenarios when they have both referenceCode and titles, and performed some code cleanups in arSolrPlugin.
Configuration menu - View commit details
-
Copy full SHA for 73e3fe6 - Browse repository at this point
Copy the full SHA 73e3fe6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3e57dc0 - Browse repository at this point
Copy the full SHA 3e57dc0View commit details -
Fix multi value fields in solr (WIP)
Update solr to only use multivalue fields where they are needed. TODO: arSolrPlugin includes a large list of fields that are being used to test field names for being multivalued. This should be updated to work via yml. arSolrMapping also has a change that sets all partial foreign type id's to be multivalued, which needs to be updated to apply that to only the required id fields.
Configuration menu - View commit details
-
Copy full SHA for 4bbadf6 - Browse repository at this point
Copy the full SHA 4bbadf6View commit details -
Adds arSolrTermQuery for Term Queries using Solr. Also updated arSolrMatchQuery to be an extension of arSolrTermQuery, and updated arSolrQuery to provide more fuzzy results.
Configuration menu - View commit details
-
Copy full SHA for 71b9324 - Browse repository at this point
Copy the full SHA 71b9324View commit details -
Configuration menu - View commit details
-
Copy full SHA for a61031d - Browse repository at this point
Copy the full SHA a61031dView commit details -
Configuration menu - View commit details
-
Copy full SHA for ebfe7ef - Browse repository at this point
Copy the full SHA ebfe7efView commit details -
Add test for arSolrExistsQuery
Adds PHPUnit tests for arSolrExistsQuery class. Refactor arSolrExistsQuery to return exceptions when query generation preconditions are not met before query statement is generated.
Configuration menu - View commit details
-
Copy full SHA for 98faa5a - Browse repository at this point
Copy the full SHA 98faa5aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 740e1ff - Browse repository at this point
Copy the full SHA 740e1ffView commit details -
Add tests for arSolrTermQuery. Also update the term query for handling situations with no terms or no fields.
Configuration menu - View commit details
-
Copy full SHA for d9b50de - Browse repository at this point
Copy the full SHA d9b50deView commit details -
Configuration menu - View commit details
-
Copy full SHA for 02ba680 - Browse repository at this point
Copy the full SHA 02ba680View commit details -
Configuration menu - View commit details
-
Copy full SHA for 931d788 - Browse repository at this point
Copy the full SHA 931d788View commit details -
Fix typos in ArSolrTermQueryTest
Update ArSolrTermQueryTest messages to display 'term query' instead of 'exists query'
Configuration menu - View commit details
-
Copy full SHA for 46f722e - Browse repository at this point
Copy the full SHA 46f722eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9586710 - Browse repository at this point
Copy the full SHA 9586710View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0312789 - Browse repository at this point
Copy the full SHA 0312789View commit details -
Configuration menu - View commit details
-
Copy full SHA for 559199a - Browse repository at this point
Copy the full SHA 559199aView commit details -
Configuration menu - View commit details
-
Copy full SHA for e7fb7b6 - Browse repository at this point
Copy the full SHA e7fb7b6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 70b62a7 - Browse repository at this point
Copy the full SHA 70b62a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9ed96e2 - Browse repository at this point
Copy the full SHA 9ed96e2View commit details -
Update arSolrRangeQuery and ArSolrRangeQueryTest to account for types
Configuration menu - View commit details
-
Copy full SHA for fe40ec6 - Browse repository at this point
Copy the full SHA fe40ec6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7c72c33 - Browse repository at this point
Copy the full SHA 7c72c33View commit details -
Configuration menu - View commit details
-
Copy full SHA for 29678ea - Browse repository at this point
Copy the full SHA 29678eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for aead69c - Browse repository at this point
Copy the full SHA aead69cView commit details -
Update arSolrBoolQuery to use all query types
Update arSolrBoolQuery to use the query params for each of the clauses instead of using edismax queries to extend support for query types that do not use edismax. Also change the _addQuery method to allow all queries that are instances of arSolrAbstractQuery instead of just arSolrQuery
Configuration menu - View commit details
-
Copy full SHA for 43c1faf - Browse repository at this point
Copy the full SHA 43c1fafView commit details -
Add sorting, aggregations to arSolrBoolQuery (WIP)
Add support for sorting and aggregations to arSolrBoolQuery. TODO: - Add tests for arSolrBoolQuery
Configuration menu - View commit details
-
Copy full SHA for 5b8a622 - Browse repository at this point
Copy the full SHA 5b8a622View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6bc285f - Browse repository at this point
Copy the full SHA 6bc285fView commit details -
Add arSolrTermsQuery and associated tests. Also fix typo in a property name in arSolrTermQuery.
Configuration menu - View commit details
-
Copy full SHA for 4c2321c - Browse repository at this point
Copy the full SHA 4c2321cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 64bcce3 - Browse repository at this point
Copy the full SHA 64bcce3View commit details -
Add setType, setPostFilter to arSolrBoolQuery
Add a method to arSolrBoolQuery that sets the types for its child queries. Also add a method for setting filters for bool queries.
Configuration menu - View commit details
-
Copy full SHA for d3c6abf - Browse repository at this point
Copy the full SHA d3c6abfView commit details -
Configuration menu - View commit details
-
Copy full SHA for d4ff1f9 - Browse repository at this point
Copy the full SHA d4ff1f9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 381874b - Browse repository at this point
Copy the full SHA 381874bView commit details -
Configuration menu - View commit details
-
Copy full SHA for a376d45 - Browse repository at this point
Copy the full SHA a376d45View commit details -
Handle types for aggregations in arSolrBoolQuery
Add a method that appends types to the aggregations before the query params are generated.
Configuration menu - View commit details
-
Copy full SHA for 24777ae - Browse repository at this point
Copy the full SHA 24777aeView commit details -
Add method to remove mustClause in arSolrBoolQuery
Add a metod to remove any term queries with a given field from the must clause in arSolrBoolQuery.
Configuration menu - View commit details
-
Copy full SHA for 1159bd9 - Browse repository at this point
Copy the full SHA 1159bd9View commit details -
Add generateQuery method for arSolrPluginUtil
Change the generateQueryString method in arSolrPluginUtil to create a solr query from the input string and rename the method to generateQuery.
Configuration menu - View commit details
-
Copy full SHA for a2da573 - Browse repository at this point
Copy the full SHA a2da573View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4dcb650 - Browse repository at this point
Copy the full SHA 4dcb650View commit details -
Fix the version in arSolrResult
Set the version param from hit in arSolrResult
Configuration menu - View commit details
-
Copy full SHA for 367b7d8 - Browse repository at this point
Copy the full SHA 367b7d8View commit details -
Rename arSolrQuery to arSolrStringQuery
Change arSolrQuery to arSolrStringQuery to avoid confusion with Elastica's Query
Configuration menu - View commit details
-
Copy full SHA for 09af5eb - Browse repository at this point
Copy the full SHA 09af5ebView commit details -
Refactor solr api interfacing to arSolrClient
Refactor code in arSolrPlugin that talked to solr into arSolrClient. Also renamed the query folder to client for clarity, and fixed a bug that was skipping over autocomplete fields in arSolrResult.
Configuration menu - View commit details
-
Copy full SHA for 984c8c3 - Browse repository at this point
Copy the full SHA 984c8c3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f99278 - Browse repository at this point
Copy the full SHA 4f99278View commit details -
Configuration menu - View commit details
-
Copy full SHA for d1ceb6c - Browse repository at this point
Copy the full SHA d1ceb6cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4aef85a - Browse repository at this point
Copy the full SHA 4aef85aView commit details -
Configuration menu - View commit details
-
Copy full SHA for f0876e4 - Browse repository at this point
Copy the full SHA f0876e4View commit details
Commits on Sep 4, 2024
-
Update arSolrPluginQuery to use a bool query
Update arSolrPluginQuery to use a single bool query directly instead of using a query container and a separate boolean query.
Configuration menu - View commit details
-
Copy full SHA for af4b64a - Browse repository at this point
Copy the full SHA af4b64aView commit details
Commits on Sep 10, 2024
-
Add solr methods for updating documents
Add updateDocument and updateDocumentById to arSolrClient which enable updating existing documents in the solr index. Also add functions to arSolrPlugin to call these from AtoM.
Configuration menu - View commit details
-
Copy full SHA for d94fe35 - Browse repository at this point
Copy the full SHA d94fe35View commit details