Dev/issue 8895 search api #89

mcantelon · 2015-10-30T21:43:00Z

Added new fields to File model and add logic to index files in stored AIPs using metsrw to parse file metadata from METS files. It's related to issue #8895.

Hwesta · 2015-11-04T01:06:03Z

storage_service/metsrw/__init__.py

@@ -0,0 +1,13 @@
+from __future__ import absolute_import


Why not specify this as a pip dependency instead of vendoring it? Pip can install from git, and we can start working on getting it into PyPI. See https://github.com/artefactual/automation-tools/blob/dev/dataverse/requirements/base.txt#L4

Thanks... that works slick.

mistydemeo · 2015-11-16T17:59:28Z

One thing I'm wondering about - I'm not sure what the requirements were here. Is there any requirement to support clauses other than AND? E.g. to be able to specify NOT or OR clauses?

mistydemeo · 2015-11-20T23:07:45Z

Ping - any thoughts re: my last comment?

mcantelon · 2015-12-01T22:46:07Z

@mistydemeo Sorry, didn't see this... The requirements are pretty rudimentary... here are the searches that were anticipated: https://wiki.archivematica.org/Research_data_management#METS_parsing

jrwdunham · 2017-11-07T22:27:53Z

docs/search.rst

+
+* uuid (file UUID)
+* package (package UUID)
+* name (enter or partial filename)


Is this a typo? Should "enter" be "entire"?

Also, I think there's a typo in the line below: "PRONUM"

I fixed these typos in #262

jrwdunham · 2017-11-07T22:29:10Z

docs/search.rst

+* min_size (minimum filesize)
+* max_size (maximum filesize)
+* normalized (boolean: whether or not file was normalized)
+* valid (boolean: whether or not file data is valid or malformed)


Wording sounds contradictory: "whether or not file data is valid or malformed"

Also, I think valid should be validated here. No? If so, I think the validated attribute of File only records whether the file has been validated, not whether it is valid. (Unless the help text is wrong in models/event.py::File

Should source_package be listed as a search filter here? FileFilter seems to expose it. It's a bit confusing since it only references Transfers I believe. See the pre-existing Package.index_file_data_from_transfer_mets

I changed validated to valid a nullable boolean field where None indicates not validated, True valid, and False invalid. Migrations and help text updated accordingly in #262.

jrwdunham · 2017-11-07T22:30:12Z

docs/search.rst

+
+   http://<storage service URL>/api/v2/search/file/?min_size=29965171
+
+Here is an example JSON response:


The text above the URL doesn't appear to match what that URL actually searches for.

Call to construct non-existent class `METSWriter` should use `METSDocument`.

jrwdunham

My comments have led to changes in the PR that supersedes this one: see #262

jrwdunham · 2017-11-07T22:54:52Z

storage_service/locations/api/search/router.py

+
+    class Meta:
+        model = models.File
+        fields = ['uuid', 'name', 'file_type', 'min_size', 'max_size', 'format_name', 'pronom_id', 'pipeline', 'source_package', 'normalized', 'validated', 'ingestion_time']


Should source_package in this list be package?

jrwdunham · 2017-11-07T22:58:02Z

docs/search.rst

+* min_size (minimum filesize)
+* max_size (maximum filesize)
+* normalized (boolean: whether or not file was normalized)
+* valid (boolean: whether or not file data is valid or malformed)


Also, I think valid should be validated here. No? If so, I think the validated attribute of File only records whether the file has been validated, not whether it is valid. (Unless the help text is wrong in models/event.py::File

jrwdunham · 2017-11-07T23:00:47Z

storage_service/locations/api/search/router.py

+    filter_backends = (filters.DjangoFilterBackend,)
+    filter_class = FileFilter
+
+    @list_route(methods=['get'])


How is this used? TODO: find out.

jrwdunham · 2017-11-07T23:03:26Z

storage_service/locations/migrations/0016_search_api.py

+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('locations', '0015_gpg_encrypted_space'),


Needs another rebase: 0016_mirror_location_aip_replication.py migration has since been introduced.

jrwdunham · 2017-11-07T23:18:16Z

storage_service/locations/models/package.py

+                metadata['filename'] = premis_object.original_name
+
+                if len(premis_object.object_identifiers[0]):
+                    if premis_object.object_identifiers[0]['type'] == 'UUID':


Now that files in a METS file can have other types of ids (e.g., handle, PURL), we need a more robust way of getting the UUID. I think this needs to loop through all identifiers and find the first UUID one, if it exists.

jrwdunham · 2017-11-07T23:21:15Z

storage_service/locations/models/package.py

+
+                    # Indicate whether or not a file has been validated in metadata and if it passed
+                    if premis_event.event_type == 'validation':
+                        metadata['validated'] = premis_event.outcomes[0]['outcome'] == "pass"


So it seems that the validated attribute of a file does actually mean "is valid". I think maybe the name should be change to is_valid. But if that's the case, then I think we want 3 possible values: True, False, and None...

jrwdunham · 2017-11-09T18:12:36Z

docs/search.rst

+For example, if you wanted to get details about the transfer source location
+contained in the space 6d0b6cce-4372-4ef8-bf48-ce642761fd41 you could HTTP get::
+
+    http://<storage service URL>/api/v2/search/location/?space=7ec3d5d9-23ec-4fd5-b9fb-df82da8de630&purpose=TS


I noticed that if you pass a nonsense param to one of these search endpoints, you get all of the resources. Is that expected/desired? For example http://192.168.168.192:8000/api/v2/search/location/?moonbeam=figaro returns all locations. As does http://192.168.168.192:8000/api/v2/search/location/

jrwdunham · 2017-11-09T23:56:15Z

docs/search.rst

+* min_size (minimum filesize)
+* max_size (maximum filesize)
+* normalized (boolean: whether or not file was normalized)
+* valid (boolean: whether or not file data is valid or malformed)


Should source_package be listed as a search filter here? FileFilter seems to expose it. It's a bit confusing since it only references Transfers I believe. See the pre-existing Package.index_file_data_from_transfer_mets

jrwdunham · 2017-11-10T20:16:29Z

docs/search.rst

+
+* uuid (file UUID)
+* package (package UUID)
+* name (enter or partial filename)


I fixed these typos in #262

jrwdunham · 2017-11-10T20:17:48Z

docs/search.rst

+* min_size (minimum filesize)
+* max_size (maximum filesize)
+* normalized (boolean: whether or not file was normalized)
+* valid (boolean: whether or not file data is valid or malformed)


I changed validated to valid a nullable boolean field where None indicates not validated, True valid, and False invalid. Migrations and help text updated accordingly in #262.

jrwdunham · 2017-11-14T00:06:06Z

I am closing this in favour of #262

Hwesta reviewed Nov 4, 2015
View reviewed changes

Hwesta added the Request: discussion The path towards resolving the issue is unclear and opinion is sought from other community members. label Nov 18, 2015

qubot force-pushed the dev/issue-8895-search-api branch 4 times, most recently from 9dc7738 to 548ab10 Compare December 1, 2015 04:50

qubot force-pushed the dev/issue-8895-search-api branch 3 times, most recently from 6676e60 to 3c5eda6 Compare December 2, 2015 03:45

sevein added this to the 0.9.0 milestone Apr 6, 2016

qubot force-pushed the dev/issue-8895-search-api branch 2 times, most recently from a3ccc17 to 4eb3a27 Compare April 14, 2016 18:58

jhsimpson modified the milestones: 0.10.0, 0.9.0 Jul 26, 2016

jhsimpson removed this from the 0.11.0 milestone May 10, 2017

jhsimpson added this to the 0.11.0 milestone Aug 11, 2017

qubot force-pushed the dev/issue-8895-search-api branch from 4eb3a27 to d59e806 Compare October 12, 2017 03:19

sevein modified the milestones: 0.11.0, 0.12.0 Oct 16, 2017

qubot force-pushed the dev/issue-8895-search-api branch from d59e806 to 6a07894 Compare October 25, 2017 09:55

jrwdunham reviewed Nov 7, 2017

View reviewed changes

qubot force-pushed the dev/issue-8895-search-api branch from 9ae818f to 4f981e4 Compare November 8, 2017 00:14

mcantelon and others added 12 commits November 7, 2017 16:25

Added Django Rest Framework as requirement.

4a7c61f

Add Django REST Framework to settings.

a7c5c8b

Add endpoints and indexing.

94ccd06

Documenting search endpoints.

ec4697b

migration tweak

cca2981

Rename

4d1d214

migration tweak

b2daf9f

fix

7a5d832

Search API docs minor formatting mods

e0a3c57

Fixed metsrw API misuse (METSDocument)

981403b

Call to construct non-existent class `METSWriter` should use `METSDocument`.

Make FLAKE8 happy (linting)

1fafd83

Bumped search API migration

970fd10

qubot force-pushed the dev/issue-8895-search-api branch from ed899d4 to 970fd10 Compare November 8, 2017 00:28

jrwdunham added 2 commits November 8, 2017 17:56

WIP: ingestion time date range search

5103f80

Using dev branch of metsrw: dev/issue-11581-premis-parsing

e4bd212

qubot force-pushed the dev/issue-8895-search-api branch from 8708a3a to e4bd212 Compare November 10, 2017 02:42

jrwdunham mentioned this pull request Nov 10, 2017

Add API endpoints for search #262

Closed

jrwdunham reviewed Nov 13, 2017

View reviewed changes

jrwdunham closed this Nov 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dev/issue 8895 search api #89

Dev/issue 8895 search api #89

mcantelon commented Oct 30, 2015

Hwesta Nov 4, 2015

mcantelon Dec 2, 2015

mistydemeo commented Nov 16, 2015

mistydemeo commented Nov 20, 2015

mcantelon commented Dec 1, 2015

jrwdunham Nov 7, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 10, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 9, 2017

jrwdunham Nov 10, 2017

jrwdunham Nov 7, 2017

jrwdunham left a comment

jrwdunham Nov 7, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 7, 2017

jrwdunham Nov 9, 2017

jrwdunham Nov 9, 2017

jrwdunham Nov 10, 2017

jrwdunham Nov 10, 2017

jrwdunham commented Nov 14, 2017


		http://<storage service URL>/api/v2/search/file/?min_size=29965171

		Here is an example JSON response:

Dev/issue 8895 search api #89

Dev/issue 8895 search api #89

Conversation

mcantelon commented Oct 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mistydemeo commented Nov 16, 2015

mistydemeo commented Nov 20, 2015

mcantelon commented Dec 1, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrwdunham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrwdunham commented Nov 14, 2017