Timx 288 marc field method refactor #200

jonavellecuerdo · 2024-07-22T17:24:46Z

Purpose and background context

Field method refactor for transform class MARC (Part 1).

Added private methods for key elements in MARC-formatted XML records (<leader>, <controlfield tag="008">(
Added field methods and corresponding unit tests for alternate_titles and call_numbers.

Note(s):

For the time-being, I've added calls to the private methods at the beginning of get_optional_fields(), which is required by field derivations (that will eventually be moved into their own field methods).
The name get_control_field_general_info was chosen over get_fixed_length_data as there are technically multiple fixed length fields in MARC. The fixed length field that MARC depends on is for code 008, which describes "general information" about the record.

How can a reviewer manually see the effects of these changes?

Run make test and verify all unit tests are passing.

Run CLI command

pipenv run transform -i tests/fixtures/marc/marc_record_all_fields.xml -o output/marc-transformed-records.json -s alma

Output:

2024-07-22 13:23:04,806 INFO transmogrifier.cli.main(): Logger 'root' configured with level=INFO
2024-07-22 13:23:04,806 INFO transmogrifier.cli.main(): No Sentry DSN found, exceptions will not be sent to Sentry
2024-07-22 13:23:04,806 INFO transmogrifier.cli.main(): Running transform for source alma
2024-07-22 13:23:05,330 INFO transmogrifier.cli.main(): Completed transform, total records processed: 1, transformed records: 1, skipped records: 0, deleted records: 0
2024-07-22 13:23:05,330 INFO transmogrifier.cli.main(): Total time to complete transform: 0:00:00.524199

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO

What are the relevant tickets?

https://mitlibraries.atlassian.net/browse/TIMX-288

Developer

All new ENV is documented in README
All new ENV has been added to staging and production environments
All related Jira tickets are linked in commit message(s)
Stakeholder approval has been confirmed (or is not needed)

Code Reviewer(s)

The commit message is clear and follows our guidelines (not just this PR message)
There are appropriate tests covering any new functionality
The provided documentation is sufficient for understanding any new functionality introduced
Any manual tests have been performed and verified
New dependencies are appropriate or there were no changes

ghukill · 2024-07-22T17:39:25Z

One question for both @jonavellecuerdo and @ehanson8: I can't comment directly in the PR, but wondering about moving these static file mappings to methods/properties on the MarcTransformer class?

https://github.com/MITLibraries/transmogrifier/pull/200/files#diff-58573f6e57860b560075cbe825c45aace3f2830bb51d47f8d8dbf4b1b0b5d5ebR14-R30

Seems like we'd want to avoid them reading the file each time they are used, but that could be solved with cached instances, or even just read and save on class init.

ghukill

Left a question about the temporary values you noted in the PR, asking if they could just be called directly even if not part of a field method proper.

Also, maybe dovetails with a general comment about the static variables at the top.

These all seem to have one thing in common: parse some data that multiple fields will need. If so, then feels like we could do that on init for the MarcTransform class somehow, and reuse that data throughout via self. As the first part of the MARC refactoring, might be a good time to consider that pattern.

ghukill · 2024-07-22T18:16:17Z

transmogrifier/sources/xml/marc.py

+        leader_field = Marc._get_leader_field(source_record)
+        control_field_general_info = Marc._get_control_field_general_info(source_record)


I see your thinking a bit in having these temporarily here for use in field that don't have their own methods yet... but what about just calling these new and/or renamed methods from those places directly?

e.g. I see leader_field used, but could call directly:

# content_type if content_type := Marc.json_crosswalk_code_to_name( self._get_leader_field(source_record)[6:7], marc_content_type_crosswalk, source_record_id, "Leader/06", ): fields["content_type"] = [content_type]

What I think this might help guide is how often these are used. If often, it might be worth caching the results on those sub-methods. If not often, it's probably okay to invoke them directly 1-2 times. I'm a big fan of caching -- where you call the method, and it reuses data if safe to cache and already called -- but there is a bit of overhead there.

Either way, might avoid needing these temporary values at all, and would give some insight into usage during future parts of this refactor. Just a thought.

Updated Marc to call the private methods as needed but as discussed, caching is not needed at this time!

ehanson8

Great work, a few minor suggestions!

ehanson8 · 2024-07-22T18:48:00Z

tests/sources/xml/test_marc.py

+        leader_field_insert (str): A string representing a MARC fixed length 'leader'
+            XML element. Defaults to a dummy value.
+        control_field_general_info_insert (str): A string representing a MARC fixed length
+            'general info control field' (i.e., code 008) XML element.
+            Defaults to a dummy value.


Smart change!

Thank you! Heads up: Renamed control_field_general_info_insert -> control_field_insert.

ehanson8 · 2024-07-22T18:55:12Z

tests/sources/xml/test_marc.py

+def test_get_control_field_general_info_if_field_blank():
+    source_record = create_marc_source_record_stub(
+        control_field_general_info_insert='<controlfield tag="008"></controlfield>'
+    )
+    with pytest.raises(
+        SkippedRecordEvent,
+        match=(
+            'Record skipped because key information is missing: <controlfield tag="008">.'
+        ),
+    ):
+        Marc._get_control_field_general_info(source_record)
+
+
+def test_get_control_field_general_info_if_field_missing():


Add raises_exception to test names?

Actually looks like we've used _raises_skipped_record_event in other transforms

Good catch! I renamed the test as suggested.

ehanson8 · 2024-07-22T19:08:19Z

tests/sources/xml/test_marc.py

@@ -894,11 +968,7 @@ def test_marc_record_missing_leader_logs_error(caplog):
    output_records = Marc("alma", marc_xml_records)


I'd update this test name since it no longer logs an error

I updated the test to say skips_record for now, but I think these two tests may be part of the old tests we'll want to deprecate eventually!

ehanson8 · 2024-07-22T19:08:42Z

tests/sources/xml/test_marc.py

@@ -908,11 +978,7 @@ def test_marc_record_missing_008_logs_error(caplog):
    output_records = Marc("alma", marc_xml_records)
    assert len(list(output_records)) == 0
    assert output_records.processed_record_count == 1


Same here with the test name

See previous comment above.

jonavellecuerdo · 2024-07-22T20:29:27Z

One question for both @jonavellecuerdo and @ehanson8: I can't comment directly in the PR, but wondering about moving these static file mappings to methods/properties on the MarcTransformer class?

https://github.com/MITLibraries/transmogrifier/pull/200/files#diff-58573f6e57860b560075cbe825c45aace3f2830bb51d47f8d8dbf4b1b0b5d5ebR14-R30

Seems like we'd want to avoid them reading the file each time they are used, but that could be solved with cached instances, or even just read and save on class init.

As @ghukill and I discussed, we opted to only make the following change: assign 'crosswalk' variables as attributes of Marc transform class for the following reasons:

Setting them as class attributes has the same effect as having them as "loose" global variables in the module in that they are instantiated only once when the Marc module is imported; whereas calling a class / property method for each one would be called for every transformed record.
The idea of having these variables set on the instance via a __post_attrs_init__ was also floated, but transmogrifier does not currently use attrs, this would be a heavier lift. We also spoke about having a custom __init__ method but realized that the base Transformer.__init__ method is "final" and can't be overridden.

ehanson8 · 2024-07-22T20:39:14Z

Setting them as class attributes has the same effect as having them as "loose" global variables in the module in that they are instantiated only once when the Marc module is imported; whereas calling a class / property method for each one would be called for every transformed record.

Good summary and I agree that seems like the best path!

ehanson8 · 2024-07-22T20:44:04Z

transmogrifier/sources/xml/marc.py

+    country_code_crosswalk = load_external_config("config/loc-countries.xml", "xml")
+    holdings_collection_crosswalk = load_external_config(
+        "config/holdings_collection_crosswalk.json", "json"
+    )
+    holdings_format_crosswalk = load_external_config(
+        "config/holdings_format_crosswalk.json", "json"
+    )
+    holdings_location_crosswalk = load_external_config(
+        "config/holdings_location_crosswalk.json", "json"
+    )
+    language_code_crosswalk = load_external_config("config/loc-languages.xml", "xml")
+    marc_content_type_crosswalk = load_external_config(
+        "config/marc_content_type_crosswalk.json", "json"
+    )
+


Perfection!

ghukill · 2024-07-23T13:18:50Z

One question for both @jonavellecuerdo and @ehanson8: I can't comment directly in the PR, but wondering about moving these static file mappings to methods/properties on the MarcTransformer class?
https://github.com/MITLibraries/transmogrifier/pull/200/files#diff-58573f6e57860b560075cbe825c45aace3f2830bb51d47f8d8dbf4b1b0b5d5ebR14-R30
Seems like we'd want to avoid them reading the file each time they are used, but that could be solved with cached instances, or even just read and save on class init.

As @ghukill and I discussed, we opted to only make the following change: assign 'crosswalk' variables as attributes of Marc transform class for the following reasons:

Setting them as class attributes has the same effect as having them as "loose" global variables in the module in that they are instantiated only once when the Marc module is imported; whereas calling a class / property method for each one would be called for every transformed record.

The idea of having these variables set on the instance via a __post_attrs_init__ was also floated, but transmogrifier does not currently use attrs, this would be a heavier lift. We also spoke about having a custom __init__ method but realized that the base Transformer.__init__ method is "final" and can't be overridden.

Thanks for making this change. Kind of minor, but a bit of nice code organization I think. Also, while I was definitely using "loose" in our conversations, it occurred to me those kind of variables are probably most accurately described as "module level" variables. Maybe that's a good term we can use going forward to have a collective understanding of what we're talking about.

ghukill

As a part 1, looks great to me!

ehanson8 · 2024-07-23T13:24:18Z

Thanks for making this change. Kind of minor, but a bit of nice code organization I think. Also, while I was definitely using "loose" in our conversations, it occurred to me those kind of variables are probably most accurately described as "module level" variables. Maybe that's a good term we can use going forward to have a collective understanding of what we're talking about.

I like "module level" and am always in favor of standardizing the vocabulary for our coding!

Why these changes are being introduced: * These updates are required to implement the architecture described in the following ADR: https://github.com/MITLibraries/transmogrifier/blob/main/docs/adrs/0005-field-methods.md How this addresses that need: * Add field methods and corresponding unit tests: alternate_titles, call_numbers * Add private methods for key MARC elements: leader and control field '008' * Rename 'xml' -> 'source_record' * Update dependencies Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/TIMX-288

jonavellecuerdo self-assigned this Jul 22, 2024

jonavellecuerdo force-pushed the TIMX-288-marc-field-method-refactor branch from afa2320 to cdad188 Compare July 22, 2024 17:25

jonavellecuerdo requested review from ghukill and ehanson8 July 22, 2024 17:26

jonavellecuerdo force-pushed the TIMX-288-marc-field-method-refactor branch 2 times, most recently from 2c8c701 to e0d7b44 Compare July 22, 2024 18:07

ghukill reviewed Jul 22, 2024

View reviewed changes

ehanson8 reviewed Jul 22, 2024

View reviewed changes

jonavellecuerdo force-pushed the TIMX-288-marc-field-method-refactor branch from d1c30e6 to 692c090 Compare July 22, 2024 20:17

jonavellecuerdo requested review from ghukill and ehanson8 July 22, 2024 20:32

jonavellecuerdo force-pushed the TIMX-288-marc-field-method-refactor branch from 692c090 to c2a8ff9 Compare July 22, 2024 20:35

ehanson8 approved these changes Jul 22, 2024

View reviewed changes

ghukill approved these changes Jul 23, 2024

View reviewed changes

jonavellecuerdo force-pushed the TIMX-288-marc-field-method-refactor branch from c2a8ff9 to fedc35a Compare July 23, 2024 18:03

jonavellecuerdo force-pushed the TIMX-288-marc-field-method-refactor branch from fedc35a to 61db0f7 Compare July 23, 2024 18:04

jonavellecuerdo merged commit d83c86d into main Jul 23, 2024
5 checks passed

jonavellecuerdo deleted the TIMX-288-marc-field-method-refactor branch July 23, 2024 18:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Timx 288 marc field method refactor #200

Timx 288 marc field method refactor #200

jonavellecuerdo commented Jul 22, 2024 •

edited

Loading

ghukill commented Jul 22, 2024

ghukill left a comment

ghukill Jul 22, 2024

jonavellecuerdo Jul 22, 2024

ehanson8 left a comment

ehanson8 Jul 22, 2024

jonavellecuerdo Jul 22, 2024

ehanson8 Jul 22, 2024

ehanson8 Jul 22, 2024

jonavellecuerdo Jul 22, 2024

ehanson8 Jul 22, 2024

jonavellecuerdo Jul 22, 2024

ehanson8 Jul 22, 2024

jonavellecuerdo Jul 22, 2024

jonavellecuerdo commented Jul 22, 2024

ehanson8 commented Jul 22, 2024

ehanson8 Jul 22, 2024

ghukill commented Jul 23, 2024

ghukill left a comment

ehanson8 commented Jul 23, 2024

		leader_field = Marc._get_leader_field(source_record)
		control_field_general_info = Marc._get_control_field_general_info(source_record)

		@@ -894,11 +968,7 @@ def test_marc_record_missing_leader_logs_error(caplog):
		output_records = Marc("alma", marc_xml_records)

Timx 288 marc field method refactor #200

Timx 288 marc field method refactor #200

Conversation

jonavellecuerdo commented Jul 22, 2024 • edited Loading

Purpose and background context

How can a reviewer manually see the effects of these changes?

Includes new or updated dependencies?

Changes expectations for external applications?

What are the relevant tickets?

Developer

Code Reviewer(s)

ghukill commented Jul 22, 2024

ghukill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehanson8 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonavellecuerdo commented Jul 22, 2024

ehanson8 commented Jul 22, 2024

Choose a reason for hiding this comment

ghukill commented Jul 23, 2024

ghukill left a comment

Choose a reason for hiding this comment

ehanson8 commented Jul 23, 2024

jonavellecuerdo commented Jul 22, 2024 •

edited

Loading