Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IMPORTV3] Check Import Occhab #3077

Closed
wants to merge 485 commits into from
Closed

[IMPORTV3] Check Import Occhab #3077

wants to merge 485 commits into from

Conversation

jacquesfize
Copy link
Contributor

@jacquesfize jacquesfize commented May 28, 2024 •
edited by bouttier
Loading

duplicates #3019

What's done

  • Debug the mapping field form
  • Refactorisation of checks in Occhab
    • add docstring and type annotations
    • split the check_transient_data in multiple methods
    • add checks for boolean typed data columns
    • add check to verify consistency between differents row for one entity
    • add the possibility to generate an uuid based on a id_source in generate_missing_uuid()
  • Rename geometry related functions named geography_xxx by "geometry_xxx"
  • integrate inter-field requirements in the jsonschema in the field mapping creation/update routes
  • A mapping between columns (file > db) can now be removed in the update
  • Debug the entity count
  • Error Code are now available in an enum ImportCodeError

TODO

  • Update existing test file --> refresh error columns
  • Complete existing test
  • Debug un-mapping action on the UI between form for different entities
  • clean the code and the branch history
  • indicate in PR changes brought to init_rows and generate_missing_uuid + change in SyntheseMixin+OcchabMixin !!!
  • Re-enable the check parent entity
    • drop query at the beginning of check_transient_data
    • re-enable check_parent_entity in check_transient_data
    • restore method in parent.py
  • Move init_rows_validity, check_orphan_rows in check_transient_data (in synthese mixin and occhab mixin)
  • restore mapping.update(request.json)
  • Create an Enum for all error code
  • integrate @bouttier PR : [IMPORT] Feat/occhab check review #3120
  • integrate @Pierre-Narcisi PR : [IMPORT] Feat/required #3119
  • Empêcher de supprimer un import si une station importée contient des habitats saisie manuellement afin de ne pas perdre ces derniers (lever une 400 avec un message expliquant qu’il faut manuellement supprimer les habitats suivants en donnant leur ids) -> [IMPORT] Safe deletion of muli-entities import #3151
  • Linéariser les migrations alembic
  • Fonctionnel mais non conforme aux discussion : lorsqu’une station est dupliquée, les erreurs ne sont levées que sur la première occurrence de la station et non pour l’ensemble des lignes.
  • Supprimer la CI sur les branches de dev
  • [Doublons UUID ] erreur silencieuse si doublons sur uuid gn_module_import#459 -> fix(import): per dataset uuid check #3152
  • corrections des tests frontend

@jacquesfize jacquesfize mentioned this pull request May 28, 2024
Closed
Pierre-Narcisi and others added 18 commits June 4, 2024 16:24
@Pierre-Narcisi
Adding duplicate check
4c7da33
@Pierre-Narcisi
add comments
7b2f8c9
@jacquesfize
wip
7ca7113
@Pierre-Narcisi
feat(check) improve check consitancy
7da7116
@jacquesfize
fix uuid_generation (in progress)
5ba9ddc
@jacquesfize
debug field mapping
fd3aec4
@jacquesfize
change update mapping
9d0f370
@jacquesfize
wip
b452a24
@jacquesfize
update preview valid data route to show list of unique entities
250b2a9
@jacquesfize
fix(import,report): correct number of entities imported
0036c2e
@jacquesfize
fix(import,check) : debug missing uuid
587eeb1
@jacquesfize
fix(import,check): fix boolean check
cba916f
@jacquesfize
update test to work with new id checks
28a2d93
@jacquesfize
feat(import-occhab, check) : start of refactorisation
db95e98
@jacquesfize
feat(import, doc): add docstring and type annotation
723b9b6
@jacquesfize
fix(import, naming) : drop 'geography' in geometry related function n…
0a07d70 
…ames
@jacquesfize
feat(import, mapping) : predefined mapping selection is hidden if non…
f42073c 
…e exists, add placeholder, rename form to clarify its function
@jacquesfize
feat(import, check): add docstring + simplify a method signature
d05d168
edelclaux and others added 3 commits July 25, 2024 16:50
@edelclaux @jacquesfize
fix: statistics tooltip display
e6282af 
fix: remove import_count from improt description

feat: add observation_count in synthese statistics

add a todo

feat: update line imported display in import list

feat: update line imported display in import report

fix: "Observations" -> "lignes"

feat(import): setup frontend / e2e tests

fix frontend test
@andriacap @jacquesfize
feat(import,synthese): add the possibility to declare jdd in the impo…
4087502 
…rt file

- Revision alembic to add column unique_dataset_id and src_unique_dataset_id
- Add into bib_fields and into cor_entity_fields
- change the model Synthese Geonature to add unique_dataset_id

Reviewed-by: andriacap

fix: change type of "eng_label" and "desc_label"

In database these two fields are nullable then
change type to allow "None" value

Reviewed-by: andriacap

fix(test): errors with unique_dataset_id

Fix errors based on adding unique_dataset_id

Reviewed-by: andriacap

fix: change revision id (after rebase feat/import)

Reviewed-by: andriacap

fix: nomenclature raise load error

Reviewed-by: andriac

refact: test synthese import jdd

Based on Elie's review

Reviewed-by: andriac

add missing line of code in synthese mixin
@jacquesfize
fix rebase
fd291fc 
update RefGeo to 1.5.3

fix revision

fix rebase develop

fix package lock
jacquesfize and others added 14 commits July 26, 2024 14:30
@bouttier bouttier mentioned this pull request Aug 10, 2024
Merged
47 tasks
@bouttier
fix(import): per dataset uuid check
41867e9 
See PnX-SI/gn_module_import#459
@bouttier
fix(occhab): add server_default=FetchedValue()
975ddd9 
This allows to add an habitat without requiring to set the field value as database default value will be used.
@bouttier
import: add Entity.childs relationship
52b5723
@bouttier
import occhab: check foreign habs before deletion
0475f62 
Prohibit deletion of import with stations including habitats not originating from the same import.
@bouttier
Merge remote-tracking branch 'origin/feat/import' into feat/occhab-check
2de725b
@bouttier bouttier changed the base branch from feat/import to develop August 14, 2024 13:36
@bouttier
Copy link
Contributor

Voir #2833

@bouttier bouttier closed this Aug 14, 2024
@bouttier bouttier deleted the feat/occhab-check branch August 14, 2024 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jacquesfize jacquesfize

Labels
None yet
Projects
Import V3
Status: En cours
Milestone
2.15
Development

Successfully merging this pull request may close these issues.

[IMPORT] Contrôle de données pour la destination occhab
6 participants
@jacquesfize @bouttier @Pierre-Narcisi @andriacap @VincentCauchois @edelclaux