My personal thoughts as OSS Maintainer and real directions for GEDCOM format

[webysther]

I started with genealogy for personal interest 8 years ago, I started like most people, using family search, an amazing tool, it helped me a lot and I learned intuitively over the years. But I'm also a programmer and I was very interested in knowing more about how to keep everything I was storing on another server: photos, audio, documents and data.

I discovered GEDCOM and saw its entire history, with ups and downs. I was even more impressed with the limited vision of the possibilities using more modern technology (gedcomx) and the construction of a specification that is really friendly and that can grow along with the community, in addition to allowing a real packaging of everything that was with the genealogy, in addition to allowing the construction of a format that really allows the stable communication of information, and does not work through standards and sub-standards or specifications that do not have real adherence to a wide community.

So to be very objective, here I am listing what I can call OpenGED because COM makes no sense anymore:

• Adoption of the container model used by epub, with this it would no longer be a text file but a compressed file with support for multiple files, solving the following problems:

  • Format would become genealogy.oged which will be an OCF Zip Container with the following mimetype application/ged+zip
  • Support image, audio, and documents inside an internal folder (maybe named as obj?) through SHA-256 hash (as done by git) as name and with their details referenced inside the ged file.
  • The ged format changes internally and becomes a 100% OpenAPI compliant json for internal media linking.

• Use the MIT over Apache, trademark use is so last year these days.

• Donate the format for a real open community like Linux foundation or Apache foundation. Inside the FamilySearch domain, the adoption will be near zero, take gedcomx as an example.

• Creation of group UID for reference to the same person to make easy merge genealogy:

  • UID-1, UID-67 can be the same person, no need to merge, only use GID = UID-1|UID-67, this is important to make back compatible with old files and no need to create a new UID of sync.

• Support for library like (by using HASH SHA-256) for fixed genealogy, like genealogy in books, to make possible exchange inside the format standard genealogy, example:

  • Names and Genealogy of Ancient Greek Gods and Goddess will the some /library/unique-hash-of-gods-book-genealogy.oged

• Support delta genealogy, when supporting library like genealogy, delta is a MUST because the library genealogy needs updates with new discoveries when inserted in the current genealogy and this can be made with delta support changes in the current genealogy:

  • /delta/unique-hash-of-gods-book-genealogy.oged when changes are made on library genealogy

• Support for a GEDCOM and OpenGED online validator as W3C has.

• MAYBE support for internal versioning, add bare git with git LFS as archive enabled inside the container.

• MAYBE start additional support for a database format like SQLite inside the format (db/types.sqlite, db/events.sqlite) to use common data tables, no need anymore crazy conversion table in specs (ref), only a clear table of information that is released.

• Create a public API / UI for conversation from GEDCOM to OpenGED and libraries provided with docker.

In my humble opinion, this would be a final format that could only have improved attributes as with GEDCOM, but in a way that we could have a compatible document that supports improvements and merges real safely.

I post here because gedcomx is a dead project at this point.

Why the guys (@fisharebest) from webtrees and another real open source project are not in committee?
Why FamilySeach doesn't open the entire website as OpenSource?
Why are there a lot of seeking money companies in the committee and not more technical ones?

References:

• GEDCOMX libraries
• awesome-gedcom "GEDCOM utilities that will ease the pain"
• The GEDCOM X File Format
• GEDCOM 7 GEDZIP

[tychonievich]

I've moved this to a discussion as it does not appear to be proposing any changes to GEDCOM.

Taking your comments in order

• Yes, GEDCOM is one of the oldest structured data file formats, and as such has an interesting history.

• GEDCOM 7 already has GEDZIP, which is similar to epub but with GEDCOM, not XML, as the underlying data language. Is there something more about epub you think we should adopt?

• Everyone has their own license preference. "So last year" is an argumentum novitatis; do you have any specific concerns with the Apache license that might make a case for a change with the legal department at FamilySearch?

• You claim that stewardship of the standard will impact adoption. As a representative of FHISO on the GEDCOM steering committee, I appreciate the appeal of an open community, but I can also say that (1) to date the only family history file formats to gain traction have been stewarded by FamilySearch (some under its previous name, the Family History Department of The Church of Jesus Christ of Latter-day Saints); and (2) the content of the specification is currently maintained by a steering committee the majority of whom are not representatives of FamilySearch.

• UIDs are already part of the spec, and the spec already has a way of marking a structure as having multiple UIDs (by using the UID tag multiple times). Non-destructive merges are also already in the spec under the ALIA tag. Non-destructive merges carry with them a set of nuanced challenges (primarily related to transitivity, scale, and disagreements between researchers) that have made ALIA unpopular in most current applications. I believe there are improvements possible, but those improvements would require a full discussion of semantics, including scalability for large shared trees merging thousands of records of the same famous historical person and handling researcher disagreement and inconsistent "same"/"not same" claims.

  Non-destructive merges are a personal interest of mine; if you have ideas in this space, I'd love to see them!

• I'm not sure I understand the hash-based point. Are you proposing a hash of the genealogical data itself, or of the source? In either case, how is the receiving application to know what is meant by the hash? Is there to be some standard repository of these hashed data where their meanings can be looked up?

  In all of this, why is a hash preferable to a URI? Usually hashes are used to remove the chance of modification, but book genealogies have to be interpreted to be represented in a computer-accessible format and that interpretation admits errors, and hence anticipated updates.

• Deltas are an interesting idea. I've seen some applications use diff/patch for this internally, as GEDCOM is a line-based text format that works well with the same tools used for deltas in programming. I've not yet seen much demand for cross-application patches, though, because there does not seem to be much confidence that the applications share the same starting point to apply the patch on top of. I had a conversation with a few developers about possibly doing block-chain-like hash+patch distributed edit trees at RootsTech several years back, but while it was a fun discussion we didn't see a clear path to adding it to applications or user workflows.

• We have a few third-party validators already present, and the steering committee is currently working to finish defining some metadata to allow the creation of more robust validator toolchains.

• We've discussed internal versioning, and agree with you that this is a "maybe": an interesting idea with some practical value, but with implementation hurdles for most applications and not a current top priority.

• We've discussed alternate serialization formats, but haven't (yet) considered sqlite. Those we have considered we've decided against because GEDCOM's serialization format is already very widely supported in the genealogical community and has features that don't map perfectly to other formats.

• We have a few third-party 5.5.1-to-7.0 converters already present. The steering committee is currently focused on the spec itself, but would welcome additional tool support here if you'd like to help.

• GEDOCM-X is not dead; it is still used by the FamilySearch API and thus implemented by every application that communicates with FamilySearch in that way. It is relatively stable (only a few commits a year) but its file format has not received much attention or adoption: it's primarily used as a wire format.

• If someone wishes to join the steering committee, they should contact @clarkegj or [email protected]

• If by "the website" you mean https://gedcom.io, it is open. If you mean https://familysearch.org, you should ask them, not the GEDCOM community.

• The contributors section of 7.0.0 includes people who reviewed prerelease drafts of the specification, primarily to provide feedback on how readily it could be adopted. Some early designs proved incompatible with many existing tools, and one of our goals was to make a version that could be used by current tools soon. As such, tools with large user bases were particularly of interest, and as you note many of those are commercial companies with the advertising and user support revenue that that business model provides. Since the release of 7.0.0, those companies no longer have any more influence on the spec than others: their representatives may participate here and/or in the project teams and/or ask to join the steering committee, as can any other individuals.

  You mention "not more technical ones". Are there specific companies you think should be represented?

I think your goal in posting this is to propose a new file format, not GEDCOM. There is definitely a place for experimenting with new file formats! FHISO, which I represent on the GEDCOM steering committee, has discussed several, some ideas of which made it into GEDCOM 7.0 and others of which did not. There are also many other file format proposals that I've admired technically, but that have not achieved a path to widespread adoption: GenTech, DeadEnds, Gramps XML, Trepo, decentralized, and so on.
The RootsDev community also periodically revisits these ideas, most recently discussing graph databases as a base model.
I look forward to seeing your contributions to that space, and hope that your results include both small updates we can incorporate into GEDCOM without major disruption to existing tools and larger ideas that be put on the longer-term feature pipelines!

[webysther]

Thanks, @tychonievich, I'll analyze all the new information that you provide and go back in a few days, thanks a lot!