Relationship between Assets-Masters and Assets-Production

[BenjamenMeyer]

We need to understand the relationship between Assets Masters and Assets Production.

I had thought that Assets Masters was an upstream of Assets Production; however there is a lot of stuff that is in Assets Production but not in Assets Masters.

Questions:

• What is the relationship?
• Is Assets Masters just another export of the SVN data?
• Which should be authoritative? One? Both?
• Do we even need Assets Masters?

Depending on the answers we may want to document it in the https://github.com/vegastrike/Vega-Strike-Engine-Source/wiki

[Loki1950]

As far as I can see Assets Masters are the source files for each in game asset while Production is how they are in game ie: binary mesh and compressed textures.

[nabaco]

@pyramid3d might shed more light on this :)

[nabaco]

I would like to just comment that if Assets Master is as the raw material of Asset Production, mean that the actual development happens in Master, and then using some procedure it turns into Production, then I think we should remove the Production repo, and just supply it as a tarball.

[BenjamenMeyer]

@nabaco if that's the case then let's make sure all the parts are in assets masters sober can trace between them, Python included

[ministerofinformation]

Assets Masters was indeed intended as the place to store the source files for generating all production assets. However, since we didn't create a Masters repo until well into the project's lifetime, many existing production assets are currently orphaned.

[Loki1950]

Indeed creating those master blend files is my self imposed task.

[pyramid3d]

@pyramid3d might shed more light on this :)

I can only reiterate what has been already mentioned by @Loki1950 and @ministerofinformation
Some production files do not possess source assets, since those were previously not submitted by the original artists. This applies to all types of assets, music, sounds, models, images, textures, ...

The Assets-Production repo is currently linked as data to the engine repo so that when someone clones the engine repo, he will automatically also clone the assets and after compilation be ready to start exploring the universe. I would advise against removing the assets-production repo or tarbvalling it, as it would make the clone&compile&run process slightly more tedious.

[nabaco]

The Assets-Production repo is currently linked as data to the engine repo so that when someone clones the engine repo, he will automatically also clone the assets and after compilation be ready to start exploring the universe. I would advise against removing the assets-production repo or tarbvalling it, as it would make the clone&compile&run process slightly more tedious.

Not necessarily. We could configure CMake to download the latest assets tarball in compilation

[stephengtuggy]

@nabaco Yes. And the tarball, if zipped up, would likely be smaller than the individual assets. So it might actually fit on GitHub (or somewhere else).

[nabaco]

Yes. We could manage it as a release of Assets-Master. Github does the zipping automatically, and you do clear versioning between Engine versions and Assets versions.
The main question is how does that fit with our thoughts of the git clone remodeling (AKA make the engine a submodule of the assets).

[nabaco]

I would also add to that, and maybe @pyramid3d or @ministerofinformation might shed more light on this, we still don't understand the process of "production", AKA how to turn Assets-Masters into Assets-Production? Is there a script to run? Or each asset should be rendered manually to reach Assets Production?

In the former case, it'll be easy to hook it up to Travis-CI and automate the process. In the latter, we'll have to think how to optimize it.

[stephengtuggy]

You know, reflecting some more on this, I don't really think the engine should be a submodule of the assets either. I think the two should be (more-or-less) completely interchangeable. Both conceptually and in practice, at the source code level. But released together, as VS-UtCS version X.X.X.

On my system, at least, Vega-Strike-Engine-Source and Assets-Production live at the same directory level. That seems to work best in my experience.

Thoughts?

[Loki1950]

The creation of Assets-Masters was very much for future use as there where very few master art source files in the svn repo creating those master .blend files is still an ongoing task.

[ministerofinformation]

There's no process for any degree of automation in getting things from Masters to Production. This doesn't mean that there shouldn't be a means of eventually having a formal and automation-supported process for relating them, but that this will have to be a longer-term goal.

Moreover, the most important role for the Masters repo is that it needs to provide for as much future-proofing as is practical: That is, in a project that is ~2 decades old already, the notions of "appropriate level of detail", "appropriate resolution", and "things the GPU will take care of" have already gone through some major shifts, and, while we can always generate a lower-poly or lower-res texture set that matches what hardware can do at any given time, the reverse is obviously not true.

Some history:

The (very, very) first VS models were made in ... Emacs, iirc (Daniel typed in the coordinates in a text editor).

As the community grew, and we needed more assets, we started getting donations of art in various formats, but often not sources, in any meaningful fashion. Moreover, a fair fraction of these were not responses to any particular intended art need, leading to a heterogeneous collection of styles, file formats, etc. filtered by what we could get into engine-compatible format. One key related challenge is that finalizing import was further complicated by any engine-specific features, ranging from the simple (locations of hardpoints, engine thrust indicators, etc. had to be added by hand with some trial and error), to the fundamental (model coordinate systems, scale of intended vessel, texturing resolution, etc. interact in awkward ways when they don't align well), to the unfortunate (the engine supports much greater modularity/chimerism than most models were defined with).

When we were no longer so starved for models that we could afford to become pickier without having to gaze upon a sea of platonic solids for our stand-in models, we finally did what we should have done in the first place and created a repository specifically for asset sources - but by this time, we had an entire set of production assets with almost no source actually still around.

Hopefully, we get to the right place eventually, even if we are prone to letting expedience derail us at times.

[nabaco]

Thank you very much @ministerofinformation for the detailed explanation! That gives us exactly the context we were missing.
So if I would boil it down to a plan, I'd suggest the following:

• Map all the assets in Assets-Production that don't have a source file and start an activity to recreate sources for those
• Start putting an automatic process to deploy to production Assets-Master

I would also add the following step:

• Move all "orphaned" assets from Assets-Production to Assets-Master, and have all the following development in Assets-Master.
  This would avoid confusion such as we have now, of where did we put what change.

I would consider doing another step, and move all text (AKA python code) files from the Assets repo to a separate repo, as to separate code and art. I'm thinking this since I wouldn't like to deploy the code in a simple tarball, unlike the art files.

Any thoughts?

[stephengtuggy]

@nabaco I like it, except that separating out the Python code might be a lot of work. Also, I believe the py code in the assets is usually customized for a particular universe.

This could still be done, perhaps. Just weighing the pros and cons.

[Loki1950]

@nabaco I like it, except that separating out the Python code might be a lot of work. Also, I believe the py code in the assets is usually customized for a particular universe.

Very much so including differing GUI elements, story missions and fixers (NPC's you interact with) it's all done on the py side.

[BenjamenMeyer]

I'd keep the Python code in masters too; the delta in git isn't going to be that bad there; otherwise fully agree we need to start mapping things out that need to be rebuilt in masters and moving things as appropriate, generating scripts, etc to push stuff over as part of a CI/CD methodology.

That also means we need unit testing around stuff where possible too.

Again, agree with @ministerofinformation that this needs to be a long term goal.

Finally, through 0.8.x we'll keep the engine and assets in sync for versions; but we also need to reverse the relationship - making the assets dependent on the engine and not the other way around -that's technically the case in the code/assets right now, just not reflect in the repository relationships. Sometime after 0.8.x - either starting with 0.9.x or perhaps after that - we can then start releasing the assets on their own schedule as makes sense for the work being done there.

[ministerofinformation]

Long-term, I think some of the organization for the Python scripts may be clearer/cleaner if/when the Python code gets refactored to move data elements (strings of news stories, flight group compositions with asset-specific keys, etc.) into separate files that are loaded in instead of coded as constants into particular scripts. This will make it easier to understand which Python functions are generic/common, and thus likely useful as the base set of functionalities for any asset set, and which are more tightly bound to a particular asset set's necessary functionality.