feature(api): single API to handle small and larger (> 1 chunk) files in a unified manner.

[happybeing]

I believe that there is a one-time opportunity for a relatively small API change that will deliver:

• storage cost reductions of around 60%
• network load reductions by a similar amount

I think this should be seriously considered a high priority for the first Autonomi API because:

• storage is a core feature used by almost all Autonomi apps
• the benefits would be enormous
• the implementation relatively simple
• much/all the benefits may be lost if done later

The benefits depend on very early implementation because it would I think be a breaking change to content addressing and the storage APIs. So that any apps using the earlier API may have to re-upload stored data in order to take advantage of the new API, or maintain a backward compatible 'back end' interface to access older data. Newer apps could well also be unable to access data uploaded using apps built with earlier APIs leading to usability and interoperability issues between similar apps along with confusion for developers and app users alike.

There's a forum topic here, in which I erroneously suggested this could be an enhancement to Self-encryption. In fact it would be part of the higher level file/data API which would in turn either use Self-encryption (for larger files) or store encrypted data within the datamap for small files (< 1 chunk).

Currently the APIs default to Self-encryption which generates a minimum of three chunks per file, regardless of whether it is small enough to fit in the data map. In earlier implementations small files were in fact stored in the data map, but this feature was removed at some point. For small files this represents a reduction of 66% in the number of chunks per file and similar reductions in the storage cost of those files. The reduction in the number of chunks needed to be handled by nodes would also be reduced dramatically, making the network considerably more efficient.

I've been unable to locate documentation of the reasons for reverting the earlier implementation to use only Self-encryption (though it may well exist), but from the discussion it appears to be feasible and the points above seem to be valid.

I note that applications can choose to just upload a chunk for a file which is small enough to fit in it, but this will not achieve the benefits mentioned because it would be app dependent, not universal and few apps are likely to implement it. Even those who do will do so in different ways (eg using / not using encrypt, encrypting in different ways).

So this really is a now or probably never choice: a simple API change now, with enormous benefits to the network and its cost effectiveness for storing data benefiting almost all users, especially those using it for personal data or regular apps, which is after all the priority required by the fundamentals of the network (Secure Access For Everyone).