[FEATURE] Client Download Stats

[dblock]

Is your feature request related to a problem?

We'd like to understand usage of OpenSearch clients over time.

What solution would you like?

A dashboard that groups clients together (as in projects under https://github.com/opensearch-project/OpenSearch-clients) and shows their downloads and such.

As an example, we collected some spot data with @nhtruong and @Xtansia:

• NET, according to nuget:
  • OpenSearch.Client:
  • 3.6M lifetime downloads.
  • 218.6K current version downloads (released Apr 18th 2024).
• OpenSearch.Net:
  • 4.0M lifetime downloads.
  • 228.7K current version downloads (released Apr 18th 2024).
  • GH repo has 2.2K views past 14days.
• Rust, according to crates.io:
  • 485K lifetime downloads.
  • 152K current version downloads (released Nov 7th 2023).
  • GH repo has 250 views past 14days.
• Java:
  • Maven Central does not publicise any download metrics.
  • GH repo has 12K views past 14days.
• Hadoop:
  • Maven Central does not publicise any download metrics.
  • GH repo has 1.6K views past 14days.
• Node.JS, according to npmjs: 445,568 weekly download
• Ruby according to rubygems: 19,923,118 lifetime downloads. Latest version, released July 11 have had 218,662 downloads
• Python according to pypistats: 25,185,690 downloads in the past month
• PHP according to packagist: 7 192 319 lifetime downloads

[prudhvigodithi]

[Triage]
Thanks @dblock , its good to have some dashboards on client download and usage numbers, we can observe the trends and patterers for each version and derive some action items.
Adding @getsaurabh02 @bshien

[prudhvigodithi]

Using the following APIs and for the list of clients from this link, we can collect daily total download numbers, index them as new documents each day, and leverage OpenSearch to analyze trends and growth metrics. To begin with we can collect the total download numbers combining for all versions.

Example:

{
  "id": "opensearch-py",
  "total_downloads": 590593208,
}

We can use this number and index daily and using OpenSearch It should be possible to see the incremental growth

Nuget:

https://api.nuget.org/v3/index.json

Npm:

https://api.npmjs.org/downloads/point

Ruby:

https://rubygems.org/api/v1/gems

Python:

https://api.pepy.tech/api/v2/projects

Rust:

https://crates.io/api/v1/crates/opensearch

PHP:

https://packagist.org/packages/opensearch-project/opensearch-php/stats.json

Maven:

Maven does not provide direct download numbers (https://search.maven.org/solrsearch/select?q=g:%22org.opensearch.client%22+AND+a:%22opensearch-java%22&rows=20&wt=json), from GitHub side we collect the repo stats (but wont save the history, the documents will be updated daily with new numbers. Possible to re-index the same data with new _id for the trends)

  "_source": {
    "id": 388880322,
    "description": "Java Client for OpenSearch",
    "forks_count": 182,
    "stargazers_count": 119,
    "watchers_count": 119,
    "size": 27949,
    "open_issues_count": 134,
    "network_count": 182,
    "subscribers_count": 27,
    "transformed_at": "2024-10-11T15:45:15.482189423Z",
    "repository": "opensearch-java"
  },

We can also ingest the repo traffic https://docs.github.com/en/rest/metrics/traffic?apiVersion=2022-11-28#about-repository-traffic.

@dblock @getsaurabh02 WDYT?

[prudhvigodithi]

Go lang

https://pkg.go.dev/github.com/opensearch-project/opensearch-go/v2
Coming from here at present go lang does not provide any metrics/stats that shows the total downloads
golang/go#41805 (comment).

Terraform

https://registry.terraform.io/v1/providers/opensearch-project/opensearch

[prudhvigodithi]

Once the data flows and once we have right visualizations created, similar to https://opensearch.org/release-dashboard, we can have https://opensearch.org/client-stats. :)