Autogenerate java docs #29

dgarcia360 · 2020-08-07T11:58:22Z

@lauranovich @tzach Before merging this PR, please merge #30

Added Sphinx support: fixed links, fixed warnings, changed the folder structure, and added build scripts.
Added sphinx-scylladb-theme.
Auto-build Sphinx docs and doxygen using GitHub Actions.
Added multiversion support. Now only builds docs for the latest branch, but conf.py script can be edited to build other documentation versions from this commit. See Multiversion support.

Note: Versions tagged previously to this PR do not have a Sphinx project installed, so it won't be possible to autogenerate docs for those.

This is required to allow new code owned by ScyllaDB. Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Part of Connection initialization will be obtaining info about shards in the host so Connection needs a reference to the host to be able to set up this info. Signed-off-by: Piotr Jastrzebski <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]>

This field will track the details of shards available at the host. It will be set during connection initialization. Multiple such initializations can occure at the same time but each should be setting the same number of shards. Signed-off-by: Piotr Jastrzebski <[email protected]>

This field will be set during initialization and will describe the shard in the host that a connection is associated to. Signed-off-by: Piotr Jastrzebski <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]>

This field will be used for paging queries to fetch all pages from the same host if possible. Signed-off-by: Piotr Jastrzebski <[email protected]>

This policy will try to fetch all pages from the same host if possible. Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]>

HostConnectionPool.cleanupIdleConnections is called from a recurring task. It can race with the initialization of the pool leading to NPE and other concurrency problems. Signed-off-by: Piotr Jastrzebski <[email protected]>

By default the driver has the ability to coalesce schem refresh queries to try and reduce the number of queries sent. In case a view and a table refresh is needed a refresh for the keyspace would be generated. While in most cases thisoptimization is helpfull - in case there are 100s of tables in a single keyspace this will cause fetching all keyspace information which will cause extra load on the system (in scylla's case a single shard). Add the ability to disable this by reusing setting of maxPendingRefreshSchmaRequests value. If its 1 disable coalescing. Signed-off-by: Shlomi Livne <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]>

Make sure groupId is set everywhere to com.scylladb and artifactId has scylla- prefix instead of cassandra- prefix. Signed-off-by: Piotr Jastrzebski <[email protected]>

Those metrics expose the information the driver has about number of shards each node has. New metric is called shard-awareness-info and its value is a Map<com.datastax.driver.core.Host, Integer>. For each host (which represents a node in a cluster) it presents either the number of shards the host has or null if the node does not provide sharding information. Signed-off-by: Piotr Jastrzebski <[email protected]>

Fixes scylladb#14 Signed-off-by: Piotr Jastrzebski <[email protected]>

Signed-off-by: Piotr Jastrzebski <[email protected]>

Previously the pool wasn't available for users before we have created requested number of connections to each shard. This was realized by not making a future returned from initAsync ready. Some users were reporting the initialization taking too long and timing out. This commit changes the way initAsync works. From now on: 1. initAsync will try to open 2 * |number of shards| connections 2. Then it will close connections that are excessive for each shard 3. For shards that didn't get enough connections, connection tasks will be scheduled to create missing connections. Also we are limiting the number of connections each ConnectionTask holds to no more than the number of shards. Signed-off-by: Piotr Jastrzebski <[email protected]>

if connections for a target shard are still being initialized. Signed-off-by: Piotr Jastrzebski <[email protected]>

Previously each ConnectionTask was keeping its own excessive connections (connections that are not to the shard it wants). This patch changes the behaviour so that all ConnectionTasks share excessive connections. This way we not only better limit number of excessive connections but also will quicker find the right connection for each task (some other task may open a connection for it). Signed-off-by: Piotr Jastrzebski <[email protected]>

docs/source/manual/compression/index.md

dgarcia360 · 2020-09-10T15:35:37Z

@tzach I ended overriding the markdown parser to have more flexible relative links: https://github.com/scylladb/java-driver/pull/29/files#diff-1ec61223727a10e5e4c74c68336809fcR80-R101

If you like the solution, we could move part of the logic tot he sphinx_scylladb_theme to make this functionality available to all the documentation projects.

tzach · 2020-09-14T07:54:34Z

@tzach I ended overriding the markdown parser to have more flexible relative links: https://github.com/scylladb/java-driver/pull/29/files#diff-1ec61223727a10e5e4c74c68336809fcR80-R101

Nice. Can you trying pushing this to upstream, so we do not have to manage this code?

tzach · 2020-09-14T07:55:15Z

@dgarcia360 can we automate the READM -> Index rename, so we do not have to do it for each project and sync with the driver upstram?

dgarcia360 · 2020-09-14T17:20:13Z

@tzach Edited the PR to keep all the docs files in their original folders by adding symbolic links and removed the index references from all the documents to make it easier once we rename the files from index to README.

can we automate the READM -> Index rename, so we do not have to do it for each project and sync with the driver upstram?

I'm on it. Already managed to achieve this for the make preview and make dirhml command, but the solution is not working with make multiversion because it's not possible to run custom commands between version builds.

I have already raised the issue here sphinx-contrib/multiversion#45 and submitted a possible solution Holzhaus/sphinx-multiversion#46. Let's wait some days hoping this is merged, if not we'll evaluate other alternatives to rename the files.

Nice. Can you trying pushing this to upstream, so we do not have to manage this code?

The upstream repo datastax/java-driver doesn't have a conf.py file. The code won't be accepted in the original recommonmark extension because is removing the functionality that validates if the relative link file exists to suit the use case. What we can do is moving part of the logic to sphinx_scylladb_theme to make this functionality available to all the documentation projects once we merge this PR.

dgarcia360 · 2020-09-21T13:32:18Z

@tzach @lauranovich The PR is ready to be reviewed. The latest commits automate the README -> Index using the forked repository https://github.com/dgarcia360/sphinx-multiversion. Additionally, I have modified the redirections scripts to support multi-version. Some of these changes will be added to the main theme once this PR is accepted.

lauranovich · 2020-09-21T14:26:00Z

Doc renders nicely and theme is applied.
the only thing I see which needs to be fixed (in a second PR)
is that the docs are branded Datastax.
It needs to be Scylla.

I'll handle this in a separate PR. - @tzach please take a look

haaawk · 2020-09-25T16:22:45Z

Something's wrong with this PR. Why does it have 54 commits - some of them already existing in the repository?

dgarcia360 · 2020-09-28T09:49:33Z

@haaawk The PR was issued when scylla-3.7.1 was tagged as the main branch (Nov 2019). Then, when the lastest branch "latest" was created, I had to update the PR to reflect the newer commits added (2020).

From the files changes, nothing other than the java docs generation has been added. You could either squash and merge this PR to make it one commit, or I could issue a separate PR with just one commit. Please, let me know which option do you prefer.

haaawk · 2020-09-28T11:49:26Z

A separate PR with just one commit would be perfect, @dgarcia360. Thank you!

dgarcia360 · 2020-09-28T16:55:10Z

Closed in favor of #38

haaawk and others added 30 commits August 7, 2019 14:47

Remove license maven plugin

c18a1aa

This is required to allow new code owned by ScyllaDB. Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Add Host field to Connection

61b89b2

Part of Connection initialization will be obtaining info about shards in the host so Connection needs a reference to the host to be able to set up this info. Signed-off-by: Piotr Jastrzebski <[email protected]>

Add ShardingInfo class to store details of shards on host

d5705ce

Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Add ProtocolVersion.isShardingSupported

6ccd1db

Signed-off-by: Piotr Jastrzebski <[email protected]>

Add shardId field to Connection

f9f6dc1

This field will be set during initialization and will describe the shard in the host that a connection is associated to. Signed-off-by: Piotr Jastrzebski <[email protected]>

Fetch sharding info when initializing the connection

bc42e31

Signed-off-by: Piotr Jastrzebski <[email protected]>

Add shard awareness to HostConnectionPool

2bd2e25

Signed-off-by: Piotr Jastrzebski <[email protected]>

Signal usage of optimized driver

702d5f2

Signed-off-by: Piotr Jastrzebski <[email protected]>

Add lastHost field to Statement

1e6338d

This field will be used for paging queries to fetch all pages from the same host if possible. Signed-off-by: Piotr Jastrzebski <[email protected]>

Add PagingOptimizingLoadBalancingPolicy

4106ab1

This policy will try to fetch all pages from the same host if possible. Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Use PagingLoadBalancingPolicy in Cluster

6511f1a

Signed-off-by: Piotr Jastrzebski <[email protected]>

Add PagingOptimizingLatencyTracker

46d29d2

Signed-off-by: Piotr Jastrzebski <[email protected]> Signed-off-by: Piotr Jastrzebski <[email protected]>

Use PagingOptimizingLatencyTracker

565d4ef

Signed-off-by: Piotr Jastrzebski <[email protected]>

Prevent schema.local full scans

972e3f5

Signed-off-by: Piotr Jastrzebski <[email protected]>

Log channel creation exceptions

4e10d5d

Signed-off-by: Piotr Jastrzebski <[email protected]>

Stop cleaning uninitialized pool

b6507ec

HostConnectionPool.cleanupIdleConnections is called from a recurring task. It can race with the initialization of the pool leading to NPE and other concurrency problems. Signed-off-by: Piotr Jastrzebski <[email protected]>

Limit number of opened connections

934fd94

Signed-off-by: Piotr Jastrzebski <[email protected]>

Prepare for publishing to Maven repo

e411a57

Make sure groupId is set everywhere to com.scylladb and artifactId has scylla- prefix instead of cassandra- prefix. Signed-off-by: Piotr Jastrzebski <[email protected]>

[maven-release-plugin] prepare release 3.7.1-scylla-0

2521e74

[maven-release-plugin] prepare for next development iteration

edad52b

Update sharding info when node restarts

9c28fb3

Fixes scylladb#14 Signed-off-by: Piotr Jastrzebski <[email protected]>

[maven-release-plugin] prepare release 3.7.1-scylla-1

354d13d

[maven-release-plugin] prepare for next development iteration

ab9dd27

HostConnectionPool: track number of opened connections per shard

d14ef5d

Signed-off-by: Piotr Jastrzebski <[email protected]>

HostConnectionPool: use connection for any shard

ba3443c

if connections for a target shard are still being initialized. Signed-off-by: Piotr Jastrzebski <[email protected]>

dgarcia360 added 2 commits August 14, 2020 12:47

Fixed merge

1ea9b03

Clean PR

26ebfee

tzach reviewed Aug 16, 2020

View reviewed changes

docs/source/manual/compression/index.md Outdated Show resolved Hide resolved

Replace index before building

254b124

dgarcia360 requested a review from tzach August 28, 2020 14:22

dgarcia360 added 2 commits September 10, 2020 17:29

Overwrite md relative link parser

7d2b859

Add original markers

5341cba

dgarcia360 added 3 commits September 14, 2020 14:47

Keep original folder structure

f192325

Fixed links

b15ff8a

Remove index references from toctree

7fc6f21

dgarcia360 and others added 7 commits September 21, 2020 11:01

Rollback index.md to README.md

9b9b5fb

Fixed multiversion

c56731c

Fixed doxygen multiversion

2883628

Fixed redirection multiversion

ec0148d

Fixed conf.py

848fe5e

Fixed multiversion redirects

8d3be89

Fixed spaces

0320099

lauranovich mentioned this pull request Sep 21, 2020

Clean gh-pages branch #30

Merged

dgarcia360 mentioned this pull request Sep 28, 2020

Autogenerate Java docs (II) #38

Merged

dgarcia360 closed this Sep 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Autogenerate java docs #29

Autogenerate java docs #29

dgarcia360 commented Aug 7, 2020 •

edited

Loading

dgarcia360 commented Sep 10, 2020

tzach commented Sep 14, 2020

tzach commented Sep 14, 2020

dgarcia360 commented Sep 14, 2020 •

edited

Loading

dgarcia360 commented Sep 21, 2020

lauranovich commented Sep 21, 2020

haaawk commented Sep 25, 2020

dgarcia360 commented Sep 28, 2020

haaawk commented Sep 28, 2020

dgarcia360 commented Sep 28, 2020

Autogenerate java docs #29

Autogenerate java docs #29

Conversation

dgarcia360 commented Aug 7, 2020 • edited Loading

dgarcia360 commented Sep 10, 2020

tzach commented Sep 14, 2020

tzach commented Sep 14, 2020

dgarcia360 commented Sep 14, 2020 • edited Loading

dgarcia360 commented Sep 21, 2020

lauranovich commented Sep 21, 2020

haaawk commented Sep 25, 2020

dgarcia360 commented Sep 28, 2020

haaawk commented Sep 28, 2020

dgarcia360 commented Sep 28, 2020

dgarcia360 commented Aug 7, 2020 •

edited

Loading

dgarcia360 commented Sep 14, 2020 •

edited

Loading