Maven openrewrite plugin, can we skip the "Resolving poms" phase (or at least debug it where the repository locations come from?)

[aaime]

Hi,
I'm learning how to use OpenRewrite in preparation for the GeoServer migration from Wicket 9 to Wicket 10.

As a test, I'm trying to run a simple recipe, the removal of unnecessary semicolons. The code base is large and has many modules, which in turn have many dependencies. Several of the modules are not in the default build profile, and have to be included using profiles.
Now, I'm trying to do a full codebase cleanup, so I'm running something like this:

git clone https://github.com/geoserver/geoserver.git
cd geoserver/src
mvn -nsu -T1C org.openrewrite.maven:rewrite-maven-plugin:run \
  -Drewrite.recipeArtifactCoordinates=org.openrewrite.recipe:rewrite-static-analysis:RELEASE \
  -Drewrite.activeRecipes=org.openrewrite.staticanalysis.RemoveExtraSemicolons \
  -Drewrite.plainTextMasks="*.java" \
  -Drewrite.checkstyleDetectionEnabled=false \
  -DsizeThresholdMb=1 \
  -Drewrite.exclusions="**/src/test/resources/*,**/*.xsd,**/*.xml,**/*.properties,**/*.yml,**/*.json,**/*.md" \
  -Drewrite.runPerSubmodule=true \
  -Dsun.net.client.defaultReadTimeout=5000 \
  -Dsun.net.client.defaultConnectTimeout=5000 \
  -Prelase,communityRelease

As you can see, I'm trying to minimize the files inspected by OpenRewrite, trying to parallelize as much of the work as possible, as a full build is 200+ modules (which I did with some success with an upstream project I'm also working on, GeoTools).

Now, the build is either stuck, or just waiting for too many HTTP timeouts, during the "Resolving Poms" phase... which seems to be downloading poms from maven repositories, recursively. This starts returning a number of error messages that are somewhat baffling:

[INFO] Project [GeoServer] Resolving Poms...
[WARNING] Failed to access maven repository https://central.maven.org/maven2 due to: central.maven.org
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository dav:http://download.osgeo.org/webdav/geotools/ due to: Repository dav:http://download.osgeo.org/webdav/geotools/ is not HTTP(S).
[WARNING] Failed to access maven repository http://snapshots.repository.codehaus.org due to: snapshots.repository.codehaus.org
[WARNING] Failed to access maven repository http://snapshots.repository.codehaus.org due to: snapshots.repository.codehaus.org
[WARNING] Failed to access maven repository http://maven.dyndns.org/glassfish/ due to: Connect timed out

Surprising bits:

• The repositories cited in the logs are often not even in the Geoserve pom files. And not even in the immediate dependencies we manage (geotools, geowebcache, jai-ext, imageio-ext). I tried adding -X but I don't see log statements telling me where OpenRewrite found these repository links
• The webdav is used for mvn deploy only, it's not even listed for dependency lookup
• While I can imagine some rewrites might need a full pom dependency resolution, I'm imagining common recipes do not need such information (like removing unnecessary semicolons)... could this phase simply be skipped?

I've also noticed another surprising behavior. If I remove the release and communityRelease from the build, only a couple dozen modules are left in the reactor. This way the pom resolution phase, while long, completes, and the build eventually does too. Now, unnecessary semicolons are removed... but also in modules not part of the reactor! In particular, modules insrc/extension (activated by -Prelease) and src/community (activated by -PcommunityRelease) have had semicolons removed. So ... the maven plugin is ignoring the project's modular structure, and including in the analysis modules that were not part of the build?

[timtebeek]

hi @aaime ; For large OSS projects we recommend the Moderne CLI; that saves the LST model to disk, such that you only need to build up the model once as opposed to once per recipe run. That should save you considerable time as you iterate through modernizing your project. We make the Moderne CLI available for free to OSS projects. That is likely your best path forward for larger projects, and I hope that's helpful to you there! :)

[aaime]

Yes, I've noticed, and indeed I've spent the morning trying to use the moderne CLI: I don't think I'm going to get anywhere close to it anytime soon.

The CLI took a lot longer to initially load the project, because it's not parallelizing anything. On GeoTools I get to apply two rewrites in a row (with full LTS computation, 3 minutes each) before the CLI is even done computing the LTS (over 6 minutes).
If I have to run a few recipes without recomputing the full LST that will save me time in the longer run, so ... let's put up with it?
Then I try to run recipes, but it has none... I sync the moderne recipes, it downloads 50 families of recipes when I only need 2.... and eventually fails downloading one.
After the sync, possibly because of that single download failure, it claims there are not recipes... I've used another command (don't remember which one) to install the static recipes by name, that works, I can finally run the recipe.
I run it... and it finds no semicolons to remove? (the maven plugin found 60+)
I eventually gave up and went back to the maven plugin out of frustration (I managed to get much more done with it).

Since with GeoServer I'm hitting a wall with the maven plugin too, I've now tried Moderne CLI again.
And it loads the full GeoServer project without checking the poms and doing remote repository searches (so was that actually necessary in the maven plugin?).
It's quick... but gives me no control over which part of the code base I want to apply the changes (remember the maven profiles selectively enabling some subdirectories?).

I give it a try anyways, again, I'm running the RemoveExtraSemicolons recipe.
Runs relatively quickly (35s), and it applies the changes. Good, but I notice I had leftovers of another change. I clean everything (git reset --hard), run the recipe again, and nothing is changed this time... I'm baffled.
Then I guess that maybe LST has been updated during the first run and try to do a "mod build" again.
Takes the same time as the first build. I run the recipe... again no changes....

Is there some magic command? Ah yes, "mod clean"... tried cleaning both builds and runs. "mod build" and "mod run" again, and ... nope, no changes to speak of. Again, I give up.

What can I do to get the maven plugin work, please?

[aaime]

Found out why the recipes were changing files... unlike other refactoring tool Moderne CLI keeps the diff in a file, and one has to do a mod git apply in order to actually change the files.

Looking at the quickstart, it suggests to run other git commands through mod as well (add, commit, push).
I find that disruptive of my usual workflow, and would rather use git directly.
Is it possible, or the commands also interact with the LST and one is supposed to run them in order to apply further rewrites?

[timtebeek]

There's no need to use git though mod; you can just apply the patch as normal. The reason we recommend it in the CLI is because that's inherently intended to run across multiple repositories, in which case it's quite convenient to use the CLI to execute those commands in each git repository. In your case you're likely to start with just a single project, so then regular Git would work just as well.

[aaime]

Phew, that's nice. But so... how does the CLI keep the LST up to date across recipe applications? One might run a few recipes, and decide to commit only a few, and the sources will be modified as a consequence. What re-alignes the LST to the new, up to date, sources?
In a similar way, say that on day 1 I run some recipes, then get changes from other developers of the same project, and decide to run recipe 2... how does it account for the commits that I've received in the meantime?

[timtebeek]

As long as there's no conflicts, or new files you want to target you can reuse the same LST as before. We typically see folks rebuild their LSTs nightly for instance, or after large sweeping changes. The goal is to considerably speed up your recipe runs, which typically do not overlap much in terms of changes. I hope that helps you there! There's a ton of best practices recipes you should now be able to run in seconds and see their proposed changes.