I think we should do all 3 of:

• src must have at least X lines of Julia code
• test has a runtests.jl file and at least Y lines of code
• readme or docs or docstrings have at least Z lines of text or code

Currently, I haven't found a source-code counter that works on Julia docstrings correctly, so I think we could omit the docs one to start.

In terms of the numbers, on Slack @rmsrosa suggested X > 5, I suggested 10 for all of them, @StefanKarpinski suggested 20, and @GunnarFarneback suggested calibrating against the existing packages to see the cutoffs.

In https://julialang.org/blog/2021/08/general-survey/ we did some of that but we don't have all the info we'd want there, and also the number of packages has grown a lot since then. But if we do go off of those numbers, only 84% of packages have at least 20 lines of tests, and 89% have at least 10 lines. We would have to look at some examples to see if those are "OK" or should be considered "non-compliant" with the intention of the new rules.

I think also we should require this only for new packages (not new versions). Why?

• New packages are the most likely violators
• New versions generally add lines overall rather than remove them
• If we want to manually override the check (e.g. for some light wrapper package, say), then we only have to do it once, for the new package registration, rather than for every version

IMO we could add new versions later, if we figure out an "overrides" system to persist an allowlist that a package is OK that it doesn't have enough lines of src/test/whatever. But to start with I think new packages is already something useful.

Yeah, new packages are the most common place where we notice people registering essentially empty packages.

In terms of implementation, I already started preparing for this when implementing the license check. In particular, during the AutoMerge process, we place a copy of the source code here:

We could implement it as follows:

1. Add a dependency on PackageAnalyzer.jl, or vendor whatever code we need from there
2. Add a "guideline" for each of the checks we want to do in https://github.com/JuliaRegistries/RegistryCI.jl/blob/master/src/AutoMerge/guidelines.jl. This would just be a function that calls analyze(path_to_package_code) and passes/fails based on the results.
3. Add the guidelines to the list here, after guideline_code_can_be_downloaded (which actually downlaods the code)
4. Add unit tests for the guideline functions, maybe similar to the license check ones: https://github.com/JuliaRegistries/RegistryCI.jl/blob/master/test/automerge-unit.jl#L512

I have a fork with a first cut at this here:
https://github.com/MarkNahabedian/RegistryCI.jl

The various thresholds are specified by environment variables to make them easy to control from the registration workflow. Does that seem reasonable?

I wrote a unit test that works if I run it by hand. From Pkg.test though it gets a file system error.

Even before I made any changes, Pkg.test fails in futime on a directory. I don't know if this is because I'm running on MSWindows.

I'd like to try running the unit tests on GitHub, but I don't know which workflow to enable, or even if GitHub will let me enable them individually.

The various thresholds are specified by environment variables to make them easy to control from the registration workflow. Does that seem reasonable?

So far we've mostly handled this via argumetns passed into the "public" functions:

I think ENV variables are mostly reasonable since this is likely to be used from CI environments, but that we shouldn't have 2 ways of configuration, and since we already have configuration via arguments, we should continue to do so. That is, I think we shouldn't use ENV variables here.

I wrote a unit test that works if I run it by hand. From Pkg.test though it gets a file system error.
Even before I made any changes, Pkg.test fails in futime on a directory. I don't know if this is because I'm running on MSWindows.
I'd like to try running the unit tests on GitHub, but I don't know which workflow to enable, or even if GitHub will let me enable them individually.

The easiest way to have the tests run automatically (and the easiest way to get review feedback) is to open a pull request. That will cause this repositories CI to automatically run (except for integration tests, which a maintainer will need to trigger via Bors). Please open a pull request from your repository to this one.

Those names look reasonable to me

Pull request: #492

I'm stuck. The pre-Julia 1.6 builds are failing because PackageAnalyzer is incopatible. Please suggest how I should work around that. Without thinking, I put a version test in the "applicability"" test for the new guideline, but the failure happens long before then, during cmpatibility checking.