using BasisMatrices in QuantEcon

[oyamad]

I was trying to include some code that is using BasisMatrices to here, but it fails. The situation is the same as that described in "Circular modules and precompilation errors".

The code is to solve dynamic programs with continuous states by the collocation method using BasisMatrices: Preliminary notebook.

Is there any way to resolve this?

[sglyon]

Very cool notebook! If/when we want to bring something like this into QuantEcon I'll have some tips to offer for how to make even better use of BasisMatrices potential performance.

If we do decide that we want to have BasisMatrices be a dependency of QuantEcon, then we will need to move the functions BasisMatrices currently imports from QuantEcon into BasisMatrices itself. We can then import these routines into QuantEcon so anyone who has been using them from the QuantEcon namespace can still do so.

[oyamad]

@sglyon Thanks. Tips from the author of the library will be much appreciated.

According to this line, BasisMatrices imports gridmake, gridmake!, ckron, fix, fix! from QuantEcon, which are mostly used in quad.jl. As an alternative to moving these to BasisMatrices, would it make sense to setup a new package, like GridTools.jl?

[sglyon]

I think that the "most Julian" thing to do would be to split them into a third package. But... that being said I still wonder if it would be worth the cost of maintaining another package. I wonder if there are any users of those routines outside QuantEcon and BasisMatrices. Any ideas or thoughts on that?

[oyamad]

Alternatively, we could

1. create a package ContinuousDP.jl (or ContinuousDPs.jl?) which will be using BasisMatrices and QuantEcon.

Or

2. define a general ContinuousDP type in QuantEcon and create a package like DPCollocationSolve.jl (the collocation method is not the only way to solve a continuous state DP).

The short run solution should be 1, where we don't make any change to BasisMatrices and QuantEcon. What would be the medium run or long run solution?

[sglyon]

That's totally true and is definitely in line with how much of the Julia package ecosystem works. I think it is a good short run solution.

WARNING: what follows is my very biased opinion, so discount appropriately.

If we make the argument to have continuous state dynamic programming in a separate package, then we could say something similar for DiscreteDP, LQ/RBLQ, quad, MarkovChains, etc. My question is where to draw the line? Do we really want to have our packages become so modular?

One argument in favor of doing this is that each package can have the minimal set of dependencies needed to support that feature. This could be nice for people who only use the current QuantEcon for small pieces. We could definitely make QuantEcon the "umbrella" package that simply imports and re-exports all the functionality from the more modular sub packages.

On the other hand, to me this seems like an extra burden to be placed on the QuantEcon team. I have a hard enough time managing the dozen or so Julia packages I currently maintain, if we were to break quantecon into 6+ package that tension would increase.

I'm totally open to hearing from the rest of the team. Does anyone else have suggestions/ideas for how the future of QuantEcon.jl should be shaped?

[oyamad]

@sglyon I share your concern. And my preference is to put as many routines into QuantEcon as possible; it will be best if I can get everything by one line using QuantEcon.

My point was:

• We don't know yet if my code is worth adding into QuantEcon.
• We cannot evaluate it unless we have working code.
• The usual process is to put the code into QuantEcon and open a PR, but it doesn't work in this case because of the dependencies issue.
• An option that doesn't bother anybody to resolve this issue is to create a (probably temporary) separate working package and discuss whether we want to include it to QuantEcon.

If more functions that use BasisMatrices are expected to come to QuantEcon in near future, then we might discuss resolving the issue (i.e., moving the relevant routines out of QuantEcon) right now.

[sglyon]

Great, I'm glad you are on board.

As I mentioned above, I totally agree with your suggestion that the best plan in the short run for this material is to use a third package that depends on both QuantEcon and BasisMatrices. If/when we decide that QuantEcon is the right home for this code, we will definitely have to resolve the issue of circular dependencies.

[jstac]

+1 for a unified package, as QuantEcon.jl now stands.

@oyamad That's a great notebook! Will you publish it somewhere?

[oyamad]

There is working code now in https://github.com/QuantEcon/ContinuousDPs.jl.

Comments/suggestions are welcome.

[greimel]

It is now possible to register multiple package in (multiple subdirectories of) one repository. (One widely used package that makes use of this is https://github.com/JuliaPlots/Makie.jl.) I managed to split out some sub-packages of QuantEcon in my fork https://github.com/greimel/QuantEconX.jl (I even registered smaller packages in a private package registry for some private project.)

I think it would be more "Julian" to have isolated functionality in individual packages. Maintainers of Julia packages tend to be very hesitant to take additional dependencies, so they are unlikely to use MarkovChains or DDPs from this package when this requires them to depend on all of QuantEcon's dependencies like NLOpt and Optim.

Having all the sub-packages in one repo would probably not really increase the cost of maintenance that @sglyon feared above.

EDIT: Existing code using QuantEcon would not be affected because QuantEcon still reexports the sub-packages. But users can choose to only depend on sub-packages, when they don't need the full package.

[sglyon]

@greimel this is a neat idea! Thanks for sharing. @oyamad (and others) do you feel like BasisMatrices belongs here?