Automatically wrapping with `Clang.jl` #149

jkozdon · 2021-07-15T23:29:54Z

Experimenting to play with #147

Todo

jkozdon · 2021-07-15T23:30:37Z

Doesn't work with everything yet but you can call initialize and finalize.

jkozdon · 2021-07-15T23:41:14Z

Input:

# For some reason you need to initialize MPI before using multiple libraries.
# Need to debug this
using MPI
MPI.Initialized() || MPI.Init()
using PETSc
for petsclib in PETSc.LibPETSc.petsclibs
    println(splitpath(petsclib.petsc_library)[end])
    @show PETSc.initialized(petsclib)
    @show PETSc.initialize(petsclib)
    @show PETSc.initialized(petsclib)
    @show PETSc.finalized(petsclib)
    @show PETSc.finalize(petsclib)
    @show PETSc.initialized(petsclib)
    @show PETSc.finalized(petsclib)
    println()
end

Output:

libpetsc_double_real_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true

libpetsc_single_real_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true

libpetsc_double_complex_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true

libpetsc_single_complex_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true

jkozdon · 2021-07-16T02:07:34Z

I guess I still need to sort out the types (But I think it should be possible with more walking of the syntax tree).

psanan · 2021-07-20T07:57:25Z

Lazy question, but what's the high-level procedure here? What is now automatic and what still has to be maintained "by hand"? Are the specific things here related to various PETSc components all of the "by hand" flavor, or are those things that have been generated automatically and are then checked in?

vchuravy · 2021-07-20T09:13:32Z

I think everything in lib is autogenerated.

jkozdon · 2021-07-20T15:19:25Z

As @vchuravy said, the file in lib is generated from the PETSc header files with the code in gen. These produce pretty raw wrappers, that I am then expanding on to make it all a little more friendly to work with.

There is a bunch of stuff we do not currently wrap (look at generator.toml), some of it will need to be sorted out and some of it won't (I think I know / have a plan for how to support most of what will be needed, just waiting to fix it until I am ready to use it / write the test).

There probably some gotchas left in there with respect to system / build dependent types that will need to be sorted out, but right now it does support multiple PETSc builds living side-by-side in Julia.

@psanan shoot me an email if you'd like to VTC to chat about what I am doing, I'd be more than happy to have feedback.

psanan · 2021-08-05T09:02:00Z

Docs suggestion, re our chat our yesterday. Perhaps somewhere like here one could add language like "PETSc.jl's high-level, easy-to-use interface exposes only some of the functionality of PETSc. However, PETSc.jl includes an automatically-generated wrapper layer which directly exposes more of the PETSc API, which may be of use to existing expert users. See (link to minimal example/test of using the wrappers, directly) for an example".

jkozdon · 2021-08-05T15:05:54Z

Thanks for the recommendation. I've put it as part of the PR task list.

ViralBShah · 2024-10-16T14:23:51Z

I'm going to try to just get this branch working without worrying about rebasing. I am already able to generate the full petsc_library.jl but it doesn't load yet. You can try it out on this branch by running generator.jl.

I don't think major changes have happened since this branch was created and now.

boriskaus · 2024-10-16T14:27:23Z

fine with me; ideally we would change the existing tests that are currently in main to only use the auto-generated wrappers, which would simplify maintaining it.

boriskaus · 2024-10-16T15:28:28Z

We don't have to strictly follow what was done before, since Clang.jl has presumably improved a lot over the last few years.

If I simplify generator.toml to:

[general]
library_name = "$petsc_library"

use_julia_native_enum_type = true
use_deterministic_symbol = true

prologue_file_path = "./prologue.jl"
# epilogue_file_path = "./epilogue.jl"

auto_mutability = true              # Create mutable structures:
auto_mutability_with_new = true     # Allow creating empty structs
auto_mutability_ignorelist = []
auto_mutability_includelist = []

extract_c_comment_style = "doxygen"

export_symbol_prefixes = ["CX", "clang_"]

It essentially compiles the whole library with a few exceptions:

julia> include("generator.jl")
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscbag.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsys.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscconf.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscconf_poison.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfix.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmacros.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscversion.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsc/private/petscadvancedmacros.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsystypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmath.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscerror.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscviewertypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscoptions.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsclog.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctime.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscbt.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscstring.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsclogtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsclogdeprecated.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmatlab.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdraw.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdrawtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdevice.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdevicetypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsf.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsftypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscvec.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscis.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsectiontypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscistypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscviewer.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsection.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmda.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdm.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmat.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmatcoarsen.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfetypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdstypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmlabel.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdt.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmdatypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpf.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscao.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfe.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscspace.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdualspace.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmcomposite.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmpatch.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplex.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpartitioner.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplextypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfv.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfvtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmfield.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplextransform.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplextransformtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmredundant.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmshell.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmsliced.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmswarm.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmstag.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmproduct.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscds.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsccharacteristic.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscts.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsnes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscksp.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpc.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpctypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscconvest.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctao.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctaolinesearch.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctao_deprecations.h
[ Info: Parsing headers...
/Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsys.h:98:6: error: "PETSc was configured with Open MPI but now appears to be compiling using a non-Open MPI mpi.h"
[ Info: Processing header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsc.h
[ Info: Building the DAG...
┌ Warning: [CollectDependentSystemNode]: found symbols in the system headers: [:MPI_Info, :MPI_Op, :MPI_Status, :MPI_Status, :MPI_Datatype, :off_t, :__darwin_off_t, :MPI_Comm, :MPI_Aint, :MPI_File, :ADIOI_FileD, :MPI_Request, :MPI_Group, :MPI_Offset, :MPI_Win]
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/passes.jl:95
[ Info: Emit Julia expressions...
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESObjectiveComputeFunctionDefaultFD, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESObjectiveComputeFunctionDefaultFD(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESComputeJacobianDefault, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESComputeJacobianDefault(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESComputeJacobianDefaultColor, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESComputeJacobianDefaultColor(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeFunction, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeFunction(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeMFFunction, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeMFFunction(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeJacobian, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeJacobian(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:TSComputeRHSFunctionLinear, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) TSComputeRHSFunctionLinear(TS, PetscReal, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:TSComputeRHSJacobianConstant, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) TSComputeRHSJacobianConstant(TS, PetscReal, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
[ Info: [ProloguePrinter]: print to /Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl
[ Info: [GeneralPrinter]: print to /Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl
[ Info: [EpiloguePrinter]: print to /Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl
[ Info: Done!
"/Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl"

boriskaus · 2024-10-16T16:17:25Z

@ViralBShah
I've looked at the tests that are marked with FIXME, and most of the changes are due to PETSc changing the function calls (slightly) during more recent versions. We can port that manually but it probably makes more sense to try and autowrap the full library and use that instead (surely easier to maintain).

Also, this PR is very large. Perhaps it is simpler to start a new one from main?

ViralBShah · 2024-10-16T16:34:11Z

Yes, the simpler thing would be to start the generator on a new PR in main, and just get that running first. And then piecemeal port things over to the new generated API. But not being a user of PETSc, that is where I will start running into my limitations here. Let's push one step forward at least.

I do feel that on main, it would be nice to fix all the FIXME things first so that everything that was working in the past is working again now, and tagged - giving a solid baseline for the new stuff.

boriskaus · 2024-10-16T17:06:16Z

I’m a PETSc user (well at least of part of the library). Yet I’m only a beginner when it comes to using clang to port it all.
I did notice in a recent project that it is possible to automatically make julia docs out of doxygen strings (which every routine in petsc has).
It would be fantastic if we can automatically provide help for all of the routines (even when the help is strictly speaking only valid for the c routines)

If you leave out the functions that have ccalls in the current main, there are only a limited number of additional functions remaining that are mainly there to make using petsc more julia-like. Those ones could be gradually expanded to include more functionally, but as long as we call autowrapped routines throughout it will be much easier to maintain this.

ViralBShah · 2024-10-16T17:11:38Z

So with your generated library, did you try doing include petsc_library.jl? Getting it to load cleanly is the first step. I assume you used the generator.toml that is in this branch (which has a lot of stuff to exclude and such).

I see what you are saying about the auto-generated comments for the wrappers. Wouldn't we still need the docs that are in the Julian interface as it stands right now - or are you thinking those could be completely replaced with this auto-generated stuff?

boriskaus · 2024-10-16T17:16:42Z

So with your generated library, did you try doing include petsc_library.jl? Getting it to load cleanly is the first step. I assume you used the generator.toml that is in this branch (which has a lot of stuff to exclude and such).

I didn’t manage to load it just now; I was using a simpler generator.toml shown in one of the comments

I see what you are saying about the auto-generated comments for the wrappers. Wouldn't we still need the docs that are in the Julian interface as it stands right now - or are you thinking those could be completely replaced with this auto-generated stuff?

I think it can be replaced with autogenerated stuff. Multiple dispatch will take care of the rest.

boriskaus · 2024-10-28T14:04:53Z

@ViralBShah, @vchuravy

One of the reasons for the wrapper file to be rather complex is that the different versions of the PETSc libraries have different definitions of PetscInt, PetscScalar, etc. which can be 32bit, 64bit etc.
In the main branch, we handle this by interpolating the values of $PetscInt, $PetscScalar into function calls as can be seen here

Yet, this doesn't work for structures, which are instead defined as:

mutable struct DMSTAGSTENCIL{PetscInt}
    loc::DMStagStencilLocation
    i::PetscInt
    j::PetscInt
    k::PetscInt
    c::PetscInt
end

At the time the initial wrapping was done, it wasn't possible to automatically wrap C structures to give something like above. Instead Clang gives something like:

mutable struct DMStagStencil
    loc::DMStagStencilLocation
    i::PetscInt
    j::PetscInt
    k::PetscInt
    c::PetscInt
    DMStagStencil() = new()
end

Question: do you know whether there is a way to automatically add types to the structures using Clang?
If yes, it would greatly simplify the generator.toml file.

EDIT: @albert-de-montserrat pointed out that this is actually done already in wrap.jl. The disadvantage is that it has to be specified manually for every struct as:

(x, s) -> add_struct_type(x, s, ("MatStencil",), ("PetscInt",)),

boriskaus · 2024-11-05T22:28:57Z

@ViralBShah it now loads again and some of the tests work. Quite a bit of work remains to automatically wrap the structs (if that is possible at all).

boriskaus · 2024-11-06T05:54:12Z

In principle yes, `structs` do get wrapped automatically with `Clang`, Yet, I had to deactivate a large number of them to get the package to load (plus all functions that call unwrapped `structs`). I suppose that this is what was done before as well. Having a way to automatize this process would certainly help.

ViralBShah · 2024-11-06T10:19:03Z

Perhaps @Gnimuc can chime in, if they have the time to look at this.

This adds an auto-wrapping tool that creates more julia-friendly functions for the PETSc functions created by the Clang.jl. It is tested on the DMSTAG routines where most of the routines defined in the petscdmstag.h header file are now translated, including their doc strings. There is still some stuff that needs manual work, but that is mostly indicated by "TODO" comments. Tests are also provided for most routines Since functions rarely change between different PETSc versions, it is unlikely that much needs to be re-wrapped between different PETSc releases. I expect that the "wrap_functions.jl" routine will also be helpful in many of the other petsc header files, but have not tested this yet.

boriskaus · 2024-11-15T17:27:19Z

took a bit of time but I now added a (mostly) automatic wrapper tool that creates Julia-friendly functions of the functions generated by Clang.jl. This includes docstrings and tests, which makes the whole experience of using PETSc from Julia much more pleasant.
This is currently tested for the DMSTAG routines (see dmstag_wrapper.jl).

ViralBShah · 2024-11-15T17:35:25Z

When I looked at this last, it felt like redoing this on top of master picemeal may be easier than trying to rebase this PR to master (which at least for me felt really complex).

boriskaus · 2024-11-15T18:06:22Z

When I looked at this last, it felt like redoing this on top of master picemeal may be easier than trying to rebase this PR to master (which at least for me felt really complex).

Not sure I understand what you mean.

I am trying to port the tests from main into this branch bit by bit, so we can ensure that we have at least the same functionality

ViralBShah · 2024-11-15T20:39:15Z

Ah - I suppose the merge conflicts will then get resolved when all tests are migrated.

jkozdon force-pushed the jek/gen branch from 985a085 to 89a1184 Compare July 16, 2021 21:17

jkozdon mentioned this pull request Jul 16, 2021

Use CBindings.jl or Clang.jl to build raw interface? #147

Open

jkozdon force-pushed the jek/gen branch 3 times, most recently from 1e39b30 to 97e39a4 Compare July 19, 2021 23:32

This was referenced Jul 20, 2021

DM and DMDA interface #120

Merged

Revisit MPI after / with #149 #150

Open

jkozdon force-pushed the jek/gen branch 3 times, most recently from 16b5f8e to fe80c8f Compare July 20, 2021 04:03

jkozdon force-pushed the jek/gen branch 2 times, most recently from 53fc7ab to aa8e660 Compare July 21, 2021 23:09

jkozdon force-pushed the jek/gen branch 4 times, most recently from afbe26f to 0f22d51 Compare August 3, 2021 19:55

boriskaus marked this pull request as ready for review August 3, 2021 21:49

jkozdon force-pushed the jek/gen branch 4 times, most recently from 3e3e0f9 to 1850ac4 Compare August 4, 2021 17:53

jkozdon force-pushed the jek/gen branch from 1850ac4 to 082a191 Compare August 5, 2021 15:06

boriskaus added 2 commits November 5, 2024 23:17

loads again (but that's about it; the rest is deactivated

52a35f0

activate some tests again

b6dc09a

Merge remote-tracking branch 'origin/jek/dmstag' into jek/gen

44ae7b3

boriskaus added 8 commits November 6, 2024 15:32

add dmstag changes

70f3eed

fixed a few tests

0debe97

activate more tests

78d3251

few additional tests

abdd1e4

more (DMStag) functions and structs included

65ef1dc

missing file

fb32fc1

wrap more functions

5015b81

boriskaus added 3 commits November 16, 2024 00:07

activate more dmstag tests

63e5fc3

typo

f38bc75

activate dmda tests

9238f21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically wrapping with `Clang.jl` #149

Automatically wrapping with `Clang.jl` #149

jkozdon commented Jul 15, 2021 •

edited

Loading

jkozdon commented Jul 15, 2021

jkozdon commented Jul 15, 2021 •

edited

Loading

jkozdon commented Jul 16, 2021

psanan commented Jul 20, 2021

vchuravy commented Jul 20, 2021

jkozdon commented Jul 20, 2021

psanan commented Aug 5, 2021

jkozdon commented Aug 5, 2021

ViralBShah commented Oct 16, 2024

boriskaus commented Oct 16, 2024

boriskaus commented Oct 16, 2024

boriskaus commented Oct 16, 2024 •

edited

Loading

ViralBShah commented Oct 16, 2024

boriskaus commented Oct 16, 2024

ViralBShah commented Oct 16, 2024 •

edited

Loading

boriskaus commented Oct 16, 2024

boriskaus commented Oct 28, 2024 •

edited

Loading

boriskaus commented Nov 5, 2024

boriskaus commented Nov 6, 2024 via email •

edited

Loading

ViralBShah commented Nov 6, 2024

boriskaus commented Nov 15, 2024

ViralBShah commented Nov 15, 2024

boriskaus commented Nov 15, 2024

ViralBShah commented Nov 15, 2024

Automatically wrapping with Clang.jl #149

Are you sure you want to change the base?

Automatically wrapping with Clang.jl #149

Conversation

jkozdon commented Jul 15, 2021 • edited Loading

jkozdon commented Jul 15, 2021

jkozdon commented Jul 15, 2021 • edited Loading

jkozdon commented Jul 16, 2021

psanan commented Jul 20, 2021

vchuravy commented Jul 20, 2021

jkozdon commented Jul 20, 2021

psanan commented Aug 5, 2021

jkozdon commented Aug 5, 2021

ViralBShah commented Oct 16, 2024

boriskaus commented Oct 16, 2024

boriskaus commented Oct 16, 2024

boriskaus commented Oct 16, 2024 • edited Loading

ViralBShah commented Oct 16, 2024

boriskaus commented Oct 16, 2024

ViralBShah commented Oct 16, 2024 • edited Loading

boriskaus commented Oct 16, 2024

boriskaus commented Oct 28, 2024 • edited Loading

boriskaus commented Nov 5, 2024

boriskaus commented Nov 6, 2024 via email • edited Loading

ViralBShah commented Nov 6, 2024

boriskaus commented Nov 15, 2024

ViralBShah commented Nov 15, 2024

boriskaus commented Nov 15, 2024

ViralBShah commented Nov 15, 2024

Automatically wrapping with `Clang.jl` #149

Automatically wrapping with `Clang.jl` #149

jkozdon commented Jul 15, 2021 •

edited

Loading

jkozdon commented Jul 15, 2021 •

edited

Loading

boriskaus commented Oct 16, 2024 •

edited

Loading

ViralBShah commented Oct 16, 2024 •

edited

Loading

boriskaus commented Oct 28, 2024 •

edited

Loading

boriskaus commented Nov 6, 2024 via email •

edited

Loading