Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues with update from 0.5.5 to 0.5.6 #611

Closed
moyner opened this issue Oct 23, 2024 · 4 comments · Fixed by #612
Closed

Issues with update from 0.5.5 to 0.5.6 #611

moyner opened this issue Oct 23, 2024 · 4 comments · Fixed by #612

Comments

@moyner
Copy link

moyner commented Oct 23, 2024

Hi,

I'm noticing that the update from 0.5.5 to 0.5.6 broke my package serialization.

If I add my package to an environment on 1.11 and add [email protected] this example does not work, if I downgrade to 0.5.5 it works. The serialized data should be basic Julia datastructures, but with some degree of nesting. Here is my reproducer:

using Jutul
using Test

function test_io()
    sys = ScalarTestSystem()
    D = ScalarTestDomain()
    model = SimulationModel(D, sys)
    source = ScalarTestForce(1.0)
    forces = setup_forces(model, sources = source)
    state0 = setup_state(model, Dict(:XVar=>0.0))

    out = tempname()
    sim = Simulator(model, state0 = state0)
    states, reports = simulate(sim, [1.0, 2.0], forces = forces, output_states = true, output_path = out, info_level = -1)
    states2, reports2 = read_results(out)
end

test_io()

Error stacktrace:

LoadError: JLD2.InvalidDataException("")
Stacktrace:
  [1] jlread(io::JLD2.MmapIO, ::Type{JLD2.GlobalHeap})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/global_heaps.jl:101
  [2] read_heap_object(f::JLD2.JLDFile{JLD2.MmapIO}, hid::JLD2.GlobalHeapID, rr::JLD2.SameRepr{UInt8})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/global_heaps.jl:134
  [3] jlconvert
    @ ~/.julia/packages/JLD2/IyyZz/src/data/writing_datatypes.jl:365 [inlined]
  [4] jlconvert
    @ ~/.julia/packages/JLD2/IyyZz/src/data/specialcased_types.jl:173 [inlined]
  [5] macro expansion
    @ ~/.julia/packages/JLD2/IyyZz/src/io/dataio.jl:125 [inlined]
  [6] macro expansion
    @ ./simdloop.jl:77 [inlined]
  [7] read_array!(v::Vector{Symbol}, f::JLD2.JLDFile{JLD2.MmapIO}, rr::JLD2.MappedRepr{Symbol, JLD2.Vlen{String}})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/io/dataio.jl:122
  [8] read_array(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, rr::JLD2.ReadRepresentation, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:239
  [9] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, rr::Any, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, JLD2.DataLayout, JLD2.FilterPipeline}, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:98
 [10] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, dt::JLD2.H5Datatype, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:84
 [11] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:48
 [12] jlconvert(::JLD2.MappedRepr{Vector{Symbol}, JLD2.RelOffset}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, ::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/data/writing_datatypes.jl:338
 [13] macro expansion
    @ ~/.julia/packages/JLD2/IyyZz/src/data/reconstructing_datatypes.jl:660 [inlined]
 [14] jlconvert(::JLD2.MappedRepr{OrderedCollections.OrderedDict{Symbol, Any}, JLD2.OnDiskRepresentation{(0, 8, 16, 24, 32, 40), Tuple{Vector{Int32}, Vector{Symbol}, Vector{Any}, Int64, Int64, Bool}, Tuple{JLD2.RelOffset, JLD2.RelOffset, JLD2.RelOffset, Int64, Int64, Bool}, 41}}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/data/reconstructing_datatypes.jl:652
 [15] read_scalar(f::JLD2.JLDFile{JLD2.MmapIO}, rr::Any, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/io/dataio.jl:107
 [16] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, rr::Any, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, JLD2.DataLayout, JLD2.FilterPipeline}, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:96
 [17] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, dt::JLD2.H5Datatype, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:84
 [18] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:48
 [19] jlconvert
    @ ~/.julia/packages/JLD2/IyyZz/src/data/writing_datatypes.jl:338 [inlined]
 [20] macro expansion
    @ ~/.julia/packages/JLD2/IyyZz/src/io/dataio.jl:125 [inlined]
 [21] macro expansion
    @ ./simdloop.jl:77 [inlined]
 [22] read_array!(v::Vector{Any}, f::JLD2.JLDFile{JLD2.MmapIO}, rr::JLD2.MappedRepr{Any, JLD2.RelOffset})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/io/dataio.jl:122
 [23] read_array(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, rr::JLD2.ReadRepresentation, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:239
 [24] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.MappedRepr{Any, JLD2.RelOffset}, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, JLD2.DataLayout, JLD2.FilterPipeline}, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:136
--- the above 8 lines are repeated 1 more time ---
 [33] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, dt::JLD2.H5Datatype, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:84
 [34] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:48
 [35] jlconvert(::JLD2.MappedRepr{Vector{Any}, JLD2.RelOffset}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, ::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/data/writing_datatypes.jl:338
 [36] macro expansion
    @ ~/.julia/packages/JLD2/IyyZz/src/data/reconstructing_datatypes.jl:660 [inlined]
 [37] jlconvert(::JLD2.MappedRepr{OrderedCollections.OrderedDict{Symbol, Any}, JLD2.OnDiskRepresentation{(0, 8, 16, 24, 32, 40), Tuple{Vector{Int32}, Vector{Symbol}, Vector{Any}, Int64, Int64, Bool}, Tuple{JLD2.RelOffset, JLD2.RelOffset, JLD2.RelOffset, Int64, Int64, Bool}, 41}}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/data/reconstructing_datatypes.jl:652
 [38] read_scalar(f::JLD2.JLDFile{JLD2.MmapIO}, rr::Any, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/io/dataio.jl:107
 [39] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, rr::Any, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, JLD2.DataLayout, JLD2.FilterPipeline}, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:96
 [40] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, dt::JLD2.H5Datatype, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:84
 [41] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/datasets.jl:48
 [42] getindex(g::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, name::String)
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/groups.jl:99
 [43] getindex
    @ ~/.julia/packages/JLD2/IyyZz/src/JLD2.jl:349 [inlined]
 [44] (::Jutul.var"#533#534"{Bool, Bool, Int64, String})(file::JLD2.JLDFile{JLD2.MmapIO})
    @ Jutul ~/git/Jutul/src/utils.jl:678
 [45] jldopen(::Function, ::String, ::Vararg{String}; kws::@Kwargs{})
    @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/loadsave.jl:4
 [46] jldopen
    @ ~/.julia/packages/JLD2/IyyZz/src/loadsave.jl:1 [inlined]
 [47] read_restart(pth::String, i::Int64; read_state::Bool, read_report::Bool)
    @ Jutul ~/git/Jutul/src/utils.jl:671
 [48] read_restart
    @ ~/git/Jutul/src/utils.jl:668 [inlined]
 [49] read_results(pth::String; read_states::Bool, states::Vector{Dict{Symbol, Any}}, read_reports::Bool, reports::Vector{Any}, range::UnitRange{Int64}, name::Nothing, verbose::Bool)
    @ Jutul ~/git/Jutul/src/utils.jl:638
 [50] retrieve_output!(states::Vector{Dict{Symbol, Any}}, reports::Vector{Any}, config::JutulConfig, n::Int64; read_reports::Bool, read_states::Bool)
    @ Jutul ~/git/Jutul/src/simulator/io.jl:19
 [51] retrieve_output!(states::Vector{Dict{Symbol, Any}}, reports::Vector{Any}, config::JutulConfig, n::Int64)
    @ Jutul ~/git/Jutul/src/simulator/io.jl:11
 [52] retrieve_output!
    @ ~/git/Jutul/src/simulator/io.jl:8 [inlined]
 [53] simulate!(sim::Simulator{Jutul.DefaultExecutor, SimulationModel{ScalarTestDomain, ScalarTestSystem, FullyImplicitFormulation, DefaultContext}, JutulStorage{Nothing}}, timesteps::Vector{Float64}; forces::@NamedTuple{sources::ScalarTestForce}, config::Nothing, initialize::Bool, restart::Nothing, state0::Nothing, parameters::Nothing, forces_per_step::Bool, start_date::Nothing, kwarg::@Kwargs{output_states::Bool, output_path::String, info_level::Int64})
    @ Jutul ~/git/Jutul/src/simulator/simulator.jl:241
 [54] simulate!
    @ ~/git/Jutul/src/simulator/simulator.jl:143 [inlined]
 [55] simulate
    @ ~/git/Jutul/src/simulator/simulator.jl:123 [inlined]
 [56] test_io()
    @ Main ~/jld_error.jl:14
 [57] top-level scope
    @ ~/jld_error.jl:18
 [58] eval
    @ ./boot.jl:430 [inlined]
 [59] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:2643
 [60] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
    @ Base ./essentials.jl:1055
 [61] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:1052
 [62] inlineeval(m::Module, code::String, code_line::Int64, code_column::Int64, file::String; softscope::Bool)
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.125.1/scripts/packages/VSCodeServer/src/eval.jl:271
 [63] (::VSCodeServer.var"#69#74"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.125.1/scripts/packages/VSCodeServer/src/eval.jl:181
 [64] withpath(f::VSCodeServer.var"#69#74"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams}, path::String)
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.125.1/scripts/packages/VSCodeServer/src/repl.jl:276
 [65] (::VSCodeServer.var"#68#73"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.125.1/scripts/packages/VSCodeServer/src/eval.jl:179
 [66] hideprompt(f::VSCodeServer.var"#68#73"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.125.1/scripts/packages/VSCodeServer/src/repl.jl:38
 [67] #67
    @ ~/.vscode/extensions/julialang.language-julia-1.125.1/scripts/packages/VSCodeServer/src/eval.jl:150 [inlined]
 [68] with_logstate(f::VSCodeServer.var"#67#72"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams}, logstate::Base.CoreLogging.LogState)
    @ Base.CoreLogging ./logging/logging.jl:522
 [69] with_logger
    @ ./logging/logging.jl:632 [inlined]
 [70] (::VSCodeServer.var"#66#71"{VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.125.1/scripts/packages/VSCodeServer/src/eval.jl:263
 [71] #invokelatest#2
    @ ./essentials.jl:1055 [inlined]
 [72] invokelatest(::Any)
    @ Base ./essentials.jl:1052
@JonasIsensee
Copy link
Collaborator

Hm, that is unfortunate.

Can you read results with 1.10 that were written with 1.11 ?
That would clarify whether the error really occurs druring loading of maybe writing.

@moyner
Copy link
Author

moyner commented Oct 24, 2024

Archive.zip

I did some testing - the issue happens on both 1.10 and 1.11. I'm attaching two files, one written from 1.10 and one from 1.11. Calling JLD2.load on either file in either Julia version results in the same error.

JonasIsensee added a commit that referenced this issue Oct 24, 2024
@JonasIsensee
fix #611
db264d1
@JonasIsensee JonasIsensee mentioned this issue Oct 24, 2024
Merged
@JonasIsensee
Copy link
Collaborator

Ah, I meant to say read/ write on the different JLD2 versions.
Thanks for the files! That helped identify the issue.
A fix with regression test is in #612.
A new release should hopefully be out today.

JonasIsensee added a commit that referenced this issue Oct 24, 2024
@JonasIsensee
fix #611
fce896f
@moyner
Copy link
Author

moyner commented Oct 24, 2024

Thanks a lot - very much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix #611 new version 0.5.7 JuliaIO/JLD2.jl
2 participants
@moyner @JonasIsensee