Segmentation Fault with combineTools using HybridNew in Combine-v10/CMSSW_14_1_0_pre4

[sergicastells9]

I've recently upgraded from CMSSW_11_3_4 to CMSSW_14_1_0_pre4 with Combine v10 and am trying to run HybridNew. I am doing a parametric analysis that uses RooPdfs for the signal/background models. My setup used to work with CMSSW_11_3_4, but now, with CMSSW_14_1_0_pre4, I get a segmentation fault when running combine -M HybridNew -m 125 -d ../combine/15_GeV/h4g_15_GeV_datacard.txt -n 15GeV --LHCmode LHC-limits --saveHybridResult. I've gone through both the ROOT and Combine changelogs and haven't found anything that might explain this behavior.

I have no issues with text2workspace, FitDiagnostics, Significance, or -H HybridNew (until it gets to the HybridNew part). The full stack is below. I have also tested running combine datacard-2-template-analysis.txt -M HybridNew --LHCmode LHC-limits --rMax 2.0 which did not have any issues, where the datacard/workspace are in HiggsAnalysis/CombinedLimit/data/tutorials/CAT23001/...

 <<< v10.0.2 >>>
>>> Random number generator seed is 123456
>>> using the Profile Likelihood test statistic modified for upper limits (Q_LHC)
>>> Method used is HybridNew

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f94144d89fa in wait4 () from /lib64/libc.so.6
#1  0x00007f941444b243 in do_system () from /lib64/libc.so.6
#2  0x00007f941636743d in TUnixSystem::StackTrace() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre4/external/el9_amd64_gcc12/lib/libCore.so
#3  0x00007f9416366df4 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre4/external/el9_amd64_gcc12/lib/libCore.so
#4  <signal handler called>
#5  0x00007f9417ea1ef7 in RooFit::Constrain(RooArgSet const&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre4/external/el9_amd64_gcc12/lib/libRooFitCore.so
#6  0x00007f9418c6cde0 in HybridNew::create(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, RooAbsCollection const&, HybridNew::Setup&) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#7  0x00007f9418c6ec88 in HybridNew::eval(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, RooAbsCollection const&, bool, double) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#8  0x00007f9418c6f5bf in HybridNew::eval(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double, bool, double) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#9  0x00007f9418c70db8 in HybridNew::runLimit(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double&, double&, double const*) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#10 0x00007f9418c6a3f1 in HybridNew::run(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double&, double&, double const*) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#11 0x00007f9418bc29a1 in Combine::mklimit(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double&, double&) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#12 0x00007f9418bcb704 in Combine::run(TString, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double&, double&, int&, TTree*, int) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#13 0x000000000040ce31 in main ()
===========================================================


The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
preferably using the command (.forum bug) in the ROOT prompt.
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs or (preferably) using the command (.gh bug) in
the ROOT prompt. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f9417ea1ef7 in RooFit::Constrain(RooArgSet const&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre4/external/el9_amd64_gcc12/lib/libRooFitCore.so
#6  0x00007f9418c6cde0 in HybridNew::create(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, RooAbsCollection const&, HybridNew::Setup&) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#7  0x00007f9418c6ec88 in HybridNew::eval(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, RooAbsCollection const&, bool, double) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#8  0x00007f9418c6f5bf in HybridNew::eval(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double, bool, double) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#9  0x00007f9418c70db8 in HybridNew::runLimit(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double&, double&, double const*) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#10 0x00007f9418c6a3f1 in HybridNew::run(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double&, double&, double const*) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#11 0x00007f9418bc29a1 in Combine::mklimit(RooWorkspace*, RooStats::ModelConfig*, RooStats::ModelConfig*, RooAbsData&, double&, double&) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#12 0x00007f9418bcb704 in Combine::run(TString, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double&, double&, int&, TTree*, int) () from /afs/crc.nd.edu/user/s/scastel2/Private/CMSSW_14_1_0_pre4/lib/el9_amd64_gcc12/libHiggsAnalysisCombinedLimit.so
#13 0x000000000040ce31 in main ()
===========================================================

Can anyone provide any insight into what might be going wrong? I'm totally at a loss here. If I'm missing any information or can upload my workspace for reference then just let me know. Many thanks in advance.

[anigamova]

Hi @sergicastells9, could you please share the datacards with the workspaces? It could be something related to pdfs you are using, but it is hard to say more without workspaces.

[sergicastells9]

Hi, thanks for the quick response. I've attached my datacard (and the compiled one for completeness) and my workspace for your reference. Let me know if I can provide anything else that might help.

For some more context, I use HiggsAnalysis-CombinedLimit with tag v10.0.2 and CombineHarvester with tag v3.0.0.

datacard_workspace.zip

[anigamova]

Hi @sergicastells9, I think the segfault happens here because you are using --LHCmode LHC-limits (which sets --fitNuisances 1) with the datacard without nuisance parameters (systematic block is empty), which does not make much sense. I'm a bit surprised that it still worked with combine v9, are you sure about this? But I think we should also be catching these issues properly, so let's keep this issue open for now

[sergicastells9]

Hi @anigamova, I'm still learning how everything works with Combine so in my tests I had left nuisance parameters empty. I see now why the --LHCmode LHC-limits would fail, but then I am also sure that it worked in v9. Whether I'm doing everything correctly who knows, but I definitely was able to calculate limits in v9 based on this exact setup. Thanks!

[sergicastells9]

A quick note that might help understanding why this happens more: I tried running HybridNew again with the lumi nuisance parameter uncommented to provide at least one so that the command might work. I still get the seg fault in this case. Is it possible that HybridNew is not the problem?

[anigamova]

A quick note that might help understanding why this happens more: I tried running HybridNew again with the lumi nuisance parameter uncommented to provide at least one so that the command might work. I still get the seg fault in this case. Is it possible that HybridNew is not the problem?

In this attempt did you run with combine v9 or v10? I think one of the reasons it might be not working with v9 is because you created workspace.root file with RooBernsteinFast based on combine v10, and unfortunately it is not backwards compatible.