FATAL segmentation violation when reading output file of PodioOutput
Dear Experts,
I was using some simulation samples recently produced with the latest version of CEPCSW, but it returns errors in [1] when I call ReconstructedParticle::getClusters(i). An example of the file I read can be found in [2]. It was created with the script in [3], combining PodioInput + CyberPFAlg + FinalPIDAlg + PodioOutput. I tried a few of them, and the issue remains.
Moreover, I didn't encounter this issue when reading the file in [4], which was created with the same modules, and the only difference is that in PodioOutput, only MCParticle, CyberPFO, CyberPFOPID and CompleteTracks are kept.
At last, the issue didn't happen with the samples created last week.
I am wondering if this issue also happens for other people, and if it is worth debugging it. Thanks very much!
Yours Sincerely,
Geliang
[1]
Click to expand the full log
ApplicationMgr SUCCESS
====================================================================================================================================
Welcome to ApplicationMgr (GaudiCoreSvc v37r2)
running on lxlogin002.ihep.ac.cn on Mon Jan 20 23:16:26 2025
====================================================================================================================================
ApplicationMgr INFO Application Manager Configured successfully
EventLoopMgr WARNING Unable to locate service "EventSelector"
EventLoopMgr WARNING No events will be processed from external input.
ApplicationMgr INFO Application Manager Initialized successfully
ApplicationMgr INFO Application Manager Started successfully
TUnixSystem::Di... FATAL segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0 0x00007f76752d89fa in wait4 () from /lib64/libc.so.6
#1 0x00007f767524b243 in do_system () from /lib64/libc.so.6
#2 0x00007f7663cfb47c in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.30.02-fb5be/x86_64-el9-gcc11-opt/lib/libCore.so
#3 0x00007f7663cf8b95 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.30.02-fb5be/x86_64-el9-gcc11-opt/lib/libCore.so
#4 <signal handler called>
#5 0x00007f763fbae6a4 in GenMatch::execute() () from /cefs/higgs/liugeliang/CEPC/202501/CEPCSW/InstallArea/x86_64-el9-gcc11-opt/lib/libGenMatch.so
#6 0x00007f76425b29c7 in Gaudi::details::LegacyAlgorithmAdapter::execute(EventContext const&) const () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/k4FWCore/x86_64-el9-gcc11-opt/lib/libk4FWCorePlugins.so
#7 0x00007f766577ea2f in Gaudi::Algorithm::sysExecute(EventContext const&) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#8 0x00007f76658a80ec in MinimalEventLoopMgr::executeEvent(EventContext&&) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#9 0x00007f76611a8b8c in EventLoopMgr::executeEvent(EventContext&&) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiCoreSvc.so
#10 0x00007f76611a92c1 in EventLoopMgr::nextEvent(int) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiCoreSvc.so
#11 0x00007f76658a7dc3 in MinimalEventLoopMgr::executeRun(int) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#12 0x00007f766117da27 in ApplicationMgr::executeRun(int) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiCoreSvc.so
#13 0x00007f76657b00de in Gaudi::Application::run() () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#14 0x00007f76657b0793 in _py_Gaudi__Application__run () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#15 0x00007f7667438052 in ffi_call_unix64 () from /cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-el9-gcc11-opt/lib64/libffi.so.8
#16 0x00007f7667436b65 in ffi_call_int () from /cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-el9-gcc11-opt/lib64/libffi.so.8
#17 0x00007f7667450655 in _call_function_pointer (argtypecount=<optimized out>, argcount=1, resmem=0x7fff5ea61000, restype=<optimized out>, atypes=<optimized out>, avalues=0x7fff5ea60ff0, pProc=0x7f76657b0770 <_py_Gaudi__Application__run>, flags=4353) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/_ctypes/callproc.c:920
#18 _ctypes_callproc (pProc=pProc
entry=0x7f76657b0770 <_py_Gaudi__Application__run>, argtuple=argtuple
entry=0x7f7665fc8460, flags=4353, argtypes=argtypes
entry=0x7f7665fc8520, restype=restype
entry=0x11b54e0, checker=checker
entry=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/_ctypes/callproc.c:1263
#19 0x00007f766744a7b9 in PyCFuncPtr_call (self=<optimized out>, inargs=<optimized out>, kwds=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/_ctypes/_ctypes.c:4201
#20 0x00007f76756b8588 in _PyObject_MakeTpCall (tstate=0xfb5470, callable=0x7f7665fd2640, args=<optimized out>, nargs=1, keywords=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:191
#21 0x00007f767566e209 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7f7665fe9b80, callable=0x7f7665fd2640, tstate=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:116
#22 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7f7665fe9b80, callable=0x7f7665fd2640, tstate=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:103
#23 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x7f7665fe9b80, callable=0x7f7665fd2640) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#24 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#25 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3489
#26 0x00007f7675665bab in _PyEval_EvalFrame (throwflag=0, f=0x7f7665fe9a00, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#27 function_code_fastcall (tstate=0xfb5470, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:330
#28 0x00007f767566d770 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x12a5240, callable=0x7f76674a5f70, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:118
#29 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x12a5240, callable=0x7f76674a5f70) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#30 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3506
#32 0x00007f7675665bab in _PyEval_EvalFrame (throwflag=0, f=0x12a5080, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#33 function_code_fastcall (tstate=0xfb5470, co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:330
#34 0x00007f767566d770 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f76671248d0, callable=0x7f7667176b80, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:118
#35 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f76671248d0, callable=0x7f7667176b80) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#36 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#37 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3506
#38 0x00007f767579b7fe in _PyEval_EvalFrame (throwflag=0, f=0x7f7667124740, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#39 _PyEval_EvalCode (tstate=tstate
entry=0xfb5470, _co=_co
entry=0x7f76674163a0, globals=<optimized out>, locals=locals
entry=0x0, args=<optimized out>, argcount=3, kwnames=0x0, kwargs=0x102c2a8, kwcount=0, kwstep=1, defs=0x7f766740cfb8, defcount=1, kwdefs=0x0, closure=0x0, name=0x7f76754a9ab0, qualname=0x7f76674153b0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4329
#40 0x00007f76756b8311 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:396
#41 0x00007f767566d770 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x102c290, callable=0x7f7667176a60, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:118
#42 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x102c290, callable=0x7f7667176a60) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#43 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#44 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3506
#45 0x00007f767579b7fe in _PyEval_EvalFrame (throwflag=0, f=0x102c120, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#46 _PyEval_EvalCode (tstate=0xfb5470, _co=_co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, args=args
entry=0x0, argcount=argcount
entry=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4329
#47 0x00007f767579bb3e in _PyEval_EvalCodeWithName (_co=_co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, args=args
entry=0x0, argcount=argcount
entry=0, kwnames=kwnames
entry=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4361
#48 0x00007f767579bb8b in PyEval_EvalCodeEx (_co=_co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, args=args
entry=0x0, argcount=argcount
entry=0, kws=kws
entry=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4377
#49 0x00007f767579bbbb in PyEval_EvalCode (co=co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:828
#50 0x00007f76757dbd1e in run_eval_code_obj (locals=0x7f767519df40, globals=0x7f767519df40, co=0x7f76677ae240, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:1221
#51 run_mod (mod=<optimized out>, filename=filename
entry=0x7f76677f3670, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, flags=flags
entry=0x7fff5ea61ad8, arena=arena
entry=0x7f76754a6910) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:1242
#52 0x00007f76757dd733 in pyrun_file (flags=0x7fff5ea61ad8, closeit=<optimized out>, locals=0x7f767519df40, globals=0x7f767519df40, start=257, filename=0x7f76677f3670, fp=0xfb0ea0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:1140
#53 pyrun_simple_file (flags=0x7fff5ea61ad8, closeit=<optimized out>, filename=0x7f76677f3670, fp=0xfb0ea0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:450
#54 PyRun_SimpleFileExFlags (fp=fp
entry=0xfb0ea0, filename=<optimized out>, closeit=closeit
entry=1, flags=flags
entry=0x7fff5ea61ad8) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:483
#55 0x00007f76757ddd6c in PyRun_AnyFileExFlags (fp=fp
entry=0xfb0ea0, filename=<optimized out>, closeit=closeit
entry=1, flags=flags
entry=0x7fff5ea61ad8) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:92
#56 0x00007f76757fad08 in pymain_run_file (cf=0x7fff5ea61ad8, config=0xfb30b0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:373
#57 pymain_run_python (exitcode=0x7fff5ea61ad0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:598
#58 Py_RunMain () at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:677
#59 0x00007f76757fb1c7 in pymain_main (args=0x7fff5ea61bc0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:707
#60 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:731
#61 0x00007f7675229590 in __libc_start_call_main () from /lib64/libc.so.6
#62 0x00007f7675229640 in __libc_start_main_impl () from /lib64/libc.so.6
#63 0x0000000000401075 in _start ()
===========================================================
The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
preferably using the command (.forum bug) in the ROOT prompt.
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs or (preferably) using the command (.gh bug) in
the ROOT prompt. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5 0x00007f763fbae6a4 in GenMatch::execute() () from /cefs/higgs/liugeliang/CEPC/202501/CEPCSW/InstallArea/x86_64-el9-gcc11-opt/lib/libGenMatch.so
#6 0x00007f76425b29c7 in Gaudi::details::LegacyAlgorithmAdapter::execute(EventContext const&) const () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/k4FWCore/x86_64-el9-gcc11-opt/lib/libk4FWCorePlugins.so
#7 0x00007f766577ea2f in Gaudi::Algorithm::sysExecute(EventContext const&) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#8 0x00007f76658a80ec in MinimalEventLoopMgr::executeEvent(EventContext&&) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#9 0x00007f76611a8b8c in EventLoopMgr::executeEvent(EventContext&&) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiCoreSvc.so
#10 0x00007f76611a92c1 in EventLoopMgr::nextEvent(int) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiCoreSvc.so
#11 0x00007f76658a7dc3 in MinimalEventLoopMgr::executeRun(int) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#12 0x00007f766117da27 in ApplicationMgr::executeRun(int) () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiCoreSvc.so
#13 0x00007f76657b00de in Gaudi::Application::run() () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#14 0x00007f76657b0793 in _py_Gaudi__Application__run () from /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/105.0.0/Gaudi/x86_64-el9-gcc11-opt/lib/libGaudiKernel.so
#15 0x00007f7667438052 in ffi_call_unix64 () from /cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-el9-gcc11-opt/lib64/libffi.so.8
#16 0x00007f7667436b65 in ffi_call_int () from /cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-el9-gcc11-opt/lib64/libffi.so.8
#17 0x00007f7667450655 in _call_function_pointer (argtypecount=<optimized out>, argcount=1, resmem=0x7fff5ea61000, restype=<optimized out>, atypes=<optimized out>, avalues=0x7fff5ea60ff0, pProc=0x7f76657b0770 <_py_Gaudi__Application__run>, flags=4353) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/_ctypes/callproc.c:920
#18 _ctypes_callproc (pProc=pProc
entry=0x7f76657b0770 <_py_Gaudi__Application__run>, argtuple=argtuple
entry=0x7f7665fc8460, flags=4353, argtypes=argtypes
entry=0x7f7665fc8520, restype=restype
entry=0x11b54e0, checker=checker
entry=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/_ctypes/callproc.c:1263
#19 0x00007f766744a7b9 in PyCFuncPtr_call (self=<optimized out>, inargs=<optimized out>, kwds=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/_ctypes/_ctypes.c:4201
#20 0x00007f76756b8588 in _PyObject_MakeTpCall (tstate=0xfb5470, callable=0x7f7665fd2640, args=<optimized out>, nargs=1, keywords=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:191
#21 0x00007f767566e209 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7f7665fe9b80, callable=0x7f7665fd2640, tstate=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:116
#22 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7f7665fe9b80, callable=0x7f7665fd2640, tstate=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:103
#23 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x7f7665fe9b80, callable=0x7f7665fd2640) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#24 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#25 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3489
#26 0x00007f7675665bab in _PyEval_EvalFrame (throwflag=0, f=0x7f7665fe9a00, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#27 function_code_fastcall (tstate=0xfb5470, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:330
#28 0x00007f767566d770 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x12a5240, callable=0x7f76674a5f70, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:118
#29 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x12a5240, callable=0x7f76674a5f70) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#30 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3506
#32 0x00007f7675665bab in _PyEval_EvalFrame (throwflag=0, f=0x12a5080, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#33 function_code_fastcall (tstate=0xfb5470, co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:330
#34 0x00007f767566d770 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f76671248d0, callable=0x7f7667176b80, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:118
#35 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f76671248d0, callable=0x7f7667176b80) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#36 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#37 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3506
#38 0x00007f767579b7fe in _PyEval_EvalFrame (throwflag=0, f=0x7f7667124740, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#39 _PyEval_EvalCode (tstate=tstate
entry=0xfb5470, _co=_co
entry=0x7f76674163a0, globals=<optimized out>, locals=locals
entry=0x0, args=<optimized out>, argcount=3, kwnames=0x0, kwargs=0x102c2a8, kwcount=0, kwstep=1, defs=0x7f766740cfb8, defcount=1, kwdefs=0x0, closure=0x0, name=0x7f76754a9ab0, qualname=0x7f76674153b0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4329
#40 0x00007f76756b8311 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Objects/call.c:396
#41 0x00007f767566d770 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x102c290, callable=0x7f7667176a60, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:118
#42 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x102c290, callable=0x7f7667176a60) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/cpython/abstract.h:127
#43 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:5077
#44 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:3506
#45 0x00007f767579b7fe in _PyEval_EvalFrame (throwflag=0, f=0x102c120, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Include/internal/pycore_ceval.h:40
#46 _PyEval_EvalCode (tstate=0xfb5470, _co=_co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, args=args
entry=0x0, argcount=argcount
entry=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4329
#47 0x00007f767579bb3e in _PyEval_EvalCodeWithName (_co=_co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, args=args
entry=0x0, argcount=argcount
entry=0, kwnames=kwnames
entry=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4361
#48 0x00007f767579bb8b in PyEval_EvalCodeEx (_co=_co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, args=args
entry=0x0, argcount=argcount
entry=0, kws=kws
entry=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:4377
#49 0x00007f767579bbbb in PyEval_EvalCode (co=co
entry=0x7f76677ae240, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/ceval.c:828
#50 0x00007f76757dbd1e in run_eval_code_obj (locals=0x7f767519df40, globals=0x7f767519df40, co=0x7f76677ae240, tstate=0xfb5470) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:1221
#51 run_mod (mod=<optimized out>, filename=filename
entry=0x7f76677f3670, globals=globals
entry=0x7f767519df40, locals=locals
entry=0x7f767519df40, flags=flags
entry=0x7fff5ea61ad8, arena=arena
entry=0x7f76754a6910) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:1242
#52 0x00007f76757dd733 in pyrun_file (flags=0x7fff5ea61ad8, closeit=<optimized out>, locals=0x7f767519df40, globals=0x7f767519df40, start=257, filename=0x7f76677f3670, fp=0xfb0ea0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:1140
#53 pyrun_simple_file (flags=0x7fff5ea61ad8, closeit=<optimized out>, filename=0x7f76677f3670, fp=0xfb0ea0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:450
#54 PyRun_SimpleFileExFlags (fp=fp
entry=0xfb0ea0, filename=<optimized out>, closeit=closeit
entry=1, flags=flags
entry=0x7fff5ea61ad8) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:483
#55 0x00007f76757ddd6c in PyRun_AnyFileExFlags (fp=fp
entry=0xfb0ea0, filename=<optimized out>, closeit=closeit
entry=1, flags=flags
entry=0x7fff5ea61ad8) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Python/pythonrun.c:92
#56 0x00007f76757fad08 in pymain_run_file (cf=0x7fff5ea61ad8, config=0xfb30b0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:373
#57 pymain_run_python (exitcode=0x7fff5ea61ad0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:598
#58 Py_RunMain () at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:677
#59 0x00007f76757fb1c7 in pymain_main (args=0x7fff5ea61bc0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:707
#60 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.12/src/Python/3.9.12/Modules/main.c:731
#61 0x00007f7675229590 in __libc_start_call_main () from /lib64/libc.so.6
#62 0x00007f7675229640 in __libc_start_main_impl () from /lib64/libc.so.6
#63 0x0000000000401075 in _start ()
===========================================================[2] On lxlogin: /cefs/higgs/zhangkl/Production/2501/E240_nnHbb/Combined/rec_E240_nnHbb_00104.root
[3] On lxlogin: /cefs/higgs/zhangkl/Production/2501/E240_nnHbb/job/rec_00104.py
[4] On lxlogin: /cefs/higgs/zhangkl/Production/2501/E240_nnHbb/Reco/ana_E240_nnHbb_00104.root
