Title: A machine learning-based study of open-charm hadrons in proton-proton collisions at the Large Hadron Collider

Abstract: In proton-proton and heavy-ion collisions, the study of charm hadrons plays a pivotal role in understanding the QCD medium and provides an undisputed testing ground for the theory of strong interaction, as they are mostly produced in the early stages of collisions via hard partonic interactions. The lightest open-charm, $D^{0}$ meson ($c\bar{u}$), can originate from two separate sources. The prompt $D^{0}$ originates from direct charm production or decay of excited open charm states, and the nonprompt stems from the decay of beauty hadrons. In this paper, using different machine learning (ML) algorithms such as XGBoost, CatBoost, and Random Forest, an attempt has been made to segregate the prompt and nonprompt production modes of $D^{0}$ meson signal from its background. The ML models are trained using the invariant mass ($M_{\pi K}$) through its hadronic decay channel, i.e., $D^{0}\rightarrow\pi^{+} K^{-}$, pseudoproper time ($t_{z}$), pseudoproper decay length ($c\tau$), and distance of closest approach of $D^{0}$ meson, using PYTHIA8 simulated $pp$ collisions at $\sqrt{s}=13~\rm{TeV}$. The ML models used in this analysis are found to retain the pseudorapidity, transverse momentum, and collision energy dependence. In addition, we report the ratio of nonprompt to prompt $D^{0}$ yield, the self-normalized yield of prompt and nonprompt $D^{0}$ and explore the charmonium, $J/\psi$ to open-charm, $D^{0}$ yield ratio as a function of transverse momenta and normalized multiplicity. The observables studied in this manuscript are well predicted by all the ML models compared to the simulation.