Studies of new Higgs boson interactions through nonresonant HH production in the b(b)over-barγγ final state in pp collisions at √s=13 TeV with the ATLAS detector

A search for nonresonant Higgs boson pair production in the b (b) over bar gamma gamma final state is performed using 140 fb(-1) of proton-proton collisions at a centre-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. This analysis supersedes and expands upon the previous nonresonant ATLAS results in this final state based on the same data sample. The analysis strategy is optimised to probe anomalous values not only of the Higgs (H) boson self-coupling modifier kappa(lambda) but also of the quartic HHVV (V = W, Z) coupling modifier kappa(2V). No significant excess above the expected background from Standard Model processes is observed. An observed upper limit mu(HH) < 4.0 is set at 95% confidence level on the Higgs boson pair production cross-section normalised to its Standard Model prediction. The 95% confidence intervals for the coupling modifiers are -1.4 < kappa(lambda) < 6.9 and -0.5 < kappa(2V) < 2.7, assuming all other Higgs boson couplings except the one under study are fixed to the Standard Model predictions. The results are interpreted in the Standard Model effective field theory and Higgs effective field theory frameworks in terms of constraints on the couplings of anomalous Higgs boson (self-)interactions.