When the Roman rite's other public prayers moved from Greek into Latin during the early centuries of the Western church, this phrase alone was left as it was, a decision no single surviving document explains but that liturgical historians generally connect to its origin as a congregational response within a longer litany, a form of prayer structured as a deacon or cantor reciting a series of specific petitions to which the assembly responds with a fixed short phrase. In the Eastern churches that structure survives largely intact, extended litanies with kyrie eleison repeated after each of many named petitions. In the West, over centuries, the surrounding petitions dropped away from the ordinary Mass while the responses themselves remained, contracting a once lengthy litany down to the compact ninefold Kyrie, Christe, Kyrie pattern familiar from the Latin Mass, so what looks like a short fixed formula is, from a liturgical historian's standpoint, the surviving skeleton of a longer form whose flesh has been stripped away.
The phrase itself draws on Greek vocabulary attested throughout the Psalms and Gospels, eleison sharing a root with the everyday Greek word for mercy or pity, so its content was never linguistically obscure to Greek speaking congregations, and its retention in Latin worship as untranslated Greek reflects reverence for its liturgical antiquity rather than any sense that its Greek wording carried an untranslatable meaning the way some Sanskrit bijas are treated. It survived because of where it sat in an evolving ritual structure rather than because of anything intrinsic to the Greek words themselves.