Rome's own account of its origin is a straight line: Christ gave the keys to Peter, Peter died at Rome, and the bishop of Rome has held what Peter held ever since. The historical account is less tidy, and the two are worth setting side by side.
Almost no historian disputes that Peter was at Rome and died there. What is disputed is when the bishop of Rome became the pope in the sense the word now carries. The earliest Roman succession lists are later compilations; the second-century Roman church appears to have been led by a body of presbyters rather than a single monarchical bishop; and the move from a precedence of honour among the ancient sees to a jurisdiction over them is visible unfolding across the fourth to eleventh centuries rather than present at the start. John Henry Newman's Essay on the Development of Christian Doctrine (1845) answers that a seed is not less itself for growing: a doctrine can unfold under fidelity to what it already held without that unfolding making it a novelty. Catholic theology has broadly followed Newman's argument since, while conceding the developmental point itself, that jurisdiction was not present in its later form at the start, which older polemics had denied.
The sharpest form of the claim is recent. Papal infallibility was defined at the First Vatican Council in 1870, not in the apostolic age, and not without opposition inside the council, whose minority left Rome rather than vote against it. A body of German-speaking Catholics refused the definition outright and became the Old Catholic churches. Since 1870 the power has been exercised in a way all parties agree on exactly once, in the definition of the bodily Assumption of Mary in 1950.
Two ruptures followed from the broader claim to Roman primacy and jurisdiction traced above, not from the 1870 infallibility definition itself: the Greek East had already refused that claim centuries earlier, the mutual excommunications of 1054 marking the parting long before Vatican I. The Reformers of the sixteenth century rejected it root and branch, and the Council of Trent, sitting between 1545 and 1563, answered them by defining the disputed points rather than conceding any of them.