Rejection of the papal claim comes from two directions and they are unlike each other.
The Orthodox objection is not to the existence of the office. Orthodox churches acknowledge that the bishop of Rome held a primacy of honour in the undivided church and was named first among the ancient patriarchates. What they reject is that primacy of honour became a jurisdiction: that the bishop of Rome may govern other churches, hear appeals from them, appoint their bishops, or define doctrine without a council. On the Orthodox account each autocephalous church governs itself and the whole church speaks only through a council received by the faithful. The 1870 definitions made the disagreement formal, and the fact that Orthodox churches themselves dispute jurisdiction among each other, notably over Ukraine since 2019, shows that the underlying question of who may decide for whom is unresolved on that side too.
The Reformation objection is different in kind. It denies that the office has the standing claimed for it at all, and reads the claim as a historical accretion rather than a development of something apostolic. That is a dispute about origins rather than about the scope of an acknowledged primacy.
There is also an internal history. The definitions of 1870 were opposed by a substantial minority of bishops at the council itself, on the grounds that they were inopportune rather than false, and a number of German-speaking Catholics separated afterwards to form the Old Catholic churches rather than accept them.
Catholic scholarship does not generally contest that the developed office is not visible in the earliest sources. The disagreement is about what follows from that, and framing it as a dispute over the evidence rather than over its interpretation misrepresents both sides.