Heresy as a working category was built case by case, and four early disputes did most of the building.
Gnosticism, a diverse second-century movement holding that the material world is the flawed product of a lesser being and that salvation comes through secret knowledge (gnosis) of one's true divine origin, prompted Irenaeus of Lyons to write Against Heresies (about 180), one of the first sustained efforts to define orthodoxy by cataloguing what it excluded, anchored in apostolic succession and a rule of faith.
Arianism, the position of the Alexandrian priest Arius that the Son was created by the Father and therefore not co-eternal with him, was condemned at the Council of Nicaea in 325, which produced the creed's homoousios, "of one substance," precisely to close the loophole Arius had used. The controversy did not end there; Arian and pro-Nicene parties contended for decades afterward, including among ruling emperors.
Pelagianism, associated with the British theologian Pelagius, held that human beings can achieve righteousness by will and effort without a strict necessity of grace; Augustine's opposition, and the doctrine's condemnation at councils in the early fifth century, shaped Western teaching on original sin and grace for the rest of its history.
Catharism, a dualist movement in medieval southern France holding matter to be evil, drew a different institutional response: the Albigensian Crusade (1209-1229), a military campaign rather than a theological refutation, and the establishment of the papal Inquisition afterward specifically to detect and try surviving Cathars, the first standing institution built for that purpose.