Three arguments attach to Babylon and each is usually stated too confidently.
The tower. Etemenanki was a real ziggurat, rebuilt to enormous scale by Nabopolassar and Nebuchadnezzar II, and it is the most plausible thing Genesis 11 has in mind. That is as far as it goes. The biblical narrative is a short theological story about human presumption and the scattering of languages; it names no king, gives no date, and describes no architecture. Babylonian sources describe the building and say nothing about a confusion of tongues. The connection is one of setting, not of reporting, and the popular treatment that reads Genesis as an eyewitness account of Etemenanki's construction has the relationship backwards.
The gate. Robert Koldewey excavated from 1899 to 1917 and shipped the glazed brick of the Ishtar Gate and a section of the Processional Way to Berlin, where they were reconstructed inside the Pergamon Museum in the 1930s. This was done under Ottoman-era permits, which is the legal argument for retention, and it removed the single most recognisable object of Mesopotamian civilisation from the country that produced it, which is the argument against. Iraq has requested its return. The German position has been that the gate as displayed is a museum reconstruction incorporating substantial new material and could not be moved intact, a technical answer to a political question.
The base. In April 2003 Coalition forces established Camp Alpha on the site. Over the following months helicopter landing zones were built and used, trenches and revetments were cut, heavy vehicles moved across archaeological deposits, and thousands of sandbags and earth-filled barriers were packed with soil taken from the site itself, soil containing pottery and brick fragments, which destroys the context that makes such material informative. Sections of the Processional Way were damaged by tracked vehicles, and fragments were broken from the dragon reliefs of the Ishtar Gate. UNESCO documented the damage in a 2009 assessment, and the episode became the standard modern example of military occupation of a heritage site.
Between these sits Saddam Hussein's reconstruction, begun in the 1980s, which raised new walls directly on ancient foundations using bricks stamped with his own name in deliberate imitation of Nebuchadnezzar's practice. It damaged the archaeology and is now a stratum of it, and it complicates every conservation decision taken since, because removing it would destroy evidence of its own period.
Babylon was inscribed on the World Heritage list in 2019, after a long delay in which the reconstruction and the military damage were both cited as obstacles.