Jewish and Christian tradition assigns the whole Torah to Moses, and the attribution is old enough to be treated as part of the text rather than as a claim about it. Genesis makes no such statement about itself: it names no author anywhere, and unlike the prophetic books it carries no superscription.
Readers noticed difficulties early. Genesis 36 lists kings who reigned in Edom before any king reigned over Israel, which is written from a vantage point after there were Israelite kings. Places are called by names the narrative elsewhere says were given later. The twelfth-century commentator Abraham ibn Ezra gestured at a set of such verses without spelling out the conclusion, telling the reader who understands to keep silent. Spinoza spelled it out in 1670 and was answered with a ban rather than an argument.
The technical observation that opened the modern discussion was simpler than any of that: the book alternates between two names for God, and the passages using each name differ in vocabulary, in style and sometimes in the story they tell. Two accounts of creation sit in the first two chapters; the flood narrative counts the animals twice, and differently. A mid-eighteenth-century French physician, Jean Astruc, proposed that Moses had worked from earlier written memoirs, which was intended as a defence of Mosaic authorship and became the seed of its dismantling.
Julius Wellhausen synthesised a century of that work into the documentary hypothesis, which dated the sources against a reconstructed history of Israelite religion rather than the reverse. The particular four-source scheme is now contested from several directions at once. That the book is composite, and later than Moses, is not.