The council is described in the Vinaya of several schools and in later chronicles, all compiled a long time after the events they narrate, and all with an interest in the outcome, since a canon agreed by five hundred arahants immediately after the founder's death is the strongest possible warrant for the texts a school holds. Scholars have treated the account with corresponding caution.
The doubts are specific rather than general. The number five hundred is a conventional figure. The idea that the whole of the discourse and discipline collections as they now stand could have been recited and agreed in a single rains retreat is difficult, given their length. The versions preserved by different schools disagree on details, including which texts were recited and what happened to Ananda. And the second council at Vesali, a century later over ten disputed monastic practices, is generally regarded as better attested than the first.
What the doubts do not touch is the practice that lies behind the story. Early Buddhist communities did transmit their texts by communal recitation, they did organise them for memorisation with the repetitions and formulas that make the discourses distinctive on the page, and groups of reciters specialised in particular collections. Whether or not one formal assembly took place at Rajagaha, the method the account describes is how these texts were actually preserved for several centuries before anyone wrote them down.