At least for the first few
years of life. It's the
natural state of the
human being to not believe in any
religion.
However, for most children, their parents soon indoctrinate them with their belief system, resulting in the children learning it as a fact before they are old enough to actually think about it and question it.
I must be clear here that being atheist simply means not believing in any gods. Surely a newborn baby doesn't believe in any.