Memory's a funny thing. Sometimes it's deceptive, serving up events that never could have happened, like a dead grandparent attending a birthday party. Sometimes it's selective, choosing to remember all the state capitals but not the name of your date. And sometimes, it simply has holes in it -- like for the first three years of your life.
That refusal of your memory to dish anything from babyhood is called childhood amnesia. We used to think that people couldn't remember infancy because the part of the brain that deals with memory wasn't developed yet. Not so. It turns out that even a three-month-old can make a memory. But a memory like birth doesn't stick.
Why not? Part of it has to do with brain structure. The prefrontal cortex, which is important to memory encoding retrieval, isn't quite baked yet.
Another possible explanation is that we don't have language abilities as babies, and talking about our memories is one way to make them stick. Infants also lack a sense of time, which makes it difficult to place a memory in context. Not having a sense of self yet comes into play, too -- children may not have an idea of self until they're 2 years old.
For more on memory, try Can a person remember being born?