Most adults will agree they cannot remember much, if anything, prior to around age four. This is usually called, in a term coined by Sigmund Freud, infantile amnesia.
What is remembered often comes in "spot" images, floating in an otherwise gray area of uncertainty. Yet when we look at a two or three year-old, perception and basic cognition appear to be in place. Why are those years lost to us?
One possibility is that memory in humans requires a narrative context. "Narrative context" means that it isn't enough to provide, say, a random visual image to the brain in order for it to form a working memory: the image needs to be put in place alongside a number of other images, with which it helps to establish a pattern.
The very first time you see a tree, your brain can't do anything with it yet. The information may simply be stored until it can be connected to other things, such as the backyard where you saw it, its size and color, or events of significance that occurred at the same time.
Once enough relationships are in place, it then becomes possible to remember not just "tree" as an abstraction, but as "the big green tree I saw when I was walking this morning behind the house."
Learning words probably aids in this process, but is not strictly necessary. Still, the brain may provide context for memories by telling itself miniature stories, complete with setting, characters, and relationships.
According to this idea, early childhood events occur at a time before we have developed enough narrative skills to tell the stories necessary for the formation of memories. No narrative context, no long-term memory.
Update as of 2016
Recently, scientists have come up with another theory for why people experience "infantile amnesia" i.e. missing memories from when people were younger. They think this could also be due to neurogenesis. While the above describes psychological theories, there are also probably biological reasons for this phenomenon.
Neurogenesis is a process where where new neurons in the brain rearrange existing connections and destabilize memories in the brain. Scientists studied the effects of neurogenesis by monitoring and performing experiments on mice, another animal that also experiences infantile amnesia.
Tests performed on mice utilized fear and exercise. The mice were first taught to fear things and then they exercised (a process believed to stimulate neurogenesis). The mice who exercised were less afraid of the things and areas they were taught to fear.
Learn More about Neurogenesis and Infantile Amnesia
- Canadian Association for Neuroscience. "Cause of infantile amnesia revealed: New neuron formation could increase capacity for new learning, at expense of old memories." ScienceDaily. (accessed October 27, 2016)
- Josselyn, Sheena A., and Frankland, Paul W. "Infantile Amnesia: A Neurogenic Hypothesis." Learning & Memory. May 2, 2012. Accessed October 27, 2016.
- Kim, Meeri. "New Brain Cells Could Be Blanking out Old Memories, Research on Rodents Suggests." The Washington Post. May 17, 2014. Accessed October 26, 2016.
- Raffensperger, Lisa. "How Was My Childhood? I Forget." Discover. November 26, 2014. Accessed October 27, 2016.