Why do otherwise clever people consistently underestimate how long it will take to do something, even when they really should know better? This problem is rampant in the software business. Even veteran developers, people with years of programming experience, will look you in the eye and name an insanely optimistic timeline for their next project. Science Daily (found via LifeHacker) cites some recent research on this topic:
Why Do We Overcommit? Study Suggests We Think We’ll Have More Time In The Future Than We Have Today. I like this summary.
Zauberman and Lynch continue, “People are consistently surprised to be so busy today. Lacking knowledge of what specific tasks will compete for their time in the future, they act as if new demands will not inevitably arise that are as pressing as those faced today.” In short, the future is ideal: The fridge is stocked, the weather clear, the train runs on schedule and meetings end on time. Today, well, stuff happens.
The more I think about it, the less this result surprises me. Compared to today, there is more time in the future. That’s where all the time comes from. That’s where the time factory is, and factory outlets are always the cheapest suppliers. As a result, everything gets discounted in the future, including the price you’ll pay for being wrong. The “you” in the future is not the you of right now, and that person can take the hit for any bad prediction. It’s the same reason you’re willing to make lifestyle choices now (fatty food, no exercise) that imperil your future self: Hey! That person isn’t you. Yet.
Rambles Blog at Star Chamber 
I asked Melissa how she always gets her scheduling so accurate, and it seems she used to keep painful track of how long it took to do everything on another project — and learned from that. Learning this type of estimation skill requires that you are capable of decomposing a new task into similar chunks that you have already experienced in the past, and then applying the rules internalized from the past; and that you know how much better or worse you will be at doing the same or similar thing over again in a new context. I’m fascinated by this problem in general, but especially in software, where there are many moving parts to be integrated. Project management is the hardest discipline of all, I think (in terms of the problems to be solved).
There’s another factor, of course, behind the time estimate problem in software– they aren’t always realistic, even knowingly, because of various meta-reasons: overpromising gets you the cool project or VC funding, and ridiculous over-estimates of worktime might be delivered by someone not wanting to touch a project with a ten foot pole, so they’re meant to discourage. We (I) sometimes did this in France when consulting, and I’ve seen engineers do it as a passive aggressive strategy with marketing requests.
Then there are those of us whose time estimates depend on whether they’ve conceptually solved a problem. As long as the solution remains elusive, there can be no estimate but, once it gels in the mind, the project is all but complete.