If you're not attending the festival for the entire 12 days, the answer to this question really depends on what the main objective of your visit is.
The first two days of the festival (Tuesday - Wednesday) are fairly quiet as many of the film industry people don't start arriving until Thursday/Friday. If you're in Cannes for the first time, Wednesday/Thursday can be useful to pick up your badge, get settled in and get your head around the festival landscape.
The first weekend (Friday - Sunday) is the busiest period of the festival. This is where you'll encounter the biggest crowds, the most stars etc, which can be great if you simply want to soak up the atmosphere.
Things tend to start to quieten down as the shadows get longer on the second week. Everything is still pretty manic Monday and Tuesday, but by Friday many people have packed up and gone home. That's not to say you can expect to see tumbleweeds rolling down the Croisette - it will still be busy by anyone's standards. If you're an independent filmmaker looking to set up meetings with people in Cannes, later in the second week is often the best time (as you're more likely to find a slot in someone's diary).
In recent years, the Festival has tried to pep up the closing weekend by moving the awards ceremony to the Saturday night (previously it was held on the Sunday and most people had left by that point). This has been met with mixed success, possibly because many people are just too tired at this point after 12 days of festival. The last weekend is a good chance to catch a film or take advantage of the fact you're in the south of France before you return home.
For the full run-down on attending the Cannes Film Festival, make sure you pick up a copy of Cannes - A Festival Virgin's Guide.