The fun factor is definitely a lot more important to me than the challenge! Of course, everybody likes a good challenge, but some games have a bit too much of it (I'm looking at you, Super Meat Boy and Bit.Trip Runner), making the fun almost non-existent, because of the high levels of frustration induced by the game. It really depends on my mood, as well. Sometimes I can handle extremely hard games, while at other times I just get so annoyed, that I switch to a more easy and relaxing one. Games are, above all, supposed to be fun and enjoyable, so difficulty should not be the most important factor when deciding what to play.