It's not exactly all roses if you decide to do that. Try navigating 2000+ albums using Media Center, and you'll learn that it's not pleasant. Even search on large collections (where large is something greater than a couple hundred albums) of music is quite painfully slow.

If you're going to have a large collection of music using Media Center, my recommendation is to organize it hierarchically on the filesystem, and use something like Musical Folders.