Darniaq has a good post up entitled "Mob Spawning and Immersion".(bad link removed) I put some actual thought into a response that is shared below. Enjoy!
Encounters in MMORPGs are for the most part a small puzzle. It reminds me of the arguments of the "dumb" enemies in the "brilliant" Metal Gear series for consoles. The guards are completely stupid, can't see 10 feet in front of them, and fall for the dumbest tricks in the book (moving box anyone). The game has always been about a complex visual puzzle that you complete via an advanced tool also know as Snake. The story is dramatic and driven by movie like plots, but the gameplay will never be "movielike". Still the MG games are still IMMERSIVE! Read on to find out why.
Back to MMORPGs where you have dungeons ala Everquest 1 where it was more about being in the right places at the right time to "finish" the dungeon. You had to camp out the "boss" which lead to fair bit of "PvP" group/guild social dynamics.
WoW dungeons play the "everything is setup when you enter". You have to start at the beginning of the puzzle and get to the end for the "cheese". While the dungeons all aren't linear there is definite "fast" and "better" ways to do them... the difference between a veteran sodoku and a novice. After some experience you can do them relatively fast and move on to the next "harder" set.
A random encounter in the outdoors in WoW or EQ though is a puzzle in itself also. There is defined rules that you have to follow. Is it linked? Is the target out of "call for help range of others"? Is this something I can take? Then add in a group. Pull this... CC this... kill this... offtank that... etc.
So what we have is a bunch of puzzles with tools to complete them. How can you make a puzzle immersive? Variables! The more you toss into the puzzle the more the player has to manage. The more they have to manage the less they have to think about where or what this monster is doing in front of them.
By having monsters that are just there... standing like numb nuts without a purpose removes a lot of variables. Players have few "unknowns" because they can see what is going to happen before they pull. While you need a certain level of this to maintain a game aspect you don't need a lot of it. Players need to know they can go kill a few Orcs without having 1,000 Orcs charging out of the fortress and hunting them down.
Putting unknowns into the fight makes it a better experience IMHO. Instead of having a room with pirates just standing there waiting to be pulled you could have a trigger that has a half dozen more pirates drop in from a hidden upper ledge.
But lets try to get away from triggered events because that is another big block towards immersion. Well they won't attack me here, but I know if I move another foot they will start coming. This is essentially how most FPS games operate. As you advance to a new "room" (throwback to MUDs baby) you trigger the next set of baddies to fight. Now you could have a Doom 3 like experience (few variables) or a Half Life 2 experience (lots of variables), but still its setup on triggered events. Great for advancing story plots... bad for immersive multiplayer.
Randomness in encounters is key. Another key is removing static spawns. Dynamic spawning will be the future and Vanguard shows some promise of this in their "NPCs will fight each other". Its a method seen in WoW a little and some other games where a wolf chases a rabbit, but nothing on a large scale.
I am talking about something like the once planned Strongholds in Star Wars Galaxies. Structures that would "spawn" in the world and the NPCs would start building them up. If players did nothing about them then they kept growing until the players would have to deal with them. Sadly it never happened.
You can take that concept and work a bit with it having multiple factions of NPCs building. Each faction reacts differently to each other. So a Jawa Faction Stronghold would war with a Tusken Raider Stronghold that has grown to large right next door. Then from these strongholds you start spawning "layers". All NPCs would actually appear from a building until it was destroyed so in a way you have a visual key of where they are coming from. Don't want more Jawa Sandcrawlers spawning? Distract the Jawa Guards and send a crack team in to destroy the Sandcrawler factory. Every layer offers easier to defeat "groups" that are dispatched out like scouting teams, harvesting groups, trading groups, social groups (meant to give quests to players), etc. As you gain more layers the inner layers grow in difficulty.
Now players can go in and destroy it, perform diplomacy, trade, and all sorts of things. You can build up faction to get into a stronghold here or there while risking out right war with others. This is really all inspired by Will Wright's upcoming Spore (which will be one of; if not the best game ever). The whole game operates in this degree somehow, but much more advanced. A simpler system will come to an MMORPG soon enough.
I don't think "spawning" will ever die because after all players need something to kill and nothing breaks FUN worse than having nothing to kill. There is still ideas out there to make it much more immersive. I just provided one I've built from enjoying games and game design ideas I've experienced. There are much smarter people in actual positions to achieve such things in a game.
Update: 15 Nov, 2009 - Edited spelling, applied label, and removed broken link.
Guild Wars 2 Heart of Thorns - Via the Twitters: Guild Wars 2 expansion, Heart of Thorns, announced. To which I say:
4 weeks ago