Quake 3 engine, while probably being enough, is just getting too old. Half-Life 2 Source engine is supposedly highly moddable, and easily so. If you're going to include physics, it's also a better solution than Quake 3. The HL2 AI can navigate around/or interact with physics object in a sophisticated manner (ie. the stalker creature arriving at a bridge that blocks his way, where it just crawls under the bridge instead of trying to blow it up, and the leg movement is completely physics-driven) The levels can be 64 times bigger than those in HL1, currently, allowing for quite the huge zoo. But hell, there are more possibilities.. but considering Valve's modsupport, HL2 might prove the easiest choice..
