I just instrumented the theme rout index.php script to output the memory_get_peak_usage() value after the page has loaded. Comes out to about 2.5mb for most pages in the basic theme. Should fit comfortably in 128 mb.

But, if you are caching images, there will be scripts running parallel that may take a bunch of memory if your images are large.

Yes, it's most of the time the images respectively their processing causing memory trouble. And to remind "large"does not refer to the file size but to the dimensions.