The Net's full of the same question and mostly to do with Nightfire lol, unfortunately they are all as frustrated as you and their does not seem to be an exact science to the answer.
These may work;
Out of memory Failing request of .... [xxxxxxxxxxxxxxx] bytes
Change the screen resolution to a lower setting, try 800x600 to start with.
pagefile.sys (Windows virtual memory), make sure that it is set to a reasonable amount and that the drive it's on (Normally C) has enough free space for it.
How to set performance options in Windows XP
Using MS Guidelines the recommended minimum size is equivalent to 1.5 times the RAM on your computer, and 3 times that figure for the maximum size. (min of 6gb, max of 12gb for you, though that sounds a bit over the top)
Make sure that you have the latest drives for your graphics card, Direct X and that you can play the movies in Nightfire.