If you have scripts that you know require large amounts of memory it's perfectly safe, and needed, to set the limit high. Problems can occur if the limit is set to high for the system in regards to available amount of memory and other running applications.
Also keep in mind that a faulty script that might end up in an infinite loop processing some data would often come to a halt quite fast if the memory limit is 64 MB or something like that. Having the memory limit at 4 GB in this scenario would leave you with a system running at full tilt in an infinite loop for way longer and might cause your entire system to become unresponsive.
But I have had large data crunching applications with memory limits of around 2-4 GB without any issues on systems with 4-8 GB of memory. Note that, as stated by other users, these applications was ONE instance. If you have applications that run several instances of your scripts (like most web-applications do) you will run into major problems by using a lot of memory in each instance. You will run out of memory and further clients will not be able to use your application until memory has been freed.
But don't just ramp up the memory limit to hide issues. Inspect why your application requires that much memory and see if you can rectify that first.
So oh god!!! now my system is slowing down because of that limit??? (only for Clearing the Cache Process apparently)
– Bursarmemory_limit
isPHP_INI_ALL
changeable, so if you have a specific script that needs 512MB then callini_set('memory_limit', '512M');
at the top of the script. – Unpaid