A website lite Umbraco consists of many different parts that consumes memory, the most typical except the iis service itself would be
- Logging
- Web Services
- Search service (that it self may be a web service)
- Scheduled jobs
- Data caching
- Custom asp.net code, ie if you have any custom dlls or code in the app_code folder
- ...
I'd start by looking in the Event Viewer seeing if there are any exceptions thrown by your web applications, if there is the web applications will typically collect lots of garbage information that is released on iisreset.
Next step is to check all custom code and tune the logging levels. When done I'd profile the application with a memory profiler, there are a few but red-gate is know for their high quality tools (www.red-gate.com/Memory_Profiling).
And... 300-500 Mb memory isn't that much from a website running .net with content caching, image transforms and so on activated, keep your expectations real. Spontaneously I'd say you are within the normal range.