Jenkins need more RAM. alternatively, you might try and restart the server every few days. restarting a server releases the memory.
Do you know what is the system requirement to run Jenkins.
I am using windoes Server 2012 (64 bit) , 4GB RAM, I manually allocated 2048M of memory to JVM but this cause my system to hang.