First: The key agent are supposed to exit when you log out, by design. Because of security issues, you don't want your login to be open if you log out by mistake. So, you really should NOT use ssh key agent for this, it is not the way to use it.
Second: It is posible to have very restricted user with few rights, to store/recive backup files on /srv/backup or where ever it has access. To log in to that user you have a ssh certificate whithout any certificate password, so you can log in without password if client has the right certificate.
You could then even restrict which machines are allowed to connect and what programs to runt in ~/.ssh, look that up in the manual.