What I ended up doing was having vim save backups and temp files to ~/.vim/backup and ~/.vim/tmp by adding this to my .vimrc:

    set backupdir=~/.vim/backup
    set directory=~/.vim/tmp

Of course these can also be relative paths so you can have a project save these to a local .backup directory too if you want to keep them near the files they relate to.