Technology in terms you understand. Sign up for the Confident Computing newsletter for weekly solutions to make your life easier. Click here and get The Ask Leo! Guide to Staying Safe on the Internet — FREE Edition as my thank you for subscribing!

How can I automate an SFTP transfer between two servers?

Question:

I’m trying to set up an automated sftp transfer from one Linux box to another. I understand that you have to create a key with ssh-keygen, then put the key file on the other machine. But sftp still prompts me for the password. I read that the users on both machines must be the same… is that correct?

No, not correct.

As it turns out, this is something I do regularly with ssh, as well as both sftp and rsync, as part of my backup and load balancing approaches for Ask Leo! Let me walk you through what I’ve done.

Become a Patron of Ask Leo! and go ad-free!

SSH Configuration

To begin with, most of this relies on a the configuration of sshd, the SSH (Secure SHell) daemon running on the server you’re attempting to connect to (we’ll call it “server2.com”). Check the “sshd_config” on that server, typically in /etc/ssh. In some cases, these settings are not always present or set the way we need:

RSAAuthentication yes
PubkeyAuthentication yes

This enables the public/private key authentication mechanism we’re about to use.

Public/Private Key Generation

We’ll generate the keypair on the Linux box that you want to connect from. We’ll call that “server1.com”. It’s that box on which you plan to run ssh, sftp or rsync.

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
c1:21:e 3:01:26:0d:f7:ec:52:0e:0c:90:9b:6e:d8:47 user1@server1.com

What I’ve done with the command above is generated a public/private key pair. I responded to each prompt by hitting Return.

…mere possession of the private key is sufficient to gain access to what ever resources into which you’ve placed the corresponding public key.

Note that I did NOT enter a passphrase. That’s kind of important, because if you do enter a passphrase you’ll need to enter it in order to use the private key. Since we’re looking for an automated solution, the private key must not have a passphrase.

This is important: by not placing a passphrase on your private key, the security implication is that mere possession of the private key is sufficient to gain access to what ever resources into which you’ve placed the corresponding public key. Safeguard your private key.

My private key was placed in /home/user1/.ssh/id_rsa. This needs to be kept secure, because of the security implication above, but also needs to be available to the process attempting to make an ssh, sftp or rsync connection. If these tools are run under the ‘user1’ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.

My public key is in /home/user1/.ssh/id_rsa.pub. This is the key that gets distributed to those places that want to grant you access.

Planting the public key

On the “remote” server, server2.com, pick an account – ANY account – that you want to connect as. In that account’s home directory, create a “.ssh” subdirectory, and in that directory create a new text file called “authorized_keys”. If it already exists, that’s fine, use the existing file.

If you create the file and/or directory, I recommend that the directory be chmod 700, and the file 600. In other words, only the owner can access the directory, and the file within it.

Add to that file the contents of the id_rsa.pub file created above. That would be a *single line* that looks something like this:

ssh-rsa <lots of characters> user1@server1.com

Once saved anyone in possession of the private key that matches this public key can now login as this account.

sftp

I planted the public key in the account user2 on server2.com. So now, on my server, server1.com, logged in as user1, and where the private key is stored as described above, an sftp session looks like this:

sftp user2@server2.com

“user2” specifies the remote account on server2.com to login as.

That’s it. Magic happens, and I’m authenticated. That magic? The private key is matched to the public key, which indicates you are authorized to login to that account. An sftp session is born. No interactivity required.

(IF you did enter a passphrase on the private key, you would have been prompted to enter it here. NOTE that this is the passphrase to unlock the private key, which is local. It has nothing to do with any passwords on the remote site.)

rsync

For file copy operations, rsync rocks. It does things like intelligent compression, copy only if needed, and a whole host of other operations.

So, assuming all the keys are set up as above, this rsync command copies a file from the local machine to the remote:

rsync -e ssh file user2@server2.com:/home/user2/

Local file “file” is copied to the remote /home/user2/file after logging in as “user2” using ssh as the transport (hence the “-e ssh” option), and with that, using the private/public key pair we created for authentication. Again, no interactivity required.

Rsync supports an incredibly rich set of options for recursion, compression attribute retention, date/time stamp and so on. Well worth a look see if you’re copying anything of any significant volume.

SSH

Since we’ve gone this far, it’s worth noting that SSH itself just works as well to open up a remote shell once the keys are in place. Example:

ssh user2@server2.com

and *poof* – a remote shell on server2, logged in as user2.

Do this

Subscribe to Confident Computing! Less frustration and more confidence, solutions, answers, and tips in your inbox every week.

I'll see you there!

81 comments on “How can I automate an SFTP transfer between two servers?”

  1. Hi!! I’m doing all the steps, but the sftp always asks me for the password. I don’t know what is happning, do you?

    Please, help!

    An

    Reply
  2. your public/private keys are probably not compatible with the ssh server. you are probably using either OpenSSH or Putty and your server is probably using an ssh.com based package. If this is the case drop putty and use openssh, create the public and private keys then run ssh-keygen -e -f key.pub > key2.pub (where key is the name of your key) this creates a new copy of the public key which can be used on an ssh.com server. Rename the key2.pub file to key.pub (which probably means you will have to delete existing key.pub) then use the private key (which has not changed) with the new public key (ie upload the new public key). Also don’t forget to change the authorisation file on the server (/home/username/.ssh or .ssh2) to accept your new key.

    Reply
  3. This article is incredible Leo. Thank You.

    I have seen the sftp usage options given below,
    sftp [-1vC] [-A Password] [-b batchfile] [-L logfilename] [-osshopt=value] [user@]host[:file [file]]

    But in my SunOS 5.9 , I am not getting the options -L, -b, -A. What could be the reason. I wanted to get a file transferred from the remote server to my server after I connect through my automated connection script. My automation script connects and gives me the SFTP prompt. It’s not identifying the commends given in the batch file.

    Thanking you once again for all the help.

    Reply
  4. There are slightly different versions of sftp on different OS’s. Does “man” work on your system? If so I’d try a “man sftp” and see if the documentation includes the features you need.

    Reply
  5. Thank you Leo.

    I have checked the man pages for sftp. As you have mentioned -L, -b, -A keys are not available.

    I just wanted to connect automatically to a remote system and fetch a file from that machine.
    Please see the script I am having.

    sftp ncc_b2b@blrsun27
    get file1.txt
    exit

    I have follwed the steps given by you for automated connection. Once I am executing this, it’s automatically connecting to the remote system and I am getting the SFTP prompt. But it’s not executing the get command.

    Can you please guide me how to do this now, as the -b option is not available. Should I go for SSLFTP or any other alternative to this.

    Thanking You Leo.

    Manoj.

    Reply
  6. YOu might need to have

    get file1.txt
    exit

    in one file (commands.txt) and then use that as input for the sftp command:

    sftp < commands.txt

    Reply
  7. Thank You Leo.

    It’s working fine now. I have managed to Automatically connect to the remote machine and transfer the required file. All because of your help.

    Once Again Thanking You Leo,
    Manoj.

    Reply
  8. Hi,
    I have tried all the steps but still the automation is not working with one machine.Kindly help me out in resolving this issue

    Thanks,
    P.Rajeswari

    Reply
  9. Rajeswari,

    I have followed the steps given by Leo and it’s working fine. It’s not clear from your message what’s the actual problem. Can you please clarify, where you are facing the problem.

    Reply
  10. I have this normal ftp script that works fine, ftping files from the unix box to a windows box.
    #!/usr/bin/ksh
    ftp -v -n hostname ftp-gw
    Type:xxx@xxx.xxx.com
    Type in password when prompted
    Ftp>quit
    Put C:tempfilename.txt

    Can you help me with this sftp script.

    Reply
  11. Hi Leo,
    I am also following all the steps but SFTP is still asking me the password. I am having SunOs and Global Scape OS and I am trying to do automation from SunOS to GlobalScape. Is there any compatible problem. Please guide

    Reply
  12. I’m trying to write a script to connect to a server via sftp that has publickey+password authentication. Even when I put the password in a file and use the -b option, it still prompts me for a password. Is there any other way to pass the password to the remote server?

    Reply
  13. Hi Leo,
    Nice article. I have one question though.
    If I want to put a file from server1 to server2 – automatically – How do I do that?

    If I do sftp user2@server2.com then it goes into interactive mode of sftp. But my problem is I want to put a file from server1 to server2 – using sftp command not from interactive prompt.

    any clues on that please?

    Regards
    Kumar

    Reply
  14. Check out the gazillion command line options for sftp. You can bypass the interactive prompt by specifying everything on the command line.

    Reply
  15. Hi, We need to create a kshell script to FTP files to the destination server. We have to use sftp to do so.

    My concern is, how do I avoid getting password prompt when using sftp command. Please help, it is very urgent.

    Thanks in advance.

    Regards, Minesh Shah

    Reply
  16. Hi Leo,
    Your instructions to automate my sftp transfers worked great! I’ve only run into one problem. When I try to change user1 on server1 to point to user3 on server2, it starts prompting for a password again. i.e. sftp user2@server2.com works fine but sftp user3@server2.com doesn’t work automatically. I copied the public key from the .ssh folder on user2 over to the .ssh folder for user3 but that doesn’t seem to help. What am I missing here?

    Regards,
    James

    Reply
  17. —–BEGIN PGP SIGNED MESSAGE—–
    Hash: SHA1

    Nine times out of ten, the permissions on the copy of the .ssh files and the
    authorized_keys file that you copied are wrong. It needs to be owned by the
    account who’s .ssh directory it resides in.

    Leo
    —–BEGIN PGP SIGNATURE—–
    Version: GnuPG v1.4.6 (MingW32)

    iD8DBQFGDHLeCMEe9B/8oqERAureAJ9CmSJMmSetKZZV8UaFGH2JVXRBswCbBd4I
    xuTKYBGphXOrpANg7P3CqMY=
    =izE9
    —–END PGP SIGNATURE—–

    Reply
  18. We have the lprng daemon running. When we put a job on a queue, it calls a shell filter script that is suppose to sftp the file to the remote server.

    The problem is, the daemon tries to connect to the remote host even though we specify the user@host2 (user is qadmin and has the keys set up both sides).

    ie: scp -b qadmin@host2

    Reply
  19. Hi Leo,great advice. One additional question, if the account doesn’t have a standard login shell will this automat process work?

    Reply
  20. —–BEGIN PGP SIGNED MESSAGE—–
    Hash: SHA1

    I don’t believe so. I vaguely recall having to enable login ability for
    accounts with which I want to do this.

    Leo
    —–BEGIN PGP SIGNATURE—–
    Version: GnuPG v1.4.6 (MingW32)

    iD8DBQFGFBU3CMEe9B/8oqERAtkwAJ4t3CuIMp7Ld/5D+2CElHl7TeP+qQCfXJX4
    8P8DihofejdoYIYiwuWF1rQ=
    =7nkv
    —–END PGP SIGNATURE—–

    Reply
  21. If you want to automate sftp because for some reason scp doesn’t work in you situation you can try “expect” You need to install it. The script would look something like this:
    Replace:
    DIR=local and remote directories.
    IP-adress=remote server address
    user=user set up on remote sftp server.

    #!/usr/local/bin/expect -f -d
    # Expect script that automates sftp
    ###################################

    spawn /usr/bin/sftp user@”IP-adress”
    expect “user@IP-adress’s password: “
    sleep 7
    send Passwordr”
    expect “sftp> “
    send “lcd DIRr”
    expect “sftp> “
    send “cd DIRr”
    expect “sftp> “
    send “cd DIRr”
    expect “sftp> “
    sleep 10
    send “mput *.pgpr”
    sleep 10
    expect “sftp> “
    send “byer”

    Reply
  22. Leo, I put your steps to use and they work. Mostly.
    Here’s my situation:
    3 servers, I’ll call A, b and c.
    A: RHEL4
    b: Solaris 8 on Sparc
    c: Solaris 8 on Sparc.

    If I send from A to b or c, the automation works fine. If I send from b or c to A, I am challenged with a password request. b and c have A’s public RSA key. A has both b and c’s RSA public keys. In every case, I am the user who is also attempting the SCP connection. It is most expedient to auto send from b to A, but I can rewrite b’s script to run on A, if I must. Just wondering why its only working in one direction.

    Reply
  23. Leo, it turns out that on server A, the authorized_keys file had permission 664. Apparently if that file is writable by anyone other than the owner, you will be challenged for a password, regardless. Once I CHMODed it to 644, the process worked flawlessly.

    Reply
  24. Hi Leo,

    I generated rsa keys for system a and b and put both keys in authorized_keys for my user on system c.

    system a connects using sftp and does not prompt for a password.

    system b prompts for a password – when I provide it the connection is made.

    on all 3 systems all files in .ssh are writable only by the user.

    what should I be looking at to trouble shoot this?

    Reply
  25. Leo, excellent article!! But it has worked for me only when i try to ssh/sftp to root account on server machine (server2 in the ex.). Any other user (making all the steps for each one) fails.
    Is there any trick related to that ??
    Thanks in advance.

    Reply
  26. I am having same problem as “Manoj Das”.

    here is part of the post and your comment on it:

    script:
    sftp ncc_b2b@blrsun27
    get file1.txt
    exit

    I have followed the steps given by you for automated connection. Once I am executing this, it’s automatically connecting to the remote system and I am getting the SFTP prompt. But it’s not executing the get command.

    Can you please guide me how to do this now, as the -b option is not available.

    Posted by: Manoj Das at October 3, 2006 07:36 AM
    YOu might need to have

    get file1.txt
    exit

    in one file (commands.txt) and then use that as input for the sftp command:

    sftp

    Reply
  27. This is followup to my earlier post. I made it work sort of:

    #!/bin/ksh

    sftp -B commands.txt user1@pontoon

    in commands.txt I have
    get 1.dat 2.dat ( I want to get 1.dat and rename it to 2.dat)
    quit

    It gets 1.dat and complains can not find 2.dat on remote server.

    Here is some other info about my system:

    sftp -V
    sftp: F-Secure SSH 3.1.0 (build 12) on sparc-sun-solaris2.8

    thanks for reading my posts.

    Reply
  28. Hi Leo, I tried to follow the steps you posted here but I still cannot make it work. I am trying to use SFTP to send files from HP-UX to Windows Server 2000 with a domain account setup on the Window Server. No matter what I did, I always get promoted with password. Is the public key authentication not going to work for me since this is an AD account? What’s the best way I can troubleshoot this issue? Thanks.

    Reply
  29. This article helps me greate.
    I have one problem.
    I can sftp without password from my account to abc@srvr1, but it asks fro password while sftp from my account to xyz@srvr3 even though I have copied the same public file in .ssh directory on both these severs.These both servers have same sshd_conf files.

    Reply
  30. Very nice document. First I want to thanks U.
    1. Is there any way to write script which copy files from remote server.

    Reply
  31. Hi,

    I’m currently using the -b Batch mode reads a series of commands from an input batchfile
    eg: -b batchfile user@host

    Right now I have to add in the switch -C for the compresion. How I can do that with the using the above code as well with the batch file contain script like – Put command to upload the file from local to remote system.

    Reply
  32. Hi Leo, I wanted to know abouty sftp, and stumbled at your site, you are a great teacher.
    Accept my humble respects.
    Regards,
    Venkat

    Reply
  33. Hey Leo,

    When I pasted the address you used before into my terminal (sftp user2@server2.com) to modify it, the CR somehow got into my copy buffer, so when I pasted it I apparently ftped into that actual address. It then locked up my system for awhile and I was unable to exit.

    Do you happen to have any idea what server2.com actually is, and whether having ftped into it could have somehow compromised my system?

    Thanks, Bill

    Reply
  34. I didn’t see a response to Vasant’s post of 4/3/08, so thought this might be useful. Each “from” account must create a key-pair and append the public key to the .ssh/authorized_keys on the “to” account side. You can’t use one public key generated by one account to cover two source “from” accounts. But one “to” account can have multiple public keys in its .ssh/authorized_keys file to allow multiple “from” accounts to access that one “to” account.

    Reply
  35. Hi Leo
    I have a problem here. I have planted the keys in the remote server but when I use the scp command in a script, it does not work.
    The login to the remote server does not create a problem but the scp itself does, in a sense that the file is not getting copied to the remote server. Do I have a hope or will have to look for alternatives.

    Thanks in advance
    Saurabh

    Reply
  36. Hi Leo,
    Great article. I thought I would not be able to use this, as in the secure government environment I work in using Red Hat Enterprise Linux 5.1, the sshd_config file has both RSAAuthentication and PubkeyAuthentication commented out with #’s.

    However, as I really needed this functionality from User A on Host A to User oracle on Host DB, I gave it a try and it’s working like a charm. Not sure “why” but I’m glad it’s working.

    Many thanks for a great, useful, well-written article.

    Reply
  37. Hi Leo,
    I have a simillar requirement.
    I would like test the functionality of sftp connectivity with keys from User A on Host A to User oracle on Host B.

    Pls confirm whether
    We can connect from HOST A USER A to HOST B Oracle
    Using passworldess authentication
    when we copy the public keys of USER A to oracle’s home directory on Host B.

    Please help.

    Reply
  38. I am trying to automate scripts using pageant for winscp. but i need to manually input the passphrase. as the file will be retrieved over the night and server reboot is done daily, is there a wasy to hardcode the passphrase in the script?

    Reply
  39. I setup 2 linux servers with the information provided and it works like a charm.

    However I’m also trying to setup a Windows server using EFT server by Globalscape. Where would the authorized_keys file be created on the Windows box to allow access?

    Reply
  40. I followed the process but still password is prompted.

    from teh below text
    “If these tools are run under the ‘user1’ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.”

    could you please let me know the “command line options will need to point to the right place and key.” so that password is not prompted.

    Reply
  41. How can I make this work with an ftp server where there is no unix login available? I can ftp to that server and I have login there, but want to use sftp to automate my file transfers. Thanks!

    Reply
  42. Dear Friend ,
    I have followed the process given above .I have 3 servers . On one server the process worked fantastically and SSH is authenticated without password , but on one server it is again asking for password , not the passphrase , the actual password for login on server but on the other server the process given on the sit is working fantastically . what to do on the server which is not responding ?

    Best Regards
    Prashant

    Reply
  43. The article is well described and very easy to follow.
    I have followed the steps mentioned above but i still get a password prompt while making a sftp connection. I believe that the problem is because the files are generated under user linus and the .ssh is under root as mentioned by you “”If these tools are run under the ‘user1’ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.”

    Could you please help me as to how to make the command line options point to the right place and key.

    Reply
  44. I have a query regarding the keys.I generated the public and private keys with the userid X on server 1 and asked server2 team to add the publickey to the user id Z on server 2.I tried to SFTP using the private key with the user Y(which have access to the private key) from Server1 to Server2 with the user id Z.Iam not able to login ..Its asking for password by saying that permissions are too open.When i try to SFTP using the User X(keys generated on this id) from server 1 to Server 2.its logging in without asking for password.can u explain what the issue is..?

    Reply
  45. Hello Leo,

    The document that you given is very superb. Its very simple and useful to follow. Thank for the document.

    I followed the same with out giving the passphrase while generating the key.Its working. But my requirement is to pass the passphrase while generating keys. When I pass it, the automated process is not working. Its asking me to pass the passphrase interactively.

    Could you please help me in this.

    Thanks for ur help in advance.

    Siva

    Reply
  46. Sorry rushing too fast – should have read before posting….

    Document is fantastic but I transfer files across multiple boxes how can I set up this process so that ALL boxes are able to transfer files without password notification. Process works well with box 1 & 2 but when I put 3 in, I lose one.
    Box 1 talks to box 2. I then set up box 1 then talks to box 3 but I lose interactive connection with box 2 ?!?!
    Can you please suggest something (I have 6 boxes that interact with each other).

    Reply
  47. Hi,
    Thanks for the detailed artcile. I have one clarification though.
    “Once saved anyone in possession of the private key that matches this public key can now login as this account.”
    Can any other user, besides user1, initial a SFTP connection to server2 as user2?
    Thanks.

    Reply
  48. Thanks so much for this great article. It has helped me immensely. I can run rsync just fine without passwords from a bash script, but now I’m now trying to execute rsync as a cron task and I am getting the following errors:
    Permission denied, please try again.
    Permission denied, please try again.
    Permission denied (publickey,password).
    rsync: connection unexpectedly closed (0 bytes received so far) [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.8]
    rsync returned 12

    I’ve tried using keychain (http://www.gentoo.org/doc/en/articles/openssh-key-management-p2.xml) and get the same errors. Any ideas? Thanks, Josephine

    Reply
  49. Hi, still I’m getting the prompt for password in the remote server when I try to sftp…
    any clues please…
    it is uix to unix box…

    Reply
  50. I’m using GlobeScape’s EFT as my sftp server and the remote server is using another kind of SFTP. will this code work?

    Reply
  51. What configuration required to automate SFTP script to connect windows server above details are from Unix to Unix what if the target server is windows
    please help me i m in huge trouble….

    Reply
  52. Great tutorial Leo,

    it’s worth adding; if you’re doing this for automated script backups, you should definitely use the “ChrootDirectory” option in sshd_config on the destination server – this prevents anyone logging in as the user (if the key was ever compromised) – this will just allow them to do sftp sessions.

    You might also want to add to sshd_config:

    Match User foobar
    PasswordAuthentication no

    for similar reasons.

    Reply
  53. TO Kumar’s question, you have answered – “Check out the gazillion command line options for sftp. ” Can you please elaborate?
    and using rsync for this is suitable ?

    Reply
  54. Hi I m trying to do sftp from unix server to remote windows server by using below unix command from unix server,

    but i m getting error as mentioned below,

    $ sftp usplwedwdev01
    ssh: connect to host usplwedwdev01 port 22: Connection refused
    Connection closed

    Could you please help me on this..

    Reply
    • That host doesn’t seem to support sftp, or it’s put it on another port, or a firewall is blocking it. You’ll need to talk to the owner of that server.

      Reply
  55. You’ve added the key in such a way that shell command can be easily run, and you can forward network ports and X-Windows. You might not want to allow this activity for a file-transfer account.

    To forbid it, make these changes to your authorized keys:

    command=”/usr/libexec/openssh/sftp-server”,no-agent-forwarding,no-port-forwarding,no-x11-forwarding,no-pty,no-user-rc ssh-rsa AAAA[stuff]xyz user@host.com

    Note that scp will not work when you do this, but sftp will.

    You can also chroot() to REALLY strap down the account, but that’s a little more complex.

    Reply
  56. hello nice blog..
    i am using amazon ec2 Linux server both have different key pair i followed the steps but not working..can you please specify the commands to rsync both servers

    Reply
  57. Hi Leo

    I want to setup OPENSSH in windows 2008 server. Need to transfer files from Linux machine with password less configuration. Could you please tell me the steps to configure.

    Reply
  58. Hi leo

    I have a shell script to fetch files from source server, when we are executing it through Unix box it is fine, but when i am trying it through oracle apps i am getting error like permission denied, can you pls help me.

    Thanking you.

    Reply
  59. Hi Leo

    I just want to do a sftp to a remote server from a shell script.
    As i am new to shell scripting, i do not how to do it.
    Appreciate if you can send me the steps.

    Thanks
    Anil

    Reply
  60. Hi Leo
    I just want to do a sftp to a remote server from a shell script.
    As i am new to shell scripting, i do not how to do it.
    Appreciate if you can send me the steps.
    Thanks
    Anil

    Reply

Leave a reply:

Before commenting please:

  • Read the article.
  • Comment on the article.
  • No personal information.
  • No spam.

Comments violating those rules will be removed. Comments that don't add value will be removed, including off-topic or content-free comments, or comments that look even a little bit like spam. All comments containing links and certain keywords will be moderated before publication.

I want comments to be valuable for everyone, including those who come later and take the time to read.