How can I automate an SFTP transfer between two servers?

//

I’m trying to set up an automated sftp transfer from one Linux box to another. I understand that you have to create a key with ssh-keygen, then put the key file on the other machine. But sftp still prompts me for the password. I read that the users on both machines must be the same… is that correct?

No, not correct.

As it turns out, this is something I do regularly with ssh, as well as both sftp and rsync, as part of my backup and load balancing approaches for Ask Leo! Let me walk you through what I’ve done.

SSH Configuration

To begin with, most of this relies on a the configuration of sshd, the SSH (Secure SHell) daemon running on the server you’re attempting to connect to (we’ll call it “server2.com”). Check the “sshd_config” on that server, typically in /etc/ssh. In some cases, these settings are not always present or set the way we need:

RSAAuthentication yes
PubkeyAuthentication yes

This enables the public/private key authentication mechanism we’re about to use.

Public/Private Key Generation

We’ll generate the keypair on the Linux box that you want to connect from. We’ll call that “server1.com”. It’s that box on which you plan to run ssh, sftp or rsync.

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
c1:21:e 3:01:26:0d:f7:ec:52:0e:0c:90:9b:6e:d8:47 user1@server1.com

What I’ve done with the command above is generated a public/private key pair. I responded to each prompt by hitting Return.

…mere possession of the private key is sufficient to gain access to what ever resources into which you’ve placed the corresponding public key.

Note that I did NOT enter a passphrase. That’s kind of important, because if you do enter a passphrase you’ll need to enter it in order to use the private key. Since we’re looking for an automated solution, the private key must not have a passphrase.

This is important: by not placing a passphrase on your private key, the security implication is that mere possession of the private key is sufficient to gain access to what ever resources into which you’ve placed the corresponding public key. Safeguard your private key.

My private key was placed in /home/user1/.ssh/id_rsa. This needs to be kept secure, because of the security implication above, but also needs to be available to the process attempting to make an ssh, sftp or rsync connection. If these tools are run under the ‘user1′ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.

My public key is in /home/user1/.ssh/id_rsa.pub. This is the key that gets distributed to those places that want to grant you access.

Planting the public key

On the “remote” server, server2.com, pick an account – ANY account – that you want to connect as. In that account’s home directory, create a “.ssh” subdirectory, and in that directory create a new text file called “authorized_keys”. If it already exists, that’s fine, use the existing file.

If you create the file and/or directory, I recommend that the directory be chmod 700, and the file 600. In other words, only the owner can access the directory, and the file within it.

Add to that file the contents of the id_rsa.pub file created above. That would be a *single line* that looks something like this:

ssh-rsa <lots of characters> user1@server1.com

Once saved anyone in possession of the private key that matches this public key can now login as this account.

sftp

I planted the public key in the account user2 on server2.com. So now, on my server, server1.com, logged in as user1, and where the private key is stored as described above, an sftp session looks like this:

sftp user2@server2.com

“user2″ specifies the remote account on server2.com to login as.

That’s it. Magic happens, and I’m authenticated. That magic? The private key is matched to the public key, which indicates you are authorized to login to that account. An sftp session is born. No interactivity required.

(IF you did enter a passphrase on the private key, you would have been prompted to enter it here. NOTE that this is the passphrase to unlock the private key, which is local. It has nothing to do with any passwords on the remote site.)

rsync

For file copy operations, rsync rocks. It does things like intelligent compression, copy only if needed, and a whole host of other operations.

So, assuming all the keys are set up as above, this rsync command copies a file from the local machine to the remote:

rsync -e ssh file user2@server2.com:/home/user2/

Local file “file” is copied to the remote /home/user2/file after logging in as “user2″ using ssh as the transport (hence the “-e ssh” option), and with that, using the private/public key pair we created for authentication. Again, no interactivity required.

Rsync supports an incredibly rich set of options for recursion, compression attribute retention, date/time stamp and so on. Well worth a look see if you’re copying anything of any significant volume.

SSH

Since we’ve gone this far, it’s worth noting that SSH itself just works as well to open up a remote shell once the keys are in place. Example:

ssh user2@server2.com

and *poof* – a remote shell on server2, logged in as user2.

There are 65 comments:

  1. An Reply

    Hi!! I’m doing all the steps, but the sftp always asks me for the password. I don’t know what is happning, do you?

    Please, help!

    An

  2. Joe Reply

    your public/private keys are probably not compatible with the ssh server. you are probably using either OpenSSH or Putty and your server is probably using an ssh.com based package. If this is the case drop putty and use openssh, create the public and private keys then run ssh-keygen -e -f key.pub > key2.pub (where key is the name of your key) this creates a new copy of the public key which can be used on an ssh.com server. Rename the key2.pub file to key.pub (which probably means you will have to delete existing key.pub) then use the private key (which has not changed) with the new public key (ie upload the new public key). Also don’t forget to change the authorisation file on the server (/home/username/.ssh or .ssh2) to accept your new key.

  3. Manoj Das Reply

    This article is incredible Leo. Thank You.

    I have seen the sftp usage options given below,
    sftp [-1vC] [-A Password] [-b batchfile] [-L logfilename] [-osshopt=value] [user@]host[:file [file]]

    But in my SunOS 5.9 , I am not getting the options -L, -b, -A. What could be the reason. I wanted to get a file transferred from the remote server to my server after I connect through my automated connection script. My automation script connects and gives me the SFTP prompt. It’s not identifying the commends given in the batch file.

    Thanking you once again for all the help.

  4. Leo Notenboom Reply

    There are slightly different versions of sftp on different OS’s. Does “man” work on your system? If so I’d try a “man sftp” and see if the documentation includes the features you need.

  5. Manoj Das Reply

    Thank you Leo.

    I have checked the man pages for sftp. As you have mentioned -L, -b, -A keys are not available.

    I just wanted to connect automatically to a remote system and fetch a file from that machine.
    Please see the script I am having.

    sftp ncc_b2b@blrsun27
    get file1.txt
    exit

    I have follwed the steps given by you for automated connection. Once I am executing this, it’s automatically connecting to the remote system and I am getting the SFTP prompt. But it’s not executing the get command.

    Can you please guide me how to do this now, as the -b option is not available. Should I go for SSLFTP or any other alternative to this.

    Thanking You Leo.

    Manoj.

  6. Leo Notenboom Reply

    YOu might need to have

    get file1.txt
    exit

    in one file (commands.txt) and then use that as input for the sftp command:

    sftp < commands.txt

  7. Manoj Das Reply

    Thank You Leo.

    It’s working fine now. I have managed to Automatically connect to the remote machine and transfer the required file. All because of your help.

    Once Again Thanking You Leo,
    Manoj.

  8. Rajeswari Reply

    Hi,
    I have tried all the steps but still the automation is not working with one machine.Kindly help me out in resolving this issue

    Thanks,
    P.Rajeswari

  9. Manoj Das Reply

    Rajeswari,

    I have followed the steps given by Leo and it’s working fine. It’s not clear from your message what’s the actual problem. Can you please clarify, where you are facing the problem.

  10. Bashir Reply

    I have this normal ftp script that works fine, ftping files from the unix box to a windows box.
    #!/usr/bin/ksh
    ftp -v -n hostname ftp-gw
    Type:xxx@xxx.xxx.com
    Type in password when prompted
    Ftp>quit
    Put C:tempfilename.txt

    Can you help me with this sftp script.

  11. Amit Gupta Reply

    Hi Leo,
    I am also following all the steps but SFTP is still asking me the password. I am having SunOs and Global Scape OS and I am trying to do automation from SunOS to GlobalScape. Is there any compatible problem. Please guide

  12. Mike McGinn Reply

    I’m trying to write a script to connect to a server via sftp that has publickey+password authentication. Even when I put the password in a file and use the -b option, it still prompts me for a password. Is there any other way to pass the password to the remote server?

  13. Ian McDermid Reply

    Well written article. More of this needed on the net. My batch job now works!!!

  14. kumar Reply

    Hi Leo,
    Nice article. I have one question though.
    If I want to put a file from server1 to server2 – automatically – How do I do that?

    If I do sftp user2@server2.com then it goes into interactive mode of sftp. But my problem is I want to put a file from server1 to server2 – using sftp command not from interactive prompt.

    any clues on that please?

    Regards
    Kumar

  15. Leo Notenboom Reply

    Check out the gazillion command line options for sftp. You can bypass the interactive prompt by specifying everything on the command line.

  16. Minesh Shah Reply

    Hi, We need to create a kshell script to FTP files to the destination server. We have to use sftp to do so.

    My concern is, how do I avoid getting password prompt when using sftp command. Please help, it is very urgent.

    Thanks in advance.

    Regards, Minesh Shah

  17. James Reply

    Hi Leo,
    Your instructions to automate my sftp transfers worked great! I’ve only run into one problem. When I try to change user1 on server1 to point to user3 on server2, it starts prompting for a password again. i.e. sftp user2@server2.com works fine but sftp user3@server2.com doesn’t work automatically. I copied the public key from the .ssh folder on user2 over to the .ssh folder for user3 but that doesn’t seem to help. What am I missing here?

    Regards,
    James

  18. Leo Notenboom Reply

    —–BEGIN PGP SIGNED MESSAGE—–
    Hash: SHA1

    Nine times out of ten, the permissions on the copy of the .ssh files and the
    authorized_keys file that you copied are wrong. It needs to be owned by the
    account who’s .ssh directory it resides in.

    Leo
    —–BEGIN PGP SIGNATURE—–
    Version: GnuPG v1.4.6 (MingW32)

    iD8DBQFGDHLeCMEe9B/8oqERAureAJ9CmSJMmSetKZZV8UaFGH2JVXRBswCbBd4I
    xuTKYBGphXOrpANg7P3CqMY=
    =izE9
    —–END PGP SIGNATURE—–

  19. Rajesh Acharya Reply

    We have the lprng daemon running. When we put a job on a queue, it calls a shell filter script that is suppose to sftp the file to the remote server.

    The problem is, the daemon tries to connect to the remote host even though we specify the user@host2 (user is qadmin and has the keys set up both sides).

    ie: scp -b qadmin@host2

  20. sai Reply

    Hi Leo,great advice. One additional question, if the account doesn’t have a standard login shell will this automat process work?

  21. Leo A. Notenboom Reply

    —–BEGIN PGP SIGNED MESSAGE—–
    Hash: SHA1

    I don’t believe so. I vaguely recall having to enable login ability for
    accounts with which I want to do this.

    Leo
    —–BEGIN PGP SIGNATURE—–
    Version: GnuPG v1.4.6 (MingW32)

    iD8DBQFGFBU3CMEe9B/8oqERAtkwAJ4t3CuIMp7Ld/5D+2CElHl7TeP+qQCfXJX4
    8P8DihofejdoYIYiwuWF1rQ=
    =7nkv
    —–END PGP SIGNATURE—–

  22. Cowboy Reply

    This was a superb article. I am going to try getting it incorporated into our IT procedures. Thank you very much.

  23. macmd Reply

    If you want to automate sftp because for some reason scp doesn’t work in you situation you can try “expect” You need to install it. The script would look something like this:
    Replace:
    DIR=local and remote directories.
    IP-adress=remote server address
    user=user set up on remote sftp server.

    #!/usr/local/bin/expect -f -d
    # Expect script that automates sftp
    ###################################

    spawn /usr/bin/sftp user@”IP-adress”
    expect “user@IP-adress’s password: “
    sleep 7
    send Passwordr”
    expect “sftp> “
    send “lcd DIRr”
    expect “sftp> “
    send “cd DIRr”
    expect “sftp> “
    send “cd DIRr”
    expect “sftp> “
    sleep 10
    send “mput *.pgpr”
    sleep 10
    expect “sftp> “
    send “byer”

  24. Andy Reply

    Leo, I put your steps to use and they work. Mostly.
    Here’s my situation:
    3 servers, I’ll call A, b and c.
    A: RHEL4
    b: Solaris 8 on Sparc
    c: Solaris 8 on Sparc.

    If I send from A to b or c, the automation works fine. If I send from b or c to A, I am challenged with a password request. b and c have A’s public RSA key. A has both b and c’s RSA public keys. In every case, I am the user who is also attempting the SCP connection. It is most expedient to auto send from b to A, but I can rewrite b’s script to run on A, if I must. Just wondering why its only working in one direction.

  25. Andy Reply

    Leo, it turns out that on server A, the authorized_keys file had permission 664. Apparently if that file is writable by anyone other than the owner, you will be challenged for a password, regardless. Once I CHMODed it to 644, the process worked flawlessly.

  26. mike huber Reply

    Hi Leo,

    I generated rsa keys for system a and b and put both keys in authorized_keys for my user on system c.

    system a connects using sftp and does not prompt for a password.

    system b prompts for a password – when I provide it the connection is made.

    on all 3 systems all files in .ssh are writable only by the user.

    what should I be looking at to trouble shoot this?

  27. Leonardo Reply

    Leo, excellent article!! But it has worked for me only when i try to ssh/sftp to root account on server machine (server2 in the ex.). Any other user (making all the steps for each one) fails.
    Is there any trick related to that ??
    Thanks in advance.

  28. Anju Reply

    I am having same problem as “Manoj Das”.

    here is part of the post and your comment on it:

    script:
    sftp ncc_b2b@blrsun27
    get file1.txt
    exit

    I have followed the steps given by you for automated connection. Once I am executing this, it’s automatically connecting to the remote system and I am getting the SFTP prompt. But it’s not executing the get command.

    Can you please guide me how to do this now, as the -b option is not available.

    Posted by: Manoj Das at October 3, 2006 07:36 AM
    YOu might need to have

    get file1.txt
    exit

    in one file (commands.txt) and then use that as input for the sftp command:

    sftp

  29. Anju Reply

    This is followup to my earlier post. I made it work sort of:

    #!/bin/ksh

    sftp -B commands.txt user1@pontoon

    in commands.txt I have
    get 1.dat 2.dat ( I want to get 1.dat and rename it to 2.dat)
    quit

    It gets 1.dat and complains can not find 2.dat on remote server.

    Here is some other info about my system:

    sftp -V
    sftp: F-Secure SSH 3.1.0 (build 12) on sparc-sun-solaris2.8

    thanks for reading my posts.

  30. Francis Reply

    Can this be applied to Windows 2003 to Solaris?
    That is automated connection from Windows 2003 to Solaris?

    Thanks

  31. Yvonne Reply

    Hi Leo, I tried to follow the steps you posted here but I still cannot make it work. I am trying to use SFTP to send files from HP-UX to Windows Server 2000 with a domain account setup on the Window Server. No matter what I did, I always get promoted with password. Is the public key authentication not going to work for me since this is an AD account? What’s the best way I can troubleshoot this issue? Thanks.

  32. Vasant Reply

    This article helps me greate.
    I have one problem.
    I can sftp without password from my account to abc@srvr1, but it asks fro password while sftp from my account to xyz@srvr3 even though I have copied the same public file in .ssh directory on both these severs.These both servers have same sshd_conf files.

  33. Gurdeep Singh Reply

    Very nice document. First I want to thanks U.
    1. Is there any way to write script which copy files from remote server.

  34. sftp compression Reply

    Hi,

    I’m currently using the -b Batch mode reads a series of commands from an input batchfile
    eg: -b batchfile user@host

    Right now I have to add in the switch -C for the compresion. How I can do that with the using the above code as well with the batch file contain script like – Put command to upload the file from local to remote system.

  35. Venkat Reply

    Hi Leo, I wanted to know abouty sftp, and stumbled at your site, you are a great teacher.
    Accept my humble respects.
    Regards,
    Venkat

  36. Bill Reply

    Hey Leo,

    When I pasted the address you used before into my terminal (sftp user2@server2.com) to modify it, the CR somehow got into my copy buffer, so when I pasted it I apparently ftped into that actual address. It then locked up my system for awhile and I was unable to exit.

    Do you happen to have any idea what server2.com actually is, and whether having ftped into it could have somehow compromised my system?

    Thanks, Bill

  37. Dickster Reply

    I didn’t see a response to Vasant’s post of 4/3/08, so thought this might be useful. Each “from” account must create a key-pair and append the public key to the .ssh/authorized_keys on the “to” account side. You can’t use one public key generated by one account to cover two source “from” accounts. But one “to” account can have multiple public keys in its .ssh/authorized_keys file to allow multiple “from” accounts to access that one “to” account.

  38. Saurabh Reply

    Hi Leo
    I have a problem here. I have planted the keys in the remote server but when I use the scp command in a script, it does not work.
    The login to the remote server does not create a problem but the scp itself does, in a sense that the file is not getting copied to the remote server. Do I have a hope or will have to look for alternatives.

    Thanks in advance
    Saurabh

  39. Terry Porter Reply

    Hi Leo,
    Great article. I thought I would not be able to use this, as in the secure government environment I work in using Red Hat Enterprise Linux 5.1, the sshd_config file has both RSAAuthentication and PubkeyAuthentication commented out with #’s.

    However, as I really needed this functionality from User A on Host A to User oracle on Host DB, I gave it a try and it’s working like a charm. Not sure “why” but I’m glad it’s working.

    Many thanks for a great, useful, well-written article.

  40. Srinivasan Reply

    Hi Leo,
    I have a simillar requirement.
    I would like test the functionality of sftp connectivity with keys from User A on Host A to User oracle on Host B.

    Pls confirm whether
    We can connect from HOST A USER A to HOST B Oracle
    Using passworldess authentication
    when we copy the public keys of USER A to oracle’s home directory on Host B.

    Please help.

  41. veeru Reply

    Hi, HOw can we connect using user id and password without changing the keys in sftp server.

  42. Kenny Reply

    I am trying to automate scripts using pageant for winscp. but i need to manually input the passphrase. as the file will be retrieved over the night and server reboot is done daily, is there a wasy to hardcode the passphrase in the script?

  43. sftpuser Reply

    Thanks, Leo. Just ran what you wrote in the article and works perfect. Your suggestions run smooth as usual.

  44. Tracy Reply

    I setup 2 linux servers with the information provided and it works like a charm.

    However I’m also trying to setup a Windows server using EFT server by Globalscape. Where would the authorized_keys file be created on the Windows box to allow access?

  45. vams Reply

    I followed the process but still password is prompted.

    from teh below text
    “If these tools are run under the ‘user1′ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.”

    could you please let me know the “command line options will need to point to the right place and key.” so that password is not prompted.

  46. Adenwala Reply

    How can I make this work with an ftp server where there is no unix login available? I can ftp to that server and I have login there, but want to use sftp to automate my file transfers. Thanks!

  47. Prashant Singh Reply

    Dear Friend ,
    I have followed the process given above .I have 3 servers . On one server the process worked fantastically and SSH is authenticated without password , but on one server it is again asking for password , not the passphrase , the actual password for login on server but on the other server the process given on the sit is working fantastically . what to do on the server which is not responding ?

    Best Regards
    Prashant

  48. shivkumar Reply

    how can i put time format so that only the latest file gets fetched(csv) when i make sftp to server
    thanks
    shiv

  49. Priyanka Reply

    The article is well described and very easy to follow.
    I have followed the steps mentioned above but i still get a password prompt while making a sftp connection. I believe that the problem is because the files are generated under user linus and the .ssh is under root as mentioned by you “”If these tools are run under the ‘user1′ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.”

    Could you please help me as to how to make the command line options point to the right place and key.

  50. kaladhar Reply

    I have a query regarding the keys.I generated the public and private keys with the userid X on server 1 and asked server2 team to add the publickey to the user id Z on server 2.I tried to SFTP using the private key with the user Y(which have access to the private key) from Server1 to Server2 with the user id Z.Iam not able to login ..Its asking for password by saying that permissions are too open.When i try to SFTP using the User X(keys generated on this id) from server 1 to Server 2.its logging in without asking for password.can u explain what the issue is..?

  51. Siva Reply

    Hello Leo,

    The document that you given is very superb. Its very simple and useful to follow. Thank for the document.

    I followed the same with out giving the passphrase while generating the key.Its working. But my requirement is to pass the passphrase while generating keys. When I pass it, the automated process is not working. Its asking me to pass the passphrase interactively.

    Could you please help me in this.

    Thanks for ur help in advance.

    Siva

  52. Ken Reply

    Sorry rushing too fast – should have read before posting….

    Document is fantastic but I transfer files across multiple boxes how can I set up this process so that ALL boxes are able to transfer files without password notification. Process works well with box 1 & 2 but when I put 3 in, I lose one.
    Box 1 talks to box 2. I then set up box 1 then talks to box 3 but I lose interactive connection with box 2 ?!?!
    Can you please suggest something (I have 6 boxes that interact with each other).

  53. Murali K Reply

    Hi,
    Thanks for the detailed artcile. I have one clarification though.
    “Once saved anyone in possession of the private key that matches this public key can now login as this account.”
    Can any other user, besides user1, initial a SFTP connection to server2 as user2?
    Thanks.

  54. Josephine Reply

    Thanks so much for this great article. It has helped me immensely. I can run rsync just fine without passwords from a bash script, but now I’m now trying to execute rsync as a cron task and I am getting the following errors:
    Permission denied, please try again.
    Permission denied, please try again.
    Permission denied (publickey,password).
    rsync: connection unexpectedly closed (0 bytes received so far) [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.8]
    rsync returned 12

    I’ve tried using keychain (http://www.gentoo.org/doc/en/articles/openssh-key-management-p2.xml) and get the same errors. Any ideas? Thanks, Josephine

  55. kris laks Reply

    Hi, still I’m getting the prompt for password in the remote server when I try to sftp…
    any clues please…
    it is uix to unix box…

  56. A. Tuscano Reply

    I’m using GlobeScape’s EFT as my sftp server and the remote server is using another kind of SFTP. will this code work?

  57. Swapnil Patel Reply

    What configuration required to automate SFTP script to connect windows server above details are from Unix to Unix what if the target server is windows
    please help me i m in huge trouble….

Leave a reply:

Before commenting please:

  • Read the article. Seriously. You'd be shocked at how many people make comments that prove they didn't.
  • Comment only on the article. If you have a new, unrelated question start with the search box at the top of the page.
  • Don't post personal information. Email addresses, phone numbers and such will be removed.

VERY IMPORTANT: because of a rise an comment spam that's making it through our filters any comments that do not add to the discussion - typically off topic or content-free comments - run a very high risk of being flagged as spam and removed.

If you have a new question unrelated to the article above, ask it on the Ask Leo! ask-a-question page.