I'm trying to set up an automated sftp transfer from one Linux box to another. I understand that you have to create a key with ssh-keygen, then put the key file on the other machine. But sftp still prompts me for the password. I read that the users on both machines must be the same... is that correct?
No, not correct.
As it turns out, this is something I do regularly with ssh, as well as both sftp and rsync, as part of my backup and load balancing approaches for Ask Leo! Let me walk you through what I've done.
Become a Patron of Ask Leo! and go ad-free!
SSH Configuration
To begin with, most of this relies on a the configuration of sshd, the SSH (Secure SHell) daemon running on the server you're attempting to connect to (we'll call it "server2.com"). Check the "sshd_config" on that server, typically in /etc/ssh. In some cases, these settings are not always present or set the way we need:
RSAAuthentication yes
PubkeyAuthentication yes
This enables the public/private key authentication mechanism we're about to use.
Public/Private Key Generation
We'll generate the keypair on the Linux box that you want to connect from. We'll call that "server1.com". It's that box on which you plan to run ssh, sftp or rsync.
ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
c1:21:e 3:01:26:0d:f7:ec:52:0e:0c:90:9b:6e:d8:47 user1@server1.com
What I've done with the command above is generated a public/private key pair. I responded to each prompt by hitting Return.
Note that I did NOT enter a passphrase. That's kind of important, because if you do enter a passphrase you'll need to enter it in order to use the private key. Since we're looking for an automated solution, the private key must not have a passphrase.
This is important: by not placing a passphrase on your private key, the security implication is that mere possession of the private key is sufficient to gain access to what ever resources into which you've placed the corresponding public key. Safeguard your private key.
My private key was placed in /home/user1/.ssh/id_rsa. This needs to be kept secure, because of the security implication above, but also needs to be available to the process attempting to make an ssh, sftp or rsync connection. If these tools are run under the 'user1' account, the tools will automatically look in the ".ssh" directory and I won't need to specify the private key location. Otherwise, command line options will need to point to the right place and key.
My public key is in /home/user1/.ssh/id_rsa.pub. This is the key that gets distributed to those places that want to grant you access.
Planting the public key
On the "remote" server, server2.com, pick an account - ANY account - that you want to connect as. In that account's home directory, create a ".ssh" subdirectory, and in that directory create a new text file called "authorized_keys". If it already exists, that's fine, use the existing file.
If you create the file and/or directory, I recommend that the directory be chmod 700, and the file 600. In other words, only the owner can access the directory, and the file within it.
Add to that file the contents of the id_rsa.pub file created above. That would be a *single line* that looks something like this:
ssh-rsa <lots of characters> user1@server1.com
Once saved anyone in possession of the private key that matches this public key can now login as this account.
sftp
I planted the public key in the account user2 on server2.com. So now, on my server, server1.com, logged in as user1, and where the private key is stored as described above, an sftp session looks like this:
sftp user2@server2.com
"user2" specifies the remote account on server2.com to login as.
That's it. Magic happens, and I'm authenticated. That magic? The private key is matched to the public key, which indicates you are authorized to login to that account. An sftp session is born. No interactivity required.
(IF you did enter a passphrase on the private key, you would have been prompted to enter it here. NOTE that this is the passphrase to unlock the private key, which is local. It has nothing to do with any passwords on the remote site.)
rsync
For file copy operations, rsync rocks. It does things like intelligent compression, copy only if needed, and a whole host of other operations.
So, assuming all the keys are set up as above, this rsync command copies a file from the local machine to the remote:
rsync -e ssh file user2@server2.com:/home/user2/
Local file "file" is copied to the remote /home/user2/file after logging in as "user2" using ssh as the transport (hence the "-e ssh" option), and with that, using the private/public key pair we created for authentication. Again, no interactivity required.
Rsync supports an incredibly rich set of options for recursion, compression attribute retention, date/time stamp and so on. Well worth a look see if you're copying anything of any significant volume.
SSH
Since we've gone this far, it's worth noting that SSH itself just works as well to open up a remote shell once the keys are in place. Example:
ssh user2@server2.com
and *poof* - a remote shell on server2, logged in as user2.
Hi!! I’m doing all the steps, but the sftp always asks me for the password. I don’t know what is happning, do you?
Please, help!
An
your public/private keys are probably not compatible with the ssh server. you are probably using either OpenSSH or Putty and your server is probably using an ssh.com based package. If this is the case drop putty and use openssh, create the public and private keys then run ssh-keygen -e -f key.pub > key2.pub (where key is the name of your key) this creates a new copy of the public key which can be used on an ssh.com server. Rename the key2.pub file to key.pub (which probably means you will have to delete existing key.pub) then use the private key (which has not changed) with the new public key (ie upload the new public key). Also don’t forget to change the authorisation file on the server (/home/username/.ssh or .ssh2) to accept your new key.
This article is incredible Leo. Thank You.
I have seen the sftp usage options given below,
sftp [-1vC] [-A Password] [-b batchfile] [-L logfilename] [-osshopt=value] [user@]host[:file [file]]
But in my SunOS 5.9 , I am not getting the options -L, -b, -A. What could be the reason. I wanted to get a file transferred from the remote server to my server after I connect through my automated connection script. My automation script connects and gives me the SFTP prompt. It’s not identifying the commends given in the batch file.
Thanking you once again for all the help.
There are slightly different versions of sftp on different OS’s. Does “man” work on your system? If so I’d try a “man sftp” and see if the documentation includes the features you need.
Thank you Leo.
I have checked the man pages for sftp. As you have mentioned -L, -b, -A keys are not available.
I just wanted to connect automatically to a remote system and fetch a file from that machine.
Please see the script I am having.
sftp ncc_b2b@blrsun27
get file1.txt
exit
I have follwed the steps given by you for automated connection. Once I am executing this, it’s automatically connecting to the remote system and I am getting the SFTP prompt. But it’s not executing the get command.
Can you please guide me how to do this now, as the -b option is not available. Should I go for SSLFTP or any other alternative to this.
Thanking You Leo.
Manoj.
YOu might need to have
get file1.txt
exit
in one file (commands.txt) and then use that as input for the sftp command:
sftp < commands.txt
Thank You Leo.
It’s working fine now. I have managed to Automatically connect to the remote machine and transfer the required file. All because of your help.
Once Again Thanking You Leo,
Manoj.
Hi,
I have tried all the steps but still the automation is not working with one machine.Kindly help me out in resolving this issue
Thanks,
P.Rajeswari
Rajeswari,
I have followed the steps given by Leo and it’s working fine. It’s not clear from your message what’s the actual problem. Can you please clarify, where you are facing the problem.
I have this normal ftp script that works fine, ftping files from the unix box to a windows box.
#!/usr/bin/ksh
ftp -v -n hostname ftp-gw
Type:xxx@xxx.xxx.com
Type in password when prompted
Ftp>quit
Put C:tempfilename.txt
Can you help me with this sftp script.
Hi Leo,
I am also following all the steps but SFTP is still asking me the password. I am having SunOs and Global Scape OS and I am trying to do automation from SunOS to GlobalScape. Is there any compatible problem. Please guide
I’m trying to write a script to connect to a server via sftp that has publickey+password authentication. Even when I put the password in a file and use the -b option, it still prompts me for a password. Is there any other way to pass the password to the remote server?
Well written article. More of this needed on the net. My batch job now works!!!
Hi Leo,
Nice article. I have one question though.
If I want to put a file from server1 to server2 – automatically – How do I do that?
If I do sftp user2@server2.com then it goes into interactive mode of sftp. But my problem is I want to put a file from server1 to server2 – using sftp command not from interactive prompt.
any clues on that please?
Regards
Kumar
Check out the gazillion command line options for sftp. You can bypass the interactive prompt by specifying everything on the command line.
Hi, We need to create a kshell script to FTP files to the destination server. We have to use sftp to do so.
My concern is, how do I avoid getting password prompt when using sftp command. Please help, it is very urgent.
Thanks in advance.
Regards, Minesh Shah
Hi Leo,
Your instructions to automate my sftp transfers worked great! I’ve only run into one problem. When I try to change user1 on server1 to point to user3 on server2, it starts prompting for a password again. i.e. sftp user2@server2.com works fine but sftp user3@server2.com doesn’t work automatically. I copied the public key from the .ssh folder on user2 over to the .ssh folder for user3 but that doesn’t seem to help. What am I missing here?
Regards,
James
—–BEGIN PGP SIGNED MESSAGE—–
Hash: SHA1
Nine times out of ten, the permissions on the copy of the .ssh files and the
authorized_keys file that you copied are wrong. It needs to be owned by the
account who’s .ssh directory it resides in.
Leo
—–BEGIN PGP SIGNATURE—–
Version: GnuPG v1.4.6 (MingW32)
iD8DBQFGDHLeCMEe9B/8oqERAureAJ9CmSJMmSetKZZV8UaFGH2JVXRBswCbBd4I
xuTKYBGphXOrpANg7P3CqMY=
=izE9
—–END PGP SIGNATURE—–
We have the lprng daemon running. When we put a job on a queue, it calls a shell filter script that is suppose to sftp the file to the remote server.
The problem is, the daemon tries to connect to the remote host even though we specify the user@host2 (user is qadmin and has the keys set up both sides).
ie: scp -b qadmin@host2
Hi Leo,great advice. One additional question, if the account doesn’t have a standard login shell will this automat process work?
—–BEGIN PGP SIGNED MESSAGE—–
Hash: SHA1
I don’t believe so. I vaguely recall having to enable login ability for
accounts with which I want to do this.
Leo
—–BEGIN PGP SIGNATURE—–
Version: GnuPG v1.4.6 (MingW32)
iD8DBQFGFBU3CMEe9B/8oqERAtkwAJ4t3CuIMp7Ld/5D+2CElHl7TeP+qQCfXJX4
8P8DihofejdoYIYiwuWF1rQ=
=7nkv
—–END PGP SIGNATURE—–
This was a superb article. I am going to try getting it incorporated into our IT procedures. Thank you very much.
If you want to automate sftp because for some reason scp doesn’t work in you situation you can try “expect” You need to install it. The script would look something like this:
Replace:
DIR=local and remote directories.
IP-adress=remote server address
user=user set up on remote sftp server.
#!/usr/local/bin/expect -f -d
# Expect script that automates sftp
###################################
spawn /usr/bin/sftp user@”IP-adress”
expect “user@IP-adress’s password: “
sleep 7
send Passwordr”
expect “sftp> “
send “lcd DIRr”
expect “sftp> “
send “cd DIRr”
expect “sftp> “
send “cd DIRr”
expect “sftp> “
sleep 10
send “mput *.pgpr”
sleep 10
expect “sftp> “
send “byer”
Leo, I put your steps to use and they work. Mostly.
Here’s my situation:
3 servers, I’ll call A, b and c.
A: RHEL4
b: Solaris 8 on Sparc
c: Solaris 8 on Sparc.
If I send from A to b or c, the automation works fine. If I send from b or c to A, I am challenged with a password request. b and c have A’s public RSA key. A has both b and c’s RSA public keys. In every case, I am the user who is also attempting the SCP connection. It is most expedient to auto send from b to A, but I can rewrite b’s script to run on A, if I must. Just wondering why its only working in one direction.
Leo, it turns out that on server A, the authorized_keys file had permission 664. Apparently if that file is writable by anyone other than the owner, you will be challenged for a password, regardless. Once I CHMODed it to 644, the process worked flawlessly.
Hi Leo,
I generated rsa keys for system a and b and put both keys in authorized_keys for my user on system c.
system a connects using sftp and does not prompt for a password.
system b prompts for a password – when I provide it the connection is made.
on all 3 systems all files in .ssh are writable only by the user.
what should I be looking at to trouble shoot this?
Hi Leo, Thanks for the tip, great info!
Leo, excellent article!! But it has worked for me only when i try to ssh/sftp to root account on server machine (server2 in the ex.). Any other user (making all the steps for each one) fails.
Is there any trick related to that ??
Thanks in advance.
I am having same problem as “Manoj Das”.
here is part of the post and your comment on it:
script:
sftp ncc_b2b@blrsun27
get file1.txt
exit
I have followed the steps given by you for automated connection. Once I am executing this, it’s automatically connecting to the remote system and I am getting the SFTP prompt. But it’s not executing the get command.
Can you please guide me how to do this now, as the -b option is not available.
Posted by: Manoj Das at October 3, 2006 07:36 AM
YOu might need to have
get file1.txt
exit
in one file (commands.txt) and then use that as input for the sftp command:
sftp
This is followup to my earlier post. I made it work sort of:
#!/bin/ksh
sftp -B commands.txt user1@pontoon
in commands.txt I have
get 1.dat 2.dat ( I want to get 1.dat and rename it to 2.dat)
quit
It gets 1.dat and complains can not find 2.dat on remote server.
Here is some other info about my system:
sftp -V
sftp: F-Secure SSH 3.1.0 (build 12) on sparc-sun-solaris2.8
thanks for reading my posts.
Can this be applied to Windows 2003 to Solaris?
That is automated connection from Windows 2003 to Solaris?
Thanks
Hi Leo, I tried to follow the steps you posted here but I still cannot make it work. I am trying to use SFTP to send files from HP-UX to Windows Server 2000 with a domain account setup on the Window Server. No matter what I did, I always get promoted with password. Is the public key authentication not going to work for me since this is an AD account? What’s the best way I can troubleshoot this issue? Thanks.
This article helps me greate.
I have one problem.
I can sftp without password from my account to abc@srvr1, but it asks fro password while sftp from my account to xyz@srvr3 even though I have copied the same public file in .ssh directory on both these severs.These both servers have same sshd_conf files.
Very nice document. First I want to thanks U.
1. Is there any way to write script which copy files from remote server.
Hi,
I’m currently using the -b Batch mode reads a series of commands from an input batchfile
eg: -b batchfile user@host
Right now I have to add in the switch -C for the compresion. How I can do that with the using the above code as well with the batch file contain script like – Put command to upload the file from local to remote system.
Hi Leo, I wanted to know abouty sftp, and stumbled at your site, you are a great teacher.
Accept my humble respects.
Regards,
Venkat
Hey Leo,
When I pasted the address you used before into my terminal (sftp user2@server2.com) to modify it, the CR somehow got into my copy buffer, so when I pasted it I apparently ftped into that actual address. It then locked up my system for awhile and I was unable to exit.
Do you happen to have any idea what server2.com actually is, and whether having ftped into it could have somehow compromised my system?
Thanks, Bill
I didn’t see a response to Vasant’s post of 4/3/08, so thought this might be useful. Each “from” account must create a key-pair and append the public key to the .ssh/authorized_keys on the “to” account side. You can’t use one public key generated by one account to cover two source “from” accounts. But one “to” account can have multiple public keys in its .ssh/authorized_keys file to allow multiple “from” accounts to access that one “to” account.
Hi Leo
I have a problem here. I have planted the keys in the remote server but when I use the scp command in a script, it does not work.
The login to the remote server does not create a problem but the scp itself does, in a sense that the file is not getting copied to the remote server. Do I have a hope or will have to look for alternatives.
Thanks in advance
Saurabh
Hi Leo,
Great article. I thought I would not be able to use this, as in the secure government environment I work in using Red Hat Enterprise Linux 5.1, the sshd_config file has both RSAAuthentication and PubkeyAuthentication commented out with #’s.
However, as I really needed this functionality from User A on Host A to User oracle on Host DB, I gave it a try and it’s working like a charm. Not sure “why” but I’m glad it’s working.
Many thanks for a great, useful, well-written article.
Hi Leo,
I have a simillar requirement.
I would like test the functionality of sftp connectivity with keys from User A on Host A to User oracle on Host B.
Pls confirm whether
We can connect from HOST A USER A to HOST B Oracle
Using passworldess authentication
when we copy the public keys of USER A to oracle’s home directory on Host B.
Please help.
Hi, HOw can we connect using user id and password without changing the keys in sftp server.
I am trying to automate scripts using pageant for winscp. but i need to manually input the passphrase. as the file will be retrieved over the night and server reboot is done daily, is there a wasy to hardcode the passphrase in the script?
Thanks, Leo. Just ran what you wrote in the article and works perfect. Your suggestions run smooth as usual.
I setup 2 linux servers with the information provided and it works like a charm.
However I’m also trying to setup a Windows server using EFT server by Globalscape. Where would the authorized_keys file be created on the Windows box to allow access?
Leo, your article works for me too, beautifully written
I followed the process but still password is prompted.
from teh below text
“If these tools are run under the ‘user1’ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.”
could you please let me know the “command line options will need to point to the right place and key.” so that password is not prompted.
How can I make this work with an ftp server where there is no unix login available? I can ftp to that server and I have login there, but want to use sftp to automate my file transfers. Thanks!
Dear Friend ,
I have followed the process given above .I have 3 servers . On one server the process worked fantastically and SSH is authenticated without password , but on one server it is again asking for password , not the passphrase , the actual password for login on server but on the other server the process given on the sit is working fantastically . what to do on the server which is not responding ?
Best Regards
Prashant
how can i put time format so that only the latest file gets fetched(csv) when i make sftp to server
thanks
shiv
You may also want to check out GoAnywhere from Linoma Software. They let you automate SFTP transfers with their built-in scheduler. Check it out at http://www.GoAnywhereMFT.com
The article is well described and very easy to follow.
I have followed the steps mentioned above but i still get a password prompt while making a sftp connection. I believe that the problem is because the files are generated under user linus and the .ssh is under root as mentioned by you “”If these tools are run under the ‘user1’ account, the tools will automatically look in the “.ssh” directory and I won’t need to specify the private key location. Otherwise, command line options will need to point to the right place and key.”
Could you please help me as to how to make the command line options point to the right place and key.
Thanks Leo. Worked perfectly.
I have a query regarding the keys.I generated the public and private keys with the userid X on server 1 and asked server2 team to add the publickey to the user id Z on server 2.I tried to SFTP using the private key with the user Y(which have access to the private key) from Server1 to Server2 with the user id Z.Iam not able to login ..Its asking for password by saying that permissions are too open.When i try to SFTP using the User X(keys generated on this id) from server 1 to Server 2.its logging in without asking for password.can u explain what the issue is..?
Hello Leo,
The document that you given is very superb. Its very simple and useful to follow. Thank for the document.
I followed the same with out giving the passphrase while generating the key.Its working. But my requirement is to pass the passphrase while generating keys. When I pass it, the automated process is not working. Its asking me to pass the passphrase interactively.
Could you please help me in this.
Thanks for ur help in advance.
Siva
Sorry rushing too fast – should have read before posting….
Document is fantastic but I transfer files across multiple boxes how can I set up this process so that ALL boxes are able to transfer files without password notification. Process works well with box 1 & 2 but when I put 3 in, I lose one.
Box 1 talks to box 2. I then set up box 1 then talks to box 3 but I lose interactive connection with box 2 ?!?!
Can you please suggest something (I have 6 boxes that interact with each other).
Hi,
Thanks for the detailed artcile. I have one clarification though.
“Once saved anyone in possession of the private key that matches this public key can now login as this account.”
Can any other user, besides user1, initial a SFTP connection to server2 as user2?
Thanks.
Thanks so much for this great article. It has helped me immensely. I can run rsync just fine without passwords from a bash script, but now I’m now trying to execute rsync as a cron task and I am getting the following errors:
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.8]
rsync returned 12
I’ve tried using keychain (http://www.gentoo.org/doc/en/articles/openssh-key-management-p2.xml) and get the same errors. Any ideas? Thanks, Josephine
Hi, still I’m getting the prompt for password in the remote server when I try to sftp…
any clues please…
it is uix to unix box…
Perfect!!! this is how things should be explained :)
Very nice explanation !!! :)
I’m using GlobeScape’s EFT as my sftp server and the remote server is using another kind of SFTP. will this code work?
This is an excellent post. Works flawless.
What configuration required to automate SFTP script to connect windows server above details are from Unix to Unix what if the target server is windows
please help me i m in huge trouble….
I ended up using sshpass, I could not get the above to work
sshpass -p “password” scp ………
Hi Leo,
Can we have the same steps to establish SFTP connection between LIUNX and Mainframe (MVS)? Please advise.
Thanks!
Great tutorial Leo,
it’s worth adding; if you’re doing this for automated script backups, you should definitely use the “ChrootDirectory” option in sshd_config on the destination server – this prevents anyone logging in as the user (if the key was ever compromised) – this will just allow them to do sftp sessions.
You might also want to add to sshd_config:
Match User foobar
PasswordAuthentication no
for similar reasons.
TO Kumar’s question, you have answered – “Check out the gazillion command line options for sftp. ” Can you please elaborate?
and using rsync for this is suitable ?
Hi I m trying to do sftp from unix server to remote windows server by using below unix command from unix server,
but i m getting error as mentioned below,
$ sftp usplwedwdev01
ssh: connect to host usplwedwdev01 port 22: Connection refused
Connection closed
Could you please help me on this..
That host doesn’t seem to support sftp, or it’s put it on another port, or a firewall is blocking it. You’ll need to talk to the owner of that server.
You’ve added the key in such a way that shell command can be easily run, and you can forward network ports and X-Windows. You might not want to allow this activity for a file-transfer account.
To forbid it, make these changes to your authorized keys:
command=”/usr/libexec/openssh/sftp-server”,no-agent-forwarding,no-port-forwarding,no-x11-forwarding,no-pty,no-user-rc ssh-rsa AAAA[stuff]xyz user@host.com
Note that scp will not work when you do this, but sftp will.
You can also chroot() to REALLY strap down the account, but that’s a little more complex.
Special thanks to Leo for making this perfect post.
Its look like treasure for us….. :-)
hello nice blog..
i am using amazon ec2 Linux server both have different key pair i followed the steps but not working..can you please specify the commands to rsync both servers
Hi Leo
I want to setup OPENSSH in windows 2008 server. Need to transfer files from Linux machine with password less configuration. Could you please tell me the steps to configure.
I need to sftp files from aix server to Windows server; please guide me .
Hi Leo and all.
I followed steps above. It works from Ubuntu to redhat 6.7 but not Redhat to Ubuntu.
{public key removed}
Thanks for the simple explanation! Helped a lot!
Hi leo
I have a shell script to fetch files from source server, when we are executing it through Unix box it is fine, but when i am trying it through oracle apps i am getting error like permission denied, can you pls help me.
Thanking you.
Sorry, no. I have zero Oracle experience.
Hi Leo
I just want to do a sftp to a remote server from a shell script.
As i am new to shell scripting, i do not how to do it.
Appreciate if you can send me the steps.
Thanks
Anil
Hi Leo
I just want to do a sftp to a remote server from a shell script.
As i am new to shell scripting, i do not how to do it.
Appreciate if you can send me the steps.
Thanks
Anil