Forum Moderators: phranque

Message Too Old, No Replies

Uploading large volumes of files

         

hawkerz

3:50 pm on Jun 19, 2007 (gmt 0)

10+ Year Member



I use WS FTP Pro for all of my ftp needs. At the moment, I am trying to upload 500 megs worth (57,000 files) to my hosting account, and WS FTP seems to have trouble handling the volume. Any suggestions? It keeps crashing on me

physics

5:34 pm on Jun 19, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I use rsync for that sort of thing. This is a linux tool so you'll need to install cygwin [cygwin.com] to run it on Windows though. Wait, is your hosting account Windows or Linux?

hawkerz

6:50 pm on Jun 19, 2007 (gmt 0)

10+ Year Member



hosting account is linux

sun818

9:17 pm on Jun 19, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If you have shell access to your Linux account, zip or tar/gzip your files and uncompress then on the server. If you have to do this repetitively, you can set up a cron job to run a script that uncompress the uploaded file for you.

physics

7:05 am on Jun 21, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



OK, if it is linux just check that rsync is installed on the remote machine.
ssh in and then do

which rsync

Then, from your local machine (after you've installed cygwin) do

cd /cydrive/c/path/to/your/dirthathasthestuffyouwanttoupload


rsync -ave ssh * user@your.server.com:./pathrelativetoyourhomedir/

If it gets interrupted just run this again. Rsync will only try to upload files that were not uploaded previously.

vincevincevince

7:19 am on Jun 21, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you use linux on your home machine, and only have FTP access to the server, then use ncftpput from the command line.