How to sync files to a ftp server with the use of ftpsync on FreeBSD
In the ports of FreeBSD I found a nice tool to sync a local directory to a remote FTP server.
First install ftpsync:
cd /usr/ports/ftp/ftpsync make install clean
Next create a config file to configure ftpsync for synchronization:
localdir=/home/mariusvw/backups ftpserver=rstore.kitara.nl ftpdir=/ ftpuser=username ftppasswd=password
Next you have two options for syncing...
Get-sync from server:
ftpsync.pl -g cfg=financenetwerk_eu-v1.cfg
Put-sync to server:
ftpsync.pl -n cfg=financenetwerk_eu-v1.cfg
You can add the -p flag to not-prune directories on the target.
I don't use this program anymore for syncing but I can tell you, it works great if you want to easy upload a website to a live server.
How to backup from a Western Digital Sharespace to another Sharespace via rsync and FreeBSD
This script simply syncs from a WD ShareSpace drive to another WD ShareSpace drive. It gets started from a other server which is in my case FreeBSD.
You might need this: How to enable SSH on a Western Digital ShareSpace
Create the following files on FreeBSD:
/backup/sharespace/ssh-wrapper
#!/bin/sh sh /backup/run &
/backup/sharespace/run
#!/bin/sh # WD ShareSpace sync script v1.1 # By Marius van Witzenburg <info@kitara.nl> # http://kitara.nl # # Works with busybox 1.1.1 and newer versions. # # Settings source="192.168.8.4" target="192.168.8.9" mailto="info@kitara.nl" mailfrom="noreply@kitara.nl" daemon="yes" force="no" waittime=5 maxlogs=48 beforetime=6 aftertime=20 # # DO NOT CHANGE BELOW! # PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin PID=$$ rarg="-aWvz" curdir=`pwd` cd /backup/ echo ${PID} > /backup/run.pid cleanup() { rm /backup/key cd ${curdir} ssh -ax root@${target} "test -f /backup/rsyncd.pid && kill `cat /backup/rsyncd.pid`" return $? } s_hup() { logger -s -p daemon.notice -t 'backup' "*** SIGHUP! Ignoring... ***" #exit $? } s_int() { logger -s -p daemon.notice -t 'backup' "*** SIGINT! Exiting... ***" cleanup echo -e "From: ${mailfrom}nTo: ${mailto}nSubject: ShareSpace backup.nnSIGINT! Exiting..." | /usr/sbin/msmtp ${mailto} rm /backup/run.pid kill ${JOB} exit $? } s_term() { logger -s -p daemon.notice -t 'backup' "*** SIGTERM! Exiting... ***" cleanup echo -e "From: ${mailfrom}nTo: ${mailto}nSubject: ShareSpace backup.nnSIGTERM! Exiting..." | /usr/sbin/msmtp ${mailto} rm /backup/run.pid kill ${JOB} exit $? } trap s_hup SIGHUP trap s_int SIGINT trap s_term SIGTERM if [ "${daemon}" = "yes" ] then logger -s -p daemon.notice -t 'backup' "Having a rest for the first run..." sleep 30 & JOB=$! wait ${JOB} fi while [ "true" ] do # Time to backup? hour=`date "+%H"` if [ "${hour}" -lt "${beforetime}" ] || [ "$hour" -gt "${aftertime}" ] then # Change the 1 to 1-3 depending on which load you want loadavg=`uptime | sed 's/.*average: ([0-9]).*/1/g'` if [ "${loadavg}" -eq 0 ] || [ "${force}" = "yes" ] then alive=`ssh -ax -q -q -o "BatchMode=yes" -o "ConnectTimeout 5" root@${target} "echo 2>&1" && return 0 || echo 1` if [ "${alive}" -eq 0 ] then user=`echo $(head /dev/urandom)$(date +%Y%m%d%H%M%S%N) | md5sum | uuencode -m -| sed -n 2p | cut -c3-43 | cut -c1-${1:-16}` sleep 1 pass=`echo $(head /dev/urandom)$(date +%Y%m%d%H%M%S%N) | md5sum | uuencode -m -| sed -n 2p | cut -c3-43 | cut -c1-${1:-16}` echo ${pass} > /backup/key test -d /backup/log || mkdir /backup/log test -d /backup/server || mkdir /backup/server test -f /backup/log/run.${maxlogs} && rm -f /backup/log/run.${maxlogs} lcur=${maxlogs} while [ ${lcur} -gt 1 ] do lpre=${lcur} lcur=`expr ${lcur} - 1` test -f /backup/log/run.${lcur}.gz && mv /backup/log/run.${lcur}.gz /backup/log/run.${lpre}.gz done test -f /backup/log/run && mv /backup/log/run /backup/log/run.1 test -f /backup/log/run.1 && gzip /backup/log/run.1 touch /backup/log/run logger -s -p daemon.notice -t 'backup' "Creating rsync server configuration" cat > /backup/server/rsyncd.conf < <EOF pid file = /backup/rsyncd.pid [shares] path = /shares/ comment = Rsync Server uid = root gid = jewab use chroot = no read only = no list = no auth users = ${user} secrets file = /backup/rsync.users EOF cat > /backup/server/rsync.users < <EOF ${user}:${pass} EOF cat > /backup/server/start < <EOF #!/bin/sh rsync --daemon --config=/backup/rsyncd.conf & EOF # Transfer server files rsync -e 'ssh -ax' ${rarg} --timeout=30 --delete /backup/server/ ${target}:/backup/ >> /backup/log/run 2>&1 & JOB=$! wait ${JOB} # Start rsync server ssh -ax root@${target} "sh /backup/start > /dev/null 2>&1; exit" JOB=$! wait ${JOB} # Sync internal shares and exclude usb shares logger -s -p daemon.notice -t 'backup' "Syncing internal shares..." rsync ${rarg} --timeout=60 --password-file=/backup/key --exclude="/usb[1-3]-1share1" --delete /shares/ rsync://${user}@${target}/shares/ >> /backup/log/run 2>&1 & JOB=$! wait ${JOB} tail -2 /backup/log/run | logger -s -p daemon.info -t 'backup' # Sync usb shares if connected for x in `ssh -ax root@${target} ls -l /shares/| grep "^d" | awk '{ print $9 }'` do if [ ! -z "`expr ${x} : '(usb[0-9]-[0-9]share[0-9])'`" ] then if [ -d "/shares/${x}/" ] then logger -s -p daemon.notice -t 'backup' "Syncing ${x}..." rsync ${rarg} --timeout=60 --password-file=/backup/key --delete /shares/${x}/ rsync://${user}@${target}/shares/${x}/ >> /backup/log/run 2>&1 & JOB=$! wait ${JOB} tail -2 /backup/log/run | logger -s -p daemon.info -t 'backup' fi fi done sleep 5 echo -e "From: ${mailfrom}nTo: ${mailto}nSubject: ShareSpace backup.nnCycle finished..." | /usr/sbin/msmtp ${mailto} # Shutdown rsync server cleanup else logger -s -p daemon.notice -t 'backup' "No access to '${target}' - aborting" fi else logger -s -p daemon.notice -t 'backup' "Load average on '${source}' is ${loadavg} - aborting" fi else logger -s -p daemon.notice -t 'backup' "Not the time to make backups - sleeping" fi # Daemonize or stop? if [ "${daemon}" != "yes" ] then logger -s -p daemon.notice -t 'backup' "Backup finished" break fi # Cycle logger -s -p daemon.notice -t 'backup' "Waiting for cycle (${waittime} minutes)..." sleep `expr ${waittime} * 60` & JOB=$! wait ${JOB} logger -s -p daemon.notice -t 'backup' "Running cycle..." done # EOF
/backup/sync.sh
#!/usr/local/bin/bash # # Sync backup system on ShareSpace drive # ip="192.168.8.4" # Fix permissions find /backup/sharespace/ -type d -exec chmod 700 '{}' ; find /backup/sharespace/ -type f -exec chmod 600 '{}' ; echo "-- Shutting down current backup" ssh -ax root@${ip} "test -f /backup/run.pid && kill `cat /backup/run.pid`" sleep 15 echo "-- Syncing new files" rsync -e 'ssh -ax' -aWv --timeout=30 --exclude '.svn' --delete --delete-excluded /backup/sharespace/ root@${ip}:/backup/ echo "-- Restarting backup" ssh -ax root@${ip} "sh /backup/ssh-wrapper > /dev/null 2>&1; exit" # EOF
After creating these files set these permissions and start the script:
chmod 700 /backup/sync.sh
/backup/sync.sh
Keep in mind that you setup SSH-keys before you use these scripts
Have fun!
How to backup from a Western Digital Sharespace to another Sharespace (v2)
I earlier wrote a script to sync two Western Digital ShareSpace drives. Since that was written in a hurry I thought lets have another good look at it...
I changed the checks for USB drives and sync the complete the shares directory instead of only the shares.
The following script is a new rewritten version of the old script which works faster for my use.
You may modify it to suit your situation but please leave the credits in the file
You might need this: How to enable SSH on a Western Digital ShareSpace
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 | #!/bin/sh # WD ShareSpace sync script v1.1 # By Marius van Witzenburg <info@kitara.nl> # http://kitara.nl # # Works with busybox 1.1.1 and newer versions. # # Settings src_ip="172.20.20.22" mailto="marius@kitara.nl" mailfrom="noreply@kitara.nl" daemon="yes" force="no" waittime=5 maxlogs=48 # # DO NOT CHANGE BELOW! # PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin PID=$$ pwd=`pwd` root=`dirname $0` logdir=${root}/logs logfile=`basename $0` cd ${root} cleanup() { #rm -f ${log} return $? } s_hup() { logger -s -p daemon.notice -t ${logfile} "*** SIGHUP! Exiting... ***" cleanup #exit $? } s_int() { logger -s -p daemon.notice -t ${logfile} "*** SIGINT! Exiting... ***" echo -e "From: ${mailfrom}nTo: ${mailto}nSubject: ShareSpace backup.nnSIGINT! Exiting..." | /usr/sbin/msmtp ${mailto} kill ${JOB} exit $? } s_term() { logger -s -p daemon.notice -t ${logfile} "*** SIGTERM! Exiting... ***" echo -e "From: ${mailfrom}nTo: ${mailto}nSubject: ShareSpace backup.nnSIGTERM! Exiting..." | /usr/sbin/msmtp ${mailto} kill ${JOB} exit $? } trap s_hup SIGHUP trap s_int SIGINT trap s_term SIGTERM if [ "${daemon}" = "yes" ] then logger -s -p daemon.notice -t ${logfile} "Having a rest for the first run..." sleep 30 & JOB=$! wait ${JOB} fi # Loop and loop and loop... while [ "true" ] do # Check if ssh host is up and running. alive=`ssh -ax -c blowfish -q -q -o "BatchMode=yes" -o "ConnectTimeout 5" root@${src_ip} "echo 2>&1" && return 0 || echo 1` if [ "${alive}" -eq 0 ] then # Change the 1 to 1-3 depending on which load you want #loadavg=`ssh -ax -c blowfish root@${src_ip} uptime | sed -e "s/.*load average: (.*...), (.*...), (.*...)/1/" -e "s/ //g"` loadavg=`ssh -ax -c blowfish root@${src_ip} uptime | sed 's/.*average: ([0-9]).*/1/g'` if [ "${loadavg}" -eq 0 ] || [ "${force}" = "yes" ] then # Be sure we have a log dir test -d ${logdir} || mkdir ${logdir} # Shift logs test -f ${logdir}/${logfile}.${maxlogs} && rm -f ${logdir}/${logfile}.${maxlogs} lcur=${maxlogs} while [ ${lcur} -gt 1 ] do lpre=${lcur} lcur=`expr ${lcur} - 1` test -f ${logdir}/${logfile}.${lcur}.gz && mv ${logdir}/${logfile}.${lcur}.gz ${logdir}/${logfile}.${lpre}.gz done test -f ${logdir}/${logfile} && mv ${logdir}/${logfile} ${logdir}/${logfile}.1 test -f ${logdir}/${logfile}.1 && gzip ${logdir}/${logfile}.1 logger -s -p daemon.notice -t ${logfile} "Starting backup from '${src_ip}'" # Sync internal shares and exclude usb shares logger -s -p daemon.notice -t ${logfile} "Syncing internal shares..." rsync -e 'ssh -ax -c blowfish -l root' -aWvz --timeout=30 --exclude="/usb[1-3]-1share1" --delete --stats ${src_ip}:/shares/ /shares/ > ${logdir}/${logfile} 2>&1 & JOB=$! wait ${JOB} # Sync usb shares if connected for x in `ssh -ax -c blowfish root@${src_ip} ls -l /shares/| grep "^d" | awk '{ print $9 }'` do if [ ! -z "`expr ${x} : '(usb[0-9]-[0-9]share[0-9])'`" ] then if [ -d "/shares/${x}/" ] then logger -s -p daemon.notice -t ${logfile} "Syncing ${x}..." rsync -e 'ssh -ax -c blowfish -l root' -aWvz --timeout=30 --delete --stats ${src_ip}:/shares/${x}/ /shares/${x}/ >> ${logdir}/${logfile} 2>&1 & JOB=$! wait ${JOB} fi fi done tail -2 ${logdir}/${logfile} | logger -s -p daemon.info -t ${logfile} else logger -s -p daemon.notice -t ${logfile} "Load average on '${src_ip}' is ${loadavg} - aborting" fi else logger -s -p daemon.notice -t ${logfile} "No access to '${src_ip}' - aborting" fi if [ "${daemon}" != "yes" ] then logger -s -p daemon.notice -t ${logfile} "Backup finished" break fi logger -s -p daemon.notice -t ${logfile} "Waiting for cycle (${waittime} minutes)..." sleep `expr ${waittime} * 60` & JOB=$! wait ${JOB} logger -s -p daemon.notice -t ${logfile} "Running cycle..." done # Go back to old location cd ${pwd} # EOF |
How to backup from a Western Digital Sharespace to another Sharespace
There is a newer version of this script available here.
You might need this: How to enable SSH on a Western Digital ShareSpace
For a script running from a FreeBSD server check this script.
~~~
Lately I've been busy with making a backup script to automate the backup of a WD Sharespace drive.
This script has been made to backup between two Sharespace drives.
The script works quite simple. In a endless loop it keeps on running when you start it to the background with: ./backup.sh &
It scans for the shares on your master drive and it will Rsync it to your slave drive on the share backup.
Put this script on your slave drive in /root/backup.sh
If you decide to modify this script please leave my name in the script.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | #!/bin/sh time=5 src_ip="192.168.3.3" log="/root/backup.log" # Written by Marius van Witzenburg # http://kitara.nl PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin PID=$$ cleanup() { rm -f ${log} return $? } s_hup() { echo -e "n*** SIGHUP! Ignoring ***n" cleanup #exit $? } s_int() { echo -en "n*** SIGINT! Exiting ***n" kill ${JOB} exit $? } s_term() { echo -en "n*** SIGTERM! Exiting ***n" kill ${JOB} exit $? } trap s_hup SIGHUP trap s_int SIGINT trap s_term SIGTERM echo "Having a rest for the first run..." > ${log} sleep 30 & JOB=$! wait ${JOB} echo "Running backup..." >> ${log} while [ "true" ] do for x in `ssh ${src_ip} ls -l /shares/| grep "^d" | awk '{ print $9 }'` do if [ ! -z ${x} ] then output='' if [ ! -z "`expr ${x} : '(usb[0-9]-[0-9]share[0-9])'`" ] then if [ -d "/shares/${x}" ] then echo -e "USB Syncing '/shares/${x}' to '/shares/${x}'n" >> $log rsync -qau --no-t --checksum --delete ${src_ip}:/shares/${x}/ /shares/${x}/ & 2>> ${log} 1> /dev/null JOB=$! wait ${JOB} echo -e "n${JOB} -- finished.nn" >> ${log} else echo -e "Skipping USB for '/shares/${x}'n" >> ${log} fi else echo -e "Syncing '/shares/${x}' to '/shares/backup/${x}'n" >> $log rsync -qau --no-t --checksum --delete ${src_ip}:/shares/${x}/ /shares/backup/${x}/ & 2>> ${log} 1> /dev/null JOB=$! wait ${JOB} echo -e "n${JOB} -- finished.nn" >> ${log} fi fi done message=`cat ${log}` echo -e "From: noreply@kitara.nlnTo: info@kitara.nlnSubject: ShareSpace backup output.nn${message}" | /usr/sbin/msmtp info@kitara.nl echo -e "Waiting for cycle (${time} minutes)...n" >> ${log} sleep `expr ${time} * 60` & JOB=$! wait ${JOB} echo "Running cycle..." > ${log} done # EOF |
Comments and donations are more than welcome




