Duply-code
From ftplicity
Contents |
duply sourcecode
duply's svn repository is not the most recent source for the code. Usually the svn only gets updated on releases. But not always. Development snapshots are taken daily. These are not guaranteed to work, but might contain unreleased fixes. Check the changelog accordingly.
SVN access
Browse the repository
http://ftplicity.svn.sourceforge.net/viewvc/ftplicity/duply/trunk/
SVN access
svn co https://ftplicity.svn.sourceforge.net/svnroot/ftplicity/duply/trunk duply
SVN Log
http://ftplicity.svn.sourceforge.net/viewvc/ftplicity/duply/?view=log
Latest Development Snapshot
mod time 2013-04-29
plain/text -> http://duply.net/tmp/duply.sh
1: #!/usr/bin/env bash
2: #
3: ###############################################################################
4: # duply (grown out of ftplicity), is a shell front end to duplicity that #
5: # simplifies the usage by managing settings for backup jobs in profiles. #
6: # It supports executing multiple commands in a batch mode to enable single #
7: # line cron entries and executes pre/post backup scripts. #
8: # Since version 1.5.0 all duplicity backends are supported. Hence the name #
9: # changed from ftplicity to duply. #
10: # See http://duply.net or http://ftplicity.sourceforge.net/ for more info. #
11: # (c) 2006 Christiane Ruetten, Heise Zeitschriften Verlag, Germany #
12: # (c) 2008-2012 Edgar Soldin (changes since version 1.3) #
13: ###############################################################################
14: # LICENSE: #
15: # This program is licensed under GPLv2. #
16: # Please read the accompanying license information in gpl.txt. #
17: ###############################################################################
18: # TODO/IDEAS/KNOWN PROBLEMS:
19: # - possibility to restore time frames (incl. deleted files)
20: # realizable by listing each backup and restore from
21: # oldest to the newest, problem: not performant
22: # - search file in all backups function and show available
23: # versions with backups date (list old avail since 0.6.06)
24: # - edit profile opens conf file in vi
25: # - implement log-fd interpretation
26: # - add a duplicity option check against the options pending
27: # deprecation since 0.5.10 namely --time-separator
28: # --short-filenames
29: # --old-filenames
30: # - add 'exclude_<command>' list usage eg. exclude_verify
31: # - a download/install duplicity option
32: # - bug: on key import it tries to import again and fails because
33: # of already existing key, probably caused by legacy gpgkey
34: # - hint on install software if a piece is missing
35: # - featreq 2995409: Prevent concurrent runs for same profile
36: # - featreq 3042778: check success of commands and react in batches
37: # e.g. backup_AND_verify_AND_purge, pre_and_bkp_and_post
38: # - import/export profile from/to .tgz function !!!
39: #
40: #
41: # CHANGELOG:
42: # 1.5.11 ()
43: # - purge-incr command for remove-all-inc-of-but-n-full feature added
44: # patch provided by Moritz Augsburger, thanks!
45: #
46: # 1.5.10 (26.03.2013)
47: # - minor indent and documentation fixes
48: # - bugfix: exclude filter failed on ubuntu, mawk w/o posix char class support
49: # - bugfix: fix url_decoding generally and for python3
50: # - bugfix 3609075: wrong script results in status line (thx David Epping)
51: #
52: # 1.5.9 (22.11.2012)
53: # - bugfix 3588926: filter --exclude* params for restore/fetch ate too much
54: # - restore/fetch now also ignores --include* or --exclude='foobar'
55: #
56: # 1.5.8 (26.10.2012)
57: # - bugfix 3575487: implement proper cloud files support
58: #
59: # 1.5.7 (10.06.2012)
60: # - bugfix 3531450: Cannot use space in target URL (file:///) anymore
61: #
62: # 1.5.6 (24.5.2012)
63: # - commands purge, purge-full have no default value anymore for security
64: # reasons; instead max value can be given via cmd line or must be set
65: # in profile; else an error is shown.
66: # - minor man page modifications
67: #
68: # versioning scheme will be simplified to [major].[minor].[patch] version
69: # with the next version raise
70: #
71: # 1.5.5.5 (4.2.2012)
72: # - bugfix 3479605: SEL context confused profile folder's permission check
73: # - colon ':' in url passphrase got ignored, added python driven url_decoding
74: # for user & pass to better process special chars
75: #
76: # 1.5.5.4 (16.10.2011)
77: # - bugfix 3421268: SFTP passwords from conf ignored and always prompted for
78: # - add support for separate sign passphrase (needs duplicity 0.6.14+)
79: #
80: # 1.5.5.3 (1.10.2011)
81: # - bugfix 3416690: preview threw echo1 error
82: # - fix unknown cmds error usage & friends if more than 2 params were given
83: #
84: # 1.5.5.2 (23.9.2011)
85: # - bugfix 3409643: ssh key auth did ask for passphrase (--ssh-askpass ?)
86: # - bugfix: mawk does not support \W and did not split multikey definitions
87: # - all parameters should survive single (') and double (") quotes now
88: #
89: # 1.5.5.1 (7.6.2011)
90: # - featreq 3311881: add ftps as supported by duplicity 0.6.13 (thx mape2k)
91: # - bugfix 3312208: signing detection broke symmetric gpg test routine
92: #
93: # 1.5.5 (2.5.2011)
94: # - bugfix: fetch problem with space char in path, escape all params
95: # containing non word chars
96: # - list available profiles, if given profile cannot be found
97: # - added --use-agent configuration hint
98: # - bugfix 3174133: --exclude* params in conf DUPL_PARAMS broke
99: # fetch/restore
100: # - version command now prints out 'using installed' info
101: # - featreq 3166169: autotrust imported keys, based on code submitted by
102: # Martin Ellis - imported keys are now automagically trusted ultimately
103: # - new txt2man feature to create manpages for package maintainers
104: #
105: # 1.5.4.2 (6.1.2011)
106: # - new command changelog
107: # - bugfix 3109884: freebsd awk segfaulted on printf '%*', use print again
108: # - bugfix: freebsd awk hangs on 'awk -W version'
109: # - bugfix 3150244: mawk does not know '--version'
110: # - minor help text improvements
111: # - new env vars CMD_PREV,CMD_NEXT replacing CMD env var for scripts
112: #
113: # 1.5.4.1 (4.12.2010)
114: # - output awk, python, bash version now in prolog
115: # - shebang uses /usr/bin/env now for freebsd compatibility, bash not in /bin/bash
116: # - new --disable-encryption parameter, to override profile encr settings for one run
117: # - added exclude-if-present setting to conf template
118: # - bug 3126972: GPG_PW only needed for signing/symmetric encryption (even though duplicity still needs it)
119: #
120: # 1.5.4 (15.11.2010)
121: # - as of 1.5.3 already, new ARCH_DIR config option
122: # - multiple key support
123: # - ftplicity-Feature Requests-2994929: separate encryption and signing key
124: # - key signing of symmetric encryption possible (duplicity patch committed)
125: # - gpg tests disable switch
126: # - gpg tests now previewable and more intelligent
127: #
128: # 1.5.3 (1.11.2010)
129: # - bugfix 3056628: improve busybox compatibility, grep did not have -m param
130: # - bugfix 2995408: allow empty password for PGP key
131: # - bugfix 2996459: Duply erroneously escapes '-' symbol in username
132: # - url_encode function is now pythonized
133: # - rsync uses FTP_PASSWORD now if duplicity 0.6.10+ , else issue warning
134: # - feature 3059262: Make pre and post aware of parameters,
135: # internal parameters + CMD of pre or post
136: #
137: # 1.5.2.3 (16.4.2010)
138: # - bugfix: date again, should now work virtually anywhere
139: #
140: # 1.5.2.2 (3.4.2010)
141: # - minor bugfix: duplicity 0.6.8b version string now parsable
142: # - added INSTALL.txt
143: #
144: # 1.5.2.1 (23.3.2010)
145: # - bugfix: date formatting is awked now and should work on all platforms
146: #
147: # 1.5.2 (2.3.2010)
148: # - bugfix: errors print to STD_ERR now, failed tasks print an error message
149: # - added --name=duply_<profile> for duplicity 0.6.01+ to name cache folder
150: # - simplified & cleaned profileless commands, removed second instance
151: # - generalized separator time routines
152: # - added support for --no-encryption (GPG_KEY='disabled'), see conf examples
153: # - minor fixes
154: #
155: # 1.5.1.5 (5.2.2010)
156: # - bugfix: added special handling of credentials for rsync, imap(s)
157: #
158: # 1.5.1.4 (7.1.2010)
159: # - bugfix: nsecs defaults now to zeroes if date does not deliver [0-9]{9}
160: # - check if ncftp binary is available if url protocol is ftp
161: # - bugfix: duplicity output is now printed to screen directly to resolve
162: # 'mem alloc problem' bug report
163: # - bugfix: passwords will not be in the url anymore to solve the 'duply shows
164: # sensitive data in process listing' bug report
165: #
166: # 1.5.1.3 (24.12.2009) 'merry xmas'
167: # - bugfix: gpg pass now apostrophed to allow space and friends
168: # - bugfix: credentials are now url encoded to allow special chars in them
169: # a note about url encoding has been added to the conf template
170: #
171: # 1.5.1.2 (1.11.2009)
172: # - bugfix: open parenthesis in password broke duplicity execution
173: # - bugfix: ssh/scp backend does not always need credentials e.g. key auth
174: #
175: # 1.5.1.1 (21.09.2009)
176: # - bugfix: fixed s3[+http] TARGET_PASS not needed routine
177: # - bugfix: TYPO in duply 1.5.1 prohibited the use of /etc/duply
178: # see https://sourceforge.net/tracker/index.php?func=detail&
179: # aid=2864410&group_id=217745&atid=1041147
180: #
181: # 1.5.1 (21.09.2009) - duply (fka. ftplicity)
182: # - first things first: ftplicity (being able to support all backends since
183: # some time) will be called duply (fka. ftplicity) from now on. The addendum
184: # is for the time being to circumvent confusion.
185: # - bugfix: exit code is 1 (error) not 0 (success), if at least on duplicity
186: # command failed
187: # - s3[+http] now supported natively by translating user/pass to access_key/
188: # secret_key environment variables needed by duplicity s3 boto backend
189: # - bugfix: additional output lines do not confuse version check anymore
190: # - list command supports now age parameter (patch by stefan on feature
191: # request tracker)
192: # - bugfix: option/param pairs are now correctly passed on to duplicity
193: # - bugfix: s3[+http] needs no TARGET_PASS if command is read only
194: #
195: # 1.5.0.2 (31.07.1009)
196: # - bugfix: insert password in target url didn't work with debian mawk
197: # related to previous bug report
198: #
199: # 1.5.0.1 (23.07.2009)
200: # - bugfix: gawk gensub dependency raised an error on debian's default mawk
201: # replaced with match/substr command combination (bug report)
202: # https://sf.net/tracker/?func=detail&atid=1041147&aid=2825388&
203: # group_id=217745
204: #
205: # 1.5.0 (01.07.2009)
206: # - removed ftp limitation, all duplicity backends should work now
207: # - bugfix: date for separator failed on openwrt busybox date, added a
208: # detecting workaround, milliseconds are not available w/ busybox date
209: #
210: # 1.4.2.1 (14.05.2009)
211: # - bugfix: free temp space detection failed with lvm, fixed awk parse routine
212: #
213: # 1.4.2 (22.04.2009)
214: # - gpg keys are now exported as gpgkey.[id].asc , the suffix reflects the
215: # armored ascii nature, the id helps if the key is switched for some reason
216: # im/export routines are updated accordingly (import is backward compatible
217: # to the old profile/gpgkey files)
218: # - profile argument is treated as path if it contains slashes
219: # (for details see usage)
220: # - non-ftplicity options (all but --preview currently) are now passed
221: # on to duplicity
222: # - removed need for stat in secure_conf, it is ls based now
223: # - added profile folder readable check
224: # - added gpg version & home info output
225: # - awk utility availability is now checked, because it was mandatory already
226: # - tmp space is now checked on writability and space requirement
227: # test fails on less than 25MB or configured $VOLSIZE,
228: # test warns if there is less than two times $VOLSIZE because
229: # that's required for --asynchronous-upload option
230: # - gpg functionality is tested now before executing duplicity
231: # test drive contains encryption, decryption, comparison, cleanup
232: # this is meant to detect non trusted or other gpg errors early
233: # - added possibility of doing symmetric encryption with duplicity
234: # set GPG_KEY="" or simply comment it out
235: # - added hints in config template on the depreciation of
236: # --short-filenames, --time-separator duplicity options
237: #
238: # new versioning scheme 1.4.2b => 1.4.2,
239: # beta b's are replaced by a patch count number e.g. 1.4.2.1 will be assigned
240: # to the first bug fixing version and 1.4.2.2 to the second and so on
241: # also the releases will now have a release date formatted (Day.Month.Year)
242: #
243: # 1.4.1b1 - bugfix: ftplicity changed filesystem permission of a folder
244: # named exactly as the profile if existing in executing dir
245: # - improved plausibility checking of config and profile folder
246: # - secure_conf only acts if needed and prints a warning now
247: #
248: # 1.4.1b - introduce status (duplicity collection-status) command
249: # - pre/post script output printed always now, not only on errors
250: # - new config parameter GPG_OPTS to pass gpg options
251: # added examples & comments to profile template conf
252: # - reworked separator times, added duration display
253: # - added --preview switch, to preview generated command lines
254: # - disabled MAX_AGE, MAX_FULL_BACKUPS, VERBOSITY in generated
255: # profiles because they have reasonable defaults now if not set
256: #
257: # 1.4.0b1 - bugfix: incr forces incremental backups on duplicity,
258: # therefore backup translates to pre_bkp_post now
259: # - bugfix: new command bkp, which represents duplicity's
260: # default action (incr or full if full_if_older matches
261: # or no earlier backup chain is found)
262: #
263: # new versioning scheme 1.4 => 1.4.0, added new minor revision number
264: # this is meant to slow down the rapid version growing but still keep
265: # versions cleanly separated.
266: # only additional features will raise the new minor revision number.
267: # all releases start as beta, each bugfix release will raise the beta
268: # count, usually new features arrive before a version 'ripes' to stable
269: #
270: # 1.4.0b
271: # 1.4b - added startup info on version, time, selected profile
272: # - added time output to separation lines
273: # - introduced: command purge-full implements duplicity's
274: # remove-all-but-n-full functionality (patch by unknown),
275: # uses config variable $MAX_FULL_BACKUPS (default = 1)
276: # - purge config var $MAX_AGE defaults to 1M (month) now
277: # - command full does not execute pre/post anymore
278: # use batch command pre_full_post if needed
279: # - introduced batch mode cmd1_cmd2_etc
280: # (in turn removed the bvp command)
281: # - unknown/undefined command issues a warning/error now
282: # - bugfix: version check works with 0.4.2 and older now
283: # 1.3b3 - introduced pre/post commands to execute/debug scripts
284: # - introduced bvp (backup, verify, purge)
285: # - bugfix: removed need for awk gensub, now mawk compatible
286: # 1.3b2 - removed pre/post need executable bit set
287: # - profiles now under ~/.ftplicity as folders
288: # - root can keep profiles in /etc/ftplicity, folder must be
289: # created by hand, existing profiles must be moved there
290: # - removed ftplicity in path requirement
291: # - bugfix: bash < v.3 did not know '=~'
292: # - bugfix: purge works again
293: # 1.3 - introduces multiple profiles support
294: # - modified some script errors/docs
295: # - reordered gpg key check import routine
296: # - added 'gpg key id not set' check
297: # - added error_gpg (adds how to setup gpg key howto)
298: # - bugfix: duplicity 0.4.4RC4+ parameter syntax changed
299: # - duplicity_version_check routine introduced
300: # - added time separator, shortnames, volsize, full_if_older
301: # duplicity options to config file (inspired by stevie
302: # from http://weareroot.de)
303: # 1.1.1 - bugfix: encryption reactivated
304: # 1.1 - introduced config directory
305: # 1.0 - first release
306: ###############################################################################
307:
308:
309: # important definitions #######################################################
310:
311: ME_LONG="$0"
312: ME="$(basename $0)"
313: ME_NAME="${ME%%.*}"
314: ME_VERSION="1.5.10"
315: ME_WEBSITE="http://duply.net"
316:
317: # default config values
318: DEFAULT_SOURCE='/path/of/source'
319: DEFAULT_TARGET='scheme://user[:password]@host[:port]/[/]path'
320: DEFAULT_TARGET_USER='_backend_username_'
321: DEFAULT_TARGET_PASS='_backend_password_'
322: DEFAULT_GPG_KEY='_KEY_ID_'
323: DEFAULT_GPG_PW='_GPG_PASSWORD_'
324:
325: # function definitions ##########################
326: function set_config { # sets config vars
327: local CONFHOME_COMPAT="$HOME/.ftplicity"
328: local CONFHOME="$HOME/.duply"
329: local CONFHOME_ETC_COMPAT="/etc/ftplicity"
330: local CONFHOME_ETC="/etc/duply"
331:
332: # confdir can be delivered as path (must contain /)
333: if [ `echo $FTPLCFG | grep /` ] ; then
334: CONFDIR=$(readlink -f $FTPLCFG 2>/dev/null || \
335: ( echo $FTPLCFG|grep -v '^/' 1>/dev/null 2>&1 \
336: && echo $(pwd)/${FTPLCFG} ) || \
337: echo ${FTPLCFG})
338: # or DEFAULT in home/.duply folder (NEW)
339: elif [ -d "${CONFHOME}" ]; then
340: CONFDIR="${CONFHOME}/${FTPLCFG}"
341: # or in home/.ftplicity folder (OLD)
342: elif [ -d "${CONFHOME_COMPAT}" ]; then
343: CONFDIR="${CONFHOME_COMPAT}/${FTPLCFG}"
344: warning_oldhome "${CONFHOME_COMPAT}" "${CONFHOME}"
345: # root can put profiles under /etc/duply (NEW) if path exists
346: elif [ -d "${CONFHOME_ETC}" ] && [ "$EUID" -eq 0 ]; then
347: CONFDIR="${CONFHOME_ETC}/${FTPLCFG}"
348: # root can keep profiles under /etc/ftplicity (OLD) if path exists
349: elif [ -d "${CONFHOME_ETC_COMPAT}" ] && [ "$EUID" -eq 0 ]; then
350: CONFDIR="${CONFHOME_ETC_COMPAT}/${FTPLCFG}"
351: warning_oldhome "${CONFHOME_ETC_COMPAT}" "${CONFHOME_ETC}"
352: # hmm no profile folder there, then use default for error later
353: else
354: CONFDIR="${CONFHOME}/${FTPLCFG}" # continue, will fail later in main
355: fi
356:
357: # remove trailing slash, get profile name etc.
358: CONFDIR="${CONFDIR%/}"
359: NAME="${CONFDIR##*/}"
360: CONF="$CONFDIR/conf"
361: PRE="$CONFDIR/pre"
362: POST="$CONFDIR/post"
363: EXCLUDE="$CONFDIR/exclude"
364: KEYFILE="$CONFDIR/gpgkey.asc"
365:
366: }
367:
368: function version_info { # print version information
369: cat <<END
370: $ME version $ME_VERSION
371: ($ME_WEBSITE)
372: END
373: }
374:
375: function version_info_using {
376: cat <<END
377: $(version_info)
378:
379: $(using_info)
380: END
381: }
382:
383: function using_info {
384: duplicity_version_get
385: # freebsd awk (--version only), debian mawk (-W version only), deliver '' so awk does not wait for input
386: AWK_VERSION=$((awk --version '' 2>/dev/null || awk -W version '' 2>/dev/null) | awk '/.+/{sub(/^[Aa][Ww][Kk][ \t]*/,"",$0);print $0;exit}')
387: PYTHON_VERSION=$(python -V 2>&1| awk '{print tolower($0);exit}')
388: GPG_INFO=`gpg --version 2>/dev/null| awk '/^gpg/{v=$1" "$3};/^Home/{print v" ("$0")"}'`
389: BASH_VERSION=$(bash --version | awk '/^GNU bash, version/{sub(/GNU bash, version[ ]+/,"",$0);print $0}')
390: echo -e "Using installed duplicity version ${DUPL_VERSION:-(not found)}${PYTHON_VERSION+, $PYTHON_VERSION}\
391: ${GPG_INFO:+, $GPG_INFO}${AWK_VERSION:+, awk '${AWK_VERSION}'}${BASH_VERSION:+, bash '${BASH_VERSION}'}."
392: }
393:
394: function usage_info { # print usage information
395:
396: cat <<USAGE_EOF
397: VERSION:
398: $(version_info)
399:
400: DESCRIPTION:
401: Duply deals as a wrapper for the mighty duplicity magic.
402: It simplifies running duplicity with cron or on command line by:
403:
404: - keeping recurring settings in profiles per backup job
405: - enabling batch operations eg. backup_verify_purge
406: - executing pre/post scripts for every command
407: - precondition checking for flawless duplicity operation
408:
409: For each backup job one configuration profile must be created.
410: The profile folder will be stored under '~/.${ME_NAME}/<profile>'
411: (where ~ is the current users home directory).
412: Hint:
413: If the folder '/etc/${ME_NAME}' exists, the profiles for the super
414: user root will be searched & created there.
415:
416: USAGE:
417: first time usage (profile creation):
418: $ME <profile> create
419:
420: general usage in single or batch mode (see EXAMPLES):
421: $ME <profile> <command>[_<command>_...] [<options> ...]
422:
423: Non $ME options are passed on to duplicity (see OPTIONS).
424: All conf parameters can also be defined in the environment instead.
425:
426: PROFILE:
427: Indicated by a path or a profile name (<profile>), which is resolved
428: to '~/.${ME_NAME}/<profile>' (~ expands to environment variable \$HOME).
429:
430: Superuser root can place profiles under '/etc/${ME_NAME}'. Simply create
431: the folder manually before running $ME as superuser.
432: Note:
433: Already existing profiles in root's profile folder will cease to work
434: unless there are moved to the new location manually.
435:
436: example 1: $ME humbug backup
437:
438: Alternatively a _path_ might be used e.g. useful for quick testing,
439: restoring or exotic locations. Shell expansion should work as usual.
440: Hint:
441: The path must contain at least one path separator '/',
442: e.g. './test' instead of only 'test'.
443:
444: example 2: $ME ~/.${ME_NAME}/humbug backup
445:
446: COMMANDS:
447: usage get usage help text
448:
449: create creates a configuration profile
450: backup backup with pre/post script execution (batch: pre_bkp_post),
451: full (if full_if_older matches or no earlier backup is found)
452: incremental (in all other cases)
453: pre/post execute '<profile>/$(basename "$PRE")', '<profile>/$(basename "$POST")' scripts
454: bkp as above but without executing pre/post scripts
455: full force full backup
456: incr force incremental backup
457: list [<age>]
458: list all files in backup (as it was at <age>, default: now)
459: status prints backup sets and chains currently in repository
460: verify list files changed since latest backup
461: restore <target_path> [<age>]
462: restore the complete backup to <target_path> [as it was at <age>]
463: fetch <src_path> <target_path> [<age>]
464: fetch single file/folder from backup [as it was at <age>]
465: purge [<max_age>] [--force]
466: list outdated backup files (older than \$MAX_AGE)
467: [use --force to actually delete these files]
468: purge-full [<max_full_backups>] [--force]
469: list outdated backup files (\$MAX_FULL_BACKUPS being the number of
470: full backups and associated incrementals to keep, counting in
471: reverse chronological order)
472: [use --force to actually delete these files]
473: purge-incr [<max_fulls_with_incrs>] [--force]
474: list outdated incremental backups (\$MAX_FULLS_WITH_INCRS being
475: the number of full backups which associated incrementals will be
476: kept, counting in reverse chronological order)
477: [use --force to actually delete these files]
478: cleanup [--force]
479: list broken backup chain files archives (e.g. after unfinished run)
480: [use --force to actually delete these files]
481:
482: changelog print changelog / todo list
483: txt2man feature for package maintainers - create a manpage based on the
484: usage output. download txt2man from http://mvertes.free.fr/, put
485: it in the PATH and run '$ME txt2man' to create a man page.
486:
487: OPTIONS:
488: --force passed to duplicity (see commands: purge, purge-full, cleanup)
489: --preview do nothing but print out generated duplicity command lines
490: --disable-encryption
491: disable encryption, overrides profile settings
492:
493: PRE/POST SCRIPTS:
494: All internal duply variables will be readable in the scripts.
495: Some of interest might be
496:
497: CONFDIR, SOURCE, TARGET_URL_<PROT|HOSTPATH|USER|PASS>,
498: GPG_<KEYS_ENC|KEY_SIGN|PW>, CMD_<PREV|NEXT>
499:
500: The CMD_* variables were introduced to allow different actions according to
501: the command the scripts were attached to e.g. 'pre_bkp_post_pre_verify_post'
502: will call the pre script two times, with CMD_NEXT variable set to 'bkp'
503: on the first and to 'verify' on the second run.
504:
505: EXAMPLES:
506: create profile 'humbug':
507: $ME humbug create (now edit the resulting conf file)
508: backup 'humbug' now:
509: $ME humbug backup
510: list available backup sets of profile 'humbug':
511: $ME humbug status
512: list and delete obsolete backup archives of 'humbug':
513: $ME humbug purge --force
514: restore latest backup of 'humbug' to /mnt/restore:
515: $ME humbug restore /mnt/restore
516: restore /etc/passwd of 'humbug' from 4 days ago to /root/pw:
517: $ME humbug fetch etc/passwd /root/pw 4D
518: (see "duplicity manpage", section TIME FORMATS)
519: a one line batch job on 'humbug' for cron execution:
520: $ME humbug backup_verify_purge --force
521:
522: FILES:
523: in profile folder '~/.${ME_NAME}/<profile>' or '/etc/${ME_NAME}'
524: conf profile configuration file
525: pre,post pre/post scripts (see above for details)
526: gpgkey.*.asc exported GPG key files
527: exclude a globbing list of included or excluded files/folders
528: (see "duplicity manpage", section FILE SELECTION)
529:
530: $(hint_profile)
531:
532: SEE ALSO:
533: duplicity man page:
534: duplicity(1) or http://duplicity.nongnu.org/duplicity.1.html
535: USAGE_EOF
536: }
537:
538: # to check call 'duply txt2man | man -l -'
539: function usage_txt2man {
540: usage_info | \
541: awk '/^^[^[:lower:][:space:]][^[:lower:]]+$/{gsub(/[^[:upper:]]/," ",$0)}{print}' |\
542: txt2man -t"$(toupper "${ME_NAME}")" -s1 -r"${ME_NAME}-${ME_VERSION}" -v'User Manuals'
543: }
544:
545: function changelog {
546: cat $ME_LONG | awk '/^#####/{on=on+1}(on==3){sub(/^#( )?/,"",$0);print}'
547: }
548:
549: function create_config {
550: if [ ! -d "$CONFDIR" ] ; then
551: mkdir -p "$CONFDIR" || error "Couldn't create config '$CONFDIR'."
552: # create initial config file
553: cat <<EOF >"$CONF"
554: # gpg encryption settings, simple settings:
555: # GPG_KEY='disabled' - disables encryption alltogether
556: # GPG_KEY='<key1>[,<key2>]'; GPG_PW='pass' - encrypt with keys, sign
557: # with key1 if secret key available and use GPG_PW for sign & decrypt
558: # GPG_PW='passphrase' - symmetric encryption using passphrase only
559: GPG_KEY='${DEFAULT_GPG_KEY}'
560: GPG_PW='${DEFAULT_GPG_PW}'
561: # gpg encryption settings in detail (extended settings)
562: # the above settings translate to the following more specific settings
563: # GPG_KEYS_ENC='<keyid1>,[<keyid2>,...]' - list of pubkeys to encrypt to
564: # GPG_KEY_SIGN='<keyid1>|disabled' - a secret key for signing
565: # GPG_PW='<passphrase>' - needed for signing, decryption and symmetric
566: # encryption. If you want to deliver different passphrases for e.g.
567: # several keys or symmetric encryption plus key signing you can use
568: # gpg-agent. Add '--use-agent' to the duplicity parameters below.
569: # also see "A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING" in duplicity manpage
570: # notes on en/decryption
571: # private key and passphrase will only be needed for decryption or signing.
572: # decryption happens on restore and incrementals (compare archdir contents).
573: # for security reasons it makes sense to separate the signing key from the
574: # encryption keys. https://answers.launchpad.net/duplicity/+question/107216
575: #GPG_KEYS_ENC='<pubkey1>,<pubkey2>,...'
576: #GPG_KEY_SIGN='<prvkey>'
577: # set if signing key passphrase differs from encryption (key) passphrase
578: # NOTE: available since duplicity 0.6.14, translates to SIGN_PASSPHRASE
579: #GPG_PW_SIGN='<signpass>'
580:
581: # gpg options passed from duplicity to gpg process (default='')
582: # e.g. "--trust-model pgp|classic|direct|always"
583: # or "--compress-algo=bzip2 --bzip2-compress-level=9"
584: # or "--personal-cipher-preferences AES256,AES192,AES..."
585: #GPG_OPTS=''
586:
587: # disable preliminary tests with the following setting
588: #GPG_TEST='disabled'
589:
590: # credentials & server address of the backup target (URL-Format)
591: # syntax is
592: # scheme://[user:password@]host[:port]/[/]path
593: # probably one out of
594: # # for cloudfiles backend user id is CLOUDFILES_USERNAME, password is
595: # # CLOUDFILES_APIKEY, you might need to set CLOUDFILES_AUTHURL manually
596: # cf+http://[user:password@]container_name
597: # file://[relative|/absolute]/local/path
598: # ftp[s]://user[:password]@other.host[:port]/some_dir
599: # hsi://user[:password]@other.host/some_dir
600: # hsi://user[:password]@other.host/some_dir
601: # imap[s]://user[:password]@host.com[/from_address_prefix]
602: # rsync://user[:password]@host.com[:port]::[/]module/some_dir
603: # # rsync over ssh (only keyauth)
604: # rsync://user@host.com[:port]/[relative|/absolute]_path
605: # # for the s3 user/password are AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
606: # s3://[user:password@]host/bucket_name[/prefix]
607: # s3+http://[user:password@]bucket_name[/prefix]
608: # # scp and sftp are aliases for the ssh backend
609: # ssh://user[:password]@other.host[:port]/[/]some_dir
610: # tahoe://alias/directory
611: # # for Ubuntu One set TARGET_PASS to oauth access token
612: # # "consumer_key:consumer_secret:token:token_secret"
613: # # if non given credentials will be prompted for and one will be created
614: # u1://host_is_ignored/volume_path
615: # u1+http:///volume_path
616: # webdav[s]://user[:password]@other.host/some_dir
617: # ATTENTION: characters other than A-Za-z0-9.-_.~ in user,password,path have
618: # to be replaced by their url encoded pendants, see
619: # http://en.wikipedia.org/wiki/Url_encoding
620: # if you define the credentials as TARGET_USER, TARGET_PASS below
621: # duply will try to url_encode them for you if need arises
622: TARGET='${DEFAULT_TARGET}'
623: # optionally the username/password can be defined as extra variables
624: # setting them here _and_ in TARGET results in an error
625: #TARGET_USER='${DEFAULT_TARGET_USER}'
626: #TARGET_PASS='${DEFAULT_TARGET_PASS}'
627:
628: # base directory to backup
629: SOURCE='${DEFAULT_SOURCE}'
630:
631: # exclude folders containing exclusion file (since duplicity 0.5.14)
632: # Uncomment the following two lines to enable this setting.
633: #FILENAME='.duplicity-ignore'
634: #DUPL_PARAMS="\$DUPL_PARAMS --exclude-if-present '\$FILENAME'"
635:
636: # Time frame for old backups to keep, Used for the "purge" command.
637: # see duplicity man page, chapter TIME_FORMATS)
638: #MAX_AGE=1M
639:
640: # Number of full backups to keep. Used for the "purge-full" command.
641: # See duplicity man page, action "remove-all-but-n-full".
642: #MAX_FULL_BACKUPS=1
643:
644: # Number of full backups for which incrementals will be kept for.
645: # Used for the "purge-incr" command.
646: # See duplicity man page, action "remove-all-inc-of-but-n-full".
647: #MAX_FULLS_WITH_INCRS=1
648:
649: # activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
650: # forces a full backup if last full backup reaches a specified age, for the
651: # format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
652: # Uncomment the following two lines to enable this setting.
653: #MAX_FULLBKP_AGE=1M
654: #DUPL_PARAMS="\$DUPL_PARAMS --full-if-older-than \$MAX_FULLBKP_AGE "
655:
656: # sets duplicity --volsize option (available since v0.4.3.RC7)
657: # set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
658: # VOLSIZE must be number of MB's to set the volume size to.
659: # Uncomment the following two lines to enable this setting.
660: #VOLSIZE=50
661: #DUPL_PARAMS="\$DUPL_PARAMS --volsize \$VOLSIZE "
662:
663: # verbosity of output (error 0, warning 1-2, notice 3-4, info 5-8, debug 9)
664: # default is 4, if not set
665: #VERBOSITY=5
666:
667: # temporary file space. at least the size of the biggest file in backup
668: # for a successful restoration process. (default is '/tmp', if not set)
669: #TEMP_DIR=/tmp
670:
671: # Modifies archive-dir option (since 0.6.0) Defines a folder that holds
672: # unencrypted meta data of the backup, enabling new incrementals without the
673: # need to decrypt backend metadata first. If empty or deleted somehow, the
674: # private key and it's password are needed.
675: # NOTE: This is confidential data. Put it somewhere safe. It can grow quite
676: # big over time so you might want to put it not in the home dir.
677: # default '~/.cache/duplicity/duply_<profile>/'
678: # if set '\${ARCH_DIR}/<profile>'
679: #ARCH_DIR=/some/space/safe/.duply-cache
680:
681: # DEPRECATED setting
682: # sets duplicity --time-separator option (since v0.4.4.RC2) to allow users
683: # to change the time separator from ':' to another character that will work
684: # on their system. HINT: For Windows SMB shares, use --time-separator='_'.
685: # NOTE: '-' is not valid as it conflicts with date separator.
686: # ATTENTION: only use this with duplicity < 0.5.10, since then default file
687: # naming is compatible and this option is pending depreciation
688: #DUPL_PARAMS="\$DUPL_PARAMS --time-separator _ "
689:
690: # DEPRECATED setting
691: # activates duplicity --short-filenames option, when uploading to a file
692: # system that can't have filenames longer than 30 characters (e.g. Mac OS 8)
693: # or have problems with ':' as part of the filename (e.g. Microsoft Windows)
694: # ATTENTION: only use this with duplicity < 0.5.10, later versions default file
695: # naming is compatible and this option is pending depreciation
696: #DUPL_PARAMS="\$DUPL_PARAMS --short-filenames "
697:
698: # more duplicity command line options can be added in the following way
699: # don't forget to leave a separating space char at the end
700: #DUPL_PARAMS="\$DUPL_PARAMS --put_your_options_here "
701:
702: EOF
703:
704: # Hints on first usage
705: cat <<EOF
706:
707: Congratulations. You just created the profile '$FTPLCFG'.
708: The initial config file has been created as
709: '$CONF'.
710: You should now adjust this config file to your needs.
711:
712: $(hint_profile)
713:
714: EOF
715: fi
716:
717: }
718:
719: # used in usage AND create_config
720: function hint_profile {
721: cat <<EOF
722: IMPORTANT:
723: Copy the _whole_ profile folder after the first backup to a safe place.
724: It contains everything needed to restore your backups. You will need
725: it if you have to restore the backup from another system (e.g. after a
726: system crash). Keep access to these files restricted as they contain
727: _all_ informations (gpg data, ftp data) to access and modify your backups.
728:
729: Repeat this step after _all_ configuration changes. Some configuration
730: options are crucial for restoration.
731:
732: EOF
733: }
734:
735: function separator {
736: echo "--- $@ ---"
737: }
738:
739: function inform {
740: echo -e "\nINFO:\n\n$@\n"
741: }
742:
743: function warning {
744: echo -e "\nWARNING:\n\n$@\n"
745: }
746:
747: function warning_oldhome {
748: local old=$1 new=$2
749: warning " ftplicity changed name to duply since you created your profiles.
750: Please rename the old folder
751: '$old'
752: to
753: '$new'
754: and this warning will disappear.
755: If you decide not to do so profiles will _only_ work from the old location."
756: }
757:
758: function error_print {
759: echo -e "$@" >&2
760: }
761:
762: function error {
763: error_print "\nSorry. A fatal ERROR occured:\n\n$@\n"
764: exit -1
765: }
766:
767: function error_gpg {
768: [ -n "$2" ] && local hint="\n $2\n\n "
769:
770: error "$1
771:
772: Hint${hint:+s}:
773: ${hint}Maybe you have not created a gpg key yet (e.g. gpg --gen-key)?
774: Don't forget the used _password_ as you will need it.
775: When done enter the 8 digit id & the password in the profile conf file.
776:
777: The key id can be found doing a 'gpg --list-keys'. In the example output
778: below the key id would be FFFFFFFF for the public key.
779:
780: pub 1024D/FFFFFFFF 2007-12-17
781: uid duplicity
782: sub 2048g/899FE27F 2007-12-17
783: "
784: }
785:
786: # TODO
787: function error_gpg_key {
788: local KEY_ID=$1
789: local KIND=$2
790: error_gpg "${KIND} gpg key '${KEY_ID}' cannot be found." \
791: "Doublecheck if the above key is listed by 'gpg --list-keys' or available
792: as gpg key file '$(basename "$(gpg_keyfile ${KEY_ID})")' in the profile folder.
793: If not you can put it there and $ME will autoimport it on the next run.
794: Alternatively import it manually as the user you plan to run $ME with."
795: }
796:
797: function error_gpg_test {
798: [ -n "$2" ] && local hint="\n $2\n\n "
799:
800: error "$1
801:
802: Hint${hint:+s}:
803: ${hint}This error means that gpg is probably misconfigured or not working
804: correctly. The error message above should help to solve the problem.
805: However, if for some reason $ME should misinterpret the situation you
806: can define GPG_TEST='disabled' in the conf file to bypass the test.
807: Please do not forget to report the bug in order to resolve the problem
808: in future versions of $ME.
809: "
810: }
811:
812: function error_path {
813: error "$@
814: PATH='$PATH'
815: "
816: }
817:
818: function error_to_string {
819: [ -n "$1" ] && [ "$1" -eq 0 ] && echo "OK" || echo "FAILED 'code $1'"
820: }
821:
822: function duplicity_version_get {
823: var_isset DUPL_VERSION && return
824: DUPL_VERSION=`duplicity --version 2>&1 | awk '/^duplicity /{print $2; exit;}'`
825: #DUPL_VERSION='0.6.08b' #,0.4.4.RC4,0.6.08b
826: DUPL_VERSION_VALUE=0
827: DUPL_VERSION_AWK=$(awk -v v="$DUPL_VERSION" 'BEGIN{
828: if (match(v,/[^\.0-9]+[0-9]*$/)){
829: rest=substr(v,RSTART,RLENGTH);v=substr(v,0,RSTART-1);}
830: if (pos=match(rest,/RC([0-9]+)$/)) rc=substr(rest,pos+2)
831: split(v,f,"[. ]"); if(f[1]f[2]f[3]~/^[0-9]+$/) vvalue=f[1]*10000+f[2]*100+f[3]; else vvalue=0
832: print "#"v"_"rest"("rc"):"f[1]"-"f[2]"-"f[3]
833: print "DUPL_VERSION_VALUE=\047"vvalue"\047"
834: print "DUPL_VERSION_RC=\047"rc"\047"
835: print "DUPL_VERSION_SUFFIX=\047"rest"\047"
836: }')
837: eval "$DUPL_VERSION_AWK"
838: #echo -e ",$DUPL_VERSION,$DUPL_VERSION_VALUE,$DUPL_VERSION_RC,$DUPL_VERSION_SUFFIX,"
839: }
840:
841: function duplicity_version_check {
842: if [ $DUPL_VERSION_VALUE -eq 0 ]; then
843: inform "duplicity version check failed (please report, this is a bug)"
844: elif [ $DUPL_VERSION_VALUE -le 404 ] && [ ${DUPL_VERSION_RC:-4} -lt 4 ]; then
845: error "The installed version $DUPL_VERSION is incompatible with $ME v$ME_VERSION.
846: You should upgrade your version of duplicity to at least v0.4.4RC4 or
847: use the older ftplicity version 1.1.1 from $ME_WEBSITE."
848: fi
849: }
850:
851: function duplicity_version_ge {
852: [ "$DUPL_VERSION_VALUE" -ge "$1" ]
853: }
854:
855: function duplicity_version_lt {
856: ! duplicity_version_ge "$1"
857: }
858:
859: function run_script { # run pre/post scripts
860: local ERR=0
861: local SCRIPT="$1"
862: if [ ! -z "$PREVIEW" ] ; then
863: echo $SCRIPT
864: elif [ -r "$SCRIPT" ] ; then
865: echo -n "Running '$SCRIPT' "
866: OUT=`. "$SCRIPT" 2>&1`; ERR=$?
867: [ $ERR -eq "0" ] && echo "- OK" || echo "- FAILED (code $ERR)"
868: echo -en ${OUT:+"Output: $OUT\n"} ;
869: else
870: echo "Skipping n/a script '$SCRIPT'."
871: fi
872: return $ERR
873: }
874:
875: function run_cmd {
876: # run or print escaped cmd string
877: CMD_ERR=0
878: if [ -n "$PREVIEW" ]; then
879: CMD_OUT=$( echo "$@ 2>&1" )
880: CMD_MSG="-- Run cmd '$CMD_MSG' --\n$CMD_OUT"
881: elif [ -n "$CMD_DISABLED" ]; then
882: CMD_MSG="$CMD_MSG (DISABLED) - $CMD_DISABLED"
883: else
884: CMD_OUT=` eval $@ 2>&1 `
885: CMD_ERR=$?
886: if [ "$CMD_ERR" = "0" ]; then
887: CMD_MSG="$CMD_MSG (OK)"
888: else
889: CMD_MSG="$CMD_MSG (FAILED)"
890: fi
891: fi
892: echo -e "$CMD_MSG"
893: # reset
894: unset CMD_DISABLED CMD_MSG
895: return $CMD_ERR
896: }
897:
898: function qw { quotewrap "$@"; }
899:
900: function quotewrap {
901: local param="$@"
902: # quote strings having non word chars (e.g. spaces)
903: if echo "$param" | awk '/[^A-Za-z0-9_\.\-]/{exit 0}{exit 1}'; then
904: echo "$param" | awk '{\
905: gsub(/[\047]/,"\047\\\047\047",$0);\
906: gsub(/[\042]/,"\047\\\042\047",$0);\
907: print "\047"$0"\047"}'
908: return
909: fi
910: echo $param
911: }
912:
913: function duplicity_params_global {
914: # already done? return
915: var_isset 'DUPL_PARAMS_GLOBAL' && return
916: local DUPL_ARG_ENC
917:
918: # use key only if set in config, else leave it to symmetric encryption
919: if gpg_disabled; then
920: local DUPL_PARAM_ENC='--no-encryption'
921: else
922: local DUPL_PARAM_ENC=$(gpg_prefix_keyset ' --encrypt-key ' 'GPG_KEYS_ENC')
923: gpg_signing && local DUPL_PARAM_SIGN=$(gpg_prefix_keyset ' --sign-key ' 'GPG_KEY_SIGN')
924: # interpret password settings
925: var_isset 'GPG_PW' && DUPL_ARG_ENC="PASSPHRASE=$(qw "${GPG_PW}")"
926: var_isset 'GPG_PW_SIGN' && DUPL_ARG_ENC="${DUPL_ARG_ENC} SIGN_PASSPHRASE=$(qw "${GPG_PW_SIGN}")"
927: fi
928:
929: local GPG_OPTS=${GPG_OPTS:+"--gpg-options $(qw "${GPG_OPTS}")"}
930:
931: # set name for dupl archive folder, since 0.6.0
932: if duplicity_version_ge 601; then
933: local DUPL_ARCHDIR=''
934: if var_isset 'ARCH_DIR'; then
935: DUPL_ARCHDIR="--archive-dir '${ARCH_DIR}'"
936: fi
937: DUPL_ARCHDIR="${DUPL_ARCHDIR} --name 'duply_${NAME}'"
938: fi
939:
940: DUPL_PARAMS_GLOBAL="${DUPL_ARCHDIR} ${DUPL_PARAM_ENC} \
941: ${DUPL_PARAM_SIGN} --verbosity '${VERBOSITY:-4}' \
942: ${GPG_OPTS}"
943:
944: DUPL_VARS_GLOBAL="TMPDIR='$TEMP_DIR' \
945: ${DUPL_ARG_ENC}"
946: }
947:
948: # filter the DUPL_PARAMS var from conf
949: function duplicity_params_conf {
950: # reuse cmd var from main loop
951: ## in/exclude parameters are currently not supported on restores
952: if [ "$cmd" = "fetch" ] || [ "$cmd" = "restore" ]; then
953: # filter exclude params from fetch/restore
954: echo "$DUPL_PARAMS" | awk '{gsub(/--(ex|in)clude[a-z-]*(([ \t]+|=)[^-][^ \t]+)?/,"");print}'
955: return
956: fi
957:
958: echo "$DUPL_PARAMS"
959: }
960:
961: function duplify { # the actual wrapper function
962: local PARAMSNOW DUPL_CMD DUPL_CMD_PARAMS
963:
964: # put command (with params) first in duplicity parameters
965: for param in "$@" ; do
966: # split cmd from params (everything before splitchar --)
967: if [ "$param" == "--" ] ; then
968: PARAMSNOW=1
969: else
970: # wrap in quotes to protect from spaces
971: [ ! $PARAMSNOW ] && \
972: DUPL_CMD="$DUPL_CMD $(qw $param)" \
973: || \
974: DUPL_CMD_PARAMS="$DUPL_CMD_PARAMS $(qw $param)"
975: fi
976: done
977:
978: # init global duplicity parameters same for all tasks
979: duplicity_params_global
980:
981: var_isset 'PREVIEW' && local RUN=echo || local RUN=eval
982: $RUN ${DUPL_VARS_GLOBAL} ${BACKEND_PARAMS} \
983: duplicity $DUPL_CMD $DUPL_PARAMS_GLOBAL $(duplicity_params_conf)\
984: $DUPL_CMD_PARAMS ${PREVIEW:+}
985:
986: local ERR=$?
987: return $ERR
988: }
989:
990: function secureconf { # secure the configuration dir
991: #PERMS=$(ls -la $(dirname $CONFDIR) | grep -e " $(basename $CONFDIR)\$" | awk '{print $1}')
992: local PERMS="$(ls -la "$CONFDIR/." | awk 'NR==2{print $1}')"
993: if [ "${PERMS/#drwx------*/OK}" != 'OK' ] ; then
994: chmod u+rwX,go= "$CONFDIR"; local ERR=$?
995: warning "The profile's folder
996: '$CONFDIR'
997: permissions are not safe ($PERMS). Secure them now. - ($(error_to_string $ERR))"
998: fi
999: }
1000:
1001: # params are $1=timeformatstring (default like date output), $2=epoch seconds since 1.1.1970 (default now)
1002: function date_fix {
1003: local DEFAULTFORMAT='%a %b %d %H:%M:%S %Z %Y'
1004: # gnu date with -d @epoch
1005: date=$(date ${2:+-d @$2} ${1:++"$1"} 2> /dev/null) && \
1006: echo $date && return
1007: # date bsd,osx with -r epoch
1008: date=$(date ${2:+-r $2} ${1:++"$1"} 2> /dev/null) && \
1009: echo $date && return
1010: # date busybox with -d epoch -D %s
1011: date=$(date ${2:+-d $2 -D %s} ${1:++"$1"} 2> /dev/null) && \
1012: echo $date && return
1013: ## some date commands do not support giving a time w/o setting it systemwide (irix,solaris,others?)
1014: # python fallback
1015: date=$(python -c "import time;print time.strftime('${1:-$DEFAULTFORMAT}',time.localtime(${2}))" 2> /dev/null) && \
1016: echo $date && return
1017: # awk fallback
1018: date=$(awk "BEGIN{print strftime(\"${1:-$DEFAULTFORMAT}\"${2:+,$2})}" 2> /dev/null) && \
1019: echo $date && return
1020: # perl fallback
1021: date=$(perl -e "use POSIX qw(strftime);\$date = strftime(\"${1:-$DEFAULTFORMAT}\",localtime(${2}));print \"\$date\n\";" 2> /dev/null) && \
1022: echo $date && return
1023: # error
1024: echo "ERROR"
1025: return 1
1026: }
1027:
1028: function nsecs {
1029: # only 9 digit returns, e.g. not all date(s) deliver nsecs
1030: local NSECS=$(date +%N 2> /dev/null | head -1 |grep -e "^[[:digit:]]\{9\}$")
1031: echo ${NSECS:-000000000}
1032: }
1033:
1034: function nsecs_to_sec {
1035: echo $(($1/1000000000)).$(printf "%03d" $(($1/1000000%1000)) )
1036: }
1037:
1038: function datefull_from_nsecs {
1039: date_from_nsecs $1 '%F %T'
1040: }
1041:
1042: function date_from_nsecs {
1043: local FORMAT=${2:-%T}
1044: local TIME=$(nsecs_to_sec $1)
1045: local SECS=${TIME%.*}
1046: local DATE=$(date_fix "%T" ${SECS:-0})
1047: echo $DATE.${TIME#*.}
1048: }
1049:
1050: function var_isset {
1051: if [ -z "$1" ]; then
1052: echo "ERROR: function var_isset needs a string as parameter"
1053: elif eval "[ \"\${$1}\" == 'not_set' ]" || eval "[ \"\${$1-not_set}\" != 'not_set' ]"; then
1054: return 0
1055: fi
1056: return 1
1057: }
1058:
1059: function url_encode {
1060: # utilize python, silently do nothing on error - because no python no duplicity
1061: OUT=$(python -c "
1062: try: import urllib.request as urllib
1063: except ImportError: import urllib
1064: print(urllib.${2}quote('$1'));
1065: " 2>/dev/null ); ERR=$?
1066: [ "$ERR" -eq 0 ] && echo $OUT || echo $1
1067: }
1068:
1069: function url_decode {
1070: # reuse function above with a simple string param hack
1071: url_encode "$1" "un"
1072: }
1073:
1074: function toupper {
1075: echo "$@"|awk '$0=toupper($0)'
1076: }
1077:
1078: function tolower {
1079: echo "$@"|awk '$0=tolower($0)'
1080: }
1081:
1082: function gpg_disabled {
1083: echo "${GPG_KEY}${GPG_KEYS_ENC}" | grep -iq 'disabled'
1084: }
1085:
1086: function gpg_signing {
1087: return $(echo ${GPG_KEY_SIGN} | grep -ic 'disabled')
1088: }
1089:
1090: # parameter key id, key_type
1091: function gpg_keyfile {
1092: local GPG_KEY="$1" TYPE="$2"
1093: local KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}"
1094: echo "${KEYFILE//.asc/${TYPE:+.$(tolower $TYPE)}.asc}"
1095: }
1096:
1097: # parameter key id
1098: function gpg_import {
1099: local FILE KEY_ID="$1" KEY_TYPE="$2" KEY_FP="" ERR=1
1100: local KEYFILES=( "$CONFDIR/gpgkey" "$(gpg_keyfile $KEY_ID)" "${KEY_TYPE:+$(gpg_keyfile $KEY_ID $KEY_TYPE)}" )
1101: #[ -z $KEYFILES ] && warning "No keyfile for '$KEY_ID' found in profile\n'$CONFDIR'."
1102: # Try autoimport from existing old gpgkey files and new gpgkey.XXX.asc files (since v1.4.2)
1103: for (( i = 0 ; i < ${#KEYFILES[@]} ; i++ )); do
1104: FILE=${KEYFILES[$i]}
1105: if [ -f "$FILE" ]; then
1106:
1107: CMD_MSG="Import keyfile '$FILE'"
1108: run_cmd "$GPG" --batch --import "$FILE"
1109: if [ "$CMD_ERR" != "0" ]; then
1110: warning "Import failed.${CMD_OUT:+\n$CMD_OUT}"
1111: # continue with next
1112: break
1113: fi
1114:
1115: # imported at least one; return success
1116: ERR=0
1117:
1118: # set trust automagically
1119: CMD_MSG="Autoset trust of key '$KEY_ID'to ultimate"
1120: run_cmd echo $(gpg_fingerprint $KEY_ID):6: \| "$GPG" --import-ownertrust --batch --logger-fd 1
1121: if [ "$CMD_ERR" = "0" ] && [ -z "$PREVIEW" ]; then
1122: break
1123: fi
1124:
1125: # set trust manually
1126: echo -e "For $ME to work you have to set the trust level
1127: with the command \"trust\" to \"ultimate\" (5) now.
1128: Exit the edit mode of gpg with \"quit\"."
1129: CMD_MSG="Running gpg to manually edit key '$KEY_ID'"
1130: run_cmd sleep 5\; "$GPG" --edit-key $KEY_ID
1131:
1132: fi
1133: done
1134:
1135: return $ERR
1136: }
1137:
1138: # check for 8 digits and using 0x00.. here because gpg uses substring matching by default
1139: # see 'How to specify a user ID' on gpg manpage
1140: function gpg_fingerprint {
1141: [ ${#1} -eq 8 ] \
1142: && local PRINT=$("$GPG" --fingerprint 0x"$1" 2>&1|awk -F= 'NR==2{gsub(/ /,"",$2);$2=toupper($2); if ( $2 ~ /^[A-F0-9]+$/ && length($2) == 40 ) print $2; else exit 1}') \
1143: && [ -n "$PRINT" ] && echo $PRINT && return 0
1144: return 1
1145: }
1146:
1147: function gpg_export_if_needed {
1148: local SUCCESS FILE KEY_TYPE KEY_LIST=$1
1149: local TMPFILE="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s).gpgexp"
1150: for KEY_ID in $KEY_LIST; do
1151: # check if already exported, do it if not
1152: for KEY_TYPE in PUB SEC; do
1153: FILE="$(gpg_keyfile $KEY_ID $KEY_TYPE)"
1154: if [ ! -f "$FILE" ] && eval gpg_$(tolower $KEY_TYPE)_avail $KEY_ID; then
1155: # exporting
1156: CMD_MSG="Export $KEY_TYPE key $KEY_ID"
1157: run_cmd $GPG --armor --export"$(test "SEC" = "$KEY_TYPE" && echo -secret-keys)"" $KEY_ID >> \"$TMPFILE\""
1158:
1159: if [ "$CMD_ERR" = "0" ]; then
1160: CMD_MSG="Write file '"$(basename "$FILE")"'"
1161: run_cmd " mv \"$TMPFILE\" \"$FILE\""
1162: fi
1163:
1164: if [ "$CMD_ERR" != "0" ]; then
1165: warning "Backup failed.${CMD_OUT:+\n$CMD_OUT}"
1166: else
1167: SUCCESS=1
1168: fi
1169:
1170: # cleanup
1171: rm "$TMPFILE" 1>/dev/null 2>&1
1172: fi
1173: done
1174: done
1175:
1176: [ -n "$SUCCESS" ] && inform "$ME exported new keys to your profile.
1177: You should backup your changed profile folder now and store it in a safe place."
1178: }
1179:
1180: function gpg_key_cache {
1181: local RES
1182: local PREFIX="GPG_KEY"
1183: local CACHE="PREFIX_$1_$2"
1184: if [ "$1" = "RESET" ]; then
1185: eval unset PREFIX_PUB_$2 PREFIX_SEC_$2
1186: return 255
1187: elif ! var_isset "$CACHE"; then
1188: if [ "$1" = "PUB" ]; then
1189: RES=$("$GPG" --list-key "$2" > /dev/null 2>&1; echo -n $?)
1190: elif [ "$1" = "SEC" ]; then
1191: RES=$("$GPG" --list-secret-key "$2" > /dev/null 2>&1; echo -n $?)
1192: else
1193: return 255
1194: fi
1195: eval $CACHE=$RES
1196: fi
1197: eval return \$$CACHE
1198: }
1199:
1200: function gpg_pub_avail {
1201: gpg_key_cache PUB $1
1202: }
1203:
1204: function gpg_sec_avail {
1205: gpg_key_cache SEC $1
1206: }
1207:
1208: function gpg_key_format {
1209: echo $1 | grep -q '^[0-9a-fA-F]\{8\}$'
1210: }
1211:
1212: function gpg_split_keyset {
1213: awk "BEGIN{ keys=toupper(\"$@\"); gsub(/[^A-Z0-9]/,\" \",keys); print keys }"
1214: }
1215:
1216: function gpg_join_keyset {
1217: local KEY_ID OUT
1218: for KEY_ID in $@; do
1219: [ -z "$OUT" ] && OUT=$KEY_ID || OUT=${OUT},${KEY_ID}
1220: done
1221: echo $OUT
1222: }
1223:
1224: function gpg_prefix_keyset {
1225: local KEY_ID OUT KEYSET
1226: [ -n "$2" ] && eval "local KEYSET=\"\${$2[@]}\""
1227: for KEY_ID in $KEYSET; do
1228: OUT=${OUT}${1}${KEY_ID}
1229: done
1230: echo $OUT
1231: }
1232:
1233: # grep a variable from conf text file (currently not used)
1234: function gpg_passwd {
1235: [ -r "$CONF" ] && \
1236: awk '/^[ \t]*GPG_PW[ \t=]/{\
1237: sub(/^[ \t]*GPG_PW[ \t]*=*/,"",$0);\
1238: gsub(/^[ \t]*[\047"]|[\047"][ \t]*$/,"",$0);\
1239: print $0; exit}' "$CONF"
1240: }
1241:
1242: function gpg_key_decryptable {
1243: # decryption needs pass, might be empty, but must be set
1244: var_isset 'GPG_PW' || return 1
1245: local KEY_ID
1246: for KEY_ID in ${GPG_KEYS_ENC[@]}; do
1247: gpg_sec_avail $KEY_ID && return 0
1248: done
1249: return 1
1250: }
1251:
1252: function gpg_symmetric {
1253: [ -n "$GPG_PW" ] && [ -z "${GPG_KEY}${GPG_KEYS_ENC}" ]
1254: }
1255:
1256: # start of script #######################################################################
1257:
1258: # confidentiality first, all we create is only readable by us
1259: umask 077
1260:
1261: # check if ftplicity is there & executable
1262: [ -n "$ME_LONG" ] && [ -x "$ME_LONG" ] || error "$ME missing. Executable & available in path? ($ME_LONG)"
1263:
1264: if [ ${#@} -eq 1 ]; then
1265: cmd="${1}"
1266: else
1267: FTPLCFG="${1}" ; cmd="${2}"
1268: fi
1269:
1270: # deal with command before profile validation calls
1271: # show requested version
1272: # OR requested usage info
1273: # OR create a profile
1274: # OR fall through
1275: ##if [ ${#@} -le 2 ]; then
1276: case "$cmd" in
1277: changelog)
1278: changelog
1279: exit 0
1280: ;;
1281: create)
1282: set_config
1283: if [ -d "$CONFDIR" ]; then
1284: error "The profile '$FTPLCFG' already exists in
1285: '$CONFDIR'.
1286:
1287: Hint:
1288: If you _really_ want to create a new profile by this name you will
1289: have to manually delete the existing profile folder first."
1290: exit 1
1291: else
1292: create_config
1293: exit 0
1294: fi
1295: ;;
1296: txt2man)
1297: set_config
1298: usage_txt2man
1299: exit 0
1300: ;;
1301: usage|-help|--help|-h|-H)
1302: set_config
1303: usage_info
1304: exit 0
1305: ;;
1306: version|-version|--version|-v|-V)
1307: version_info_using
1308: exit 0
1309: ;;
1310: # fallthrough.. we got a command that needs an existing profile
1311: *)
1312: # if we reach here, user either forgot profile or chose wrong profileless command
1313: if [ ${#@} -le 1 ]; then
1314: error "\
1315: Missing or wrong parameters.
1316: Only the commands
1317: changelog, create, usage, txt2man, version
1318: can be called without selecting an existing profile first.
1319: Your command was '$cmd'.
1320:
1321: Hint: Run '$ME usage' to get help."
1322: fi
1323: esac
1324: ##fi
1325:
1326:
1327: # Hello world
1328: echo "Start $ME v$ME_VERSION, time is $(date_fix '%F %T')."
1329:
1330: # check system environment
1331: DUPLICITY="$(which duplicity 2>/dev/null)"
1332: [ -z "$DUPLICITY" ] && error_path "duplicity missing. installed und available in path?"
1333: # init, exec duplicity version check info
1334: duplicity_version_get
1335: duplicity_version_check
1336:
1337: [ -z "$(which awk 2>/dev/null)" ] && error_path "awk missing. installed und available in path?"
1338:
1339: ### read configuration
1340: set_config
1341: # check validity
1342: if [ ! -d "$CONFDIR" ]; then
1343: error "Selected profile '$FTPLCFG' does not resolve to a profile folder in
1344: '$CONFDIR'.
1345:
1346: Hints:
1347: Select one of the available profiles: $(ls -1p $(dirname "$CONFDIR")| awk 'BEGIN{ORS="";OFS=""}/\/$/&&!/^\.+\/$/{print sep"\047"substr($0,0,length($0)-1)"\047";sep=","}').
1348: Use '$ME <name> create' to create a new profile.
1349: Use '$ME usage' to get usage help."
1350: elif [ ! -x "$CONFDIR" ]; then
1351: error "\
1352: Profile folder in '$CONFDIR' cannot be accessed.
1353:
1354: Hint:
1355: Check the filesystem permissions and set directory accessible e.g. 'chmod 700'."
1356: elif [ ! -f "$CONF" ] ; then
1357: error "'$CONF' not found."
1358: elif [ ! -r "$CONF" ] ; then
1359: error "'$CONF' not readable."
1360: else
1361: . "$CONF"
1362: #KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}"
1363: TEMP_DIR=${TEMP_DIR:-'/tmp'}
1364: # backward compatibility: old TARGET_PW overrides silently new TARGET_PASS if set
1365: if var_isset 'TARGET_PW'; then
1366: TARGET_PASS="${TARGET_PW}"
1367: fi
1368: fi
1369: echo "Using profile '$CONFDIR'."
1370:
1371: # secure config dir, if needed w/ warning
1372: secureconf
1373:
1374: # split TARGET in handy variables
1375: TARGET_SPLIT_URL=$(echo $TARGET | awk '{ \
1376: target=$0; match(target,/^([^\/:]+):\/\//); \
1377: prot=substr(target,RSTART,RLENGTH);\
1378: rest=substr(target,RSTART+RLENGTH); \
1379: if (credsavail=match(rest,/^[^@]*@/)){\
1380: creds=substr(rest,RSTART,RLENGTH-1);\
1381: credcount=split(creds,cred,":");\
1382: rest=substr(rest,RLENGTH+1);\
1383: # split creds with regexp\
1384: match(creds,/^([^:]+)/);\
1385: user=substr(creds,RSTART,RLENGTH);\
1386: pass=substr(creds,RSTART+1+RLENGTH);\
1387: };\
1388: # filter quotes or escape them\
1389: gsub(/[\047\042]/,"",prot);\
1390: gsub(/[\047\042]/,"",rest);\
1391: gsub(/[\047]/,"\047\\\047\047",creds);\
1392: print "TARGET_URL_PROT=\047"prot"\047\n"\
1393: "TARGET_URL_HOSTPATH=\047"rest"\047\n"\
1394: "TARGET_URL_CREDS=\047"creds"\047\n";\
1395: if(user){\
1396: gsub(/[\047]/,"\047\\\047\047",user);\
1397: print "TARGET_URL_USER=\047"user"\047\n"}\
1398: if(pass){\
1399: gsub(/[\047]/,"\047\\\047\047",pass);\
1400: print "TARGET_URL_PASS=$(url_decode \047"pass"\047)\n"}\
1401: }')
1402: eval ${TARGET_SPLIT_URL}
1403:
1404: # check if backend specific software is in path
1405: [ -n "$(echo ${TARGET_URL_PROT} | grep -i -e '^ftp://$')" ] && \
1406: [ -z "$(which ncftp 2>/dev/null)" ] && error_path "Protocol 'ftp' needs ncftp. Installed und available in path?"
1407: [ -n "$(echo ${TARGET_URL_PROT} | grep -i -e '^ftps://$')" ] && \
1408: [ -z "$(which lftp 2>/dev/null)" ] && error_path "Protocol 'ftps' needs lftp. Installed und available in path?"
1409:
1410: # fetch commmand from parameters ########################################################
1411: # Hint: cmds is also used to check if authentification info sufficient in the next step
1412: cmds="$2"; shift 2
1413:
1414: # translate backup to batch command
1415: cmds=${cmds//backup/pre_bkp_post}
1416:
1417: # complain if command(s) missing
1418: [ -z $cmds ] && error " No command given.
1419:
1420: Hint:
1421: Use '$ME usage' to get usage help."
1422:
1423: # process params
1424: for param in "$@"; do
1425: #echo !$param!
1426: case "$param" in
1427: # enable ftplicity preview mode
1428: '--preview')
1429: PREVIEW=1
1430: ;;
1431: # interpret duplicity disable encr switch
1432: '--disable-encryption')
1433: GPG_KEY='disabled'
1434: ;;
1435: *)
1436: if [ `echo "$param" | grep -e "^-"` ] || \
1437: [ `echo "$last_param" | grep -e "^-"` ] ; then
1438: # forward parameter[/option pairs] to duplicity
1439: dupl_opts["${#dupl_opts[@]}"]=${param}
1440: else
1441: # anything else must be a parameter (eg. for fetch, ...)
1442: ftpl_pars["${#ftpl_pars[@]}"]=${param}
1443: fi
1444: last_param=${param}
1445: ;;
1446: esac
1447: done
1448:
1449: # plausibility check config - VARS & KEY ################################################
1450: # check if src, trg, trg pw
1451: # auth info sufficient
1452: # gpg key, gpg pwd (might be empty) set in config
1453: # OR key in local gpg db
1454: # OR key can be imported from keyfile
1455: # OR fail
1456: if [ -z "$SOURCE" ] || [ "$SOURCE" == "${DEFAULT_SOURCE}" ]; then
1457: error " Source Path (setting SOURCE) not set or still default value in conf file
1458: '$CONF'."
1459:
1460: elif [ -z "$TARGET" ] || [ "$TARGET" == "${DEFAULT_TARGET}" ]; then
1461: error " Backup Target (setting TARGET) not set or still default value in conf file
1462: '$CONF'."
1463:
1464: elif var_isset 'TARGET_USER' && var_isset 'TARGET_URL_USER' && \
1465: [ "${TARGET_USER}" != "${TARGET_URL_USER}" ]; then
1466: error " TARGET_USER ('${TARGET_USER}') _and_ user in TARGET url ('${TARGET_URL_USER}')
1467: are configured with different values. There can be only one.
1468:
1469: Hint: Remove conflicting setting."
1470:
1471: elif var_isset 'TARGET_PASS' && var_isset 'TARGET_URL_PASS' && \
1472: [ "${TARGET_PASS}" != "${TARGET_URL_PASS}" ]; then
1473: error " TARGET_PASS ('${TARGET_PASS}') _and_ password in TARGET url ('${TARGET_URL_PASS}')
1474: are configured with different values. There can be only one.
1475:
1476: Hint: Remove conflicting setting."
1477: fi
1478:
1479: # check if authentication information sufficient
1480: if ( ( ! var_isset 'TARGET_USER' && ! var_isset 'TARGET_URL_USER' ) && \
1481: ( ! var_isset 'TARGET_PASS' && ! var_isset 'TARGET_URL_PASS' ) ); then
1482: # ok here some exceptions:
1483: # protocols that do not need passwords
1484: # s3[+http] only needs password for write operations
1485: # u1[+http] can ask for creds and create an oauth token
1486: if [ -n "$(tolower "${TARGET_URL_PROT}" | grep -e '^\(file\|tahoe\|ssh\|scp\|sftp\|u1\(\+http\)\?\)://$')" ]; then
1487: : # all is well file/tahoe do not need passwords, ssh might use key auth
1488: elif [ -n "$(tolower "${TARGET_URL_PROT}" | grep -e '^s3\(\+http\)\?://$')" ] && \
1489: [ -z "$(echo ${cmds} | grep -e '\(bkp\|incr\|full\|purge\|cleanup\)')" ]; then
1490: : # still fine, it's possible to read only access configured buckets anonymously
1491: else
1492: error " Backup target credentials needed but not set in conf file
1493: '$CONF'.
1494: Setting TARGET_USER or TARGET_PASS or the corresponding values in TARGET url
1495: are missing. Some protocols only might need it for write access to the backup
1496: repository (commands: bkp,backup,full,incr,purge) but not for read only access
1497: (e.g. verify,list,restore,fetch).
1498:
1499: Hints:
1500: Add the credentials (user,password) to the conf file.
1501: To force an empty password set TARGET_PASS='' or TARGET='prot://user:@host..'.
1502: "
1503: fi
1504: fi
1505:
1506: # GPG config plausibility check1 (disabled check) #############################
1507: if gpg_disabled; then
1508: : # encryption disabled, all is well
1509:
1510: elif [ -z "${GPG_KEY}${GPG_KEYS_ENC}${GPG_KEY_SIGN}" ] && ! var_isset 'GPG_PW'; then
1511: warning "GPG_KEY and GPG_PW are empty or not set in conf file
1512: '$CONF'.
1513: Will disable encryption for duplicity now.
1514:
1515: Hint:
1516: If you really want to use _no_ encryption you can disable this warning by
1517: setting GPG_KEY='disabled' in conf file."
1518: GPG_KEY='disabled'
1519: fi
1520:
1521: # GPG availability check (now we know if gpg is really needed)#################
1522: if ! gpg_disabled; then
1523: GPG="$(which gpg 2>/dev/null)"
1524: [ -z "$GPG" ] && error_path "gpg missing. installed und available in path?"
1525: fi
1526:
1527:
1528: # Output versions info ########################################################
1529: using_info
1530:
1531: # GPG create key settings, config check2 (needs gpg) ##########################
1532: if gpg_disabled; then
1533: : # the following tests are not necessary
1534: else
1535:
1536: # key set?
1537: if [ "$GPG_KEY" == "${DEFAULT_GPG_KEY}" ]; then
1538: error_gpg "Encryption Key GPG_KEY still default in conf file
1539: '$CONF'."
1540: fi
1541:
1542: # check gpg keys format
1543: for KEY_SET_NAME in GPG_KEY GPG_KEYS_ENC $(gpg_signing && echo -n GPG_KEY_SIGN); do
1544: eval KEY_SET="\${${KEY_SET_NAME}}"
1545: for KEY_ID in $(gpg_split_keyset "$KEY_SET"); do
1546: # test format [ ! $(echo $GPG_KEY | grep '^[0-9a-fA-F]\{8\}$') ] not set correct (8 digit ID) or
1547: gpg_key_format ${KEY_ID} || \
1548: error_gpg "GPG key '${KEY_ID}' set in '${KEY_SET_NAME}' is not \na valid 8 character hex digit string e.g. '012345AB'."
1549: done
1550: done
1551:
1552: # create enc gpg keys array, for further processing
1553: GPG_KEYS_ENC=( $(gpg_split_keyset ${GPG_KEY}) $(gpg_split_keyset ${GPG_KEYS_ENC}) )
1554:
1555: # check gpg encr public keys availability
1556: for (( i = 0 ; i < ${#GPG_KEYS_ENC[@]} ; i++ )); do
1557: KEY_ID=${GPG_KEYS_ENC[$i]}
1558: # test availability, try to import, retest
1559: if ! gpg_pub_avail ${KEY_ID}; then
1560: echo "Encryption public key '${KEY_ID}' not found. Try to import."
1561: gpg_import "${KEY_ID}" PUB
1562: gpg_key_cache RESET ${KEY_ID}
1563: gpg_pub_avail ${KEY_ID} || error_gpg_key "${KEY_ID}" "Public"
1564: fi
1565: done
1566:
1567: # gpg secret sign key availability
1568: # if none set, autoset first encryption key as sign key
1569: if ! gpg_signing; then
1570: echo "Signing disabled per configuration."
1571: # try first key, if one set
1572: elif ! var_isset 'GPG_KEY_SIGN'; then
1573: KEY_ID=${GPG_KEYS_ENC[0]}
1574: if ! gpg_key_format "${KEY_ID}"; then
1575: echo "Signing disabled. Not GPG_KEY entries in config."
1576: GPG_KEY_SIGN='disabled'
1577: else
1578: # use avail OR try import OR fail
1579: if gpg_sec_avail "${KEY_ID}"; then
1580: GPG_KEY_SIGN=${KEY_ID}
1581: else
1582: gpg_import "${KEY_ID}" SEC
1583: gpg_key_cache RESET ${KEY_ID}
1584: if gpg_sec_avail "${KEY_ID}"; then
1585: GPG_KEY_SIGN=${KEY_ID}
1586: fi
1587: fi
1588:
1589: # interpret sign key setting
1590: if var_isset 'GPG_KEY_SIGN'; then
1591: echo "Autoset found secret key of first GPG_KEY entry '${KEY_ID}' for signing."
1592: else
1593: echo "Signing disabled. First GPG_KEY entry's '${KEY_ID}' private key is missing."
1594: GPG_KEY_SIGN='disabled'
1595: fi
1596: fi
1597: else
1598: KEY_ID=${GPG_KEY_SIGN}
1599: if ! gpg_sec_avail ${KEY_ID}; then
1600: inform "Secret signing key defined in setting GPG_KEY_SIGN='${KEY_ID}' not found.\nTry to import."
1601: gpg_import "${KEY_ID}" SEC
1602: gpg_key_cache RESET ${KEY_ID}
1603: gpg_sec_avail ${KEY_ID} || error_gpg_key "${KEY_ID}" "Private"
1604: else
1605: echo "Using configured key '${KEY_ID}' as signing key."
1606: fi
1607: fi
1608:
1609: # pw set? only if symmetric or signing on
1610: if ( gpg_symmetric || gpg_signing ) \
1611: && ( ! var_isset 'GPG_PW' || [ "$GPG_PW" == "${DEFAULT_GPG_PW}" ] ); then
1612: error_gpg "Encryption Password GPG_PW (needed for signing or symmetric encryption)
1613: is not set or still default value in conf file
1614: '$CONF'." "For empty password set GPG_PW='' in conf file."
1615: fi
1616:
1617: # end GPG config plausibility check2
1618: fi
1619:
1620: # config plausibility check - SPACE ###########################################
1621: # is tmp writeable
1622: # is tmp big enough
1623: if [ ! -d "$TEMP_DIR" ]; then
1624: error "Temporary file space '$TEMP_DIR' is not a directory."
1625: elif [ ! -w "$TEMP_DIR" ]; then
1626: error "Temporary file space '$TEMP_DIR' not writable."
1627: fi
1628:
1629: # get volsize, default duplicity volume size is 25MB since v0.5.07
1630: VOLSIZE=${VOLSIZE:-25}
1631: # get free temp space
1632: TEMP_FREE="$(df $TEMP_DIR 2>/dev/null | awk 'END{pos=(NF-2);if(pos>0) print $pos;}')"
1633: # check for free space or FAIL
1634: if [ "$((${TEMP_FREE:-0}-${VOLSIZE:-0}*1024))" -lt 0 ]; then
1635: error "Temporary file space '$TEMP_DIR' free space is smaller ($((TEMP_FREE/1024))MB)
1636: than one duplicity volume (${VOLSIZE}MB).
1637:
1638: Hint: Free space or change TEMP_DIR setting."
1639: fi
1640:
1641: # check for enough async upload space and WARN only
1642: if [ $((${TEMP_FREE:-0}-2*${VOLSIZE:-0}*1024)) -lt 0 ]; then
1643: warning "Temporary file space '$TEMP_DIR' free space is smaller ($((TEMP_FREE/1024))MB)
1644: than two duplicity volumes (2x${VOLSIZE}MB). This can lead to problems when
1645: using the --asynchronous-upload option.
1646:
1647: Hint: Free space or change TEMP_DIR setting."
1648: fi
1649:
1650: # test - GPG SANITY #####################################################################
1651: # if encryption is disabled, skip this whole section
1652: if gpg_disabled; then
1653: echo -e "Test - En/Decryption skipped. (GPG disabled)"
1654: elif [ "$GPG_TEST" = "disabled" ]; then
1655: echo -e "Test - En/Decryption skipped. (Testing disabled)"
1656: else
1657:
1658: GPG_TEST="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s)"
1659: function cleanup_gpgtest {
1660: echo -en "Cleanup - Delete '${GPG_TEST}_*'"
1661: rm ${GPG_TEST}_* 2>/dev/null && echo "(OK)" || echo "(FAILED)"
1662: }
1663:
1664: # signing enabled?
1665: if gpg_signing; then
1666: CMD_PARAM_SIGN="--sign --default-key ${GPG_KEY_SIGN} --passphrase-fd 0"
1667: CMD_MSG_SIGN="Sign with ${GPG_KEY_SIGN}"
1668: fi
1669:
1670: # using keys
1671: if [ ${#GPG_KEYS_ENC[@]} -gt 0 ]; then
1672:
1673: for KEY_ID in ${GPG_KEYS_ENC[@]}; do
1674: CMD_PARAMS="$CMD_PARAMS -r ${KEY_ID}"
1675: done
1676: # check encrypting
1677: CMD_MSG="Test - Encrypt to $(gpg_join_keyset ${GPG_KEYS_ENC[@]})${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}"
1678: run_cmd echo $(qw "${GPG_PW_SIGN:-$GPG_PW}") \| $GPG $CMD_PARAM_SIGN $CMD_PARAMS --batch --status-fd 1 $GPG_OPTS -o "${GPG_TEST}_ENC" -e "$ME_LONG"
1679:
1680: if [ "$CMD_ERR" != "0" ]; then
1681: KEY_NOTRUST=$(echo "$CMD_OUT"|awk '/^\[GNUPG:\] INV_RECP 10/ { print $4 }')
1682: [ -n "$KEY_NOTRUST" ] && HINT="Key '${KEY_NOTRUST}' seems to be untrusted. If you really trust this key try to
1683: 'gpg --edit-key $KEY_NOTRUST' and raise the trust level to ultimate. If you
1684: can trust all of your keys set GPG_OPTS='--trust-model always' in conf file."
1685: error_gpg_test "Encryption failed (Code $CMD_ERR).${CMD_OUT:+\n$CMD_OUT}" "$HINT"
1686: fi
1687:
1688: # check decrypting
1689: CMD_MSG="Test - Decrypt"
1690: gpg_key_decryptable || CMD_DISABLED="No matching secret key or GPG_PW not set."
1691: run_cmd echo $(qw "${GPG_PW}") \| $GPG --passphrase-fd 0 -o "${GPG_TEST}_DEC" --batch $GPG_OPTS -d "${GPG_TEST}_ENC"
1692:
1693: if [ "$CMD_ERR" != "0" ]; then
1694: error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}"
1695: fi
1696:
1697: # symmetric only
1698: else
1699: # check encrypting
1700: CMD_MSG="Test - Encryption with passphrase${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}"
1701: run_cmd echo $(qw "${GPG_PW}") \| $GPG $CMD_PARAM_SIGN --passphrase-fd 0 -o "${GPG_TEST}_ENC" --batch $GPG_OPTS -c "$ME_LONG"
1702:
1703: if [ "$CMD_ERR" != "0" ]; then
1704: error_gpg_test "Encryption failed.${CMD_OUT:+\n$CMD_OUT}"
1705: fi
1706:
1707: # check decrypting
1708: CMD_MSG="Test - Decryption with passphrase"
1709: run_cmd echo $(qw "${GPG_PW}") \| $GPG --passphrase-fd 0 -o "${GPG_TEST}_DEC" --batch $GPG_OPTS -d "${GPG_TEST}_ENC"
1710: if [ "$CMD_ERR" != "0" ]; then
1711: error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}"
1712: fi
1713: fi
1714:
1715: # compare original w/ decryptginal
1716: CMD_MSG="Test - Compare"
1717: [ -r "${GPG_TEST}_DEC" ] || CMD_DISABLED="Nothing to compare."
1718: run_cmd "test \"\$(cat '$ME_LONG')\" = \"\$(cat '${GPG_TEST}_DEC')\""
1719:
1720: if [ "$CMD_ERR" = "0" ]; then
1721: cleanup_gpgtest
1722: else
1723: error_gpg_test "Comparision failed.${CMD_OUT:+\n$CMD_OUT}"
1724: fi
1725:
1726: fi # end disabled
1727:
1728: ## an empty line
1729: #echo
1730:
1731: # Exclude file is needed, create it if necessary
1732: [ -f "$EXCLUDE" ] || touch "$EXCLUDE"
1733:
1734: # export only used keys, if bkp not already exists ######################################
1735: gpg_export_if_needed "${GPG_KEYS_ENC[@]} $(gpg_signing && echo $GPG_KEY_SIGN)"
1736:
1737:
1738: # command execution #####################################################################
1739:
1740: # urldecode url vars into plain text
1741: var_isset 'TARGET_URL_USER' && TARGET_URL_USER="$(url_decode "$TARGET_URL_USER")"
1742: var_isset 'TARGET_URL_PASS' && TARGET_URL_PASS="$(url_decode "$TARGET_URL_PASS")"
1743:
1744: # defined TARGET_USER&PASS vars replace their URL pendants
1745: # (double defs already dealt with)
1746: var_isset 'TARGET_USER' && TARGET_URL_USER="$TARGET_USER"
1747: var_isset 'TARGET_PASS' && TARGET_URL_PASS="$TARGET_PASS"
1748:
1749: # build target backend data depending on protocol
1750: case "$(tolower "${TARGET_URL_PROT%%:*}")" in
1751: 's3'|'s3+http')
1752: BACKEND_PARAMS="AWS_ACCESS_KEY_ID='${TARGET_URL_USER}' AWS_SECRET_ACCESS_KEY='${TARGET_URL_PASS}'"
1753: BACKEND_URL="${TARGET_URL_PROT}${TARGET_URL_HOSTPATH}"
1754: ;;
1755: 'cf+http')
1756: # respect possibly set cloudfile env vars
1757: var_isset 'CLOUDFILES_USERNAME' && TARGET_URL_USER="$CLOUDFILES_USERNAME"
1758: var_isset 'CLOUDFILES_APIKEY' && TARGET_URL_PASS="$CLOUDFILES_APIKEY"
1759: # add them to duplicity params
1760: var_isset 'TARGET_URL_USER' && \
1761: BACKEND_PARAMS="CLOUDFILES_USERNAME=$(qw "${TARGET_URL_USER}")"
1762: var_isset 'TARGET_URL_PASS' && \
1763: BACKEND_PARAMS="$BACKEND_PARAMS CLOUDFILES_APIKEY=$(qw "${TARGET_URL_PASS}")"
1764: BACKEND_URL="${TARGET_URL_PROT}${TARGET_URL_HOSTPATH}"
1765: # info on missing AUTH_URL
1766: if ! var_isset 'CLOUDFILES_AUTHURL'; then
1767: echo -e "INFO: No CLOUDFILES_AUTHURL defined (in conf).\n Will use default from python-cloudfiles (probably rackspace)."
1768: else
1769: BACKEND_PARAMS="$BACKEND_PARAMS CLOUDFILES_AUTHURL=$(qw "${CLOUDFILES_AUTHURL}")"
1770: fi
1771: ;;
1772: 'file'|'tahoe')
1773: BACKEND_URL="${TARGET_URL_PROT}${TARGET_URL_HOSTPATH}"
1774: ;;
1775: 'rsync')
1776: # everything in url (this backend does not support pass in env var)
1777: # this is obsolete from version 0.6.10 (buggy), hopefully in 0.6.11
1778: # print warning older version is detected
1779: var_isset 'TARGET_URL_USER' && BACKEND_CREDS="$(url_encode "${TARGET_URL_USER}")"
1780: if duplicity_version_lt 610; then
1781: warning "\
1782: Duplicity version '$DUPL_VERSION' does not support providing the password as
1783: env var for rsync backend. For security reasons you should consider to
1784: update to a version greater than '0.6.10' of duplicity."
1785: var_isset 'TARGET_URL_PASS' && BACKEND_CREDS="${BACKEND_CREDS}:$(url_encode "${TARGET_URL_PASS}")"
1786: else
1787: var_isset 'TARGET_URL_PASS' && BACKEND_PARAMS="FTP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
1788: fi
1789: var_isset 'BACKEND_CREDS' && BACKEND_CREDS="${BACKEND_CREDS}@"
1790: BACKEND_URL="${TARGET_URL_PROT}${BACKEND_CREDS}${TARGET_URL_HOSTPATH}"
1791: ;;
1792: *)
1793: # all protocols with username in url, only username is in url,
1794: # pass is env var for secúrity, url_encode username to protect special chars
1795: var_isset 'TARGET_URL_USER' &&
1796: BACKEND_CREDS="$(url_encode "${TARGET_URL_USER}")@"
1797: # sortout backends way to handle password
1798: case "$(tolower "${TARGET_URL_PROT%%:*}")" in
1799: 'imap'|'imaps')
1800: var_isset 'TARGET_URL_PASS' && BACKEND_PARAMS="IMAP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
1801: ;;
1802: 'ssh'|'sftp'|'scp')
1803: # ssh backend wants to be told that theres a pass to use
1804: var_isset 'TARGET_URL_PASS' && \
1805: DUPL_PARAMS="$DUPL_PARAMS --ssh-askpass" && \
1806: BACKEND_PARAMS="FTP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
1807: ;;
1808: *)
1809: # rest uses FTP_PASS var
1810: var_isset 'TARGET_URL_PASS' && \
1811: BACKEND_PARAMS="FTP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
1812: ;;
1813: esac
1814: BACKEND_URL="${TARGET_URL_PROT}${BACKEND_CREDS}${TARGET_URL_HOSTPATH}"
1815: ;;
1816: esac
1817: # protect eval from special chars in url (e.g. open ')' in password,
1818: # spaces in path, quotes) happens above in duplify() via quotewrap()
1819: SOURCE="$SOURCE"
1820: BACKEND_URL="$BACKEND_URL"
1821: EXCLUDE="$EXCLUDE"
1822:
1823: # convert cmds to array, lowercase for safety
1824: CMDS=( $(awk "BEGIN{ cmds=tolower(\"$cmds\"); gsub(/_/,\" \",cmds); print cmds }") )
1825:
1826: # run cmds
1827: for cmd in ${CMDS[*]};
1828: do
1829:
1830: ## init
1831: # raise index in cmd array for pre/post param
1832: var_isset 'CMD_NO' && CMD_NO=$((++CMD_NO)) || CMD_NO=0
1833: # save start time
1834: RUN_START=$(date_fix %s)$(nsecs)
1835: # user info
1836: echo; separator "Start running command $(echo $cmd|awk '$0=toupper($0)') at $(date_from_nsecs $RUN_START)"
1837:
1838: # get prev/nextcmd vars
1839: nextno=$(($CMD_NO+1))
1840: [ "$nextno" -lt "${#CMDS[@]}" ] && CMD_NEXT=${CMDS[$nextno]} || CMD_NEXT='END'
1841: prevno=$(($CMD_NO-1))
1842: [ "$prevno" -ge 0 ] && CMD_PREV=${CMDS[$prevno]} || CMD_PREV='START'
1843:
1844: case "$cmd" in
1845: pre|post)
1846: if [ "$cmd" == 'pre' ]; then
1847: script=$PRE
1848: else
1849: script=$POST
1850: fi
1851: # script execution in a subshell, protect us from failures/var overwrites
1852: ( run_script "$script" )
1853: ;;
1854: bkp)
1855: duplify -- "${dupl_opts[@]}" --exclude-globbing-filelist "$EXCLUDE" \
1856: "$SOURCE" "$BACKEND_URL"
1857: ;;
1858: incr)
1859: duplify incr -- "${dupl_opts[@]}" --exclude-globbing-filelist "$EXCLUDE" \
1860: "$SOURCE" "$BACKEND_URL"
1861: ;;
1862: full)
1863: duplify full -- "${dupl_opts[@]}" --exclude-globbing-filelist "$EXCLUDE" \
1864: "$SOURCE" "$BACKEND_URL"
1865: ;;
1866: verify)
1867: duplify verify -- "${dupl_opts[@]}" --exclude-globbing-filelist "$EXCLUDE" \
1868: "$BACKEND_URL" "$SOURCE"
1869: ;;
1870: list)
1871: # time param exists since 0.5.10+
1872: TIME="${ftpl_pars[0]:-now}"
1873: duplify list-current-files -- -t "$TIME" "${dupl_opts[@]}" "$BACKEND_URL"
1874: ;;
1875: cleanup)
1876: duplify cleanup -- "${dupl_opts[@]}" "$BACKEND_URL"
1877: ;;
1878: purge)
1879: MAX_AGE=${ftpl_pars[0]:-$MAX_AGE}
1880: [ -z "$MAX_AGE" ] && error " Missing parameter <max_age>. Can be set in profile or as command line parameter."
1881:
1882: duplify remove-older-than "${MAX_AGE}" \
1883: -- "${dupl_opts[@]}" "$BACKEND_URL"
1884: ;;
1885: purge-full)
1886: MAX_FULL_BACKUPS=${ftpl_pars[0]:-$MAX_FULL_BACKUPS}
1887: [ -z "$MAX_FULL_BACKUPS" ] && error " Missing parameter <max_full_backups>. Can be set in profile or as command line parameter."
1888:
1889: duplify remove-all-but-n-full "${MAX_FULL_BACKUPS}" \
1890: -- "${dupl_opts[@]}" "$BACKEND_URL"
1891: ;;
1892: purge-incr)
1893: MAX_FULLS_WITH_INCRS=${ftpl_pars[0]:-$MAX_FULLS_WITH_INCRS}
1894: [ -z "$MAX_FULLS_WITH_INCRS" ] && error " Missing parameter <max_fulls_with_incrs>. Can be set in profile or as command line parameter."
1895:
1896: duplify remove-all-inc-of-but-n-full "${MAX_FULLS_WITH_INCRS}" \
1897: -- "${dupl_opts[@]}" "$BACKEND_URL"
1898: ;;
1899: restore)
1900: OUT_PATH="${ftpl_pars[0]}"; TIME="${ftpl_pars[1]:-now}";
1901: [ -z "$OUT_PATH" ] && error " Missing parameter target_path for restore.
1902:
1903: Hint:
1904: Syntax is -> $ME <profile> restore <target_path> [<age>]"
1905:
1906: duplify -- -t "$TIME" "${dupl_opts[@]}" "$BACKEND_URL" "$OUT_PATH"
1907: ;;
1908: fetch)
1909: IN_PATH="${ftpl_pars[0]}"; OUT_PATH="${ftpl_pars[1]}";
1910: TIME="${ftpl_pars[2]:-now}";
1911: ( [ -z "$IN_PATH" ] || [ -z "$OUT_PATH" ] ) && error " Missing parameter <src_path> or <target_path> for fetch.
1912:
1913: Hint:
1914: Syntax is -> $ME <profile> fetch <src_path> <target_path> [<age>]"
1915:
1916: # duplicity 0.4.7 doesnt like cmd restore in combination with --file-to-restore
1917: duplify -- --restore-time "$TIME" "${dupl_opts[@]}" \
1918: --file-to-restore "$IN_PATH" "$BACKEND_URL" "$OUT_PATH"
1919: ;;
1920: status)
1921: duplify collection-status -- "${dupl_opts[@]}" "$BACKEND_URL"
1922: ;;
1923: *)
1924: warning "Unknown command '$cmd'."
1925: ;;
1926: esac
1927:
1928: CMD_ERR=$?
1929: RUN_END=$(date_fix %s)$(nsecs) ; RUNTIME=$(( $RUN_END - $RUN_START ))
1930:
1931: # print message on error; set error code
1932: if [ "$CMD_ERR" -ne 0 ]; then
1933: error_print "$(datefull_from_nsecs $RUN_END) Task '$(echo $cmd|awk '$0=toupper($0)')' failed with exit code '$CMD_ERR'."
1934: FTPL_ERR=1
1935: fi
1936:
1937: separator "Finished state $(error_to_string $CMD_ERR) at $(date_from_nsecs $RUN_END) - \
1938: Runtime $(printf "%02d:%02d:%02d.%03d" $((RUNTIME/1000000000/60/60)) $((RUNTIME/1000000000/60%60)) $((RUNTIME/1000000000%60)) $((RUNTIME/1000000%1000)) )"
1939:
1940: done
1941:
1942: exit ${FTPL_ERR}
