Difference between revisions of "Code"
Jump to navigation
Jump to search
Line 16: | Line 16: | ||
===Latest Development Snapshot=== | ===Latest Development Snapshot=== | ||
− | mod time <nowiki>#$system:stat -c%y tmp/duply.sh| awk '{print $1}'#</nowiki> | + | mod time <!--nowiki>#$system:stat -c%y tmp/duply.sh| awk '{print $1}'#</nowiki--> |
− | <include src="stat -c%y tmp/duply.sh" shell=true nocache/> | + | <include src="stat -c%y tmp/duply.sh| awk '{print $1}'" shell=true nocache/> |
plain/text -> http://duply.net/tmp/duply.sh | plain/text -> http://duply.net/tmp/duply.sh |
Revision as of 15:44, 12 August 2019
Contents
duply sourcecode
duply's svn repository is not the most recent source for the code. Usually the svn only gets updated on releases. But not always. Development snapshots are taken daily. These are not guaranteed to work, but might contain unreleased fixes. Check the snapshot's changelog header accordingly.
SVN access
Browse the repository
https://sourceforge.net/p/ftplicity/code/HEAD/tree/duply/trunk/
SVN access
svn co https://svn.code.sf.net/p/ftplicity/code/duply/trunk duply
SVN Log
https://sourceforge.net/p/ftplicity/code/commit_browser
Latest Development Snapshot
mod time
ERROR in secure-include.php::ef_include_get_errors (line 976) - Not allowed to include remote URLs!
plain/text -> http://duply.net/tmp/duply.sh
1 #!/usr/bin/env bash 2 # 3 ################################################################################ 4 # duply (grown out of ftplicity), is a shell front end to duplicity that # 5 # simplifies the usage by managing settings for backup jobs in profiles. # 6 # It supports executing multiple commands in a batch mode to enable single # 7 # line cron entries and executes pre/post backup scripts. # 8 # Since version 1.5.0 all duplicity backends are supported. Hence the name # 9 # changed from ftplicity to duply. # 10 # See http://duply.net or http://ftplicity.sourceforge.net/ for more info. # 11 # (c) 2006 Christiane Ruetten, Heise Zeitschriften Verlag, Germany # 12 # (c) 2008-2023 Edgar Soldin (changes since version 1.3) # 13 ################################################################################ 14 # LICENSE: # 15 # This program is licensed under GPLv2. # 16 # Please read the accompanying license information in gpl-2.0.txt. # 17 ################################################################################ 18 # TODO/IDEAS/KNOWN PROBLEMS: 19 # - possibility to restore time frames (incl. deleted files) 20 # realizable by listing each backup and restore from 21 # oldest to the newest, problem: not performant 22 # - search file in all backups function and show available 23 # versions with backups date (list old avail since 0.6.06) 24 # - edit profile opens conf file in vi 25 # - implement log-fd interpretation 26 # - add a duplicity option check against the options pending 27 # deprecation since 0.5.10 namely --time-separator 28 # --short-filenames 29 # --old-filenames 30 # - add 'exclude_<command>' list usage e.g. exclude_verify 31 # - featreq 25: a download/install duplicity option 32 # - import/export profile from/to .tgz function !!! 33 # - remove url_encode, test for invalid chars n throw error instead 34 # 35 # CHANGELOG: 36 # 2.5.2 (30.11.2023) 37 # - bug #139: "ampersand (&) in gpg passphrase breaks gpg tests" 38 # 39 # 2.5.1 (4.10.2023) 40 # - quotewrap only strings with quotes ('") or spaces from now on 41 # - add --verbosity only if set in profile conf 42 # - bugfix #138: fix quoting when filtering params, thx Eric 43 # - bugfix #137: relax version parsing regex 44 # 45 # 2.5.0 (25.09.2023) 46 # - bugfix #136: "not compatible with duplicity 2.x", thx tengel, lds, Rhomeo 47 # check for duplicity 2.1+ (2.0 broke implied commands), 48 # command line ui changed incompatibly 49 # - filter in/excludes more strictly for more duplicity actions now 50 # - replace '--file-to-restore' with '--path-to-restore' 51 # - filter backup only params now 52 # 53 # 2.4.3 (05.05.2023) 54 # - bugfix #134: workaround bash 4.2 and earlier read bug (thx Tavio Wong) 55 # 56 # 2.4.2 (19.01.2023) 57 # - featreq #55: change to purgeAuto in systemd unit files (thx B.Foresman) 58 # - featreq #56: systemd files should go in /etc, not /lib (thx B.Foresman) 59 # - bugfix #133: read -N not available on macOS (thx Peter Torelli) 60 # 61 # 2.4.1 (09.09.2022) 62 # - fixup duplicity links, moved to http://duplicity.us 63 # - bugfix: duply hangs on awk version detection on OpenBSD (thx phthomas137) 64 # 65 # 2.4 (06.04.2022) 66 # - bugfix #127: date_from_nsecs ignores format string 67 # - bugfix #116: separators print date now too 68 # - featreq #48: add purgeAuto command (see man page) 69 # - replaced tab indents with 2spaces everywhere 70 # - bugfix #129,131,132: duply stumbles over 'python -s' shebang, 71 # python interpreter parse failed if duplicity is a snap app 72 # - bugfix #130: duplicity version check failed "gpg: WARNING: ..." 73 # - version output, always print PYTHONPATH, if interpreter was determined 74 # - update python references to python3 75 # 76 # 2.3.1 (11.02.2021) 77 # - bugfix 123: symmetric encryption errs out, asks for '' private key 78 # 79 # 2.3 (30.12.2020) 80 # - don't import whole key pair anymore if only pub/sec is requested 81 # - gpg import routine informs on missing key files in profile now 82 # - add check/import needed secret key for decryption 83 # - featreq 50: Disable GPG key backups, implemented/added settings 84 # GPG_IMPORT/GPG_EXPORT='disabled' to conf template 85 # 86 # 2.2.2 (24.02.2020) 87 # - bugfix 120: Failures in "Autoset trust of key" during restore 88 # because of gpg2.2 fingerprint output change 89 # 90 # 2.2.1 (22.01.2020) 91 # - featreq 46: Example systemd units & Howto, courtesy of Jozef Riha 92 # - featreq 47: Clarify message about keeping the profile, also by Jozef Riha 93 # - fix abbreviation spelling of 'e.g.' 94 # 95 # 2.2 (30.12.2018) 96 # - featreq 44: implement grouping for batch commands 97 # new separators are [] (square brackets) or groupIn/groupOut 98 # command 'backup' translates now to [pre_bkp_post] to be skipped as 99 # one block in case a condition was set in the batch instruction 100 # 101 # 2.1 (23.07.2018) 102 # - be more verbose when duplicity version detection fails 103 # - using info shows python binary's path for easier identification now 104 # - reworked python interpreter handling, it's either 105 # configured per PYTHON var 106 # unconfigured, parsed from duplicity shebang 107 # or set to current duplicity default 'python2' (was 'python' until now) 108 # - do not quotewrap strings because of slashes (e.g. paths) anymore 109 # - bugfix: improved in/exclude stripping from conf DUPL_PARAMS 110 # 111 # 2.0.4 (20.02.2018) 112 # - bugfix 114: "duply usage is not current" wrt. purgeFull/Incr 113 # - bugfix 115: typo in error message - "Not GPG_KEY entries" should be "No" 114 # - bugfix 117: no duply_ prefix when ARCH_DIR is set in conf 115 # - bugfix debian 882159: duply: occasionally shows negative runtimes 116 # 117 # 2.0.3 (29.08.2017) 118 # - bugfix: "line 2231: CMDS: bad array subscript" 119 # - bugfix 112: "env: illegal option -- u" on MacOSX 120 # 121 # 2.0.2 (23.05.2017) 122 # - bugfix: never insert creds into file:// targets 123 # - bugfix: avail profiles hint sometimes shortend the names by one char 124 # - bugfix 108: CMD_NEXT variable should ignore conditional commands (and, or) 125 # - export condition before/after next/prev command as CND_PREV,CND_NEXT now 126 # - bugfix 97: Unknown command should be ERROR, not WARNING 127 # 128 # 2.0.1 (16.11.2016) 129 # - bugfix 104: Duply 2.0 sets wrong archive dir, --name was always 'duply_' 130 # 131 # 2.0 (27.10.2016) 132 # made this a major version change, as we broke backward compatibility anyway 133 # (see last change in v1.10). got complaints that rightfully pointed out 134 # that should only come w/ a major version change. so, here we go ;) 135 # if your backend stops working w/ this version create a new profile and 136 # export the env vars needed as described in the comments of the conf file 137 # directly above the SOURCE setting. 138 # Changes: 139 # - making sure multi spaces in TARGET survive awk processing 140 # - new env var PROFILE exported to scripts 141 # - fix 102: expose a unique timestamp variable for pre/post scripts 142 # actually a featreq. exporting RUN_START nanosec unix timestamp 143 # - fix 101: GPG_AGENT_INFO is 'bogus' (thx Thomas Harning Jr.) 144 # - fix 96: duply cannot handle two consecutive spaces in paths 145 # 146 # 1.11.3 (29.5.2016) 147 # - fix wrong "WARNING: No running gpg-agent ..." when sign key was not set 148 # 149 # 1.11.2 (11.2.2016) 150 # - fix "gpg: unsafe" version print out 151 # - bugfix 91: v1.11 [r47] broke asymmetric encryption when using GPG_KEYS_ENC 152 # - bugfix 90: S3: TARGET_USER/PASS have no effect, added additional 153 # documentation about needed env vars to template conf file 154 # 155 # 1.11.1 (18.12.2015) 156 # - bugfix 89: "Duply has trouble with PYTHON-interpreter" on OSX homebrew 157 # - reverted duply's default PYTHON to 'python' 158 # 159 # 1.11 (24.11.2015) 160 # - remove obsolete --ssh-askpass routine 161 # - add PYTHON conf var to allow global override of used python interpreter 162 # - enforced usage of "python2" in PATH as default interpreter for internal 163 # use _and_ to run duplicity (setup.py changed the shebang to the fixed 164 # path /usr/bin/python until 0.7.05, which we circumvent this way) 165 # - featreq 36: support gpg-connect-agent as a means to detect if an agent is 166 # running (thx Thomas Harning Jr.), used gpg-agent for detection though 167 # - quotewrapped run_cmd parameters to protect it from spaces e.g. in TMP path 168 # - key export routine respects gpg-agent usage now 169 # 170 # 1.10.1 (19.8.2015) 171 # - bugfix 86: Duply+Swift outputs warning 172 # - bugfix 87: Swift fails without BACKEND_URL 173 # 174 # 1.10 (31.7.2015) 175 # - featreq 37: busybox issues - fix awk, grep version detection, 176 # fix grep failure because --color=never switch is unsupported 177 # (thx Thomas Harning Jr. for reporting and helping to debug/fix it) 178 # - bugfix 81: --exclude-globbing-filelist is deprecated since 0.7.03 179 # (thx Joachim Wiedorn, also for maintaining the debian package) 180 # - implemented base-/dirname as bash functions 181 # - featreq 31 " Support for duplicity Azure backend " - ignored a 182 # contributed patch by Scott McKenzie and instead opted for removing almost 183 # all code that deals with special env vars required by backends. 184 # adding and modifying these results in too much overhead so i dropped this 185 # feature. the future alternative for users is to consult the duplicity 186 # manpage and add the needed export definitions to the conf file. 187 # appended a commented example to the template conf below the auth section. 188 # 189 # 1.9.2 (21.6.2015) 190 # - bugfix: exporting keys with gpg2.1 works now (thx Philip Jocks) 191 # - documented GPG_OPTS needed for gpg2.1 to conf template (thx Troy Engel) 192 # - bugfix 82: GREP_OPTIONS=--color=always disrupted time calculation 193 # - added GPG conf var (see conf template for details) 194 # - added grep version output as it is an integral needed binary 195 # - added PYTHONPATH printout in version output 196 # 197 # 1.9.1 (13.10.2014) 198 # - export CMD_ERR now for scripts to detect if CMD_PREV failed/succeeded 199 # - bugfix: CMD_PREV contained command even if it was skipped 200 # 201 # 1.9.0 (24.8.2014) 202 # - bugfix: env vars were not exported when external script was executable 203 # - rework GPG_KEY handling, allow virtually anything now (uid, keyid etc.) 204 # see gpg manpage, section "How to specify a user ID" 205 # let gpg complain when the delivered values are invalid for whatever reason 206 # - started to rework tmp space checking, exposed folder & writable check 207 # TODO: reimplement enough file space available checking 208 # 209 # 1.8.0 (13.7.2014) 210 # - add command verifyPath to expose 'verify --file-to-restore' action 211 # - add time parameter support to verify command 212 # - add section time formats to usage output 213 # 214 # 1.7.4 (24.6.2014) 215 # - remove ubuntu one support, service is discontinued 216 # - featreq 31: add authenticated swift (contributed by Justus Seifert) 217 # 218 # 1.7.3 (3.4.2014) 219 # - bugfix: test routines, gpg2 asked for passphrase although GPG_PW was set 220 # 221 # 1.7.2 (1.4.2014 "April,April") 222 # - bugfix: debian Bug#743190 "duply no longer allows restoration without 223 # gpg passphrase in conf file" 224 # GPG_AGENT_INFO env var is now needed to trigger --use-agent 225 # - bugfix: gpg keyenc test routines didn't work if GPG_PW was not set 226 # 227 # 1.7.1 (30.3.2014) 228 # - bugfix: purge-* commands renamed to purgeFull, purgeIncr due to 229 # incompatibility with new minus batch separator 230 # 231 # 1.7.0 (20.3.2014) 232 # - disabled gpg key id plausibility check, too many valid possibilities 233 # - featreq 7 "Halt if precondition fails": 234 # added and(+), or(-) batch command(separator) support 235 # - featreq 26 "pre/post script with shebang line": 236 # if a script is flagged executable it's executed in a subshell 237 # now as opposed to sourced to bash, which is the default 238 # - bugfix: do not check if dpbx, swift credentials are set anymore 239 # - bugfix: properly escape profile name, archdir if used as arguments 240 # - add DUPL_PRECMD conf setting for use with e.g. trickle 241 # 242 # 1.6.0 (1.1.2014) 243 # - support gs backend 244 # - support dropbox backend 245 # - add gpg-agent support to gpg test routines 246 # - autoenable --use-agent if passwords were not defined in config 247 # - GPG_OPTS are now honored everywhere, keyrings or complete gpg 248 # homedir can thus be configured to be located anywhere 249 # - always import both secret and public key if avail from config profile 250 # - new explanatory comments in initial exclude file 251 # - bugfix 7: Duply only imports one key at a time 252 # 253 # 1.5.11 (19.07.2013) 254 # - purge-incr command for remove-all-inc-of-but-n-full feature added 255 # patch provided by Moritz Augsburger, thanks! 256 # - documented version command in man page 257 # 258 # 1.5.10 (26.03.2013) 259 # - minor indent and documentation fixes 260 # - bugfix: exclude filter failed on ubuntu, mawk w/o posix char class support 261 # - bugfix: fix url_decoding generally and for python3 262 # - bugfix 3609075: wrong script results in status line (thx David Epping) 263 # 264 # 1.5.9 (22.11.2012) 265 # - bugfix 3588926: filter --exclude* params for restore/fetch ate too much 266 # - restore/fetch now also ignores --include* or --exclude='foobar' 267 # 268 # 1.5.8 (26.10.2012) 269 # - bugfix 3575487: implement proper cloud files support 270 # 271 # 1.5.7 (10.06.2012) 272 # - bugfix 3531450: Cannot use space in target URL (file:///) anymore 273 # 274 # 1.5.6 (24.5.2012) 275 # - commands purge, purge-full have no default value anymore for security 276 # reasons; instead max value can be given via cmd line or must be set 277 # in profile; else an error is shown. 278 # - minor man page modifications 279 # 280 # versioning scheme will be simplified to [major].[minor].[patch] version 281 # with the next version raise 282 # 283 # 1.5.5.5 (4.2.2012) 284 # - bugfix 3479605: SEL context confused profile folder's permission check 285 # - colon ':' in url passphrase got ignored, added python driven url_decoding 286 # for user & pass to better process special chars 287 # 288 # 1.5.5.4 (16.10.2011) 289 # - bugfix 3421268: SFTP passwords from conf ignored and always prompted for 290 # - add support for separate sign passphrase (needs duplicity 0.6.14+) 291 # 292 # 1.5.5.3 (1.10.2011) 293 # - bugfix 3416690: preview threw echo1 error 294 # - fix unknown cmds error usage & friends if more than 2 params were given 295 # 296 # 1.5.5.2 (23.9.2011) 297 # - bugfix 3409643: ssh key auth did ask for passphrase (--ssh-askpass ?) 298 # - bugfix: mawk does not support \W and did not split multikey definitions 299 # - all parameters should survive single (') and double (") quotes now 300 # 301 # 1.5.5.1 (7.6.2011) 302 # - featreq 3311881: add ftps as supported by duplicity 0.6.13 (thx mape2k) 303 # - bugfix 3312208: signing detection broke symmetric gpg test routine 304 # 305 # 1.5.5 (2.5.2011) 306 # - bugfix: fetch problem with space char in path, escape all params 307 # containing non word chars 308 # - list available profiles, if given profile cannot be found 309 # - added --use-agent configuration hint 310 # - bugfix 3174133: --exclude* params in conf DUPL_PARAMS broke 311 # fetch/restore 312 # - version command now prints out 'using installed' info 313 # - featreq 3166169: autotrust imported keys, based on code submitted by 314 # Martin Ellis - imported keys are now automagically trusted ultimately 315 # - new txt2man feature to create manpages for package maintainers 316 # 317 # 1.5.4.2 (6.1.2011) 318 # - new command changelog 319 # - bugfix 3109884: freebsd awk segfaulted on printf '%*', use print again 320 # - bugfix: freebsd awk hangs on 'awk -W version' 321 # - bugfix 3150244: mawk does not know '--version' 322 # - minor help text improvements 323 # - new env vars CMD_PREV,CMD_NEXT replacing CMD env var for scripts 324 # 325 # 1.5.4.1 (4.12.2010) 326 # - output awk, python, bash version now in prolog 327 # - shebang uses /usr/bin/env now for freebsd compatibility, 328 # bash not in /bin/bash 329 # - new --disable-encryption parameter, 330 # to override profile encr settings for one run 331 # - added exclude-if-present setting to conf template 332 # - bug 3126972: GPG_PW only needed for signing/symmetric encryption 333 # (even though duplicity still needs it) 334 # 335 # 1.5.4 (15.11.2010) 336 # - as of 1.5.3 already, new ARCH_DIR config option 337 # - multiple key support 338 # - ftplicity-Feature Requests-2994929: separate encryption and signing key 339 # - key signing of symmetric encryption possible (duplicity patch committed) 340 # - gpg tests disable switch 341 # - gpg tests now previewable and more intelligent 342 # 343 # 1.5.3 (1.11.2010) 344 # - bugfix 3056628: improve busybox compatibility, grep did not have -m param 345 # - bugfix 2995408: allow empty password for PGP key 346 # - bugfix 2996459: Duply erroneously escapes '-' symbol in username 347 # - url_encode function is now pythonized 348 # - rsync uses FTP_PASSWORD now if duplicity 0.6.10+ , else issue warning 349 # - feature 3059262: Make pre and post aware of parameters, 350 # internal parameters + CMD of pre or post 351 # 352 # 1.5.2.3 (16.4.2010) 353 # - bugfix: date again, should now work virtually anywhere 354 # 355 # 1.5.2.2 (3.4.2010) 356 # - minor bugfix: duplicity 0.6.8b version string now parsable 357 # - added INSTALL.txt 358 # 359 # 1.5.2.1 (23.3.2010) 360 # - bugfix: date formatting is awked now and should work on all platforms 361 # 362 # 1.5.2 (2.3.2010) 363 # - bugfix: errors print to STD_ERR now, failed tasks print an error message 364 # - added --name=duply_<profile> for duplicity 0.6.01+ to name cache folder 365 # - simplified & cleaned profileless commands, removed second instance 366 # - generalized separator time routines 367 # - added support for --no-encryption (GPG_KEY='disabled'), see conf examples 368 # - minor fixes 369 # 370 # 1.5.1.5 (5.2.2010) 371 # - bugfix: added special handling of credentials for rsync, imap(s) 372 # 373 # 1.5.1.4 (7.1.2010) 374 # - bugfix: nsecs defaults now to zeroes if date does not deliver [0-9]{9} 375 # - check if ncftp binary is available if url protocol is ftp 376 # - bugfix: duplicity output is now printed to screen directly to resolve 377 # 'mem alloc problem' bug report 378 # - bugfix: passwords will not be in the url anymore to solve the 'duply shows 379 # sensitive data in process listing' bug report 380 # 381 # 1.5.1.3 (24.12.2009) 'merry xmas' 382 # - bugfix: gpg pass now apostrophed to allow space and friends 383 # - bugfix: credentials are now url encoded to allow special chars in them 384 # a note about url encoding has been added to the conf template 385 # 386 # 1.5.1.2 (1.11.2009) 387 # - bugfix: open parenthesis in password broke duplicity execution 388 # - bugfix: ssh/scp backend does not always need credentials e.g. key auth 389 # 390 # 1.5.1.1 (21.09.2009) 391 # - bugfix: fixed s3[+http] TARGET_PASS not needed routine 392 # - bugfix: TYPO in duply 1.5.1 prohibited the use of /etc/duply 393 # see https://sourceforge.net/tracker/index.php?func=detail& 394 # aid=2864410&group_id=217745&atid=1041147 395 # 396 # 1.5.1 (21.09.2009) - duply (fka. ftplicity) 397 # - first things first: ftplicity (being able to support all backends since 398 # some time) will be called duply (fka. ftplicity) from now on. The addendum 399 # is for the time being to circumvent confusion. 400 # - bugfix: exit code is 1 (error) not 0 (success), if at least on duplicity 401 # command failed 402 # - s3[+http] now supported natively by translating user/pass to access_key/ 403 # secret_key environment variables needed by duplicity s3 boto backend 404 # - bugfix: additional output lines do not confuse version check anymore 405 # - list command supports now age parameter (patch by stefan on feature 406 # request tracker) 407 # - bugfix: option/param pairs are now correctly passed on to duplicity 408 # - bugfix: s3[+http] needs no TARGET_PASS if command is read only 409 # 410 # 1.5.0.2 (31.07.1009) 411 # - bugfix: insert password in target url didn't work with debian mawk 412 # related to previous bug report 413 # 414 # 1.5.0.1 (23.07.2009) 415 # - bugfix: gawk gensub dependency raised an error on debian's default mawk 416 # replaced with match/substr command combination (bug report) 417 # https://sf.net/tracker/?func=detail&atid=1041147&aid=2825388& 418 # group_id=217745 419 # 420 # 1.5.0 (01.07.2009) 421 # - removed ftp limitation, all duplicity backends should work now 422 # - bugfix: date for separator failed on openwrt busybox date, added a 423 # detecting workaround, milliseconds are not available w/ busybox date 424 # 425 # 1.4.2.1 (14.05.2009) 426 # - bugfix: free temp space detection failed with lvm, fixed awk parse routine 427 # 428 # 1.4.2 (22.04.2009) 429 # - gpg keys are now exported as gpgkey.[id].asc , the suffix reflects the 430 # armored ascii nature, the id helps if the key is switched for some reason 431 # im/export routines are updated accordingly (import is backward compatible 432 # to the old profile/gpgkey files) 433 # - profile argument is treated as path if it contains slashes 434 # (for details see usage) 435 # - non-ftplicity options (all but --preview currently) are now passed 436 # on to duplicity 437 # - removed need for stat in secure_conf, it is ls based now 438 # - added profile folder readable check 439 # - added gpg version & home info output 440 # - awk utility availability is now checked, because it was mandatory already 441 # - tmp space is now checked on writability and space requirement 442 # test fails on less than 25MB or configured $VOLSIZE, 443 # test warns if there is less than two times $VOLSIZE because 444 # that's required for --asynchronous-upload option 445 # - gpg functionality is tested now before executing duplicity 446 # test drive contains encryption, decryption, comparison, cleanup 447 # this is meant to detect non trusted or other gpg errors early 448 # - added possibility of doing symmetric encryption with duplicity 449 # set GPG_KEY="" or simply comment it out 450 # - added hints in config template on the depreciation of 451 # --short-filenames, --time-separator duplicity options 452 # 453 # new versioning scheme 1.4.2b => 1.4.2, 454 # beta b's are replaced by a patch count number e.g. 1.4.2.1 will be assigned 455 # to the first bug fixing version and 1.4.2.2 to the second and so on 456 # also the releases will now have a release date formatted (Day.Month.Year) 457 # 458 # 1.4.1b1 - bugfix: ftplicity changed filesystem permission of a folder 459 # named exactly as the profile if existing in executing dir 460 # - improved plausibility checking of config and profile folder 461 # - secure_conf only acts if needed and prints a warning now 462 # 463 # 1.4.1b - introduce status (duplicity collection-status) command 464 # - pre/post script output printed always now, not only on errors 465 # - new config parameter GPG_OPTS to pass gpg options 466 # added examples & comments to profile template conf 467 # - reworked separator times, added duration display 468 # - added --preview switch, to preview generated command lines 469 # - disabled MAX_AGE, MAX_FULL_BACKUPS, VERBOSITY in generated 470 # profiles because they have reasonable defaults now if not set 471 # 472 # 1.4.0b1 - bugfix: incr forces incremental backups on duplicity, 473 # therefore backup translates to pre_bkp_post now 474 # - bugfix: new command bkp, which represents duplicity's 475 # default action (incr or full if full_if_older matches 476 # or no earlier backup chain is found) 477 # 478 # new versioning scheme 1.4 => 1.4.0, added new minor revision number 479 # this is meant to slow down the rapid version growing but still keep 480 # versions cleanly separated. 481 # only additional features will raise the new minor revision number. 482 # all releases start as beta, each bugfix release will raise the beta 483 # count, usually new features arrive before a version 'ripes' to stable 484 # 485 # 1.4.0b 486 # 1.4b - added startup info on version, time, selected profile 487 # - added time output to separation lines 488 # - introduced: command purge-full implements duplicity's 489 # remove-all-but-n-full functionality (patch by unknown), 490 # uses config variable $MAX_FULL_BACKUPS (default = 1) 491 # - purge config var $MAX_AGE defaults to 1M (month) now 492 # - command full does not execute pre/post anymore 493 # use batch command pre_full_post if needed 494 # - introduced batch mode cmd1_cmd2_etc 495 # (in turn removed the bvp command) 496 # - unknown/undefined command issues a warning/error now 497 # - bugfix: version check works with 0.4.2 and older now 498 # 1.3b3 - introduced pre/post commands to execute/debug scripts 499 # - introduced bvp (backup, verify, purge) 500 # - bugfix: removed need for awk gensub, now mawk compatible 501 # 1.3b2 - removed pre/post need executable bit set 502 # - profiles now under ~/.ftplicity as folders 503 # - root can keep profiles in /etc/ftplicity, folder must be 504 # created by hand, existing profiles must be moved there 505 # - removed ftplicity in path requirement 506 # - bugfix: bash < v.3 did not know '=~' 507 # - bugfix: purge works again 508 # 1.3 - introduces multiple profiles support 509 # - modified some script errors/docs 510 # - reordered gpg key check import routine 511 # - added 'gpg key id not set' check 512 # - added error_gpg (adds how to setup gpg key howto) 513 # - bugfix: duplicity 0.4.4RC4+ parameter syntax changed 514 # - duplicity_version_check routine introduced 515 # - added time separator, shortnames, volsize, full_if_older 516 # duplicity options to config file (inspired by stevie 517 # from http://weareroot.de) 518 # 1.1.1 - bugfix: encryption reactivated 519 # 1.1 - introduced config directory 520 # 1.0 - first release 521 ################################################################################ 522 523 # utility functions overriding binaries 524 525 # wrap grep to override possible env set GREP_OPTIONS=--color=always 526 function grep { 527 command env "GREP_OPTIONS=" grep "$@" 528 } 529 530 # implement basename in plain bash 531 function basename { 532 local stripped="${1%/}" 533 echo "${stripped##*/}" 534 } 535 536 # implement dirname in plain bash 537 function dirname { 538 echo ${1%/*} 539 } 540 541 # implement basic which in plain bash 542 function which { 543 type -p "$@" 544 } 545 546 # check availability of executables via file name or file paths 547 function lookup { 548 local bin="$1" 549 # look for file names in path via bash hash OR 550 # look for executables at given relative/absolute location 551 ( [ "${bin##*/}" == "$bin" ] && hash "$bin" 2>/dev/null ) || [ -x "$bin" ] 552 } 553 554 # important definitions ####################################################### 555 556 ME_LONG="$0" 557 ME="$(basename $0)" 558 ME_NAME="${ME%%.*}" 559 ME_VERSION="2.5.2" 560 ME_WEBSITE="https://duply.net" 561 562 # default config values 563 DEFAULT_SOURCE='/path/of/source' 564 DEFAULT_TARGET='scheme://user[:password]@host[:port]/[/]path' 565 DEFAULT_TARGET_USER='_backend_username_' 566 DEFAULT_TARGET_PASS='_backend_password_' 567 DEFAULT_GPG='gpg' 568 DEFAULT_GPG_KEY='_KEY_ID_' 569 DEFAULT_GPG_PW='_GPG_PASSWORD_' 570 571 # function definitions ########################## 572 573 function set_config { # sets global config vars 574 local CONFHOME_COMPAT="$HOME/.ftplicity" 575 local CONFHOME="$HOME/.duply" 576 local CONFHOME_ETC_COMPAT="/etc/ftplicity" 577 local CONFHOME_ETC="/etc/duply" 578 579 # confdir can be delivered as path (must contain /) 580 if [ `echo $FTPLCFG | grep /` ] ; then 581 CONFDIR=$(readlink -f $FTPLCFG 2>/dev/null || \ 582 ( echo $FTPLCFG|grep -v '^/' 1>/dev/null 2>&1 \ 583 && echo $(pwd)/${FTPLCFG} ) || \ 584 echo ${FTPLCFG}) 585 # or DEFAULT in home/.duply folder (NEW) 586 elif [ -d "${CONFHOME}" ]; then 587 CONFDIR="${CONFHOME}/${FTPLCFG}" 588 # or in home/.ftplicity folder (OLD) 589 elif [ -d "${CONFHOME_COMPAT}" ]; then 590 CONFDIR="${CONFHOME_COMPAT}/${FTPLCFG}" 591 warning_oldhome "${CONFHOME_COMPAT}" "${CONFHOME}" 592 # root can put profiles under /etc/duply (NEW) if path exists 593 elif [ -d "${CONFHOME_ETC}" ] && [ "$EUID" -eq 0 ]; then 594 CONFDIR="${CONFHOME_ETC}/${FTPLCFG}" 595 # root can keep profiles under /etc/ftplicity (OLD) if path exists 596 elif [ -d "${CONFHOME_ETC_COMPAT}" ] && [ "$EUID" -eq 0 ]; then 597 CONFDIR="${CONFHOME_ETC_COMPAT}/${FTPLCFG}" 598 warning_oldhome "${CONFHOME_ETC_COMPAT}" "${CONFHOME_ETC}" 599 # hmm no profile folder there, then use default for error later 600 else 601 CONFDIR="${CONFHOME}/${FTPLCFG}" # continue, will fail later in main 602 fi 603 604 # remove trailing slash, get profile name etc. 605 CONFDIR="${CONFDIR%/}" 606 PROFILE="${CONFDIR##*/}" 607 CONF="$CONFDIR/conf" 608 PRE="$CONFDIR/pre" 609 POST="$CONFDIR/post" 610 EXCLUDE="$CONFDIR/exclude" 611 KEYFILE="$CONFDIR/gpgkey.asc" 612 } 613 614 function version_info { # print version information 615 cat <<END 616 $ME_NAME version $ME_VERSION 617 ($ME_WEBSITE) 618 END 619 } 620 621 function version_info_using { 622 cat <<END 623 $(version_info) 624 625 $(using_info) 626 END 627 } 628 629 function using_info { 630 # init needed vars into global name space 631 lookup duplicity && { duplicity_version_get; } 632 local NOTFOUND="INVALID" 633 local AWK_VERSION GREP_VERSION PYTHON_RUNNER \ 634 PYTHON_RUNNER_RESOLVED PYTHON_VERSION PYTHON_PATH 635 # freebsd awk / GNU awk (--version), 636 # debian mawk (-W version), 637 # openbsd awk (-V, exitcode 0 when any program string is given regardless .e.g. "-W version", so place it last) 638 # some awks wait for input if they misinterpret/don't know the options, pipe '' as a precaution 639 AWK_VERSION=$( lookup awk && ( 640 echo | awk --version ||\ 641 echo | awk -V ||\ 642 echo | awk -W version ) 2>/dev/null | awk 'NR<=2&&tolower($0)~/(busybox|awk)/{success=1;print;exit} END{if(success<1) print "unknown"}' || echo "$NOTFOUND" ) 643 GREP_VERSION=$( lookup grep && grep --version 2>&1 | awk 'NR<=2&&tolower($0)~/(busybox|grep.*[0-9]+\.[0-9]+)/{success=1;print;exit} END{if(success<1) print "unknown"}' || echo "$NOTFOUND" ) 644 645 if [ -n "$PYTHON" ]; then 646 PYTHON_RUNNER=$PYTHON 647 else 648 PYTHON_RUNNER="$(duplicity_python_binary_parse)" 649 fi 650 # fetch version and resolve python 651 [ -n "$PYTHON_RUNNER" ] && { 652 PYTHON_VERSION=$($PYTHON_RUNNER -V 2>&1| awk '{print tolower($0);exit}' || echo "'$PYTHON_RUNNER' $NOTFOUND" ) 653 local PYTHON_RUNNER_ARRAY=( $PYTHON_RUNNER ) 654 PYTHON_RUNNER_RESOLVED="$(which ${PYTHON_RUNNER_ARRAY[0]})" 655 # readd params if there were 656 [ ${#PYTHON_RUNNER_ARRAY[@]} -gt 1 ] && \ 657 PYTHON_RUNNER_RESOLVED="${PYTHON_RUNNER_RESOLVED} ${PYTHON_RUNNER_ARRAY[@]:1}" 658 PYTHON_PATH="$($PYTHON_RUNNER -c "import sys;print(':'.join(sys.path));")" 659 } 660 661 local GPG_INFO=$(gpg_avail && gpg --version 2>&1| awk '/^gpg.*[0-9\.]+$/&&length(v)<1{v=$1" "$3}/^Home:/{h=" ("$0")"}END{print v""h}' || echo "gpg $NOTFOUND") 662 local BASH_VERSION=$(bash --version | awk 'NR==1{IGNORECASE=1;sub(/GNU bash, version[ ]+/,"",$0);print $0}') 663 # print out 664 echo -e "Using installed duplicity version ${DUPL_VERSION:-$NOTFOUND}\ 665 ${PYTHON_VERSION+, $PYTHON_VERSION ${PYTHON_RUNNER:+($PYTHON_RUNNER_RESOLVED)}${PYTHON_PATH:+ 'PYTHONPATH=$PYTHON_PATH'}}\ 666 ${GPG_INFO:+, $GPG_INFO}${AWK_VERSION:+, awk '${AWK_VERSION}'}${GREP_VERSION:+, grep '${GREP_VERSION}'}\ 667 ${BASH_VERSION:+, bash '${BASH_VERSION}'}." 668 } 669 670 function usage_info { # print usage information 671 672 cat <<USAGE_EOF 673 VERSION: 674 $(version_info) 675 676 DESCRIPTION: 677 Duply deals as a wrapper for the mighty duplicity magic. 678 It simplifies running duplicity with cron or on command line by: 679 680 - keeping recurring settings in profiles per backup job 681 - enabling batch operations e.g. backup_verify+purge 682 - executing pre/post scripts (different actions possible 683 depending on previous or next command or it's exit status) 684 - precondition checking for flawless duplicity operation 685 686 For each backup job one configuration profile must be created. 687 The profile folder will be stored under '~/.${ME_NAME}/<profile>' 688 (where ~ is the current users home directory). 689 Hint: 690 If the folder '/etc/${ME_NAME}' exists, the profiles for the super 691 user root will be searched & created there. 692 693 USAGE: 694 first time usage (profile creation): 695 $ME <profile> create 696 697 general usage in single or batch mode (see EXAMPLES): 698 $ME <profile> <command>[[_|+|-]<command>[_|+|-]...] [<options> ...] 699 700 For batches the conditional separators can also be written as pseudo commands 701 and(+), or(-). See SEPARATORS for details. 702 703 Non $ME options are passed on to duplicity (see OPTIONS). 704 All conf parameters can also be defined in the environment instead. 705 706 PROFILE: 707 Indicated by a path or a profile name (<profile>), which is resolved 708 to '~/.${ME_NAME}/<profile>' (~ expands to environment variable \$HOME). 709 710 Superuser root can place profiles under '/etc/${ME_NAME}'. Simply create 711 the folder manually before running $ME_NAME as superuser. 712 Note: 713 Already existing profiles in root's home folder will cease to work 714 unless they are moved to the new location manually. 715 716 example 1: $ME humbug backup 717 718 Alternatively a _path_ might be used e.g. useful for quick testing, 719 restoring or exotic locations. Shell expansion should work as usual. 720 Hint: 721 The path must contain at least one path separator '/', 722 e.g. './test' instead of only 'test'. 723 724 example 2: $ME ~/.${ME_NAME}/humbug backup 725 726 SEPARATORS: 727 _ (underscore) 728 neutral separator 729 + (plus sign), _and_ 730 conditional AND 731 the next command will only be executed if the previous succeeded 732 - (minus sign), _or_ 733 conditional OR 734 the next command will only be executed if the previous failed 735 [] (square brackets), _groupIn_/_groupOut_ 736 enables grouping of commands 737 738 example: 739 'pre+[bkp-verify]_post' translates to 740 'pre_and_groupIn_bkp_or_verify_groupOut_post' 741 742 COMMANDS: 743 usage get usage help text 744 745 and/or/groupIn/groupOut 746 pseudo commands used in batches (see SEPARATORS above) 747 748 create creates a configuration profile 749 backup backup with pre/post script execution (batch: [pre_bkp_post]), 750 full (if full_if_older matches or no earlier backup is found) 751 incremental (in all other cases) 752 pre/post execute '<profile>/$(basename "$PRE")', '<profile>/$(basename "$POST")' scripts 753 bkp as above but without executing pre/post scripts 754 full force full backup 755 incr force incremental backup 756 list [<age>] 757 list all files in backup (as it was at <age>, default: now) 758 status prints backup sets and chains currently in repository 759 verify [<age>] [--compare-data] 760 list files changed, since age if given 761 verifyPath <rel_path_in_bkp> <local_path> [<age>] [--compare-data] 762 list changes of a file or folder path in backup compared to a 763 local path, since age if given 764 restore <target_path> [<age>] 765 restore the complete backup to <target_path> [as it was at <age>] 766 fetch <src_path> <target_path> [<age>] 767 fetch single file/folder from backup [as it was at <age>] 768 purge [<max_age>] [--force] 769 list outdated backup files (older than \$MAX_AGE) 770 [use --force to actually delete these files] 771 purgeFull [<max_full_backups>] [--force] 772 list outdated backup files (\$MAX_FULL_BACKUPS being the number of 773 full backups and associated incrementals to keep, counting in 774 reverse chronological order) 775 [use --force to actually delete these files] 776 purgeIncr [<max_fulls_with_incrs>] [--force] 777 list outdated incremental backups (\$MAX_FULLS_WITH_INCRS being 778 the number of full backups which associated incrementals will be 779 kept, counting in reverse chronological order) 780 [use --force to actually delete these files] 781 purgeAuto [--force] 782 convenience batch wrapper for all purge commands above. 783 purge, purgeFull, purgeIncr are added if their conf vars were set. e.g. 784 MAX_AGE=1Y 785 MAX_FULL_BACKUPS=6 786 MAX_FULLS_WITH_INCR=3 787 in profile conf file would result in 788 [purge_purgeFull_purgeIncr] 789 cleanup [--force] 790 list broken backup chain files archives (e.g. after unfinished run) 791 [use --force to actually delete these files] 792 793 changelog print changelog / todo list 794 txt2man feature for package maintainers - create a manpage based on the 795 usage output. download txt2man from http://mvertes.free.fr/, put 796 it in the PATH and run '$ME txt2man' to create a man page. 797 version show version information of $ME_NAME and needed programs 798 799 OPTIONS: 800 --force passed to duplicity (see commands: 801 purge, purgeFull, purgeIncr, cleanup) 802 --preview do nothing but print out generated duplicity command lines 803 --disable-encryption 804 disable encryption, overrides profile settings 805 806 TIME FORMATS: 807 For all time related parameters like age, max_age etc. 808 Refer to the duplicity manpage for all available formats. Here some examples: 809 2002-01-25T07:00:00+02:00 (full date time format string) 810 2002/3/5 (date string YYYY/MM/DD) 811 12D (interval, 12 days ago) 812 1h78m (interval, 1 hour 78 minutes ago) 813 814 PRE/POST SCRIPTS: 815 Some useful internal duply variables are exported to the scripts. 816 817 PROFILE, CONFDIR, SOURCE, TARGET_URL_<PROT|HOSTPATH|USER|PASS>, 818 GPG_<KEYS_ENC|KEY_SIGN|PW>, CMD_ERR, RUN_START, 819 CMD_<PREV|NEXT> (previous/next command), 820 CND_<PREV|NEXT> (condition before/after) 821 822 The CMD_* variables were introduced to allow different actions according to 823 the command the scripts were attached to e.g. 'pre_bkp_post_pre_verify_post' 824 will call the pre script two times, with CMD_NEXT variable set to 'bkp' 825 on the first and to 'verify' on the second run. 826 CMD_ERR holds the exit code of the CMD_PREV . 827 828 EXAMPLES: 829 create profile 'humbug': 830 $ME humbug create (don't forget to edit this new conf file) 831 backup 'humbug' now: 832 $ME humbug backup 833 list available backup sets of profile 'humbug': 834 $ME humbug status 835 list and delete outdated backups of 'humbug': 836 $ME humbug purge --force 837 restore latest backup of 'humbug' to /mnt/restore: 838 $ME humbug restore /mnt/restore 839 restore /etc/passwd of 'humbug' from 4 days ago to /root/pw: 840 $ME humbug fetch etc/passwd /root/pw 4D 841 (see "duplicity manpage", section TIME FORMATS) 842 a one line batch job on 'humbug' for cron execution: 843 $ME humbug backup_verify_purge --force 844 batch job to run a full backup with pre/post scripts: 845 $ME humbug pre_full_post 846 847 FILES: 848 in profile folder '~/.${ME_NAME}/<profile>' or '/etc/${ME_NAME}' 849 conf profile configuration file 850 pre,post pre/post scripts (see above for details) 851 gpgkey.*.asc exported GPG key files 852 exclude a globbing list of included or excluded files/folders 853 (see "duplicity manpage", section FILE SELECTION) 854 855 $(hint_profile) 856 857 SEE ALSO: 858 duplicity man page duplicity(1) or http://duplicity.us/docs.html 859 USAGE_EOF 860 } 861 862 # to check call 'duply txt2man | man -l -' 863 function usage_txt2man { 864 usage_info | \ 865 awk '/^^[^[:lower:][:space:]][^[:lower:]]+$/{gsub(/[^[:upper:]]/," ",$0)}{print}' |\ 866 txt2man -t"$(toupper "${ME_NAME}")" -s1 -r"${ME_NAME}-${ME_VERSION}" -v'User Manuals' 867 } 868 869 function changelog { 870 cat $ME_LONG | awk '/^#####/{on=on+1}(on==3){sub(/^#( )?/,"",$0);print}' 871 } 872 873 function create_config { 874 if [ ! -d "$CONFDIR" ] ; then 875 mkdir -p "$CONFDIR" || error "Couldn't create config '$CONFDIR'." 876 # create initial config file 877 cat <<EOF >"$CONF" 878 # gpg encryption settings, simple settings: 879 # GPG_KEY='disabled' - disables encryption alltogether 880 # GPG_KEY='<key1>[,<key2>]'; GPG_PW='pass' - encrypt with keys, 881 # sign if secret key of key1 is available use GPG_PW for sign & decrypt 882 # Note: you can specify keys via all methods described in gpg manpage, 883 # section "How to specify a user ID", escape commas (,) via backslash (\) 884 # e.g. 'Mueller, Horst', 'Bernd' -> 'Mueller\, Horst, Bernd' 885 # as they are used to separate the entries 886 # GPG_PW='passphrase' - symmetric encryption using passphrase only 887 GPG_KEY='${DEFAULT_GPG_KEY}' 888 GPG_PW='${DEFAULT_GPG_PW}' 889 # gpg encryption settings in detail (extended settings) 890 # the above settings translate to the following more specific settings 891 # GPG_KEYS_ENC='<keyid1>[,<keyid2>,...]' - list of pubkeys to encrypt to 892 # GPG_KEY_SIGN='<keyid1>|disabled' - a secret key for signing 893 # GPG_PW='<passphrase>' - needed for signing, decryption and symmetric 894 # encryption. If you want to deliver different passphrases for e.g. 895 # several keys or symmetric encryption plus key signing you can use 896 # gpg-agent. Simply make sure that GPG_AGENT_INFO is set in environment. 897 # also see "A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING" in duplicity manpage 898 # notes on en/decryption 899 # private key and passphrase will only be needed for decryption or signing. 900 # decryption happens on restore and incrementals (compare archdir contents). 901 # for security reasons it makes sense to separate the signing key from the 902 # encryption keys. https://answers.launchpad.net/duplicity/+question/107216 903 #GPG_KEYS_ENC='<pubkey1>,<pubkey2>,...' 904 #GPG_KEY_SIGN='<prvkey>' 905 # set if signing key passphrase differs from encryption (key) passphrase 906 # NOTE: available since duplicity 0.6.14, translates to SIGN_PASSPHRASE 907 #GPG_PW_SIGN='<signpass>' 908 909 # uncomment and set a file path or name force duply to use this gpg executable 910 # available in duplicity 0.7.04 and above (currently unreleased 06/2015) 911 #GPG='/usr/local/gpg-2.1/bin/gpg' 912 913 # gpg options passed from duplicity to gpg process (default='') 914 # e.g. "--trust-model pgp|classic|direct|always" 915 # or "--compress-algo=bzip2 --bzip2-compress-level=9" 916 # or "--personal-cipher-preferences AES256,AES192,AES..." 917 # or "--homedir ~/.duply" - keep keyring and gpg settings duply specific 918 # or "--pinentry-mode loopback" - needed for GPG 2.1+ _and_ 919 # also enable allow-loopback-pinentry in your .gnupg/gpg-agent.conf 920 #GPG_OPTS='' 921 922 # disable preliminary tests with the following setting 923 #GPG_TEST='disabled' 924 # disable automatic gpg key importing altogether 925 #GPG_IMPORT='disabled' 926 # disable automatic gpg key exporting to profile folder 927 #GPG_EXPORT='disabled' 928 929 # backend, credentials & location of the backup target (URL-Format) 930 # generic syntax is 931 # scheme://[user[:password]@]host[:port]/[/]path 932 # e.g. 933 # sftp://bob:secret@backupserver.com//home/bob/dupbkp 934 # for details and available backends see duplicity manpage, section URL Format 935 # http://duplicity.us/vers8/duplicity.1.html#url-format 936 # BE AWARE: 937 # some backends (cloudfiles, S3 etc.) need additional env vars to be set to 938 # work properly, read after the TARGET definition for more details. 939 # ATTENTION: 940 # characters other than A-Za-z0-9.-_.~ in the URL have to be 941 # replaced by their url encoded pendants, see 942 # http://en.wikipedia.org/wiki/Url_encoding 943 # if you define the credentials as TARGET_USER, TARGET_PASS below $ME 944 # will try to url_encode them for you if the need arises. 945 TARGET='${DEFAULT_TARGET}' 946 947 # optionally the username/password can be defined as extra variables 948 # setting them here _and_ in TARGET results in an error 949 # ATTENTION: 950 # there are backends that do not support the user/pass auth scheme. 951 # prominent examples are S3, Azure, Cloudfiles. when in doubt consult the 952 # duplicity manpage. usually there is a NOTE section explaining if and which 953 # env vars should be set. 954 #TARGET_USER='${DEFAULT_TARGET_USER}' 955 #TARGET_PASS='${DEFAULT_TARGET_PASS}' 956 # e.g. for cloud files backend it might look like this (uncomment for use!) 957 #export CLOUDFILES_USERNAME='someuser' 958 #export CLOUDFILES_APIKEY='somekey' 959 #export CLOUDFILES_AUTHURL ='someurl' 960 # the following is an incomplete list (<backend>: comma separated env vars list) 961 # Azure: AZURE_ACCOUNT_NAME, AZURE_ACCOUNT_KEY 962 # Cloudfiles: CLOUDFILES_USERNAME, CLOUDFILES_APIKEY, CLOUDFILES_AUTHURL 963 # Google Cloud Storage: GS_ACCESS_KEY_ID, GS_SECRET_ACCESS_KEY 964 # Pydrive: GOOGLE_DRIVE_ACCOUNT_KEY, GOOGLE_DRIVE_SETTINGS 965 # S3: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY 966 # Swift: SWIFT_USERNAME, SWIFT_PASSWORD, SWIFT_AUTHURL, 967 # SWIFT_TENANTNAME OR SWIFT_PREAUTHURL, SWIFT_PREAUTHTOKEN 968 969 # base directory to backup 970 SOURCE='${DEFAULT_SOURCE}' 971 972 # a command that runs duplicity e.g. 973 # shape bandwidth use via trickle 974 # "trickle -s -u 640 -d 5120" # 5Mb up, 40Mb down" 975 #DUPL_PRECMD="" 976 977 # override the python interpreter to execute duplicity, unset by default 978 # e.g. "python3" or "/usr/bin/python3.8" 979 #PYTHON="python" 980 981 # exclude folders containing exclusion file (since duplicity 0.5.14) 982 # Uncomment the following two lines to enable this setting. 983 #FILENAME='.duplicity-ignore' 984 #DUPL_PARAMS="\$DUPL_PARAMS --exclude-if-present '\$FILENAME'" 985 986 # Time frame for old backups to keep, Used for the "purge" command. 987 # see duplicity man page, chapter TIME_FORMATS) 988 #MAX_AGE=1M 989 990 # Number of full backups to keep. Used for the "purgeFull" command. 991 # See duplicity man page, action "remove-all-but-n-full". 992 #MAX_FULL_BACKUPS=1 993 994 # Number of full backups for which incrementals will be kept for. 995 # Used for the "purgeIncr" command. 996 # See duplicity man page, action "remove-all-inc-of-but-n-full". 997 #MAX_FULLS_WITH_INCRS=1 998 999 # activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3) 1000 # forces a full backup if last full backup reaches a specified age, for the 1001 # format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS 1002 # Uncomment the following two lines to enable this setting. 1003 #MAX_FULLBKP_AGE=1M 1004 #DUPL_PARAMS="\$DUPL_PARAMS --full-if-older-than \$MAX_FULLBKP_AGE " 1005 1006 # sets duplicity --volsize option (available since v0.4.3.RC7) 1007 # set the size of backup chunks to VOLSIZE MB instead of the default 25MB. 1008 # VOLSIZE must be number of MB's to set the volume size to. 1009 # Uncomment the following two lines to enable this setting. 1010 #VOLSIZE=50 1011 #DUPL_PARAMS="\$DUPL_PARAMS --volsize \$VOLSIZE " 1012 1013 # verbosity of output (error 0, warning 1-2, notice 3-4, info 5-8, debug 9) 1014 # default is 4, if not set 1015 #VERBOSITY=5 1016 1017 # temporary file space. at least the size of the biggest file in backup 1018 # for a successful restoration process. (default is '/tmp', if not set) 1019 #TEMP_DIR=/tmp 1020 1021 # Modifies archive-dir option (since 0.6.0) Defines a folder that holds 1022 # unencrypted meta data of the backup, enabling new incrementals without the 1023 # need to decrypt backend metadata first. If empty or deleted somehow, the 1024 # private key and it's password are needed. 1025 # NOTE: This is confidential data. Put it somewhere safe. It can grow quite 1026 # big over time so you might want to put it not in the home dir. 1027 # default '~/.cache/duplicity/duply_<profile>/' 1028 # if set '\${ARCH_DIR}/<profile>' 1029 #ARCH_DIR=/some/space/safe/.duply-cache 1030 1031 # DEPRECATED setting 1032 # sets duplicity --time-separator option (since v0.4.4.RC2) to allow users 1033 # to change the time separator from ':' to another character that will work 1034 # on their system. HINT: For Windows SMB shares, use --time-separator='_'. 1035 # NOTE: '-' is not valid as it conflicts with date separator. 1036 # ATTENTION: only use this with duplicity < 0.5.10, since then default file 1037 # naming is compatible and this option is pending depreciation 1038 #DUPL_PARAMS="\$DUPL_PARAMS --time-separator _ " 1039 1040 # DEPRECATED setting 1041 # activates duplicity --short-filenames option, when uploading to a file 1042 # system that can't have filenames longer than 30 characters (e.g. Mac OS 8) 1043 # or have problems with ':' as part of the filename (e.g. Microsoft Windows) 1044 # ATTENTION: only use this with duplicity < 0.5.10, later versions default file 1045 # naming is compatible and this option is pending depreciation 1046 #DUPL_PARAMS="\$DUPL_PARAMS --short-filenames " 1047 1048 # more duplicity command line options can be added in the following way 1049 # don't forget to leave a separating space char at the end 1050 #DUPL_PARAMS="\$DUPL_PARAMS --put_your_options_here " 1051 1052 EOF 1053 1054 # create initial exclude file 1055 cat <<EOF >"$EXCLUDE" 1056 # although called exclude, this file is actually a globbing file list 1057 # duplicity accepts some globbing patterns, even including ones here 1058 # here is an example, this incl. only 'dir/bar' except it's subfolder 'foo' 1059 # - dir/bar/foo 1060 # + dir/bar 1061 # - ** 1062 # for more details see duplicity manpage, section File Selection 1063 # http://duplicity.us/vers8/duplicity.1.html#file-selection 1064 1065 EOF 1066 1067 # Hints on first usage 1068 cat <<EOF 1069 1070 Congratulations. You just created the profile '$FTPLCFG'. 1071 The initial config file has been created as 1072 '$CONF'. 1073 You should now adjust this config file to your needs. 1074 1075 $(hint_profile) 1076 1077 EOF 1078 fi 1079 1080 } 1081 1082 # used in usage AND create_config 1083 function hint_profile { 1084 cat <<EOF 1085 IMPORTANT: 1086 Copy the _whole_ profile folder after the first backup to a safe place. 1087 It contains everything (duply related) needed to restore your backups. 1088 1089 Pay attention to (possibly later added) external files such as credentials 1090 or auth files (e.g. netrc, .megarc, ssh keys) or environment variables 1091 (e.g. DPBX_ACCESS_TOKEN). 1092 It is good policy to place those in the profile folder if possible at all. 1093 e.g. in case of 'multi://' target the config .json file 1094 Env vars should be added to duply profiles' conf file. 1095 1096 Keep access to these files restricted as they contain information (gpg key, 1097 passphrases etc.) to access and modify your backups. 1098 1099 Finally: 1100 You should attempt a restore from an unrelated host to be sure you really 1101 have everything needed for restoration. 1102 1103 Repeat these steps after _all_ configuration changes. Some configuration 1104 options are crucial for restoration. 1105 1106 EOF 1107 } 1108 1109 function separator { 1110 echo "--- $@ ---" 1111 } 1112 1113 function inform { 1114 echo -e "\nINFO:\n\n$@\n" 1115 } 1116 1117 function warning { 1118 echo -e "\nWARNING:\n\n$@\n" 1119 } 1120 1121 function warning_oldhome { 1122 local old=$1 new=$2 1123 warning " ftplicity changed name to duply since you created your profiles. 1124 Please rename the old folder 1125 '$old' 1126 to 1127 '$new' 1128 and this warning will disappear. 1129 If you decide not to do so profiles will _only_ work from the old location." 1130 } 1131 1132 function error_print { 1133 echo -e "$@" >&2 1134 } 1135 1136 function error { 1137 error_print "\nSorry. A fatal ERROR occured:\n\n$@\n" 1138 exit -1 1139 } 1140 1141 function error_gpg { 1142 [ -n "$2" ] && local hint="\n $2\n\n " 1143 1144 error "$1 1145 1146 Hint${hint:+s}: 1147 ${hint}Maybe you have not created a gpg key yet (e.g. gpg --gen-key)? 1148 Don't forget the used _password_ as you will need it. 1149 When done enter the 8 digit id & the password in the profile conf file. 1150 1151 The key id can be found doing a 'gpg --list-keys'. In the example output 1152 below the key id for the public key would be FFFFFFFF. 1153 1154 pub 1024D/FFFFFFFF 2007-12-17 1155 uid duplicity 1156 sub 2048g/899FE27F 2007-12-17 1157 " 1158 } 1159 1160 function error_gpg_test { 1161 [ -n "$2" ] && local hint="\n $2\n\n " 1162 1163 error "$1 1164 1165 Hint${hint:+s}: 1166 ${hint}This error means that gpg is probably misconfigured or not working 1167 correctly. The error message above should help to solve the problem. 1168 However, if for some reason $ME_NAME should misinterpret the situation you 1169 can define GPG_TEST='disabled' in the conf file to bypass the test. 1170 Please do not forget to report the bug in order to resolve the problem 1171 in future versions of $ME_NAME. 1172 " 1173 } 1174 1175 function error_path { 1176 error "$@ 1177 PATH='$PATH' 1178 " 1179 } 1180 1181 function error_to_string { 1182 [ -n "$1" ] && [ "$1" -eq 0 ] && echo "OK" || echo "FAILED 'code $1'" 1183 } 1184 1185 function duplicity_version_get { 1186 # use cached value, just print 1187 var_isset DUPL_VERSION && return 1188 1189 local DUPL_VERSION_OUT DUPL_VERSION_AWK PYTHON_BIN CMD='duplicity' 1190 # only run with a user specific python if configured (running by default 1191 # breaks homebrew as they place a shell wrapper for duplicity in path) 1192 [ -n "$PYTHON" ] &&\ 1193 CMD="$PYTHON $(qw "$(which duplicity)")" 1194 1195 DUPL_VERSION_OUT=$($CMD --version) 1196 DUPL_VERSION=$(echo $DUPL_VERSION_OUT | awk '/^duplicity/{print $2; exit;}') 1197 #DUPL_VERSION='1.2.3' #'0.7.03' #'0.6.08b' #,0.4.4.RC4,0.6.08b 1198 DUPL_VERSION_VALUE=0 1199 DUPL_VERSION_AWK=$(awk -v v="$DUPL_VERSION" 'BEGIN{ 1200 if (match(v,/[^\.0-9]+[0-9]*$/)){ 1201 rest=substr(v,RSTART,RLENGTH);v=substr(v,0,RSTART-1);} 1202 if (pos=match(rest,/RC([0-9]+)$/)) rc=substr(rest,pos+2) 1203 split(v,f,"[. ]"); if(f[1]f[2]f[3]~/^[0-9]+$/) vvalue=f[1]*10000+f[2]*100+f[3]; else vvalue=0 1204 print "#"v"_"rest"("rc"):"f[1]"-"f[2]"-"f[3] 1205 print "DUPL_VERSION_VALUE=\047"vvalue"\047" 1206 print "DUPL_VERSION_RC=\047"rc"\047" 1207 print "DUPL_VERSION_SUFFIX=\047"rest"\047" 1208 }') 1209 eval "$DUPL_VERSION_AWK" 1210 #echo -e ",$DUPL_VERSION,$DUPL_VERSION_VALUE,$DUPL_VERSION_RC,$DUPL_VERSION_SUFFIX," 1211 1212 # doublecheck findings and report error 1213 if [ $DUPL_VERSION_VALUE -eq 0 ]; then 1214 inform "duplicity version check failed (please report, this is a bug) 1215 the command 1216 $CMD 1217 resulted in 1218 $DUPL_VERSION_OUT 1219 " 1220 elif [ $DUPL_VERSION_VALUE -le 404 ] && [ ${DUPL_VERSION_RC:-4} -lt 4 ]; then 1221 error "The installed version $DUPL_VERSION is incompatible with $ME_NAME v$ME_VERSION. 1222 You should upgrade your version of $ME_NAME to at least v0.4.4RC4 or 1223 use the older ftplicity version 1.1.1 from $ME_WEBSITE." 1224 elif [ $DUPL_VERSION_VALUE -le 20100 ] ; then 1225 error "The installed version $DUPL_VERSION is incompatible with $ME_NAME v$ME_VERSION. 1226 You should upgrade your version of duplicity to at least v2.1.0 or 1227 use the older $ME_NAME version 2.4.3 from $ME_WEBSITE." 1228 fi 1229 } 1230 1231 function duplicity_version_ge { 1232 [ "$DUPL_VERSION_VALUE" -ge "$1" ] 1233 } 1234 1235 function duplicity_version_lt { 1236 ! duplicity_version_ge "$1" 1237 } 1238 1239 # parse interpreter from duplicity shebang 1240 function duplicity_python_binary_parse { 1241 # reuse cached result 1242 var_isset 'DUPL_PYTHON_BIN' && { 1243 [ -n "$DUPL_PYTHON_BIN" ] && { 1244 echo $DUPL_PYTHON_BIN 1245 return 1246 } || return 1 1247 } 1248 1249 # no duplicity, nothing to parse 1250 lookup duplicity || return 1 1251 1252 local DUPL_BIN=$(which duplicity) 1253 # test for shebang 1254 IFS= LC_ALL=C read -rn2 shebang < "$DUPL_BIN" && [ "$shebang" != '#!' ] && { 1255 DUPL_PYTHON_BIN="" 1256 return 1 1257 } 1258 1259 # parse it or warn 1260 DUPL_PYTHON_BIN=$(awk 'NR==1&&/^#!/{sub(/^#!( *\/usr\/bin\/env *)?/,""); print}' < "$DUPL_BIN") 1261 if ! echo "$DUPL_PYTHON_BIN" | grep -q -i 'python'; then 1262 warning "Could not parse the python interpreter used from duplicity ($DUPL_BIN). Result was 1263 '$DUPL_PYTHON_BIN'. 1264 " 1265 DUPL_PYTHON_BIN="" 1266 return 1 1267 fi 1268 1269 # success 1270 echo $DUPL_PYTHON_BIN 1271 return 1272 } 1273 1274 function run_script { # run pre/post scripts 1275 local ERR=0 1276 local SCRIPT="$1" 1277 if [ ! -z "$PREVIEW" ] ; then 1278 echo "$([ ! -x "$SCRIPT" ] && echo ". ")$SCRIPT" 1279 elif [ -r "$SCRIPT" ] ; then 1280 echo -n "Running '$SCRIPT' " 1281 if [ -x "$SCRIPT" ]; then 1282 OUT=$("$SCRIPT" 2>&1) 1283 ERR=$? 1284 else 1285 OUT=$(. "$SCRIPT" 2>&1) 1286 ERR=$? 1287 fi 1288 [ $ERR -eq "0" ] && echo "- OK" || echo "- FAILED (code $ERR)" 1289 echo -en ${OUT:+"Output: $OUT\n"} ; 1290 else 1291 echo "Skipping n/a script '$SCRIPT'." 1292 fi 1293 return $ERR 1294 } 1295 1296 function run_cmd { 1297 # run or print escaped cmd string 1298 local CMD_ERR=0 1299 if [ -n "$PREVIEW" ]; then 1300 CMD_OUT=$( echo "$@ 2>&1" ) 1301 CMD_MSG="-- Run cmd -- $CMD_MSG --\n$CMD_OUT" 1302 elif [ -n "$CMD_DISABLED" ]; then 1303 CMD_MSG="$CMD_MSG (DISABLED) - $CMD_DISABLED" 1304 else 1305 echo -n -e "$CMD_MSG" 1306 CMD_OUT=` eval "$@" 2>&1 ` 1307 CMD_ERR=$? 1308 if [ "$CMD_ERR" = "0" ]; then 1309 CMD_MSG=" (OK)" 1310 else 1311 CMD_MSG=" (FAILED)" 1312 fi 1313 fi 1314 echo -e "$CMD_MSG" 1315 # reset 1316 unset CMD_DISABLED CMD_MSG 1317 return $CMD_ERR 1318 } 1319 1320 function qw { quotewrap $@; } 1321 1322 function quotewrap { 1323 local param="$@" 1324 # quote strings having non word chars (e.g. spaces) 1325 if echo "$param" | awk '/[^a-zA-Z0-9,\._\+\-@%\/]/{exit 0}{exit 1}'; then 1326 echo "$param" | awk '{\ 1327 gsub(/[\047]/,"\047\\\047\047",$0);\ 1328 gsub(/[\042]/,"\047\\\042\047",$0);\ 1329 print "\047"$0"\047"}' 1330 return 1331 fi 1332 echo $param 1333 } 1334 1335 function duplicity_params_global { 1336 # already done? return 1337 var_isset 'DUPL_PARAMS_GLOBAL' && return 1338 local DUPL_ARG_ENC 1339 1340 # use key only if set in config, else leave it to symmetric encryption 1341 if gpg_disabled; then 1342 local DUPL_PARAM_ENC='--no-encryption' 1343 else 1344 local DUPL_PARAM_ENC=$(gpg_prefix_keyset '--encrypt-key' "${GPG_KEYS_ENC_ARRAY[@]}") 1345 gpg_signing && local DUPL_PARAM_SIGN=$(gpg_prefix_keyset '--sign-key' "$GPG_KEY_SIGN") 1346 # interpret password settings 1347 var_isset 'GPG_PW' && DUPL_ARG_ENC="PASSPHRASE=$(qw "${GPG_PW}")" 1348 var_isset 'GPG_PW_SIGN' && DUPL_ARG_ENC="${DUPL_ARG_ENC} SIGN_PASSPHRASE=$(qw "${GPG_PW_SIGN}")" 1349 fi 1350 1351 local GPG_OPTS=${GPG_OPTS:+"--gpg-options $(qw "${GPG_OPTS}")"} 1352 1353 # set name for dupl archive folder, since 0.6.0 1354 if duplicity_version_ge 601; then 1355 local DUPL_ARCHDIR='' 1356 if var_isset 'ARCH_DIR'; then 1357 DUPL_ARCHDIR="--archive-dir $(qw "${ARCH_DIR}")" 1358 # reuse erronously duply_ prefixed folders from bug #117 1359 if [ -d "$ARCH_DIR/duply_${PROFILE}" ]; then 1360 DUPL_ARCHDIR="${DUPL_ARCHDIR} --name $(qw "duply_${PROFILE}")" 1361 else 1362 DUPL_ARCHDIR="${DUPL_ARCHDIR} --name $(qw "${PROFILE}")" 1363 fi 1364 else 1365 DUPL_ARCHDIR="--name $(qw "duply_${PROFILE}")" 1366 fi 1367 fi 1368 1369 DUPL_PARAMS_GLOBAL="${DUPL_ARCHDIR} ${DUPL_PARAM_ENC}\ 1370 ${DUPL_PARAM_SIGN} ${VERBOSITY:+--verbosity $VERBOSITY}\ 1371 ${GPG_OPTS}" 1372 1373 DUPL_VARS_GLOBAL="TMPDIR='$TEMP_DIR'\ 1374 ${DUPL_ARG_ENC}" 1375 } 1376 1377 # function to filter the DUPL_PARAMS var from user conf 1378 function duplicity_params_conf { 1379 local OUT="$DUPL_PARAMS" 1380 # reuse cmd var from main loop 1381 ## in/exclude parameters are currently not supported on 1382 ## cleanup, status (collection_status), list (list_current_files), purge* (remove_*), fetch/restore 1383 case $cmd in 1384 cleanup | status | list | purge* | restore | fetch ) 1385 # filter exclude params from fetch/restore/status 1386 OUT=$(eval stripXcludes $OUT) 1387 ;; 1388 esac 1389 1390 case $cmd in 1391 bkp | incr | full ) 1392 # nothing to strip, we're backing up' 1393 ;; 1394 *) 1395 OUT=$(eval stripBkpOnlyParams $OUT) 1396 ;; 1397 esac 1398 1399 # print result 1400 echo "$OUT" 1401 } 1402 1403 # strip in/exclude parameters from param string 1404 function stripXcludes { 1405 local STRIPNEXT OUT; 1406 for p in "$@"; do 1407 #echo "calc =$p=" 1408 if [ -n "$STRIPNEXT" ]; then 1409 unset STRIPNEXT 1410 # strip the value of previous parameter 1411 continue 1412 elif echo "$p" | awk '/^\-\-(in|ex)clude(\-[a-zA-Z\-]+)?$/{exit 0;}{exit 1;}'; then 1413 # strips e.g. --include /foo/bar 1414 STRIPNEXT="yes" 1415 continue 1416 elif echo "$p" | awk '/^\-\-(in|ex)clude(\-[a-zA-Z\-]+)?=/{exit 0;}{exit 1;}'; then 1417 # strips e.g. --include=/foo/bar 1418 continue 1419 fi 1420 1421 OUT="$OUT $(qw "$p")" 1422 done 1423 echo "$OUT" 1424 } 1425 1426 # strip backup only parameters from param string 1427 function stripBkpOnlyParams { 1428 local STRIPNEXT OUT; 1429 1430 for p in "$@"; do 1431 if [ -n "$STRIPNEXT" ]; then 1432 unset STRIPNEXT 1433 # strip the value of previous parameter 1434 continue 1435 elif echo "$p" | awk '/^\-\-(allow-source-mismatch|asynchronous-upload|dry-run)$/{exit 0;}{exit 1;}'; then 1436 continue 1437 elif echo "$p" | awk '/^\-\-(volsize)$/{exit 0;}{exit 1;}'; then 1438 # strips e.g. --volsize 100 1439 STRIPNEXT="yes" 1440 continue 1441 elif echo "$p" | awk '/^\-\-volsize=/{exit 0;}{exit 1;}'; then 1442 # strips e.g. --volsize=100 1443 continue 1444 fi 1445 1446 OUT="$OUT $(qw "$p")" 1447 done 1448 echo "$OUT" 1449 } 1450 1451 function duplify { # the actual wrapper function 1452 local PARAMSNOW DUPL_CMD DUPL_CMD_PARAMS 1453 1454 # put command (with params) first in duplicity parameters 1455 for param in "$@" ; do 1456 # split cmd from params (everything before splitchar --) 1457 if [ "$param" == "--" ] ; then 1458 PARAMSNOW=1 1459 else 1460 # wrap in quotes to protect from spaces 1461 [ ! $PARAMSNOW ] && \ 1462 DUPL_CMD="$DUPL_CMD $(qw "$param")" \ 1463 || \ 1464 DUPL_CMD_PARAMS="$DUPL_CMD_PARAMS $(qw "$param")" 1465 fi 1466 done 1467 1468 # init global duplicity parameters same for all tasks 1469 duplicity_params_global 1470 1471 local RUN=eval CMD=duplicity 1472 # run in cmd line preview mode if requested 1473 var_isset 'PREVIEW' && RUN=echo 1474 # only run with a user specific python if configured (running by default 1475 # breaks homebrew as they place a shell wrapper for duplicity in path) 1476 # resolve duplicity path for usage with python interpreter 1477 [ -n "$PYTHON" ] &&\ 1478 CMD="$PYTHON $(qw "$(which duplicity)")" 1479 1480 $RUN "${DUPL_VARS_GLOBAL} ${BACKEND_PARAMS}\ 1481 ${DUPL_PRECMD} $CMD $DUPL_CMD $DUPL_PARAMS_GLOBAL $(duplicity_params_conf)\ 1482 $GPG_USEAGENT $(gpg_custom_binary) $DUPL_CMD_PARAMS" 1483 1484 local ERR=$? 1485 return $ERR 1486 } 1487 1488 function secureconf { # secure the configuration dir 1489 #PERMS=$(ls -la $(dirname $CONFDIR) | grep -e " $(basename $CONFDIR)\$" | awk '{print $1}') 1490 local PERMS="$(ls -la "$CONFDIR/." | awk 'NR==2{print $1}')" 1491 if [ "${PERMS/#drwx------*/OK}" != 'OK' ] ; then 1492 chmod u+rwX,go= "$CONFDIR"; local ERR=$? 1493 warning "The profile's folder 1494 '$CONFDIR' 1495 permissions are not safe ($PERMS). Secure them now. - ($(error_to_string $ERR))" 1496 fi 1497 } 1498 1499 # params are $1=timeformatstring (default like date output), $2=epoch seconds since 1.1.1970 (default now) 1500 function date_fix { 1501 local DEFAULTFORMAT='%a %b %d %H:%M:%S %Z %Y' 1502 local date 1503 #[ "$1" == "%N" ] && return #test the no nsec test below 1504 # gnu date with -d @epoch 1505 date=$(date ${2:+-d @$2} ${1:++"$1"} 2> /dev/null) && \ 1506 echo $date && return 1507 # date bsd,osx with -r epoch 1508 date=$(date ${2:+-r $2} ${1:++"$1"} 2> /dev/null) && \ 1509 echo $date && return 1510 # date busybox with -d epoch -D %s 1511 date=$(date ${2:+-d $2 -D %s} ${1:++"$1"} 2> /dev/null) && \ 1512 echo $date && return 1513 ## some date commands do not support giving a time w/o setting it systemwide (irix,solaris,others?) 1514 # python fallback 1515 #date=$("$(python_binary)" -c "import time;print time.strftime('${1:-$DEFAULTFORMAT}',time.localtime(${2}))" 2> /dev/null) && \ 1516 # echo $date && return 1517 # awk fallback 1518 date=$(awk "BEGIN{print strftime(\"${1:-$DEFAULTFORMAT}\"${2:+,$2})}" 2> /dev/null) && \ 1519 echo $date && return 1520 # perl fallback 1521 date=$(perl -e "use POSIX qw(strftime);\$date = strftime(\"${1:-$DEFAULTFORMAT}\",localtime(${2}));print \"\$date\n\";" 2> /dev/null) && \ 1522 echo $date && return 1523 # error 1524 echo "ERROR" 1525 return 1 1526 } 1527 1528 function nsecs { 1529 local NSECS 1530 # test if date supports nanosecond output 1531 if ! var_isset NSECS_DISABLED; then 1532 NSECS=$(date_fix %N 2> /dev/null | head -1 |grep -e "^[[:digit:]]\{9\}$") 1533 [ -n "$NSECS" ] && NSECS_DISABLED=0 || NSECS_DISABLED=1 1534 fi 1535 1536 # add 9 digits, not all date(s) deliver nsecs e.g. busybox date 1537 if [ "$NSECS_DISABLED" == "1" ]; then 1538 date_fix %s000000000 1539 else 1540 date_fix %s%N 1541 fi 1542 } 1543 1544 function nsecs_to_sec { 1545 echo $(($1/1000000000)).$(printf "%03d" $(($1/1000000%1000)) ) 1546 } 1547 1548 function datefull_from_nsecs { 1549 date_from_nsecs $1 '%F %T' 1550 } 1551 1552 function date_from_nsecs { 1553 local FORMAT=${2:-%T} 1554 local TIME=$(nsecs_to_sec $1) 1555 local SECS=${TIME%.*} 1556 local DATE=$(date_fix "${FORMAT}" ${SECS:-0}) 1557 echo $DATE.${TIME#*.} 1558 } 1559 1560 function var_isset { 1561 if [ -z "$1" ]; then 1562 echo "ERROR: function var_isset needs a string as parameter" 1563 elif eval "[ \"\${$1}\" == 'not_set' ]" || eval "[ \"\${$1-not_set}\" != 'not_set' ]"; then 1564 return 0 1565 fi 1566 return 1 1567 } 1568 1569 function is_condition { 1570 local CMD=$(tolower "$1") 1571 [ "$CMD" == 'and' ] || [ "$CMD" == 'or' ] 1572 } 1573 1574 function is_groupMarker { 1575 local CMD=$(tolower "$1") 1576 [ "$CMD" == 'groupin' ] || [ "$CMD" == 'groupout' ] 1577 } 1578 1579 function is_command { 1580 local CMD=$(tolower "$1") 1581 ! is_condition "$CMD" && ! is_groupMarker "$CMD" 1582 } 1583 1584 function url_encode { 1585 # utilize python, silently do nothing on error - because no python no duplicity 1586 local PYTHON_RUNNER 1587 PYTHON_RUNNER="$(duplicity_python_binary_parse)" ||\ 1588 PYTHON_RUNNER="python" &&\ 1589 OUT=$($PYTHON_RUNNER -c " 1590 try: import urllib.request as urllib 1591 except ImportError: import urllib 1592 print(urllib.${2}quote('$1')); 1593 " 2>/dev/null ); ERR=$? 1594 [ "$ERR" -eq 0 ] && echo $OUT || echo $1 1595 } 1596 1597 function url_decode { 1598 # reuse function above with a simple string param hack 1599 url_encode "$1" "un" 1600 } 1601 1602 function toupper { 1603 echo "$@"|awk '$0=toupper($0)' 1604 } 1605 1606 function tolower { 1607 echo "$@"|awk '$0=tolower($0)' 1608 } 1609 1610 function isnumber { 1611 case "$*" in 1612 ''|*[!0-9]*) return 1;; 1613 *) return 0;; 1614 esac 1615 } 1616 1617 #function tmp_space { 1618 # 1619 # if ! isnumber $VOLSIZE; then 1620 # inform "failed to determine free space (please report, this is a bug)" 1621 # return 1622 # fi 1623 # 1624 # get free temp space 1625 # TEMP_FREE="$(df -P -k "$TEMP_DIR" 2>/dev/null | awk 'END{pos=(NF-2);if(pos>0) print $pos;}')" 1626 # # check for free space or FAIL 1627 # if [ $((${TEMP_FREE:-0}-${VOLSIZE:-0}*1024)) -lt 0-lt 0 ]; then 1628 # error "Temporary file space '$TEMP_DIR' free space is smaller ($((TEMP_FREE/1024))MB) 1629 #than one duplicity volume (${VOLSIZE}MB). 1630 # 1631 # Hint: Free space or change TEMP_DIR setting." 1632 #fi 1633 # 1634 #} 1635 1636 function gpg_disabled { 1637 echo "${GPG_KEY}" | grep -iq -e '^disabled$' 1638 } 1639 1640 # usage: join SEPARATOR "entry1" "entry2" 1641 function join { 1642 local SEP="$1" ENTRY OUT; shift; 1643 for ENTRY in "$@"; do 1644 ENTRY=${ENTRY//$SEP/\\$SEP} 1645 [ -z "$OUT" ] && OUT=$ENTRY || OUT="$OUT$SEP$ENTRY" 1646 done 1647 echo $OUT 1648 } 1649 1650 function gpg_testing { 1651 [ "$GPG_TEST" != "disabled" ] 1652 } 1653 1654 function gpg_signing { 1655 echo ${GPG_KEY_SIGN} | grep -v -q -e '^disabled$' 1656 } 1657 1658 function gpg_keytype { 1659 echo "$1" | awk '/^PUB$/{print "public"}/^SEC$/{print "secret"}' 1660 } 1661 1662 # parameter key id, key_type 1663 function gpg_keyfile { 1664 local GPG_KEY=$(gpg_key_legalize $1) TYPE="$2" 1665 local KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}" 1666 echo "${KEYFILE//.asc/${TYPE:+.$(tolower $TYPE)}.asc}" 1667 } 1668 1669 # parameter key id 1670 function gpg_import { 1671 local i FILE FOUND=0 KEY_ID="$1" KEY_TYPE="$2" KEY_FP="" ERR=0 1672 [ "$GPG_IMPORT" = "disabled" ] && { 1673 echo "Skipping import of needed $(gpg_keytype "$KEY_TYPE") key '$KEY_ID'. (GPG_IMPORT='disabled')" 1674 return 1675 } 1676 1677 # create a list of legacy key file names and current naming scheme 1678 # we always import pub and sec if they are avail in conf folder 1679 local KEYFILES=( "$CONFDIR/gpgkey" $(gpg_keyfile "$KEY_ID") \ 1680 $(gpg_keyfile "$KEY_ID" "$KEY_TYPE") ) 1681 1682 # Try autoimport from existing old gpgkey files 1683 # and new gpgkey.XXX.asc files (since v1.4.2) 1684 # and even newer gpgkey.XXX.[pub|sec].asc 1685 for (( i = 0 ; i < ${#KEYFILES[@]} ; i++ )); do 1686 FILE=${KEYFILES[$i]} 1687 if [ -f "$FILE" ]; then 1688 FOUND=1 1689 1690 CMD_MSG="Import keyfile '$FILE' to keyring" 1691 run_cmd gpg $GPG_OPTS --batch --import $(qw "$FILE") 1692 if [ "$?" != "0" ]; then 1693 warning "Import failed.${CMD_OUT:+\n$CMD_OUT}" 1694 ERR=1 1695 # continue with next 1696 continue 1697 fi 1698 fi 1699 done 1700 1701 if [ "$FOUND" -eq 0 ]; then 1702 echo "Notice: No keyfile for '$KEY_ID' found in profile folder." 1703 return 1 1704 fi 1705 1706 # try to set trust automagically 1707 CMD_MSG="Autoset trust of key '$KEY_ID' to ultimate" 1708 run_cmd echo $(gpg_fingerprint "$KEY_ID"):6: \| gpg $GPG_OPTS --import-ownertrust --batch --logger-fd 1 1709 if [ "$?" = "0" ] && [ -z "$PREVIEW" ]; then 1710 # success on all levels, we're done 1711 return $ERR 1712 fi 1713 1714 # failover: user has to set trust manually 1715 echo -e "For $ME_NAME to work you have to set the trust level 1716 with the command \"trust\" to \"ultimate\" (5) now. 1717 Exit the edit mode of gpg with \"quit\"." 1718 CMD_MSG="Running gpg to manually edit key '$KEY_ID'" 1719 run_cmd sleep 5\; gpg $GPG_OPTS --edit-key $(qw "$KEY_ID") 1720 1721 return $ERR 1722 } 1723 1724 # see 'How to specify a user ID' on gpg manpage 1725 function gpg_fingerprint { 1726 gpg $GPG_OPTS --fingerprint "$1" 2>&1 | \ 1727 awk 'NR==2{sub(/^.*=/,"");gsub(/[ \t]/,""); if ( $0 !~ /^[A-F0-9]+$/ || length($0) != 40 ) exit 1; print}' 1728 } 1729 1730 function gpg_export_if_needed { 1731 [ "$GPG_EXPORT" = 'disabled' ] && { \ 1732 echo "Skipping export of gpg keys. (GPG_EXPORT='disabled')" 1733 return 1734 } 1735 1736 local SUCCESS FILE KEY_TYPE 1737 local TMPFILE="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s).gpgexp" 1738 for KEY_ID in "$@"; do 1739 # check if already exported, do it if not 1740 for KEY_TYPE in PUB SEC; do 1741 FILE="$(gpg_keyfile "$KEY_ID" $KEY_TYPE)" 1742 if [ ! -f "$FILE" ] && eval gpg_$(tolower $KEY_TYPE)_avail \"$KEY_ID\"; then 1743 1744 # exporting 1745 CMD_MSG="Backup $(gpg_keytype "$KEY_TYPE") key '$KEY_ID' to profile." 1746 # gpg2.1 insists on passphrase here, gpg2.0- happily exports w/o it 1747 # we pipe an empty string when GPG_PW is not set to avoid gpg silently waiting for input 1748 run_cmd $(gpg_pass_pipein GPG_PW_SIGN GPG_PW) gpg $GPG_OPTS $GPG_USEAGENT $(gpg_param_passwd GPG_PW_SIGN GPG_PW) --armor --export"$(test "SEC" = "$KEY_TYPE" && echo -secret-keys)" $(qw "$KEY_ID") '>>' $(qw "$TMPFILE") 1749 CMD_ERR=$? 1750 1751 if [ "$CMD_ERR" = "0" ]; then 1752 CMD_MSG="Write file '"$(basename "$FILE")"'" 1753 run_cmd mv $(qw "$TMPFILE") $(qw "$FILE") 1754 fi 1755 1756 if [ "$CMD_ERR" != "0" ]; then 1757 warning "Backup failed.${CMD_OUT:+\n$CMD_OUT}" 1758 else 1759 SUCCESS=1 1760 fi 1761 1762 # cleanup 1763 rm $(qw "$TMPFILE") 1>/dev/null 2>&1 1764 fi 1765 done 1766 done 1767 1768 [ -n "$SUCCESS" ] && inform "$ME_NAME exported new keys to your profile. 1769 You should backup your changed profile folder now and store it in a safe place." 1770 } 1771 1772 # replace all non-alnum chars with underscore (for file operations) 1773 function gpg_key_legalize { 1774 echo $* | awk '{gsub(/[^a-zA-Z0-9]/,"_",$0); print}' 1775 } 1776 1777 function gpg_key_cache { 1778 local RES 1779 local MODE=$1 1780 shift 1781 local PREFIX="GPG_KEY" 1782 local SUFFIX=$(gpg_key_legalize "$@") 1783 local KEYID="$*" 1784 local CACHE="${PREFIX}_${MODE}_${SUFFIX}" 1785 if [ "$MODE" = "RESET" ]; then 1786 eval unset ${PREFIX}_PUB_$SUFFIX ${PREFIX}_SEC_$SUFFIX 1787 return 255 1788 elif ! var_isset "$CACHE"; then 1789 if [ "$MODE" = "PUB" ]; then 1790 RES=$(gpg $GPG_OPTS --list-key "$KEYID" > /dev/null 2>&1; echo -n $?) 1791 elif [ "$MODE" = "SEC" ]; then 1792 RES=$(gpg $GPG_OPTS --list-secret-key "$KEYID" > /dev/null 2>&1; echo -n $?) 1793 else 1794 return 255 1795 fi 1796 eval $CACHE=$RES 1797 fi 1798 eval return \$$CACHE 1799 } 1800 1801 function gpg_pub_avail { 1802 gpg_key_cache PUB "$@" 1803 } 1804 1805 function gpg_sec_avail { 1806 gpg_key_cache SEC "$@" 1807 } 1808 1809 function gpg_key_format { 1810 echo $1 | grep -q '^[0-9a-fA-F]\{8\}$' 1811 } 1812 1813 # splits a comma separated line into lines, respects escaped commas 1814 function gpg_split_keyset { 1815 local LIST 1816 LIST=$(echo "$@" | awk '{ gsub(/,/,"\n",$0); gsub(/\\\n/,",",$0); print $0 }') 1817 echo -e "$LIST" 1818 } 1819 1820 function gpg_prefix_keyset { 1821 local PREFIX="$1" OUT="" 1822 shift 1823 for KEY_ID in "$@"; do 1824 OUT="${OUT} $PREFIX $(qw ${KEY_ID})" 1825 done 1826 echo $OUT 1827 } 1828 1829 # grep a variable from conf text file (currently not used) 1830 function gpg_passwd { 1831 [ -r "$CONF" ] && \ 1832 awk '/^[ \t]*GPG_PW[ \t=]/{\ 1833 sub(/^[ \t]*GPG_PW[ \t]*=*/,"",$0);\ 1834 gsub(/^[ \t]*[\047"]|[\047"][ \t]*$/,"",$0);\ 1835 print $0; exit}' "$CONF" 1836 } 1837 1838 # return success if at least one secret key is available 1839 function gpg_key_decryptable { 1840 # no keys, no problem 1841 gpg_symmetric && return 0 1842 1843 local KEY_ID 1844 for KEY_ID in "${GPG_KEYS_ENC_ARRAY[@]}"; do 1845 gpg_sec_avail "$KEY_ID" && return 0 1846 done 1847 return 1 1848 } 1849 1850 function gpg_symmetric { 1851 [ -z "${GPG_KEY}${GPG_KEYS_ENC_ARRAY}" ] 1852 } 1853 1854 # checks for max two params if they are set, typically GPG_PW & GPG_PW_SIGN 1855 function gpg_param_passwd { 1856 var_isset GPG_USEAGENT && exit 1 1857 1858 if ( [ -n "$1" ] && var_isset "$1" ) || ( [ -n "$2" ] && var_isset "$2" ); then 1859 echo "--passphrase-fd 0 --batch" 1860 fi 1861 } 1862 1863 # select the earliest defined and create an "echo <value> |" string 1864 function gpg_pass_pipein { 1865 var_isset GPG_USEAGENT && exit 1 1866 1867 for var in "$@" 1868 do 1869 if var_isset "$var"; then 1870 echo "echo $(qw $(eval echo \$$var)) |" 1871 return 0 1872 fi 1873 done 1874 1875 return 1 1876 } 1877 1878 # checks if gpg-agent is available, returns error code 1879 # 0 on success 1880 # 1 if GPG_AGENT_INFO is not set (unused, should probably be merged w/ 3) 1881 # 2 if GPG_AGENT_INFO is stale 1882 # 3 cannot connect to gpg-agent 1883 function gpg_agent_avail { 1884 # GPG_AGENT_INFO is deprecated in gpg2.1, 1885 # first try to connect to a possibly running agent here 1886 local ERR=3 1887 gpg-agent > /dev/null 2>&1 && return 0 1888 1889 # detect stale pid in legacy GPG_AGENT_INFO env var 1890 if var_isset GPG_AGENT_INFO; then 1891 # check if a pid matching process is running at all 1892 local GPG_AGENT_PID=$(echo $GPG_AGENT_INFO|awk -F: '{print $2}') 1893 if isnumber "$GPG_AGENT_PID"; then 1894 ps -p "$GPG_AGENT_PID" > /dev/null 2>&1 || ERR=2 1895 fi 1896 fi 1897 1898 return $ERR 1899 } 1900 1901 function gpg_custom_binary { 1902 var_isset GPG && [ "$GPG" != "$DEFAULT_GPG" ] &&\ 1903 echo "--gpg-binary $(qw "$GPG")" 1904 } 1905 1906 function gpg_binary { 1907 local BIN 1908 var_isset GPG && BIN="$GPG" || BIN="$DEFAULT_GPG" 1909 echo "$BIN" 1910 } 1911 1912 function gpg_avail { 1913 lookup "$(gpg_binary)" 1914 } 1915 1916 # enforce the use of our selected gpg binary 1917 function gpg { 1918 command "$(gpg_binary)" "$@" 1919 } 1920 export -f gpg 1921 1922 # start of script ####################################################################### 1923 1924 # confidentiality first, all we create is only readable by us 1925 umask 077 1926 1927 # check if ftplicity is there & executable 1928 [ -n "$ME_LONG" ] && [ -x "$ME_LONG" ] || error "$ME missing. Executable & available in path? ($ME_LONG)" 1929 1930 if [ ${#@} -eq 1 ]; then 1931 cmd="${1}" 1932 else 1933 FTPLCFG="${1}" ; cmd="${2}" 1934 fi 1935 1936 # deal with command before profile validation calls 1937 # show requested version 1938 # OR requested usage info 1939 # OR create a profile 1940 # OR fall through 1941 ##if [ ${#@} -le 2 ]; then 1942 case "$cmd" in 1943 changelog) 1944 changelog 1945 exit 0 1946 ;; 1947 create) 1948 set_config 1949 if [ -d "$CONFDIR" ]; then 1950 error "The profile '$FTPLCFG' already exists in 1951 '$CONFDIR'. 1952 1953 Hint: 1954 If you _really_ want to create a new profile by this name you will 1955 have to manually delete the existing profile folder first." 1956 exit 1 1957 else 1958 create_config 1959 exit 0 1960 fi 1961 ;; 1962 txt2man) 1963 set_config 1964 usage_txt2man 1965 exit 0 1966 ;; 1967 usage|-help|--help|-h|-H) 1968 set_config 1969 usage_info 1970 exit 0 1971 ;; 1972 version|-version|--version|-v|-V) 1973 # profile can override GPG/PYTHON, so import it if it was given 1974 var_isset FTPLCFG && { 1975 set_config 1976 [ -r "$CONF" ] && . "$CONF" || warning "Cannot import config '$CONF'." 1977 } 1978 version_info_using 1979 exit 0 1980 ;; 1981 # fallthrough.. we got a command that needs an existing profile 1982 *) 1983 # if we reach here, user either forgot profile or chose wrong profileless command 1984 if [ ${#@} -le 1 ]; then 1985 error "\ 1986 Missing or wrong parameters. 1987 Only the commands 1988 changelog, create, usage, txt2man, version 1989 can be called without selecting an existing profile first. 1990 Your command was '$cmd'. 1991 1992 Hint: Run '$ME usage' to get help." 1993 fi 1994 esac 1995 1996 1997 # Hello world 1998 echo "Start $ME v$ME_VERSION, time is $(date_fix '%F %T')." 1999 2000 # check system environment 2001 2002 # is duplicity avail 2003 lookup duplicity || error_path "duplicity missing. installed und available in path?" 2004 # init, exec duplicity version check info 2005 duplicity_version_get 2006 2007 # check for certain important helper programs 2008 # TODO: we should probably check for duplicity and $PYTHON (if set) here too 2009 for f in awk grep ; do 2010 lookup "$f" || \ 2011 error_path "$f missing. installed und available in path?" 2012 done 2013 2014 ### read configuration 2015 set_config 2016 # check validity 2017 if [ ! -d "$CONFDIR" ]; then 2018 error "Selected profile '$FTPLCFG' does not resolve to a profile folder in 2019 '$CONFDIR'. 2020 2021 Hints: 2022 Select one of the available profiles: $(for d in "$(dirname "$CONFDIR")"/*/; do [ -e "$d" ] || [ -L "$d" ] || continue; printf "$sep'$(basename "$d")'"; sep=",";done) 2023 Use '$ME <name> create' to create a new profile. 2024 Use '$ME usage' to get usage help." 2025 elif [ ! -x "$CONFDIR" ]; then 2026 error "\ 2027 Profile folder in '$CONFDIR' cannot be accessed. 2028 2029 Hint: 2030 Check the filesystem permissions and set directory accessible e.g. 'chmod 700'." 2031 elif [ ! -f "$CONF" ] ; then 2032 error "'$CONF' not found." 2033 elif [ ! -r "$CONF" ] ; then 2034 error "'$CONF' not readable." 2035 else 2036 . "$CONF" 2037 #KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}" 2038 TEMP_DIR=${TEMP_DIR:-'/tmp'} 2039 # backward compatibility: old TARGET_PW overrides silently new TARGET_PASS if set 2040 if var_isset 'TARGET_PW'; then 2041 TARGET_PASS="${TARGET_PW}" 2042 fi 2043 fi 2044 echo "Using profile '$CONFDIR'." 2045 2046 # secure config dir, if needed w/ warning 2047 secureconf 2048 2049 # split TARGET in handy variables 2050 TARGET_SPLIT_URL=$(echo "$TARGET" | awk '{ \ 2051 target=$0; match(target,/^([^\/:]+):\/\//); \ 2052 prot=substr(target,RSTART,RLENGTH);\ 2053 rest=substr(target,RSTART+RLENGTH); \ 2054 if (credsavail=match(rest,/^[^@]*@/)){\ 2055 creds=substr(rest,RSTART,RLENGTH-1);\ 2056 credcount=split(creds,cred,":");\ 2057 rest=substr(rest,RLENGTH+1);\ 2058 # split creds with regexp\ 2059 match(creds,/^([^:]+)/);\ 2060 user=substr(creds,RSTART,RLENGTH);\ 2061 pass=substr(creds,RSTART+1+RLENGTH);\ 2062 };\ 2063 # filter quotes or escape them\ 2064 gsub(/[\047\042]/,"",prot);\ 2065 gsub(/[\047\042]/,"",rest);\ 2066 gsub(/[\047]/,"\047\\\047\047",creds);\ 2067 print "TARGET_URL_PROT=\047"prot"\047\n"\ 2068 "TARGET_URL_HOSTPATH=\047"rest"\047\n"\ 2069 "TARGET_URL_CREDS=\047"creds"\047\n";\ 2070 if(user){\ 2071 gsub(/[\047]/,"\047\\\047\047",user);\ 2072 print "TARGET_URL_USER=\047"user"\047\n"}\ 2073 if(pass){\ 2074 gsub(/[\047]/,"\047\\\047\047",pass);\ 2075 print "TARGET_URL_PASS=$(url_decode \047"pass"\047)\n"}\ 2076 }') 2077 eval "${TARGET_SPLIT_URL}" 2078 2079 # fetch commmand from parameters ######################################################## 2080 # Hint: cmds is also used to check if authentification info sufficient in the next step 2081 cmds="$2"; shift 2 2082 2083 # complain if command(s) missing 2084 [ -z $cmds ] && error " No command given. 2085 2086 Hint: 2087 Use '$ME usage' to get usage help." 2088 2089 # process params 2090 for param in "$@"; do 2091 #echo !$param! 2092 case "$param" in 2093 # enable ftplicity preview mode 2094 '--preview') 2095 PREVIEW=1 2096 ;; 2097 # interpret duplicity disable encr switch 2098 '--disable-encryption') 2099 GPG_KEY='disabled' 2100 ;; 2101 *) 2102 if [ `echo "$param" | grep -e "^-"` ] || \ 2103 [ `echo "$last_param" | grep -e "^-"` ] ; then 2104 # forward parameter[/option pairs] to duplicity 2105 dupl_opts["${#dupl_opts[@]}"]=${param} 2106 else 2107 # anything else must be a parameter (e.g. for fetch, ...) 2108 ftpl_pars["${#ftpl_pars[@]}"]=${param} 2109 fi 2110 last_param=${param} 2111 ;; 2112 esac 2113 done 2114 2115 # plausibility check config - VARS & KEY ################################################ 2116 # check if src, trg, trg pw 2117 # auth info sufficient 2118 # gpg key, gpg pwd (might be empty) set in config 2119 # OR key in local gpg db 2120 # OR key can be imported from keyfile 2121 # OR fail 2122 if [ -z "$SOURCE" ] || [ "$SOURCE" == "${DEFAULT_SOURCE}" ]; then 2123 error " Source Path (setting SOURCE) not set or still default value in conf file 2124 '$CONF'." 2125 2126 elif [ -z "$TARGET" ] || [ "$TARGET" == "${DEFAULT_TARGET}" ]; then 2127 error " Backup Target (setting TARGET) not set or still default value in conf file 2128 '$CONF'." 2129 2130 elif var_isset 'TARGET_USER' && var_isset 'TARGET_URL_USER' && \ 2131 [ "${TARGET_USER}" != "${TARGET_URL_USER}" ]; then 2132 error " TARGET_USER ('${TARGET_USER}') _and_ user in TARGET url ('${TARGET_URL_USER}') 2133 are configured with different values. There can be only one. 2134 2135 Hint: Remove conflicting setting." 2136 2137 elif var_isset 'TARGET_PASS' && var_isset 'TARGET_URL_PASS' && \ 2138 [ "${TARGET_PASS}" != "${TARGET_URL_PASS}" ]; then 2139 error " TARGET_PASS ('${TARGET_PASS}') _and_ password in TARGET url ('${TARGET_URL_PASS}') 2140 are configured with different values. There can be only one. 2141 2142 Hint: Remove conflicting setting." 2143 fi 2144 2145 # GPG config plausibility check1 (disabled check) ############################# 2146 if gpg_disabled; then 2147 : # encryption disabled, all is well 2148 elif [ -z "${GPG_KEY}${GPG_KEYS_ENC}${GPG_KEY_SIGN}" ] && ! var_isset 'GPG_PW'; then 2149 warning "GPG_KEY, GPG_KEYS_ENC, GPG_KEY_SIGN and GPG_PW are empty/not set in conf file 2150 '$CONF'. 2151 Will disable encryption for duplicity now. 2152 2153 Hint: 2154 If you really want to use _no_ encryption you can disable this warning by 2155 setting GPG_KEY='disabled' in conf file." 2156 GPG_KEY='disabled' 2157 fi 2158 2159 # GPG availability check (now we know if gpg is really needed)################# 2160 if ! gpg_disabled; then 2161 gpg_avail || error_path "gpg '$(gpg_binary)' missing. installed und available in path?" 2162 fi 2163 2164 # Output versions info ######################################################## 2165 using_info 2166 2167 # GPG create key settings, config check2 (needs gpg) ########################## 2168 if gpg_disabled; then 2169 : # the following tests are not necessary 2170 else 2171 2172 # we test this early as any invocation gpg2.1+ starts gpg-agent automatically 2173 GPG_AGENT_ERR=$(gpg_agent_avail ; echo $?) 2174 2175 # enc key still default? 2176 if [ "$GPG_KEY" == "${DEFAULT_GPG_KEY}" ]; then 2177 error_gpg "Encryption Key GPG_KEY still default in conf file 2178 '$CONF'." 2179 fi 2180 2181 # create array of gpg encr keys, for further processing 2182 OIFS="$IFS" IFS=$'\n' 2183 GPG_KEYS_ENC_ARRAY=( $( gpg_split_keyset ${GPG_KEY},${GPG_KEYS_ENC} ) ) 2184 IFS="$OIFS" 2185 2186 # pw set? 2187 # symmetric needs one, always 2188 if gpg_symmetric && ( [ -z "$GPG_PW" ] || [ "$GPG_PW" == "${DEFAULT_GPG_PW}" ] ) \ 2189 ; then 2190 error_gpg "Encryption passphrase GPG_PW (needed for symmetric encryption) 2191 is empty/not set or still default value in conf file 2192 '$CONF'." 2193 fi 2194 # this is a technicality, we can only pump one pass via pipe into gpg 2195 # but symmetric already always needs one for encryption 2196 if gpg_symmetric && var_isset GPG_PW && var_isset GPG_PW_SIGN &&\ 2197 [ -n "$GPG_PW_SIGN" ] && [ "$GPG_PW" != "$GPG_PW_SIGN" ]; then 2198 error_gpg "GPG_PW _and_ GPG_PW_SIGN are defined but not identical in config 2199 '$CONF'. 2200 This is unfortunately impossible. For details see duplicity manpage, 2201 section 'A Note On Symmetric Encryption And Signing'. 2202 2203 Tip: Separate signing keys may have empty passwords e.g. GPG_PW_SIGN=''. 2204 Tip2: Use gpg-agent." 2205 fi 2206 2207 # test - GPG KEY AVAILABILITY ################################################## 2208 2209 # check gpg public keys availability, try import if needed 2210 for (( i = 0 ; i < ${#GPG_KEYS_ENC_ARRAY[@]} ; i++ )); do 2211 KEY_ID="${GPG_KEYS_ENC_ARRAY[$i]}" 2212 # test availability, try to import, retest 2213 if ! gpg_pub_avail "${KEY_ID}"; then 2214 echo "Encryption public key '${KEY_ID}' not in keychain. Try to import from profile." 2215 gpg_import "${KEY_ID}" PUB 2216 gpg_key_cache RESET "${KEY_ID}" 2217 gpg_pub_avail "${KEY_ID}" || { \ 2218 gpg_testing && error_gpg \ 2219 "Needed public gpg key '${KEY_ID}' is not available in keychain." \ 2220 "Doublecheck if the above key is listed by 'gpg --list-keys' or available 2221 as gpg key file '$(basename "$(gpg_keyfile "${KEY_ID}")")' in the profile folder. 2222 If not you can put it there and $ME_NAME will autoimport it on the next run. 2223 Alternatively import it manually as the user you plan to run $ME_NAME with." 2224 } 2225 else 2226 echo "Public key '${KEY_ID}' found in keychain." 2227 fi 2228 done 2229 2230 # check gpg encr secret encryption keys availability and fail 2231 # if none is available after a round of importing trials 2232 gpg_key_decryptable || \ 2233 { 2234 echo "Missing secret keys for decryption in keychain." 2235 for (( i = 0 ; i < ${#GPG_KEYS_ENC_ARRAY[@]} ; i++ )); do 2236 KEY_ID="${GPG_KEYS_ENC_ARRAY[$i]}" 2237 # test availability, try to import, retest 2238 if ! gpg_sec_avail "${KEY_ID}"; then 2239 echo "Try to import secret key '${KEY_ID}' from profile." 2240 gpg_import "${KEY_ID}" SEC 2241 gpg_key_cache RESET "${KEY_ID}" 2242 fi 2243 done 2244 gpg_key_decryptable || \ 2245 { 2246 gpg_testing && error_gpg_test "None of the configured keys '$(join "','" "${GPG_KEYS_ENC_ARRAY[@]}")' \ 2247 has a secret key in the keychain. Decryption will be impossible!" 2248 } 2249 } 2250 2251 # gpg secret sign key availability 2252 # if none set, autoset first encryption key as sign key 2253 if ! gpg_signing; then 2254 echo "Signing disabled per configuration." 2255 # try first key, if one set 2256 elif ! var_isset 'GPG_KEY_SIGN'; then 2257 KEY_ID="${GPG_KEYS_ENC_ARRAY[0]}" 2258 if [ -z "${KEY_ID}" ]; then 2259 echo "Signing disabled. No GPG_KEY entries in config." 2260 GPG_KEY_SIGN='disabled' 2261 else 2262 # use avail OR try import OR fail 2263 if gpg_sec_avail "${KEY_ID}"; then 2264 GPG_KEY_SIGN="${KEY_ID}" 2265 else 2266 echo "Signing secret key '${KEY_ID}' not found." 2267 gpg_import "${KEY_ID}" SEC 2268 gpg_key_cache RESET "${KEY_ID}" 2269 if gpg_sec_avail "${KEY_ID}"; then 2270 GPG_KEY_SIGN="${KEY_ID}" 2271 fi 2272 fi 2273 2274 # interpret sign key setting 2275 if var_isset 'GPG_KEY_SIGN'; then 2276 echo "Autoset found secret key of first GPG_KEY entry '${KEY_ID}' for signing." 2277 else 2278 echo "Signing disabled. First GPG_KEY entry's '${KEY_ID}' private key is missing." 2279 GPG_KEY_SIGN='disabled' 2280 fi 2281 fi 2282 else 2283 KEY_ID="${GPG_KEY_SIGN}" 2284 if ! gpg_sec_avail "${KEY_ID}"; then 2285 inform "Secret signing key defined in setting GPG_KEY_SIGN='${KEY_ID}' not found.\nTry to import." 2286 gpg_import "${KEY_ID}" SEC 2287 gpg_key_cache RESET "${KEY_ID}" 2288 gpg_sec_avail "${KEY_ID}" || error_gpg_key "${KEY_ID}" "Private" 2289 fi 2290 fi 2291 2292 # using GPG_AGENT_ERR set early above, try to autoenable gpg-agent or issue some warnings 2293 # key enc can deal without, but might profit from gpg-agent 2294 # if GPG_PW is not set alltogether 2295 # if signing key is different from first (main) enc key (we can only pipe one pass into gpg) 2296 if ! gpg_symmetric && \ 2297 ( ! var_isset GPG_PW || \ 2298 ( gpg_signing && ! var_isset GPG_PW_SIGN && [ "$GPG_KEY_SIGN" != "${GPG_KEYS_ENC_ARRAY[0]}" ] ) ); then 2299 2300 if [ "$GPG_AGENT_ERR" -eq 1 ]; then 2301 warning "Cannot use gpg-agent. GPG_AGENT_INFO not set." 2302 elif [ "$GPG_AGENT_ERR" -eq 2 ]; then 2303 warning "Cannot use gpg-agent! GPG_AGENT_INFO contains stale pid." 2304 elif [ "$GPG_AGENT_ERR" -eq 3 ]; then 2305 warning "No running gpg-agent found although GPG_PW or GPG_PW_SIGN (enc != sign key) not set." 2306 else 2307 echo "Enable gpg-agent usage. Running gpg-agent instance found and GPG_PW or GPG_PW_SIGN (enc != sign key) not set." 2308 GPG_USEAGENT="--use-agent" 2309 fi 2310 fi 2311 2312 # end GPG config plausibility check2 2313 fi 2314 2315 # config plausibility check - SPACE ########################################### 2316 2317 # is tmp is a folder and writable 2318 CMD_MSG="Checking TEMP_DIR '${TEMP_DIR}' is a folder and writable" 2319 run_cmd test -d $(qw "$TEMP_DIR") '&&' test -w $(qw "$TEMP_DIR") 2320 if [ "$?" != "0" ]; then 2321 error "Temporary file space '$TEMP_DIR' is not a directory or writable." 2322 fi 2323 2324 2325 # get volsize, default duplicity volume size is 25MB since v0.5.07 2326 #VOLSIZE=${VOLSIZE:-25} 2327 # double if asynch is on 2328 #echo $@ $DUPL_PARAMS | grep -q -e '--asynchronous-upload' && FACTOR=2 || FACTOR=1 2329 2330 # TODO: check for enough (async= upload space and WARN only 2331 # use function tmp_space 2332 #echo TODO: reimplent tmp space check 2333 2334 2335 # test - GPG SANITY ##################################################################### 2336 # if encryption is disabled, skip this whole section 2337 if gpg_disabled; then 2338 echo -e "Test - En/Decryption skipped. (GPG='disabled')" 2339 elif ! gpg_testing; then 2340 echo -e "Test - En/Decryption skipped. (GPG_TEST='disabled')" 2341 else 2342 2343 GPG_TEST_PREFIX="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s)" 2344 function cleanup_gpgtest { 2345 echo -en "Cleanup - Delete '${GPG_TEST_PREFIX}_*'" 2346 rm "${GPG_TEST_PREFIX}"_* 2>/dev/null && echo "(OK)" || echo "(FAILED)" 2347 } 2348 2349 # signing enabled? 2350 if gpg_signing; then 2351 CMD_PARAM_SIGN="--sign --default-key $(qw ${GPG_KEY_SIGN})" 2352 CMD_MSG_SIGN="Sign with '${GPG_KEY_SIGN}'" 2353 fi 2354 2355 # using keys 2356 if [ ${#GPG_KEYS_ENC_ARRAY[@]} -gt 0 ]; then 2357 2358 for KEY_ID in "${GPG_KEYS_ENC_ARRAY[@]}"; do 2359 CMD_PARAMS="$CMD_PARAMS -r $(qw ${KEY_ID})" 2360 done 2361 # check encrypting 2362 CMD_MSG="Test - Encrypt to '$(join "','" "${GPG_KEYS_ENC_ARRAY[@]}")'${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}" 2363 run_cmd $(gpg_pass_pipein GPG_PW_SIGN GPG_PW) gpg $CMD_PARAM_SIGN $(gpg_param_passwd GPG_PW_SIGN GPG_PW) $CMD_PARAMS $GPG_USEAGENT --status-fd 1 $GPG_OPTS -o $(qw "${GPG_TEST_PREFIX}_ENC") -e $(qw "$ME_LONG") 2364 CMD_ERR=$? 2365 2366 if [ "$CMD_ERR" != "0" ]; then 2367 KEY_NOTRUST=$(echo "$CMD_OUT"|awk '/^\[GNUPG:\] INV_RECP 10/ { print $4 }') 2368 [ -n "$KEY_NOTRUST" ] && HINT="Key '${KEY_NOTRUST}' seems to be untrusted. If you really trust this key try to 2369 'gpg --edit-key "$KEY_NOTRUST"' and raise the trust level to ultimate. If you 2370 can trust all of your keys set GPG_OPTS='--trust-model always' in conf file." 2371 error_gpg_test "Encryption failed (Code $CMD_ERR).${CMD_OUT:+\n$CMD_OUT}" "$HINT" 2372 fi 2373 2374 # check decrypting 2375 CMD_MSG="Test - Decrypt" 2376 gpg_key_decryptable || CMD_DISABLED="No matching secret key available." 2377 run_cmd $(gpg_pass_pipein GPG_PW) gpg $(gpg_param_passwd GPG_PW) $GPG_OPTS -o $(qw "${GPG_TEST_PREFIX}_DEC") $GPG_USEAGENT -d $(qw "${GPG_TEST_PREFIX}_ENC") 2378 CMD_ERR=$? 2379 2380 if [ "$CMD_ERR" != "0" ]; then 2381 error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}" 2382 fi 2383 2384 # symmetric only 2385 else 2386 # check encrypting 2387 CMD_MSG="Test - Encryption with passphrase${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}" 2388 run_cmd $(gpg_pass_pipein GPG_PW) gpg $GPG_OPTS $CMD_PARAM_SIGN --passphrase-fd 0 -o $(qw "${GPG_TEST_PREFIX}_ENC") --batch -c $(qw "$ME_LONG") 2389 CMD_ERR=$? 2390 if [ "$CMD_ERR" != "0" ]; then 2391 error_gpg_test "Encryption failed.${CMD_OUT:+\n$CMD_OUT}" 2392 fi 2393 2394 # check decrypting 2395 CMD_MSG="Test - Decryption with passphrase" 2396 run_cmd $(gpg_pass_pipein GPG_PW) gpg $GPG_OPTS --passphrase-fd 0 -o $(qw "${GPG_TEST_PREFIX}_DEC") --batch -d $(qw "${GPG_TEST_PREFIX}_ENC") 2397 CMD_ERR=$? 2398 if [ "$CMD_ERR" != "0" ]; then 2399 error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}" 2400 fi 2401 fi 2402 2403 # compare original w/ decryptginal 2404 CMD_MSG="Test - Compare" 2405 [ -r "${GPG_TEST_PREFIX}_DEC" ] || CMD_DISABLED="File not found. Nothing to compare." 2406 run_cmd "test \"\$(cat '$ME_LONG')\" = \"\$(cat '${GPG_TEST_PREFIX}_DEC')\"" 2407 CMD_ERR=$? 2408 if [ "$CMD_ERR" = "0" ]; then 2409 cleanup_gpgtest 2410 else 2411 error_gpg_test "Comparision failed.${CMD_OUT:+\n$CMD_OUT}" 2412 fi 2413 2414 fi # end disabled 2415 2416 ## an empty line 2417 #echo 2418 2419 # Exclude file is needed, create it if necessary 2420 [ -f "$EXCLUDE" ] || touch "$EXCLUDE" 2421 2422 # export only used keys, if bkp not already exists ###################################### 2423 gpg_export_if_needed "${GPG_KEYS_ENC_ARRAY[@]}" "$(gpg_signing && echo $GPG_KEY_SIGN)" 2424 2425 2426 # command execution ##################################################################### 2427 2428 # urldecode url vars into plain text 2429 var_isset 'TARGET_URL_USER' && TARGET_URL_USER="$(url_decode "$TARGET_URL_USER")" 2430 var_isset 'TARGET_URL_PASS' && TARGET_URL_PASS="$(url_decode "$TARGET_URL_PASS")" 2431 2432 # defined TARGET_USER&PASS vars replace their URL pendants 2433 # (double defs already dealt with) 2434 var_isset 'TARGET_USER' && TARGET_URL_USER="$TARGET_USER" 2435 var_isset 'TARGET_PASS' && TARGET_URL_PASS="$TARGET_PASS" 2436 2437 TARGET_URL_PROT_lowercase="$(tolower "${TARGET_URL_PROT%%:*}")" 2438 2439 # issue some warnings 2440 case "$TARGET_URL_PROT_lowercase" in 2441 'cf+http') 2442 # info on missing AUTH_URL 2443 if ! var_isset 'CLOUDFILES_AUTHURL'; then 2444 inform "No CLOUDFILES_AUTHURL exported (in conf). 2445 Will use default which is probably rackspace." 2446 fi 2447 ;; 2448 'swift') 2449 # info on possibly missing AUTH_URL 2450 var_isset 'SWIFT_AUTHURL' ||\ 2451 warning "\ 2452 Swift will probably fail because the conf var SWIFT_AUTHURL was not exported!" 2453 ;; 2454 'rsync') 2455 # everything in url (this backend does not support pass in env var) 2456 # this is obsolete from version 0.6.10 (buggy), hopefully fixed in 0.6.11 2457 # print warning older version is detected 2458 duplicity_version_lt 610 && 2459 warning "\ 2460 Duplicity version '$DUPL_VERSION' does not support providing the password as 2461 env var for rsync backend. For security reasons you should consider to 2462 update to a version greater than '0.6.10' of duplicity." 2463 ;; 2464 esac 2465 2466 2467 # for all protocols we put username in url and pass into env var 2468 # for sec�rity reasons, we url_encode username to protect special chars 2469 # first sortout backends with special ways to handle password 2470 case "$TARGET_URL_PROT_lowercase" in 2471 'imap'|'imaps') 2472 var_isset 'TARGET_URL_PASS' && BACKEND_PARAMS="IMAP_PASSWORD=$(qw "${TARGET_URL_PASS}")" 2473 ;; 2474 *) 2475 # rest uses FTP_PASS var 2476 var_isset 'TARGET_URL_PASS' && \ 2477 BACKEND_PARAMS="FTP_PASSWORD=$(qw "${TARGET_URL_PASS}")" 2478 ;; 2479 esac 2480 # insert url encoded username into target url if needed 2481 if var_isset 'TARGET_URL_USER' && [ "$TARGET_URL_PROT_lowercase" != "file" ]; then 2482 BACKEND_URL="${TARGET_URL_PROT}$(url_encode "${TARGET_URL_USER}")@${TARGET_URL_HOSTPATH}" 2483 else 2484 BACKEND_URL="$TARGET" 2485 fi 2486 2487 2488 # protect eval from special chars in url (e.g. open ')' in password, 2489 # spaces in path, quotes) happens above in duplify() via quotewrap() 2490 SOURCE="$SOURCE" 2491 BACKEND_URL="$BACKEND_URL" 2492 EXCLUDE="$EXCLUDE" 2493 # since 0.7.03 --exclude-globbing-filelist is deprecated 2494 EXCLUDE_PARAM="--exclude$(duplicity_version_lt 703 && echo -globbing)-filelist" 2495 2496 # replace magic separators to command equivalents (+=and,-=or,[=groupIn,]=groupOut) 2497 cmds=$(awk -v cmds="$cmds" "BEGIN{ \ 2498 gsub(/\+/,\"_and_\",cmds);\ 2499 gsub(/\-/,\"_or_\",cmds);\ 2500 gsub(/\[/,\"_groupIn_\",cmds);\ 2501 gsub(/\]/,\"_groupOut_\",cmds);\ 2502 print cmds}") 2503 2504 # split commands by '_', preserve spaces even if not allowed :) 2505 IFS='_' read -ra CMDS_IN <<< "$(tolower "$cmds")" 2506 2507 # convert cmds to array, 2508 # post process, translate batch commands 2509 # ATTENTION: commands are lowercase from here on out 2510 declare -a CMDS 2511 for cmd in "${CMDS_IN[@]}"; do 2512 case "$cmd" in 2513 # backup -> [pre_bkp_post] 2514 'backup') 2515 CMDS=("${CMDS[@]}" groupin pre bkp post groupout) 2516 ;; 2517 # purgeAuto -> [purge purgeFull purgeIncr] depending on set conf vars 2518 'purgeauto') 2519 purgeAuto=${MAX_AGE:+ purge}${MAX_FULL_BACKUPS:+ purgefull}${MAX_FULLS_WITH_INCRS:+ purgeincr} 2520 [[ -z "$purgeAuto" ]] && error "Command 'fullAuto' was given but neither of the purge conf vars configured." 2521 CMDS=("${CMDS[@]}" groupin $purgeAuto groupout) 2522 ;; 2523 *) 2524 CMDS=("${CMDS[@]}" "$cmd") 2525 ;; 2526 esac 2527 done 2528 #echo $(IFS=',';echo "${CMDS[*]}") 2529 2530 unset FTPL_ERR 2531 2532 # run CMDS 2533 for cmd in "${CMDS[@]}"; 2534 do 2535 2536 ## init 2537 # raise index in cmd array for pre/post param 2538 var_isset 'CMD_NO' && CMD_NO=$((++CMD_NO)) || CMD_NO=0 2539 2540 unset CMD_VALUE CMD_NEXT CMD_PREV CND_NEXT CND_PREV 2541 2542 # get next cmd,cnd vars 2543 nextno=$(( $CMD_NO )) 2544 while ! var_isset 'CMD_NEXT' 2545 do 2546 nextno=$(($nextno+1)) 2547 if [ "$nextno" -lt "${#CMDS[@]}" ]; then 2548 CMD_VALUE=${CMDS[$nextno]} 2549 is_condition "$CMD_VALUE" && CND_NEXT="$CMD_VALUE" && continue 2550 is_groupMarker "$CMD_VALUE" && continue 2551 CMD_NEXT="$CMD_VALUE" 2552 else 2553 CMD_NEXT='END' 2554 fi 2555 done 2556 2557 # get prev cnd, cnds are skipped pseudocmds 2558 prevno=$(( $CMD_NO )); 2559 while ! var_isset 'CND_PREV' 2560 do 2561 prevno=$(($prevno-1)) 2562 if [ "$prevno" -ge 0 ]; then 2563 CMD_VALUE=${CMDS[$prevno]} 2564 is_condition "$CMD_VALUE" && CND_PREV="$CMD_VALUE" && break 2565 is_command "$CMD_VALUE" && break 2566 else 2567 break 2568 fi 2569 done 2570 2571 # get prev cmd command minus skipped commands, only executed 2572 prevno=$(( $CMD_NO - ${CMD_SKIPPED-0} )); 2573 while ! var_isset 'CMD_PREV' 2574 do 2575 prevno=$(($prevno-1)) 2576 if [ "$prevno" -ge 0 ]; then 2577 CMD_VALUE=${CMDS[$prevno]} 2578 is_condition "$CMD_VALUE" && CND_PREV="$CMD_VALUE" && continue 2579 is_groupMarker "$CMD_VALUE" && continue 2580 CMD_PREV="$CMD_VALUE" 2581 else 2582 CMD_PREV='START' 2583 fi 2584 done 2585 2586 function get_cmd_skip_count { 2587 # find closing bracket, get group skip count 2588 local nextno=$CMD_NO 2589 local GRP_OPEN=0 2590 local GRP_SKIP=0 2591 local CMD_VALUE 2592 while [ "$nextno" -lt "${#CMDS[@]}" ] 2593 do 2594 nextno=$(($nextno+1)) 2595 CMD_VALUE=${CMDS[$nextno]} 2596 GRP_SKIP=$(( ${GRP_SKIP} + 1 )); 2597 if is_command "$CMD_VALUE" && [ "$GRP_OPEN" -lt 1 ]; then 2598 break; 2599 elif [ "$CMD_VALUE" == 'groupin' ]; then 2600 GRP_OPEN=$(( ${GRP_OPEN} + 1 )) 2601 elif [ "$CMD_VALUE" == 'groupout' ]; then 2602 GRP_OPEN=$(( ${GRP_OPEN} - 1 )) 2603 if [ "$GRP_OPEN" -lt 1 ]; then 2604 break; 2605 fi 2606 fi 2607 done 2608 2609 echo $GRP_SKIP; 2610 } 2611 2612 # decision time: are we skipping already or dealing with condition "commands" or other non-cmds? 2613 unset SKIP_NOW 2614 if var_isset 'CMD_SKIP' && [ $CMD_SKIP -gt 0 ]; then 2615 # skip cnd/grp cmds silently 2616 is_command "$cmd" && echo -e "\n--- Skipping command $(toupper $cmd) ! ---" 2617 CMD_SKIP=$(($CMD_SKIP - 1)) 2618 SKIP_NOW="yes" 2619 elif ! var_isset 'PREVIEW' && [ "$cmd" == 'and' ] && [ "$CMD_ERR" -ne "0" ]; then 2620 CMD_SKIP=$(get_cmd_skip_count) 2621 # incl. this "cmd" 2622 CMD_SKIP=$(( $CMD_SKIP + 1 )) 2623 unset CMD_SKIPPED 2624 SKIP_NOW="yes" 2625 elif ! var_isset 'PREVIEW' && [ "$cmd" == 'or' ] && [ "$CMD_ERR" -eq "0" ]; then 2626 CMD_SKIP=$(get_cmd_skip_count) 2627 # incl. this "cmd" 2628 CMD_SKIP=$(( $CMD_SKIP + 1 )) 2629 unset CMD_SKIPPED 2630 SKIP_NOW="yes" 2631 elif is_condition "$cmd" || is_groupMarker "$cmd"; then 2632 unset 'CMD_SKIP'; 2633 SKIP_NOW="yes" 2634 fi 2635 2636 # let's do the skip now 2637 if [ -n "$SKIP_NOW" ]; then 2638 # sum up how many commands we actually skipped for the prev var routines 2639 CMD_SKIPPED=$((${CMD_SKIPPED-0} + 1)) 2640 continue 2641 fi 2642 2643 # save start time 2644 RUN_START=$(nsecs) 2645 2646 # export some useful env vars for external scripts/programs to use 2647 export PROFILE CONFDIR SOURCE TARGET_URL_PROT TARGET_URL_HOSTPATH \ 2648 TARGET_URL_USER TARGET_URL_PASS \ 2649 GPG_KEYS_ENC=$(join "\n" "${GPG_KEYS_ENC_ARRAY[@]}") GPG_KEY_SIGN \ 2650 GPG_PW CMD_PREV CMD_NEXT CMD_ERR CND_PREV CND_NEXT\ 2651 RUN_START 2652 2653 # user info 2654 echo; separator "Start running command $(toupper $cmd) at $(datefull_from_nsecs $RUN_START)" 2655 2656 case "$(tolower $cmd)" in 2657 'pre'|'post') 2658 if [ "$cmd" == 'pre' ]; then 2659 script=$PRE 2660 else 2661 script=$POST 2662 fi 2663 # script execution in a subshell, protect us from failures/var overwrites 2664 ( run_script "$script" ) 2665 ;; 2666 'bkp') 2667 duplify backup -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \ 2668 "$SOURCE" "$BACKEND_URL" 2669 ;; 2670 'incr') 2671 duplify incremental -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \ 2672 "$SOURCE" "$BACKEND_URL" 2673 ;; 2674 'full') 2675 duplify full -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \ 2676 "$SOURCE" "$BACKEND_URL" 2677 ;; 2678 'verify') 2679 TIME="${ftpl_pars[0]:+"-t ${ftpl_pars[0]}"}" 2680 duplify verify -- $TIME "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \ 2681 "$BACKEND_URL" "$SOURCE" 2682 ;; 2683 'verifypath') 2684 TIME="${ftpl_pars[2]:+"-t ${ftpl_pars[2]}"}" 2685 IN_PATH="${ftpl_pars[0]}"; OUT_PATH="${ftpl_pars[1]}"; 2686 ( [ -z "$IN_PATH" ] || [ -z "$OUT_PATH" ] ) && error " Missing parameter <rel_bkp_path> or <local_path> for verifyPath. 2687 2688 Hint: 2689 Syntax is -> $ME <profile> verifyPath <rel_bkp_path> <local_path> [<age>]" 2690 2691 duplify verify -- $TIME "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \ 2692 --path-to-restore "$IN_PATH" "$BACKEND_URL" "$OUT_PATH" 2693 ;; 2694 'list') 2695 # time param exists since 0.5.10+ 2696 TIME="${ftpl_pars[0]:+"-t ${ftpl_pars[0]}"}" 2697 duplify list-current-files -- $TIME "${dupl_opts[@]}" "$BACKEND_URL" 2698 ;; 2699 'cleanup') 2700 duplify cleanup -- "${dupl_opts[@]}" "$BACKEND_URL" 2701 ;; 2702 'purge') 2703 MAX_AGE=${ftpl_pars[0]:-$MAX_AGE} 2704 [ -z "$MAX_AGE" ] && error " Missing parameter <max_age>. Can be set in profile or as command line parameter." 2705 2706 duplify remove-older-than "${MAX_AGE}" \ 2707 -- "${dupl_opts[@]}" "$BACKEND_URL" 2708 ;; 2709 'purgefull') 2710 MAX_FULL_BACKUPS=${ftpl_pars[0]:-$MAX_FULL_BACKUPS} 2711 [ -z "$MAX_FULL_BACKUPS" ] && error " Missing parameter <max_full_backups>. Can be set in profile or as command line parameter." 2712 2713 duplify remove-all-but-n-full "${MAX_FULL_BACKUPS}" \ 2714 -- "${dupl_opts[@]}" "$BACKEND_URL" 2715 ;; 2716 'purgeincr') 2717 MAX_FULLS_WITH_INCRS=${ftpl_pars[0]:-$MAX_FULLS_WITH_INCRS} 2718 [ -z "$MAX_FULLS_WITH_INCRS" ] && error " Missing parameter <max_fulls_with_incrs>. Can be set in profile or as command line parameter." 2719 2720 duplify remove-all-inc-of-but-n-full "${MAX_FULLS_WITH_INCRS}" \ 2721 -- "${dupl_opts[@]}" "$BACKEND_URL" 2722 ;; 2723 'restore') 2724 OUT_PATH="${ftpl_pars[0]}"; TIME="${ftpl_pars[1]:-now}"; 2725 [ -z "$OUT_PATH" ] && error " Missing parameter target_path for restore. 2726 2727 Hint: 2728 Syntax is -> $ME <profile> restore <target_path> [<age>]" 2729 2730 duplify restore -- -t "$TIME" "${dupl_opts[@]}" "$BACKEND_URL" "$OUT_PATH" 2731 ;; 2732 'fetch') 2733 IN_PATH="${ftpl_pars[0]}"; OUT_PATH="${ftpl_pars[1]}"; 2734 TIME="${ftpl_pars[2]:-now}"; 2735 ( [ -z "$IN_PATH" ] || [ -z "$OUT_PATH" ] ) && error " Missing parameter <src_path> or <target_path> for fetch. 2736 2737 Hint: 2738 Syntax is -> $ME <profile> fetch <src_path> <target_path> [<age>]" 2739 2740 duplify restore -- --restore-time "$TIME" "${dupl_opts[@]}" \ 2741 --path-to-restore "$IN_PATH" "$BACKEND_URL" "$OUT_PATH" 2742 ;; 2743 'status') 2744 duplify collection-status -- "${dupl_opts[@]}" "$BACKEND_URL" 2745 ;; 2746 *) 2747 error " Unknown command '$cmd'. 2748 2749 Hint: 2750 Use '$ME usage' to get usage help." 2751 ;; 2752 esac 2753 2754 CMD_ERR=$? 2755 RUN_END=$(nsecs) 2756 RUNTIME=$(( $RUN_END - $RUN_START )) 2757 2758 # print message on error; set error code 2759 if [ "$CMD_ERR" -ne 0 ]; then 2760 error_print "$(datefull_from_nsecs $RUN_END) Task '$(echo $cmd|awk '$0=toupper($0)')' failed with exit code '$CMD_ERR'." 2761 FTPL_ERR=1 2762 fi 2763 2764 separator "Finished state $(error_to_string $CMD_ERR) at $(datefull_from_nsecs $RUN_END) - \ 2765 Runtime $(printf "%02d:%02d:%02d.%03d" $((RUNTIME/1000000000/60/60)) $((RUNTIME/1000000000/60%60)) $((RUNTIME/1000000000%60)) $((RUNTIME/1000000%1000)) )" 2766 2767 done 2768 2769 exit ${FTPL_ERR}