Using shell variables for command options

Florian Brucker 07/30/2018. 2 answers, 1.852 views
bash variable

In a Bash script, I'm trying to store the options I'm using for rsync in a separate variable. This works fine for simple options (like --recursive), but I'm running into problems with --exclude='.*':

$ find source

$ rsync -rnv --exclude='.*' source/ dest
sending incremental file list

sent 57 bytes  received 19 bytes  152.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

$ RSYNC_OPTIONS="-rnv --exclude='.*'"

$ rsync $RSYNC_OPTIONS source/ dest
sending incremental file list

sent 78 bytes  received 22 bytes  200.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

As you can see, passing --exclude='.*' to rsync "manually" works fine (.bar isn't copied), it doesn't work when the options are stored in a variable first.

I'm guessing that this is either related to the quotes or the wildcard (or both), but I haven't been able to figure out what exactly is wrong.

2 Answers

Kusalananda 08/03/2018.

In general, it's a bad idea to demote a list of separate items into a single string, no matter whether it's a list of command line options or a list of pathnames.

Using an array instead:

rsync_options=( -rnv --exclude='.*' )


rsync_options=( -r -n -v --exclude='.*' )

and later...

rsync "${rsync_options[@]}" source/ target

This way, the quoting of the individual options is maintained (as long as you double quote the expansion of ${rsync_options[@]}). It also allows you to easily manipulate the individual entries of the array, would you need to do so, before calling rsync.

In any Bourne-like shell, one may use the list of positional parameters for this:

set -- -rnv --exclude='.*'

rsync "$@" source/ target

Again, double quoting the expansion of $@ is critical here.

Tangentially related:

The issue is that when you put the two sets of option into a string, the single quotes of the --exclude option's value becomes part of that value. Hence,

RSYNC_OPTIONS='-rnv --exclude=.*'

would have worked¹... but it's better (as in safer) to use an array with individually quoted entries. Using an array also allows you to use things with spaces in them, if you would need to, and avoids having the shell perform filename generation (globbing) on the options.

¹ as long as $IFS is not modified and that there's no file whose name starts with --exclude=. in the current directory

Florian Brucker 08/05/2018.

@Kusalananda has already explained the basic problem and how to solve it, and the Bash FAQ entry linked to by @glenn jackmann also provides a lot of useful information. Here's a detailed explanation of what's happening in my problem based on these resources.

We'll use a small script that prints each of its arguments on a separate line to illustrate things (argtest.bash):


for var in "$@"
    echo "$var"

Passing options "manually":

$ ./argtest.bash -rnv --exclude='.*'

As expected, the parts -rnv and --exclude='.*' are split into two arguments, as they are separated by unquoted whitespace (this is called word splitting).

Also note that the quotes around .* have been removed: the single quotes tell the shell to pass their content without special interpretation, but the quotes themselves are not passed to the command.

If we now store the options in a variable as a string (as opposed to using an array), then the quotes are not removed:

$ OPTS="--exclude='.*'"

$ ./argtest.bash $OPTS

This is because of two reasons: the double quotes used when defining $OPTS prevent special treatment of the single quotes, so the latter are part of the value:

$ echo $OPTS

When we now use $OPTS as an argument to a command then quotes are processed before parameter expansion, so the quotes in $OPTS occur "too late".

This means that (in my original problem) rsync uses the exclude pattern '.*' (with quotes!) instead of the pattern .* -- it excludes files whose name starts with single quote followed by a dot and ends with a single quote. Obviously that's not what was intended.

A workaround would have been to omit the double quotes when defining $OPTS:

$ OPTS2=--exclude='.*'

$ ./argtest.bash $OPTS2

However, it's a good practice to always quote variable assignments because of subtle differences in more complex cases.

As @Kusalananda noted, not quoting .* would also have worked. I had added the quotes to prevent pattern expansion, but that wasn't stricly necessary in this special case:

$ ./argtest.bash --exclude=.*

It turns out that Bash does perform pattern expansion, but the pattern --exclude=.* doesn't match any file, so the pattern is passed on to the command. Compare:

$ touch some_file

$ ./argtest.bash some_*

$ ./argtest.bash does_not_exit_*

However, not quoting the pattern is dangerous, because if (for whatever reason) there was a file matching --exclude=.* then the pattern gets expanded:

$ touch -- --exclude=.special-filenames-happen

$ ./argtest.bash --exclude=.*

Finally, let's see why using an array prevents my quoting problem (in addition to the other advantages of using arrays to store command arguments).

When defining the array, word splitting and quote handling happens as expected:

$ ARRAY_OPTS=( -rnv --exclude='.*' )

$ echo length of the array: "${#ARRAY_OPTS[@]}"
length of the array: 2

$ echo first element: "${ARRAY_OPTS[0]}"
first element: -rnv

$ echo second element: "${ARRAY_OPTS[1]}"
second element: --exclude=.*

When passing the options to the command, we use the syntax "${ARRAY[@]}", which expands each element of the array into a separate word:

$ ./argtest.bash "${ARRAY_OPTS[@]}"
--exclude=.* - Download Hi-Res Songs

1 Elohim

Holding Hands flac

Elohim. 2019. Writer: Quinn XCII;Jeremy Zucker;Elohim;Danny Parra;Corey Berkowitz;Antonina Armato.
2 P!nk

Walk Me Home flac

P!nk. 2019. Writer: P!nk;Scott Harris;Nate Ruess.
3 Katy Perry

365 flac

Katy Perry. 2019. Writer: Zedd;Katy Perry;Caroline Ailin;Corey Sanders;Daniel Davidsen;Cutfather;Peter Wallevik.

Happy flac

DEAMN. 2019.
5 Ariana Grande

7 Rings flac

Ariana Grande. 2019. Writer: Ariana Grande;Richard Rodgers;TBHits;Njomza;Michael "Mikey" Foster;Kaydence;Tayla Parx;Scootie;Oscar Hammerstein II;Victoria Monét.
6 5 Seconds Of Summer

Who Do You Love flac

5 Seconds Of Summer. 2019. Writer: Andrew Taggart;Talay Riley;Oak;Sean Douglas;Luke Hemmings;Calum Hood;Ashton Irwin;Michael Clifford;Trevorious;Zaire Koalo.
7 Bonn

No Sleep flac

Bonn. 2019. Writer: Albin Nedler;Bonn;Martin Garrix.
8 Bruno Mars

Please Me flac

Bruno Mars. 2019. Writer: J. White Did It;DJ White Shadow;Philip Lawrence;Bruno Mars;Cardi B.
9 Brooks

Better When You're Gone flac

Brooks. 2019. Writer: David Guetta;Emma Lov Block;Ido Zmishlany;Jackson Foote;Jeremy Dussolliet;Brooks.
10 Ariana Grande

Imagine flac

Ariana Grande. 2019. Writer: JProof;Priscilla Renea;Happy Perez;Andrew "Pop" Wansel;Ariana Grande.
11 Avril Lavigne

Dumb Blonde flac

Avril Lavigne. 2019. Writer: Mitch Allan;Bonnie McKee;Nicki Minaj;Avril Lavigne.
12 Avril Lavigne

Birdie flac

Avril Lavigne. 2019. Writer: J.R. Rotem;Avril Lavigne.
13 Ariana Grande

Thank U, Next flac

Ariana Grande. 2019. Writer: Crazy Mike;Scootie;Victoria Monét;Tayla Parx;TBHits;Ariana Grande.
14 Ariana Grande

Fake Smile flac

Ariana Grande. 2019. Writer: Joseph Frierson;Mary Frierson;Fred Ball;Happy Perez;Andrew "Pop" Wansel;Wendy Rene;Kennedi Lykken;Priscilla Renea;Justin Tranter;Ariana Grande.
15 Avril Lavigne

Warrior flac

Avril Lavigne. 2019. Writer: Chad Kroeger;Travis Clark;Chris Baseford;Avril Lavigne.
16 Nichkhun

Home flac

Nichkhun. 2019.
17 Dua Lipa

Swan Song flac

Dua Lipa. 2019. Writer: Dua Lipa;Justin Tranter;Kennedi Lykken;Mattias Larsson;Robin Fredriksson;Junkie XL.
18 Dimitri Vegas & Like Mike

Selfish flac

Dimitri Vegas & Like Mike. 2019. Writer: Victor Thell;Michael Thivaios;Maria Smith;Jeff Porcaro;Dimitri Thivaios;David Paich;Dan Book;Will Grands;ANGEMI.
19 Jason Derulo

Let's Shut Up And Dance flac

Jason Derulo. 2019. Writer: Bongo.
20 Ariana Grande

Needy flac

Ariana Grande. 2019. Writer: Tayla Parx;TBHits;Victoria Monét;Ariana Grande.

Related questions

Hot questions


Popular Tags