Comment by dugmartin

Isn't the >> going to cause urls.txt to grow geometrically?

eg, copying http://example.com/foo and then http://example.com/bar and then http://example.com/baz I think urls.txt would look like:

    http://example.com/foo
    http://example.com/foo
    http://example.com/bar
    http://example.com/foo
    http://example.com/bar
    http://example.com/baz

EDIT: I stand corrected. I should have run the code instead of mentally executing it.

One issue I did find when I ran it. The shebang should be:

    #!/bin/bash

as [[ is a bash built-in. I'm on Ubuntu - maybe this works on Mac as /bin/sh is an alias to /bin/bash?