The Unix people have thought of these things. You can easily do them with command line tools.
> Operational Supportability: How do you monitor the operation ?
Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget.
> Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ?
wget has command line options to skip downloading files that already exist. Or you can use tail to skip the number of lines in the input file as there exist complete entries in the destination directory.
> Maintainability: Can we run the same application on our desktop as on our production servers ?
I'm not sure how this is supposed to be an argument against using the standard utilities that are on everybody's machine already.
> Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?
Again, what? Extensibility is the wheelhouse of the thing you're complaining about.
> Operational Supportability: How do you monitor the operation ?
Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget.
> Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ?
wget has command line options to skip downloading files that already exist. Or you can use tail to skip the number of lines in the input file as there exist complete entries in the destination directory.
> Maintainability: Can we run the same application on our desktop as on our production servers ?
I'm not sure how this is supposed to be an argument against using the standard utilities that are on everybody's machine already.
> Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?
Again, what? Extensibility is the wheelhouse of the thing you're complaining about.