Comment Comment by known cat 100GB_data_file_with_DUPLICATE_lines | parallel --pipe awk \'\!a[\$0]++\' > data_file_with_UNIQUE_linesis BETTER since it uses less computer resources thangawk '!a[$0]++' 100GB_data_file_with_DUPLICATE_lines > data_file_with_UNIQUE_lines parent known 10 years ago
is BETTER since it uses less computer resources than
gawk '!a[$0]++' 100GB_data_file_with_DUPLICATE_lines > data_file_with_UNIQUE_lines