Removing non-consecutive duplicate lines from a file

The uniq command will "Discard all but one of successive identical lines" from a file or input stream.

In order to remove non-consecutive duplicate lines, use awk:

awk '!x[$0]++' FILE