2 ways to remove duplicate lines from Linux files
There are many ways to remove duplicate lines from a text file on Linux, but here are two that involve the awk and uniq commands and that offer slight- Community submission by Editor (9to6linux) on
There are many ways to remove duplicate lines from a text file on Linux, but here are two that involve the awk and uniq commands and that offer slightly different results.
The first command we'll examine in this post is a very unusual awk command that systematically removes every line in the file that is encountered more than once. It leaves the first instance of the line intact, but "remembers" it and removes any duplicates encountered afterwards.