sort and uniq in Linux: Count Frequencies Fast

sort and uniq are the quiet workhorses of text processing. Separately they are simple; together they answer the question every log file eventually asks: "how often does each thing happen?"

Sorting sensibly #

Sort alphabetically, then numerically

$ sort names.txt
$ sort -n sizes.txt

The -n flag matters more than it looks: alphabetical sort puts 9 after 10, and half of all sorting bugs trace back to forgetting it. For human-readable sizes there is -h, matching the output of ls and df:

Sort du output largest-first

$ du -sh */ | sort -rh

The flags that matter #

Flag	Effect
`-n`	Numeric sort
`-h`	Human numbers (2K, 1M, 4G)
`-r`	Reverse
`-u`	Unique while sorting
`-k2`	Sort by the second field
`-t,`	Field separator for -k
`-R`	Random shuffle

uniq counts things #

uniq collapses adjacent duplicates — adjacent being the catch. Sort first, always:

Count occurrences of each line

$ grep -o 'GET /[^ ]*' access.log | sort | uniq -c | sort -rn | head

Read that pipeline right to left, or rather left to right as the machine does: extract, sort, count, rank, trim. It is the single most useful pipe in this series, and it appears in awk analyses and log reviews alike.

-c prefixes counts, -d prints only duplicated lines, -u only unique ones — three flags that cover deletion audits, duplicate detection, and set arithmetic:

Find lines that appear exactly once

$ sort emails.txt | uniq -u

For unstructured searches feeding these counts, start with grep; for cutting the columns you count, cut, tr, and paste sit right beside this pair in the terminal notes.