Patterns and Ranges

Pattern kinds

A pattern is anything that evaluates to true or false, plus the two special forms BEGIN/END and the comma range.

awk '/^#/'                  # comment lines
awk 'NF == 0'               # blank lines
awk '$3 > 100'              # third field over 100
awk 'length($0) > 80'       # long lines
awk 'NR % 2'                # odd-numbered lines

Regex patterns and the match operators

/re/ on its own tests $0. To test any other string use ~ and !~.

awk '$1 ~ /^a/'             # first field starts with a
awk '$1 !~ /^a/'            # first field does not
awk '$0 ~ "^" prefix'       # dynamic regex from a string

- awk regexes are ERE, so +, ?, |, (), {n,m} all work with no backslash. \d, \w, \s, and lookarounds do not exist, use [0-9], [[:alnum:]_], [[:space:]]. - A dynamic regex is a plain string, so a backslash needs doubling: $0 ~ "\\." matches a literal dot, $0 ~ "\." is undefined behavior. - /re/ used where a value is expected means ($0 ~ /re/), which is why x = /a/ stores 1 or 0, not the pattern.

# case-insensitive without gawk's IGNORECASE
awk 'tolower($0) ~ /error/' app.log

Combining patterns

Boolean operators are &&, ||, !, and they short-circuit.

awk '/GET/ && $9 == 200'                 # both
awk '/4[0-9][0-9]/ || /5[0-9][0-9]/'     # either
awk '!/^$/'                              # not blank
awk 'NR > 1 && NF >= 3 && $2 != "-"'     # chained

Parentheses are only needed for clarity, && binds tighter than || as usual. ! binds tightest, so !/a/ && /b/ is (!/a/) && /b/.

Ranges with a comma

A range turns on at the first record matching the left pattern and off at the next record matching the right one. Both endpoints are inclusive.

awk '/BEGIN CERT/,/END CERT/' bundle.pem     # inclusive block
awk 'NR==10,NR==20' file                     # lines 10-20
awk '/START/,0'                              # from START to EOF

Gotchas that bite:

- The right pattern is tested on the same record, so a single line matching both is a one-record range. This is the opposite of sed, where addr2 is only tested from the following line onward. - A range can fire more than once in a file, it restarts every time the left pattern matches again. - If the right pattern never matches, the range runs to end of file. - To get the block without the delimiter lines, use a flag instead:

awk '/START/ { f=1; next } /END/ { f=0 } f' file

next, nextfile, and exit

These are the flow-control statements that operate on records, not lines of code.

# skip the header, then work
awk 'FNR==1 { next } { total += $2 } END { print total }' *.csv

# just the first line of each file, cheaply
awk 'FNR==1 { print FILENAME": "$0; nextfile }' logs/*.log

# stop reading a huge file as soon as you have the answer
awk '/FATAL/ { print; exit 1 }' huge.log

nextfile is not in POSIX 2008 but works in gawk, mawk, and current macOS awk (20200816). next inside BEGIN or END is an error everywhere.

Empty-pattern shorthands

Idioms that look cryptic but are just "pattern with the default action".

awk 'NF'                    # drop blank lines
awk '!seen[$0]++'           # dedupe, keeping first occurrence
awk '$2'                    # keep lines whose 2nd field is non-empty, non-zero
awk '/x/ || /y/'            # lines matching either
awk 'END{print NR}'         # wc -l

awk '$2' drops a line whose second field is the string "0", because a field that looks numeric is tested as the number 0, which is false. If you mean "field 2 is present", write awk '$2 != ""'.