Writing in comp.lang.awk
Ed Morton ports numerous complex sed expressions to Awk:
A comp.lang.awk author ask the question:
I have a file that has a series of lists
(qqq)
aaa 111
bbb 222
and I want to make it look like
aaa 111 (qqq)
bbb 222 (qqq)
IMHO the clearest sed solution given was:
sed -e '
/^([^)]*)/{
h; # remember the (qqq) part
d
}
/ [1-9][0-9]*$/{
G; # strap the (qqq) part to the list
s/\n/ /
}
' yourfile
while the awk one was:
awk '/^\(/{ h=$0;next } { print $0,h }' file
As I've said repeatedly, sed is an excellent tool for simple
substitutions on a single line. For anything else you should use awk,
perl, etc.
Having said that, let's take a look at the awk equivalents for the
posted sed examples below that are not simple substitutions on a single
line so people can judge for themselves (i.e. quietly - this is not a
contest and not a religious war!) which code is clearer, more
consistent, and more obvious. When reading this, just imagine yourself
having to figure out what the given script does in order to debug or
enhance it or write your own similar one later.
Note that in awk as in shell there are many ways to solve a problem so
I'm trying to stick to the solutions that I think would be the most
useful to a beginner since that's who'd be reading an examples page like
this, and without using any GNU awk extensions. Also note I didn't test
any of this but it's all pretty basic stuff so it should mostly be right.
For those who know absolutely nothing about awk, I think all you need to
know to understand the scripts below is that, like sed, it loops through
input files evaluating conditions against the current input record (a
line by default) and executing the actions you specify (printing the
current input record if none specified) if those conditions are true,
and it has the following pre-defined symbols:
NR = Number or Records read so far
NF = Number of Fields in current record
FS = the Field Separator
RS = the Record Separator
BEGIN = a pattern that's only true before processing any input
END = a pattern that's only true after processing all input.
Oh, and setting RS to the NULL string (-v RS='') tells awk to read
paragraphs instead of lines as individual records, and setting FS to the
NULL string (-v FS='') tells awk to treat each individual character as a
field.
For more info on awk, see http://www.awk.info.
Introductory Examples
Double space a file:
Sed:
sed G
Awk
awk '{print $0 "\n"}'
Double space a file which already has blank lines in it. Output file
should contain no more than one blank line between lines of text.
Sed:
sed '/^$/d;G'
Awk:
awk 'NF{print $0 "\n"}'
Triple space a file
Sed:
sed 'G;G'
Awk:
awk '{print $0 "\n\n"}'
Undo double-spacing (assumes even-numbered lines are always blank):
Sed:
sed 'n;d'
Awk:
awk 'NF'
Insert a blank line above every line which matches "regex":
Sed:
sed '/regex/{x;p;x;}'
Awk:
awk '{print (/regex/ ? "\n" : "") $0}'
Insert a blank line below every line which matches "regex":
Sed:
sed '/regex/G'
Awk:
awk '{print $0 (/regex/ ? "\n" : "")}'
Insert a blank line above and below every line which matches "regex":
Sed:
sed '/regex/{x;p;x;G;}'
Awk:
awk '{print (/regex/ ? "\n" $0 "\n" : $0)}'
Numbering
Number each line of a file (simple left alignment). Using a tab (see
note on '\t' at end of file) instead of space will preserve margins:
Sed:
sed = filename | sed 'N;s/\n/\t/'
Awk:
awk '{print NR "\t" $0}'
Number each line of a file (number on left, right-aligned):
Sed:
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
Awk:
awk '{printf "%6s %s\n",NR,$0}'
Number each line of file, but only print numbers if line is not blank:
Sed:
ed '/./=' filename | sed '/./N; s/\n/ /'
Awk:
awk 'NF{print NR "\t" $0}'
Count lines (emulates "wc -l")
Sed:
sed -n '$='
Awk:
awk 'END{print NR}'
Text Conversion and Substitution
Align all text flush right on a 79-column width:
Sed:
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space
Awk:
awk '{printf "%79s\n",$0}'
Center all text in the middle of 79-column width. In method 1,
spaces at the beginning of the line are significant, and trailing
spaces are appended at the end of the line. In method 2, spaces at
the beginning of the line are discarded in centering the line, and
no trailing spaces appear at the end of lines.
Print the last line of a file (emulates "tail -1")
Sed:
sed '$!d' # method 1
sed -n '$p' # method 2
Awk:
awk 'END{print $0}'
Print the next-to-the-last line of a file
Sed:
sed -e '$!{h;d;}' -e x # for 1-line files, print blank line
sed -e '1{$q;}' -e '$!{h;d;}' -e x # for 1-line files, print the line
sed -e '1{$d;}' -e '$!{h;d;}' -e x # for 1-line files, print nothing
Awk:
awk '{prev=curr; curr=$0} END{print prev}'
Print only lines which match regular expression (emulates "grep")
Sed:
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2
Awk:
awk '/regexp/'
Print only lines which do NOT match regexp (emulates "grep -v")
Sed:
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax
Awk:
awk '!/regexp/'
Print the line immediately before a regexp, but not the line
containing the regexp
Sed:
sed -n '/regexp/{g;1!p;};h'
Awk:
awk '/regexp/{print prev} {prev=$0}'
Print the line immediately after a regexp, but not the line
containing the regexp
Sed:
sed -n '/regexp/{n;p;}'
Awk:
awk 'found{print $0} {found=(/regexp/ ? 1 : 0)}'
Print 1 line of context before and after regexp, with line number
indicating where the regexp occurred (similar to "grep -A1 -B1")
The sed utility is a stream editor, a program that reads a stream of data, makes changes to it, and passes it on. It is often used to make global changes to a large file or to a stream of data generated by a pipeline of commands. While sed is a complicated program in its own right, its most common use is to perform global substitutions in the middle of a pipeline:
command1 < orig.data | sed 's/old/new/g' | command2 > result
Here, s/old/new/g tells sed to look for the regexp old on each input line and globally replace it with the text new, i.e., all the occurrences on a line. This is similar to awk's gsub function.
The following program, awksed.awk, accepts at least two command-line arguments: the pattern to look for and the text to replace it with. Any additional arguments are treated as data file names to process. If none are provided, the standard input is used:
# awksed.awk --- do s/foo/bar/g using just print
# Thanks to Michael Brennan for the idea
function usage()
{
print "usage: awksed pat repl [files...]" > "/dev/stderr"
exit 1
}
BEGIN {
# validate arguments
if (ARGC < 3)
usage()
RS = ARGV[1]
ORS = ARGV[2]
# don't use arguments as files
ARGV[1] = ARGV[2] = ""
}
# look ma, no hands!
{
if (RT == "")
printf "%s", $0
else
print
}
The program relies on gawk's ability to have RS be a regexp, as well as on the setting of RT to the actual text that terminates the record.
The idea is to have RS be the pattern to look for. gawk automatically sets $0 to the text between matches of the pattern. This is text that we want to keep, unmodified. Then, by setting ORS to the replacement text, a simple print statement outputs the text we want to keep, followed by the replacement text.
There is one wrinkle to this scheme, which is what to do if the last record doesn't end with text that matches RS. Using a print statement unconditionally prints the replacement text, which is not correct. However, if the file did not end in text that matches RS, RT is set to the null string. In this case, we can print $0 using printf.
The BEGIN rule handles the setup, checking for the right number of arguments and calling usage if there is a problem. Then it sets RS and ORS from the command-line arguments and sets ARGV[1] and ARGV[2] to the null string, so that they are not treated as file names.
The usage function prints an error message and exits. Finally, the single rule handles the printing scheme outlined above, using print or printf as appropriate, depending upon the value of RT.
The s2a project is a sed to awk conversion utility written in awk. As input it takes sed scripts, and it outputs an equivalent awk script.
This version should be fully functional as far as the following sed commands are concerned: a,d,s,p,q,c,i,n.
Commands to be implemented in the future: {},=,h,g,N,P,r,x,y,l,H,G,D,b,t,:
Bugs
$ is not a valid line address.
Also, line continuation with '\' is not implemented.
