Awk is being used all around the world for real programming problems,
but the news is not getting out.
We are aiming to create a database of at least one hundred Awk programs which will:
Identify the tasks that Awk is really being used for
Enable analysis of the benefits of the language for practical programming
Serve as an information exchange for applications
Contribute
If you, or your colleagues or friends have
written a program which has been used for purposes small or large,
why not take five minutes to record the facts, so that others can see what you've done?
To contribute, fill in this template
and mail it to mail@awk.info with the subject line Awk 100 contribution.
PatentMatrix is an automated tool to survey patents related to large sets of genes or proteins.
The tool allows a rapid survey of patents associated
with genes or proteins in a particular area of interest as
defined by keywords. It can be efficiently used to evaluate
the IP-related novelty of scientific findings and to rank
genes or proteins according to their IP position.
VitaminA IRC bot is an experiment on what can be done with GNU AWK. It's a very simple though powerful scripting language. Using the coprocess feature, plugins can be implemented very easily and in a language-independent way as a side-effect.
The project runs only on Unix-derived systems.
Plaiter (pronounced "player") is a command line front end to command line music players.
What does Plaiter do that (say) mpg123 can't already? It queues tracks, first of all. Secondly, it understands commands like play, plause, stop, next and prev. Finally, unlike most of the command line music players out there, Plaiter can handle a play list with more than one type of audio file, selecting the proper helper app to handle each type of file you throw at it.
Awk, impelemeneted in the Java virtual machine. Very useful for extending lightweight scripting in Awk with (e.g.) network and GUI facilities from Java.
Programmers often take awk "as is", never thinking to use it as a lab in which we can explore other language extensions. This is of course, only one way to treat the Awk code base.
An alternate approach is to treat the Awk code base as a reusable library of parsers, regular expression engines, etc etc and to make modifications to the lanugage. This second approach was take by David Ladd and J. Christopher Raming in their A* system.
Aaslg and aaslr implement the Amazing Awk Syntax Language, AASL
(pro- nounced ``hassle''). Aaslg (pronounced ``hassling'') takes
an AASL specification from the concatenation of the file(s) (default
standard input) and emits the corresponding AASL table on standard
output.
The AASL implementation is not large. The scanner is 78 lines of
awk,the parser is 61 lines of AASL (using a fairly low-density
paragraphing style and a good manycomments), and the semantics pass
is 290 lines of awk. The table interpreter is 340 lines, about half
of which (and most of the complexity) can be attributed to the
automatic error recovery.
As an experiment with a more ambitious AASL specification, one for
ANSI C was written. This occupies 374 lines excluding comments and
blank lines, and with the exception of the messy details of
C declarators is mostly a fairly straightforward transcription of the
syntax given in the ANSI standard.
"aaa" (the Amazing Awk Assembler) is a primitive assembler written entirely
in awk and sed. It was done for fun, to establish whether it was possible.
It is; it works.
Using "aaa", it's very easy to adapt to a new machine, provided the machine
falls into the generic "8-bit-micro" category.
For many years, Ronald Loui has taugh AI using Awk. He writes:
Most
people are surprised when I tell them what language we use in our
undergraduate AI programming class. That's understandable. We use
GAWK.
A repeated observation in this class is that only the scripting
programmers can generate code fast enough to keep up with the
demands of the class. Even though students
were allowed to choose any language they wanted, and many had
to unlearn the Java ways of doing things in order to benefit
from scripting, there were few who could develop ideas into
code effectively and rapidly without scripting.
What I have found not only surprising but also hopeful, is that
when I have approached the AI people who still enjoy
programming, some of them are not the least bit
surprised.
Awf may not be lightning fast, and it has certain restrictions, but it does a decent job on most manual pages and simple -ms documents, and isn't subject to AT&T's brain-damaged licensing that denies many System V users any text formatter at all. It is also a text formatter that is simple enough to be tinkered with, for people who want to experiment.
The stable and cross-platform nature of Awk enabled the simple creation of a robust toolkit for teaching operating system concepts to university
students. The toolkit is much simpler/ easier to port to new platforms, than alternative and more
elaborate course ware tools.
This work was the basis for a quite prestigious publication in the IEEE Transactions on Education journal, 2008, Vol 51, Issue 4. Who said
Awk was an old-fashioned tool?
Supports the essential operations of defining strings and replacing strings in text by their definitions.
All in 110 lines.
A little awk goes a long way.
Arnold Robbins and Nelson Beebe's classic
spell checker
A powerful spell checker, and a case-study on how to best write applications using hundreds of lines of Awk.
Some must lead, some must follow, and some have to fix the typos.
A Great Auk is someone with write permission to
our repository. Since the source for this web
site is stored in that repoistory, it also means that they are
webmasters of this site. So they (try) to:
keep the code and pages in a (somewhat) consistent form,
encourage code
documentation and test suites,
watch comp.lang.awk for cool stuff to add to this site,
write little demo programs,
handle
queries about this site,
work the issue reports,
etc.
If you want to be a Great Auk, please start contributing to this site using any of the usual methods. Once it is clear that you
know what you are doing and that you
play nice with others, then you should ask a current Great Auk to nominate you. Then, all the current Great Auks will vote about giving
your write access.
"Listen to people who program, not to people who want to tell you how to program."
- Ronald P. Loui
"Good design is as little design as possible."
- Dieter Rams
"When we have on occasion rewritten an Awk program in a conventional programming language like C or C++, the result was usually much longer, and much harder to debug." - Arnold Robbins & Nelson Beebe
AWK is a superb language for testing algorithms and applications with some complexity, especially where the problem can be broken into chunks which can streamed as part of a pipe. It's an ideal tool for augmenting the features of shell programming as it is ubiquitous; found in some form on almost all Unix/Linux/BSD systems. Many problems dealing with text, log lines or symbol tables are handily solved or at the very least prototyped with awk along with the other tools found on Unix/Linux systems.
(Other languages like Perl is) a good programming language for
writing self-contained programs, but pre-compilation and
long start-up time are worth paying only if once the program has
loaded it can do everything in one go. This contrasts sharply with
the Operator-stream Paradigm, where operators are chained together
in pipelines of two, three or more programs. The overhead associated
with initializing (say) Perl at every stage of the pipeline makes pipelining
inefficient. A better way of manipulating structured ASCII
files is to use the AWK programming language, which is much smaller,
more specialized for this task, and is very fast at
startup.
Awk is a stable, cross platform computer language named for its
authors
Alfred Aho,
Peter Weinberger &
Brian Kernighan. They write:
"Awk is a convenient and expressive programming language that can be
applied to a wide variety of computing and data-manipulation tasks".
In Classic
Shell Scripting, Arnold Robbins & Nelson Beebe confess their Awk bias:
"We like it. A lot.
The simplicity and power of
Awk often make it just the right tool for the
job."
Besides the Bourne shell, Awk is the only other scripting language available in the standard Unix environment.
Implementations of AWK exist as installed software for almost all other operating systems.
Awk is a mature language- it was first implemented in the 1970s.
As a tool from the golden age, it is sometimes called primitive.
It is more accurate to call it
elemental, so tightly focused is the language on what it does best:
quickly converting this into that.
Consequently, throughout history, Awk has been the language of choice for
many famous scientists such as
Leonardo daVinci.
Many communities have a mascot, a banner that they proudly wave high. So where's the Awk mascot?
I made on up, but you gotta say, it is kinda lame:
So you have any ideas for such a mascot, please email mail@awk.info
with the subject line "suggestion for mascot".
Not to stiffle anyone's creativity but the mascot might be
based on the mantra "less, but better" or "easy is not wrong" or
"a little awk goes a long way".
Current Offerings
Chris Johnson
Chris writes "more of a logo rather than a mascot":
Markdown- an ultra lightweight HTML markup language;
Intricate:
Awk++- enables object-oriented programming in Awk;
AwkLisp- a fully functioning LISP interpreter, written in Awk.
Interestingly, without comments, the LISP interpreter is only three times longer than the HTML markup language.
This comments either on the power of Awk, the regularity of LISP's core semantics, or both.
A standard idiom in Gawk is to reset the random number generator in a BEGIN block.
BEGIN {srand() }
Sadly, when called with no arguments, this "reseeding" uses time-in-seconds. So if the same "random"
task runs multiple times in the same second, it will get the same random number seed.
Houston, We Have a Problem
"Ben" writes:
I have a Gawk script that puts random comments into a file. It is run 3
times in a row in quick succession. I found that seeding the random
number generator using gawk did not work because all 3 times it was run
was done within the same second (and it uses the time).
I was wondering if anyone could give me some suggestions as to
what can be done to get around this
problem.
Solution #1: Persistent Memory
Kenny McCormack writes:
When last I ran into this problem, what I did was to save the last value
returned by rand() to a file, then on the next run, read that in and use
that value as the arg to srand(). Worked well.
(Editor's comment: Kenny's solution does work well but incurs the cost of maintaining and reading/writing that
"last value" file.)
Solution #2: Use Bash
Tim Menzies writes:
How about setting the seed using the BASH $RANDOM variable:
If referenced multiple times in a second, it always generates
a different number.
In the above usage, if we have a seed, use it. Else, no seed so start all "random" at the same place. If you prefer to
use the default "seed from time-in-seconds" then use:
BEGIN { if (Seed) { srand(Seed) } else { srand() } }
(Editor's comment: Tim's solution incurs the overhead of additional command-line syntax. However, it does allow
the process calling Gawk to control the seed. This is important when trying to, say, debug code by recreating the sequence
of random numbers that lead to the bug.)
Solution #3: Query the OS
Thomas Weidenfeller writes:
Is that good enough (random enough) for your task?
BEGIN {
"od -tu4 -N4 -A n /dev/random" | getline
srand(0+$0)
}
(Editor's comment: Nice. Thomas' solution reminds us that "Gawk" can access a whole host of operating system
facilities.)
Solution #4: Use the Process Id
Aharon Robbins writes:
You could so something like add PROCINFO["pid"] to the value of the time,
or use that as the seed.
This web site is a front end to a
repository
of Awk code.
The site, and the code, is maintained
by the international awk community (which includes you)
so there are many ways you can contribute:
Test the page by placing it on a publicly readable site, then see if it renders ok.
Email the url of that page to mail@awk.info. Do NOT send the page.
When writing a page, please follow these guidelines:
Do not use <hr> tags: these are reserved for dividing pages in a multi-page view.
Use only one <h1> tag at the top of page. Everything else should
<h2> or below.
Try to avoid using tricky CSS/HTML styling tricks. Vanilla HMTL is best.
The page you write will end up being rendered as the
middle pane of this site (around 550 pixels wide). So don't
write wide pages.
If you include code samples, note that our CSS wraps pre-formatted
code if it gets too wide. For example, at the time
of this writing, the following pre-formatted texts
gets ugly after about 75 characters:
In the language of this site, a function file is a 100% standalone file containing one or more
functions with no dependancies on other files.
Note that if your function file depends on other files, then it becomes a package (see below).
Functions are stored in a file caled myfunc.awk.
Add a Package
In the language of this site, a package is a file that depends on other files
(and the other files may depend on yet others, recursively).
Following a recent discussion in comp.lang.awk, we say that these dependancies are commented with
#use file.awk
where file.awk is some
file (e.g. a file in the current directory).
Note that :
file.awk will be loaded before the file containing the reference to #use file.awk.
This script columnates the input file, so that columns line up like in the GNU column(1) command. Its output is like that of column -t. First, awk reads the whole file, keeps track of the maximum width of each field, and saves all the lines/records. At the END, the lines are printed in columnated format. If your terminal is not too narrow, you'll get a handsome display of the file.
Code
{ line[NR] = $0 # saves the line
for (f=1; f<=NF; f++) {
len = length($f)
if (len>max[f])
max[f] = len } # an array of maximum field widths
}
END {
for(nr=1; nr<=NR; nr++) {
nf = split(line[nr], fields)
for (f=1; f<nf; f++)
printf "%-*s", max[f]+2, fields[f]
print fields[f] } # the last field need not be padded
}
Here is some Awk code from the Rosetta Code
wiki hat multiplyes integers using only addition, doubling, and halving.
How?
Take two numbers to be multiplied and write them down at the top of two columns.
In the left-hand column repeatedly halve the last number, discarding any remainders, and write the result below the last in the same column, until you write a value of 1.
In the right-hand column repeatedly double the last number and write the result below. stop when you add a result in the same row as where the left hand column shows 1.
Examine the table produced and discard any row where the value in the left column is even.
Sum the values in the right-hand column that remain to produce the result of multiplying the original two numbers together
For example: 17 X 34
17 34
Halving the first column:
17 34
8
4
2
1
Doubling the second column:
17 34
8 68
4 136
2 272
1 544
Strike-out rows whose first cell is even:
17 34
8 --
4 ---
2 ---
1 544
Sum the remaining numbers in the right-hand column:
17 34
8 --
4 ---
2 ---
1 544
====
578
So 17 multiplied by 34, by the Ethiopian method is 578.
The task is to define three functions/methods/procedures/subroutines:
one to halve an integer,
one to double an integer, and
one to state if an integer is even.
Code
function halve(x) { return(int(x/2)) }
function double(x) { return(x*2) }
function iseven(x) { return((x%2) == 0) }
function ethiopian(plier, plicand) {
r = 0
while(plier >= 1) {
if ( !iseven(plier) ) {
r += plicand
}
plier = halve(plier)
plicand = double(plicand)
}
return(r)
}
BEGIN { print ethiopian(17, 34) }
The program is called TAWK for "tiny awk," since the problem it solves is vaguely reminiscent of the "awk" pattern-matching language found on Unix (versions have also been created for DOS).
It demonstrates one of the thornier problems in computer science: parsing and executing a
programming language.
The data-encapsulation features of C++ prove most useful here, and a recursive-descent technique is used to read arbitrarily long fields and records.
Some of the code at awk.info is somewhat historical in nature. For example, Scott Pakin's gender
predictor was written in 1991. Given that, it might be mistakenly concluded that Awk is somehow old-fashioned and not suitable
for modern tasks.
Text mining, on the other hand, could be the killer app for Awk in the 21st century.
The language excels at creating one-off reports that handle the quirks
of a particular file format.
There is a growing interest in using Awk for this kind of work.
All the examples presented below come from work conducted in 2007, 2008:
Why Text Mining?
If we could properly understand unstructured text, this would
be a result of tremendous practical importance. A recent study
concluded that:
80 percent of business is conducted on unstructured information;
85 percent of all data stored is held in an unstructured
format;
Unstructured data doubles every three months;
That is, if we can tame the text mining problem, it would
be possible to reason and learn from a much wider range of
business data than ever before.
Further, despite its maturity (polite way of saying "written a very long time ago"),
Awk still runs as fast (or faster) than supposedly more modern languages (perhaps because it its simple- fewer layers between the data
and the processing). Donald McCarthy's comparative analysis
show that Awk is not necessarily slower
than certain other script-based text miners.
Note that, in the Menzies/Marcus and Schmitt/Christianson tool kits,
Awk by itself was not enough. The two data mining toolkits mentioned
above were all intricate combinations of Awk and sed and bash and etc
end etc. Within that combination, Awk was very useful for handling
the specifics not managed by the other tools.
;
