This sub was set to restricted as all the moderators had left – so awk questions were generally ending up on /r/bash, which is not ideal. So I put in a request to take it over.

I've no great plans for this place – I just wanted to bring it back to life, so that redditors once again have a central place for questions and discussions about this venerable Unix scripting and text processing language.

Do you find in your experience that a surprisingly few number of people know how much you can do with awk, and that it makes a lot of more complex programs unnecessary?

Hey everyone!

I just published an article about using AWK in real-world scenarios based on my own experiences. I hope you'll find it helpful too! Feel free to check it out: https://0t1.me/blog/2024/09/01/practical-awk/

Thanks!

As I've done in past years, I'm doing the AoC2024 in awk. For those who want to follow along (or if you're doing the AoC in awk and want to compare your solutions with mine), I'm posting my solutions/spoilers in GitHub

I usually peter out around the A* algorithm puzzle (because A* in awk is particularly unpleasant, and it usually falls later when things get busy on the home-front), so I'm not guaranteeing that I'll finish all 25, but figured it might be of interest here.

Hi all,

I've long been a huge awk fan and use it for a lot of scripts and handy-one liners.

I built the website TypeQuicker which has code typing practice and recently added awk to it as an option.

I've always to be able to recall from memory (without having to look it up or ask an LLM) specific awk features / syntax so that I could write one-liners quickly. Practising touch typing while typing awk kinda helps me kill two birds with one stone so to speak

I'm really disappointed that r/awk has gone to sleep. (Awk is my lifeline.)

Seems to me that part of the reason is that a high proportion of the more complex Bash and command-line questions need (and get) an awk solution.

After all, awk can do almost anything that grep, sed, cut, paste and uniq can do, all in one process, and it runs about 50 times faster than shell for many things.

For my complex stuff, awk is about 5 times slower than C. Mostly, that does not much matter. Awk is way faster to develop, easier to refactor, and more portable.

Any idea how many of the 1.4k members here are actually active? What other communities do you belong to?

How about cross-posting relevant posts from Bash, command-line etc to awk solutions over here?

I used the pixels from the bird picture to recreate the zebra picture.

Source on GitHub https://github.com/zack-bitcoin/pixel-shaker

i've always used awk in my scripts, as a data extractor/transformer, but never as its own self, for direct scripting.

this week, i stumbled across zoxide, a smart cd written in rust, and thought i could write the "same idea" but using only posix shell commands. it worked and the script, ananas, can be seen here.

in the script, i used awk, since it was the simplest/fastest way to achieve what i needed.

this makes me thought : couldn't i write the whole script in awk directly, making it way efficient (in the shell script, i had to do a double swoop of the "database" file, whereas i could do everything in one go using awk).

now, it was an ultra pleasant coding session. awk is simple, fast and elegant. it makes for an amazing scripting language, and i might port other scripts i've rewritten to awk.

however, gawk shows worst performance than my shell script... i was quite disappointed, not in awk but in myself since i feel this must be my fault.

does anyone know a good time profiling (not line reached profiling a la gawk) for awk ? i would like to detect my script's bottleneck.

# shell posix
number_of_entries  average_resolution_time_ms  database_size  database_size_real
1                  9.00                        4.0K           65
10                 8.94                        4.0K           1.3K
100                9.18                        16K            14K
1000               9.59                        140K           138K
10000              13.84                       1020K          1017K
100000             50.52                       8.1M           8.1M

# mawk
number_of_entries  average_resolution_time_ms  database_size  database_size_real
1                  5.66                        4.0K           65
10                 5.81                        4.0K           1.3K
100                6.04                        16K            14K
1000               6.36                        140K           138K
10000              9.62                        1020K          1017K
100000             33.61                       8.1M           8.1M

# gawk
number_of_entries  average_resolution_time_ms  database_size  database_size_real
1                  8.01                        4.0K           65
10                 7.96                        4.0K           1.3K
100                8.19                        16K            14K
1000               9.10                        140K           138K
10000              15.34                       1020K          1017K
100000             70.29                       8.1M           8.1M

Nice information, well written too

In streaming mode, AWK processes input line by line and uses very little RAM, largely independent of file size. Stateful operations are different: they require accumulating data in memory — such as building associative arrays for deduplication (!x[$0]++) or field distribution (x[NF]++), storing records for multi-pass processing, or concatenating strings for aggregate outputs.

I compared the memory usage of basic stateful scenarios across gawk, mawk, and nawk.
Full article: https://awklab.com/memory-efficiency-awk-mawk-gawk

Dear all:

Recently I am trying to improve my TUI in awk. I've realized that there is one important difference between gawk, nawk and mawk.

After you use split function to split a variable into an array, and you want to loop over the array elements, what you would usually do it:

for (key in arr) {
    arr[key] blah
}

But I just realize that the "order" (I know the array in awk has no order, like a dictionary in python) of the for loop in nawk and mawk is actually messy. Instead of starting from 1 to the final key, it following some seemly random pattern when going through the array. gawk on the other hand is following the numerical order using this for loop syntax. Test it with the following two code blocks:

For gawk:

gawk 'BEGIN{
    str = "First\nSecond\nThird\nFourth\nFifth"
    split(str, arr, "\n");
    for (key in arr) {
	print key ", " arr[key]
    }
}'

For mawk or nawk:

mawk 'BEGIN{
    str = "First\nSecond\nThird\nFourth\nFifth"
    split(str, arr, "\n");
    for (key in arr) {
	print key ", " arr[key]
    }
}'

A complimentary way I figured it out is using the standard for loop syntax:

awk 'BEGIN{
    str = "First\nSecond\nThird\nFourth\nFifth"
    # get total number of elements in arr
    Narr = split(str, arr, "\n");
    for (key = 1; key <= Narr; key++) {
	print key ", " arr[key]
    }
}'

Hope this difference is helpful, and any comment is welcome!

Edit: To avoid confusion with GNOME Evolution, the project has been renamed,

Grub EvoDevo

I wrote EvoDevo, a highly configurable GRUB theme with scalable graphics, background blurring, and antialiased true-type fonts. I doubt many of you are into GRUB shaving, but the EvoDevo install script relies crucially on Awk to create the menu entries with smooth fonts. GRUB doesn't do font antialiasing, so the "menu entries" visible in the screenshot are actually fake entries created from the menu titles (which are text), converted to SVG and then PNG files.

I think this illustrates the continuing relevance of text-based formats and simple programmable tools (here, a POSIX shell and Awk) that combine well with one another. Something like the Unix philosophy in a GUI world :-)

Link:

https://github.com/ftonneau/grub-evodevo

Built this for myself but figured r/awk would find it interesting.

I use Awk to add missing features to my text editors and to perform the occasional ad-hoc transformation. I wanted a tool that would let me rapidly prototype against a selection of text (useful when a file needs multiple edits), an entire single file, or multiple files. Somewhere I could build up a collection of templates, snippets, and "chains" (i.e. sequence of snippets). The outcome would be to copy a one-liner, or to download a script. I didn't want an IDE; I was looking for something that fit my workflow.

I added a bunch of quality of life improvements, for example, automatic detection of the field separator (simply enough, but single-clicking this in feels good. It's nice to see a starting point being added in: BEGIN { FS = "," } { print $1, $2, $3 }), and a graphical auto-determination of field widths of a fixed column dataset. It has presets for common field patterns, these can be eye-balled before deciding or prototype your own and see the results as you change it. I included presets for strftime function, as well as the format code reference (e.g. what %Y vs %y do). And a larger general reference that summarize the gawk documentation. It's brief and reminds me without having to look things up externally. You can also assign shortcuts to your snippets/chains, and write tests for them. A neat features is being able to see the interim results of a series of transformation.

But what I enjoy most is the flow. Quickly write and apply an ad-hoc command, or search for and apply an existing snippet/chain, or chain together some existing snippets for a new purpose. It has a workspace to stash sets of work files away if your focus has to change to a new awk related task.

And it's pretty powerful as it can run other commands, too, for example 'jq'. Being able to run commands can be dangerous so I put safe guards in place. For example, preventing live-preview of commands that have side effects, blocking certain patterns entirely, or just turning off live-preview entirely to prevent premature execution of commands (ranging from mkdir to rm -rf)

Anyway, curious what people think. Let me know if you use it, or what you prefer instead (that does something similar).

Repo: https://github.com/gahrae/awk-estra

My tag line for Awk-estra (read: Orchestra) is "turn solos into symphonies".