Let's talk awk

[pidsley]

Most of you have probably heard me rave about awk. I am not an expert, by any means, but I do like to see what the language can do. Any time you find yourself scripting something with more than one call to grep or sed, there is probably an easier way to do the same thing with awk. When I see a grep/awk/sed/sed/cut command, I try to rewrite it with just awk, and I always learn something.

I came across a good example today. Say we want to show the current local ip address. There are other ways to do this, but one way is by parsing the output of "ifconfig <interface>" -- here are the first four lines:

Code: Select all

$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:0f:fe:7f:1c:5d  
          inet addr:192.168.0.163  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fd00::20f:feff:fe7f:1c5d/64 Scope:Global
          inet6 addr: fe80::20f:feff:fe7f:1c5d/64 Scope:Link

One way to get the address is like this:

Code: Select all

$ ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}' | head -1
192.168.0.163

One grep, two awks, and a head. There has to be a better way. Let's do it one step at a time. awk can grep, like this:

Code: Select all

$ ifconfig eth0 | awk '/inet addr/ {print}'
          inet addr:192.168.0.163  Bcast:192.168.0.255  Mask:255.255.255.0

This gives us just the line we want. We can tell awk to separate fields at the colon, and print just the second field:

Code: Select all

$ ifconfig eth0 | awk -F: '/inet addr/ {print $2}'
192.168.0.163  Bcast

Notice this gave us everything from the first colon to the second colon. Now we just need to get rid of the "Bcast" -- we could use sed, but awk can do this too.

Code: Select all

$ ifconfig eth0 | awk -F: '/inet addr/ {gsub (/  Bcast/, ""); print $2}'
192.168.0.163

As I said, this may not be the best way to do this, but I used it as an example and hope it showed you part of how powerful awk can be.

One more thing -- the very first thing you should ask yourself when you have a problem like this is "Is there another core command that will give me information that is closer to what I need, and therefore easier to parse?" In this example, we started with "ifconfig" -- what if we instead start with "ip addr"?

Code: Select all

$ ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0f:fe:7f:1c:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.163/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever

Much easier to parse. I will leave that for you as an experiment.

If you have something you're working on in awk and want help with, or something you did or found that seems particularly interesting, please post it here!

[machinebacon]

Thank you for this post, will be interesting as it grows. An utterly underrated program it is.

I'd like to add following references:
- http://www.grymoire.com/Unix/Awk.html
- http://linuxbbq.org/grillers/PDF/sed+%2 ... Arnold.pdf starting on page 142, but the previous chapters are interesting to understand regex.

[pidsley]

^ thank you bacon. I really need to spend some time with those references.

[machinebacon]

awk can do floating point:

Code: Select all

x=5;y=10;sum=$((x*y));echo $sum
50

x=5.5;y=10;sum=$((x*y));echo $sum
bash: 5.5: syntax error: invalid arithmetic operator (error token is ".5")

awk 'BEGIN {print 5.5*10}'
55

[pidsley]

^ great! works for division too

Code: Select all

$ awk 'BEGIN {print 100/3}'                                                                                            
33.3333

[GekkoP]

Thanks so much for this one. I need to get better with awk because I just know the basics. Great example in the first post.

[GekkoP]

Ok, using Pidsley tutorial I did something I found quite useful. Added this to my .zshrc:

Code: Select all

sizeof() {
    du -csh $1 | awk '/total/ {gsub (/ total/, ""); print $1}'
}

Result:

Code: Select all

manuel@bebop ~ sizeof videos
239M

[Alad]

Another link from Tim:

http://xmodulo.com/2014/07/use-awk-command-linux.html

The site has more interesting things as well.