Useful bash one liners

Oct 19, 2015 · 3 minute read · Comments
Devops Linux

Disclaimer: These are extremely simplified one liners that do not perform any form of input validation or character escaping. You should not use this in a ‘hostile’ environment where you have no idea what the input might be. You have been warned.

This is a list of some bash one liners I use on a daily basis during development and problem debugging. I made this list as a “you might not know this one yet” and will continue to update it every now and then.

Latest update: 2016/06/18.

Running a command on multiple files at once

This is a basic structure we will be re-using for the other examples. Run a command on all files in a directory:

for FILE in $(ls *); do command $FILE; done

Run a command for all lines in a file:

for LINE in $(cat file.txt); do command $LINE; done

Warning: As noted in the comments, this assumes there are no spaces in the lines in your file. If they do contain spaces, you need to add proper escaping using quotes.

Make certain things easier to read

Format a local XML file with proper indenting:

xmllint --format input.xml > output.xml

Run this script on all XML files in a directory:

for FILE in $(ls *.xml); do xmllint --format $FILE -o $FILE; done

Monitor new lines at the end of a log file and colorize the output (requires the package ccze):

tail -f /var/logsyslog | ccze

Find a specific text in a lot of files

Find a text inside a list of files and output the filename when a match occurs. Recurse and case-insensitive:

grep -irl foo*

Count the amount of files in a directory:

cd dir; ls -1 | wc -l

Find a filename that contains the string “foo”:

find ./ -name *foo*

Find all files modified in the last 7 days:

find ./ -mtime -7

And similar all files that have be modified more than 7 days ago:

find ./ -mtime +7

Modify files

chmod 644 all files in the current directory and below:

find . -type f -exec chmod 644 {} \;

chmod 755 all directories in the current directory and below:

find . -type d -exec chmod 755 {} \;

Commandline JSON formatting and parsing

The JQ command is a must-have for anything that returns JSON output on the command line:

curl url | jq '.'

I use this for finding the latest snapshot in a snapshot repository for elasticsearch:

curl -s -XGET "localhost:9200/_snapshot/my_backup/_all" | jq -r '.snapshots[-1:][].snapshot'

Actions against botnets and spammers

Find a list of all bots using a guestbook script to spam a site (that sadly has no captcha). I run this on the apache access_log file:

cat access_log | grep POST | grep guestbook | awk '{print $1}' | sort | uniq > ips.txt

The ips.txt file will now contain a list of unique ip addresses I want to ban with iptables:

for IP in $(cat ips.txt ); do iptables -I INPUT -s $IP -j REJECT; done

Cleanup stuff

Delete all but the 5 most recent items in a directory. I use this in Bamboo build scripts to clean up old releases during a deployment:

ls -1 --sort=time | tail -n +6 | xargs rm -rf -

That’s all for now!

Karel Bemelmans