Useful bash one liners
Oct 19, 2015 · 3 minute read · CommentsDevopsLinux
Disclaimer: These are extremely simplified one liners that do not perform any form of input validation or character escaping. You should not use this in a ‘hostile’ environment where you have no idea what the input might be. You have been warned.
This is a list of some bash one liners I use on a daily basis during development and problem debugging. I made this list as a “you might not know this one yet” and will continue to update it every now and then.
Latest update: 2016/06/18.
Running a command on multiple files at once
This is a basic structure we will be re-using for the other examples. Run a command on all files in a directory:
for FILE in $(ls *); do command $FILE; done
Run a command for all lines in a file:
for LINE in $(cat file.txt); do command $LINE; done
Warning: As noted in the comments, this assumes there are no spaces in the lines in your file. If they do contain spaces, you need to add proper escaping using quotes.
Make certain things easier to read
Format a local XML file with proper indenting:
xmllint --format input.xml > output.xml
Run this script on all XML files in a directory:
for FILE in $(ls *.xml); do xmllint --format $FILE -o $FILE; done
Monitor new lines at the end of a log file and colorize the output (requires the package ccze):
tail -f /var/logsyslog | ccze
Find a specific text in a lot of files
Find a text inside a list of files and output the filename when a match occurs. Recurse and case-insensitive:
grep -irl foo*
Count the amount of files in a directory:
cd dir; ls -1 | wc -l
Find a filename that contains the string “foo”:
find ./ -name *foo*
Find all files modified in the last 7 days:
find ./ -mtime -7
And similar all files that have be modified more than 7 days ago:
find ./ -mtime +7
Modify files
chmod 644 all files in the current directory and below:
find . -type f -exec chmod 644 {} \;
chmod 755 all directories in the current directory and below:
find . -type d -exec chmod 755 {} \;
Commandline JSON formatting and parsing
The JQ command is a must-have for anything that returns JSON output on the command line:
curl url | jq '.'
I use this for finding the latest snapshot in a snapshot repository for elasticsearch:
curl -s -XGET "localhost:9200/_snapshot/my_backup/_all" | jq -r '.snapshots[-1:][].snapshot'
Actions against botnets and spammers
Find a list of all bots using a guestbook script to spam a site (that sadly has no captcha). I run this on the apache access_log file:
cat access_log | grep POST | grep guestbook | awk '{print $1}' | sort | uniq > ips.txt
The ips.txt file will now contain a list of unique ip addresses I want to ban with iptables:
for IP in $(cat ips.txt ); do iptables -I INPUT -s $IP -j REJECT; done
Cleanup stuff
Delete all but the 5 most recent items in a directory. I use this in Bamboo build scripts to clean up old releases during a deployment:
ls -1 --sort=time | tail -n +6 | xargs rm -rf -
That’s all for now!