Use sed to add a newline to the end of a file (this will perform an in-place replacement and create a backup file, MYFILE.bak):

$ sed -i.bak '$G' MYFILE

Some programs, such as Emacs, will omit the trailing newline from a file unless configured to add one. Emacs can be configured to add a newline automatically.

Show all DNS in 192.168.0.*; only if reverse DNS is enabled on DNS server:

$ for I in `seq 1 254`; do host 192.168.0.$I | grep ^[^Host] | sed "s/.in-addr.*pointer/ points to ---->\t/"; done

If you need a process to survive your logging out of the system (or being knocked off of it), you can either screen it or wrap it with nohup. If you nohup, you'll lose control of it entirely, but the output will be written to nohup.out. With screen you can re-attach and take control later:

# nohup program -opts args &

-or-

# screen (press 'ctrl+a d' to detach after you've run the proc)

You can use awk to change the field separator in a data stream:

$ cat foo
a,b,c,d,e,f
a,b,c,d,e,f
$ awk 'BEGIN { FS = ","; OFS = ".."; } { $1 = $1; print }' foo
a..b..c..d..e..f
a..b..c..d..e..f

(You just have to touch one of the fields to get it to process the line.)

Ever been on a machine that was ailing and just wouldn't respond? As soon as you're root, lower the priority of the offending process ID(s) (in this example, 1103) by using the 'renice' command:

# renice -19 1103

There are a few different ways to convert a DOS file to Unix format, but the most available is probably this:

$ col -bx < dosfile > unixfile

Convert a man page to HTML for easy viewing online:

$ gunzip < /usr/local/man/man8/lsof.8.gz | nroff -man | man2html -title "lsof" > otmp/lsof.html

Here is a way to copy an entire filesystem without descending down its subsumed mount points. This example uses the root filesystem:

$ find / -xdev | cpio -pm /desired/location

If you need to see what queries are coming through your nameserver, toggle the querylog flag by typing:

$ rndc querylog

Tired of typing your SSL password on boot of your webserver? You can decrypt it if you're certain it's safe:

# openssl rsa -in server.key -out server.key.unsecure

You've entered a command with an option that didn't exist and somehow you now have a file that starts with a hyphen (i.e., -myfile). If you try to remove it, rm thinks that you're offering up the options m, y, f, i, l, and e. No good, but you can give this a try instead:

$ rm -- -myfile

The double-hyphen will disable option parsing and let you delete the file with no problem.

Make a recursive copy of your source tree before any changes (pristine version). E.g., copy project/ to project-pristine/.

After the changes, run this command:

$ diff -crN old new > output.diff
- or -
$ diff -crN project-pristine project > name_of_change.diff

On the target system in /path/to/project/../:

$ patch -p0 --dry-run < ./name_of_change.diff

Remove --dry-run to make it take effect.

To create a .ico file that can be used for the favicon.ico on webpages, you can use a nice little program called png2ico. There is no package for this in Debian or Ubuntu so you will need to compile it yourself. Compilation is very straightforward. You do need a libpng package. Most of the time it will be installed but if you see an error, just searching for that file will yield the package you need to get for your distribution.

The session below uses /usr/local/src for the compilation and installed to /usr/local/bin. It can be compiled anywhere and installed anywhere, but having it in your $PATH makes things easier:

$ cd /usr/local/src
$ wget http://www.winterdrache.de/freeware/png2ico/data/png2ico-src-2002-12-08.tar.gz
$ tar xvzf png2ico-src-2002-12-08.tar.gz
$ cd png2ico
$ make
$ sudo cp png2ico /usr/local/bin/
$ sudo cp doc/png2ico.1 /usr/local/man/man1/

Once compiled, upload your png images. They should be square; sizes 16x16 and 32x32 are suggested. To create the icon you would then call the following:

$ png2ico favicon.ico your_favicon16x16.png your_favicon32x32.png

Afterwards you will then have a favicon.ico file. You then place this in the webroot of your website add the following to <head></head> section of your webpages:

<link rel="shortcut icon" href="/favicon.ico" />

It's good form to include it, but most common browsers automatically look for an icon at that location.

You need to get someone into an internal machine that doesn't have a public IP? Use an SSH tunnel. For this example, machine_a is your internal machine and machine_b is external:

$ ssh -R 9000:localhost:22 you@machine_b

Once you've logged in, you should be able to run this on machine_b:

$ ssh -p 9000 you@localhost

bzip2 -c largefile.txt | ssh user@hostname "bzip2 -dc > /home/user/myDir/largefile.txt"

Bascially it bzips the file on the fly whilst sending over ssh and decompressing at the other end on the target machine. Very useful for send large files over a vpn or slow connection.

It also saves time by combining all the processes into one.

I know you can use scp -C, but this is faster as it uses bzip.

You can also increase the compression on bzip to make the transfer size smaller, though this will increase processing time however if it's a very slow connection this should not matter.

bzip -c1 will compress at the highest compression ratio

If you want to tail the errors on another terminal, just push them to a fifo:

$ mkfifo pgerror
$ psql -U user dbname < ./dbfile 2> pgerror

On your other terminal:

$ tail -f pgerror

Voila! STDOUT on the main terminal, and STDERR on the secondary.

To zip up the contents of a directory (selectively), use find and zip:

$ find . -type f -name "*.jpg" | zip -@ myimages.zip

If you need to find the process hogging your CPU, try this:

$ ps aux | awk '!/root|nobody/ { if ($4>2) {print $2}}'

An often-used concept in shell scripting is globbing. You can use this in python, as well:

import glob

for textFile in glob.glob("*.txt"):
    # Do something with 'textFile'.

You can have grep show only the filenames that matched (instead of showing all lines in those files that matched) by using the -l switch:

$ grep -l SELECT *.sql
file1.sql
file2.sql

grep will only list unique filenames (i.e. a file with two matches will only be listed once).

A shell script which runs a sequence of commands will run them all even if some of them fail. This is rarely desirable, as later commands will usually depend on the success of prior ones.

You can use && between commands to implement this dependency, of course, or you can use the -e switch on the shebang line (works with bash, may work with other shells) to cause the interpreter to halt execution after any command returns a non-zero exit code:

#!/.../bash -e

If you need to know where a line ends, where unprintable characters lurk, and the difference between white space and tabs, you can always remember to take the cat to the vet:

$ <commands> | cat -vet

This command will replace occurrences of foo with bar in the file X in-place and create a backup of X in X.old:

$ sed -i.old "s/foo/bar/g" filename

If you need to fool a machine into believing that a host:port pair is local, you can use ipchains to redirect traffic. For example, the desired destination is www.example.com:80 and you want it to go to localhost:8080:

# echo '1' > /proc/sys/net/ipv4/ip_forward
# ipchains -A input -j REDIRECT 8080 -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 80

Note: No one really uses ipchains anymore, but it can be found on older systems.

You can use this in your shell dotfile (e.g. ~/.bash_profile, ~/.zshrc) to create a screen session when you log in or reattach to an existing screen:

if [[ $TERM != 'screen' ]] ; then
    if [[ `screen -list | grep -v "No" | awk '$2 { print }' | wc -l` == 0 ]] ; then
        screen
    else
        screen -dr
    fi
fi

This works in bash and zsh.

Since lsof is tragically broken in Solaris 10, you have to try other methods of finding out which ports a given PID has opened:

# pfiles 2602 &> output

That will redirect stdout and stderr to a file (output) that will outline any files (and ports; remember, everything is a file) that the process has open.

Given a log file with a date in the first column, chop up the file into separate file:

$ awk '/^[0-9]/ {print $0 > $1".log"}' logfile.txt

If you have a string that has characters that the shell won't like, you can do a substitution on them:

$ awk '{gsub("/","_",$1); print $1 > $1".log"}' logfile.txt

If you find yourself needing to drop the case on the entire contents of a file, you can try this:

$ dd if=original of=filtered conv=lcase

-or-

$ tr '[A-Z]' '[a-z]' < original > filtered

If you need to fetch a substring, but you have more than one delimiter, throw them all in there on the -F option to awk.

Given the text <VirtualHost 192.168.1.0:80>:

$ awk -F" |>" '/^<Virtual/ {print $2}'

Would return 192.168.1.0:80.

To log into MySQL using the CLI in windows:

(in the mysql\bin directory)
c:\mysql\bin> mysql --user=<user> --pass=<pass> --port=3306

Have you ever entered a command that involved redirection and just stared at it, palms sweating, heart racing, hoping against hope that, after all the variables interpolate, it doesn't destroy any of your crucial system files sitting nearby?

Enter: noclobber

If you're using the Bash shell, you can set this option to prevent overwriting of existing files. To see your current settings:

bash-2.05$ set -o
allexport       off
braceexpand     on
errexit         off
hashall         on
histexpand      on
keyword         off
monitor         on
noclobber       off
...

To set noclobber:

bash-2.05$ set -o noclobber

If you want to open an SSH tunnel in the background, use the -N and -f switches:

ssh -Nf -L2121:localhost:21 user@host

If you want the time, and only the time:

$ date | (read u v w x y z; echo $x)
14:05:52

Add the following to your ~/.inputrc:

"\e[5~": history-search-backward
"\e[6~": history-search-forward

And of course, in order to use your ~/.inputrc you need to set your INPUTRC environment variable in ~/.bash_profile:

export INPUTRC=$HOME/.inputrc

The next time you login you will be able to run "gvim <pageup>" and the last entry that started with "gvim " will appear, <pageup> again will bring the next one up, and so on.

Note that escape codes for PageUp and PageDown vary depending on your terminal type; check out this tip for a technique on how to find out what your terminal expects.

If you have a URL that you need to crawl and you know the range of numbers in the image, you can do something like this:

$ curl -O http://www.example.com/img/samples[00-99].jpg

That should (at least attempt to) fetch the images samples00.jpg through samples99.jpg. Enjoy!

If you're using more than one range, you'll want to build your filename or a path with the --create-dirs option. For example:

$ curl http://www.example.com/imgs[00-99]/samples[00-27].jpg --create-dirs -o "#1/#2.jpg"

Alternatively, you can just be ghetto and name the files like dirname_filename.jpg:

$ curl http://www.example.com/images[00-99]/samples[00-27].jpg -o "#1_#2.jpg"

Download an entire directory tree:

$ wget -r ftp://username:password@site/path/to/suck

$ echo file.ext1.ext2 | cut -d"." -f1-2

$ echo file.ext1.ext2 | sed 's/.[^.]*$//' file.ext1

You can use tr to remove non-printable characters from a data stream:

$ tr -cd '\11\12\40-\176' < $INPUT_FILE > $OUTPUT_FILE

(This tip taken from http://www.devdaily.com/unix/edu/un010011/.)

If you find yourself on a machine without the semi-ubiquitous ImageMagick packages, you might at least have pnmscale. Starting with a PNG file, you can do the following to resize it to 450 pixels wide:

$ pngtopnm < ./firefox-upgrade.png | pnmscale -x 450 | cjpeg -smoo 100 -qual 100 > firefox-upgrade.jpg

You can use the -l <kilobits_per_sec> option with scp (NOT ssh or sftp) to restrict the bandwidth used to transfer files:

$ scp -l 200 user@host:~/files .

At the command line:

$ ruby -pe 'puts' < file.txt

#prints line number 52 from file.txt $ ruby -pe 'next unless $. == 52' < file.txt

You can use the tee program to save the contents of a pipe to a file while also viewing it on standard out:

# tail -0f /var/log/httpd/error_log | tee ~/newest_errors.txt

Note: tail -0 instructs tail to begin at the very end of the file (the default is to show the last ten lines), and -f means tail will periodically check the file for additional data and print the data to standard out.

If you're running scp -r, beware of symlinks; they are followed rather than preserved. This might be favorable behavior if you're referencing files outside what you're copying, but if you're copying symlinks which reference other parts of what you're copying, the referenced files will be duplicated on the destination host.

Take, for example, the following files on host A:

~/myfiles/
  foo/
    a.txt
  bar/ -> foo/

When you run this command on host B:

joe@B:~$ scp -r joe@A:~/myfiles .

The result on host B will be:

~/myfiles/
  foo/
    a.txt
  bar/
    a.txt

You can use this command to show duplicate lines in a file:

$ uniq -d MYFILE

This is a simple swap monitor script:

#!/bin/bash
# Notify me one time if my swap is over MAXSWAP and log the
# swap usage as well in SWAPLOG.
# Usage: ./monitor.sh &

SWAPLOG=~/logs/monitor/swap
MAXSWAP=5
INTERVAL=30

send_sms_msg () {
  if [ "$1" ]; then STRING=$1; fi
  if [ $SMS_SENT ]; then
    return 0
  else
    echo $STRING | mailx -s 'monitor msg' 5101234567@cingularme.com
    SMS_SENT=1
  fi
}

while true; do
    # parse free output, get current swap value
    swaptest=`free -m |grep Swap|perl -pe 's/Swap:\s+\S+\s+(\S+).*/$1/'`
    if [ $swaptest -ge $MAXSWAP ]; then
        echo `date` Swap is: $swaptest >> $SWAPLOG
        send_sms_msg "Swap is: $swaptest"
    fi;
    sleep $INTERVAL
done;

You have a mailbox you need to sort. You can't have it come back through your MTA, of course. Use this script to push it through the procmail filter of your choice:

#!/bin/sh

ORGMAIL=/var/spool/mail/$LOGNAME

if cd $HOME &&
  test -s $ORGMAIL &&
  lockfile -r0 -l3600 .newmail.lock 2>/dev/null
then
  trap "rm -f .newmail.lock" 1 2 3 15
  umask 077
  lockfile -l3600 -ml
  cat $ORGMAIL >>.newmail &&
  cat /dev/null >$ORGMAIL
  lockfile -mu
  formail -s procmail <.newmail &&
  rm -f .newmail
  rm -f .newmail.lock
fi
exit 0

If you would like to have syntax highlighting on a file that would otherwise not have any highlighting at all, you can symlink the file to the appropriate extension and open it:

$ ln -s example.lxp example.html
$ vi example.html # opened with pretty highlighting

*Note: This should be used sparingly and you should clean up your symlinks.

To transfer files via tar/ssh, do the following:

$ tar cvjf - * | ssh user@remote "(cd /desired/path; tar xjf -)"

Netcat is handy little utility for scripting all manners of network functionality. Here we're making sure a web server is responding as we'd expect:

$ (echo "GET / HTTP/1.1"; echo "Host: www.xinu.org"; echo) | nc www.xinu.org 80

Make sure you always specify a path free of symlinks. This can be pretty tough, though. An alternative approach is to use namei to track down symlinks:

# namei /usr/X11/bin/xterm
f: /usr/X11/bin/xterm
d /
d usr
l X11 -> X11R6
  d X11R6
d bin
- xterm

If you need to upload an entire directory structure, check out wput on sourceforge.net. It works the same way as wget only in the other direction (i.e., supporting various protocols).

http://wput.sourceforge.net/

Thanks to Aronalle for this tip!

If you need to discover on which paths a binary depends, you can sometimes run strings on it and grep for everything starting with a slash:

# strings /usr/sbin/named | grep ^/
/lib/ld-linux.so.2
/etc/named.conf
/etc/rndc.key
/etc/lwresd.conf
/etc/resolv.conf
/var/run/named/named.pid
/var/run/named/lwresd.pid
/dev/null

Sometimes you need to iterate over variable names and access their values. In shell script, you would do something like this to get the values of FOO and BAR:

$ FOO=apple
$ BAR=orange
$ VARS="FOO BAR"
$ for v in $VARS ; do echo ${!v} ; done

The above works in bash. Use this in Zsh:

$ for v in $VARS ; do echo ${(P)v} ; done

(The 'P' flag on ${v} causes a further variable lookup before ${v} is evaluated.)

Thanks to Cliff for the tip!

If you want to use tcpdump to watch initiating connections (that is, the syn flag only is set indicating we're looking at the first third of the three-way handshake) on ports 80 and 443 you could do something like this:

# tcpdump '(tcp[13] & 0x3f = 2) and (dst port 80 or dst port 443)'

If you have a bunch of files in your home directory and you want to push them into ~/sort, create the directory and then do the following:

$ find . -type f -maxdepth 1 ! -name ".?*" | xargs -I '{}' mv '{}' sort

Note: GNU xargs uses -i. BSD versions use -I.

Let's say you want to check the uptime on a list of servers. We're assuming that you've got a key on each machine otherwise you'll be entering your password often:

$ xargs -i ssh {} uptime < ./server.list
  5:46am  up 155 days, 17:49,  2 users,  load average: 0.12, 0.03, 0.01
  5:46am  up 147 days, 17:14,  2 users,  load average: 0.02, 0.05, 0.01
  5:46am  up 209 days, 17:26,  0 users,  load average: 0.00, 0.00, 0.00
  5:46am  up 89 days,  6:30,  0 users,  load average: 0.00, 0.00, 0.00
  5:46am  up 82 days,  6:40,  0 users,  load average: 0.07, 0.06, 0.01
  5:46am  up 104 days,  9:51,  0 users,  load average: 0.03, 0.03, 0.00
  5:50am  up 68 days,  9:17,  0 users,  load average: 0.00, 0.00, 0.00
  5:48am  up 68 days,  9:15,  0 users,  load average: 0.00, 0.00, 0.00