On some systems (and for some terminal types), Emacs' use of the backspace key can be confusing. The backspace key may behave like the Delete key. You can fix this either by using this elisp in your ~/.emacs as follows:

(keyboard-translate ?\C-h ?\C-?)

This shell command may work if the elisp does not:

stty erase '^?'