Modifier keys are handled quite specially, and I doubt Mathematica will be able to override the system. You probably have to do this in a layer between Mathematica and the OS. BUT, it is possible to make the key behave different depending on the application you are in. Thus with a bit of work, it MAY be possible to have the capslock key behave differently only in Mathematica.
edit: I did not see you say which operating system you had, so I've added Mac instructions.
Windows
For example, if you have Windows, you can use the program called http://www.autohotkey.com/ . It specifically has a feature where you can bind a key to a script, specifically the following script:
How can a hotkey or hotstring be made exclusive to certain program(s)?
In other words, I want a certain key to act as it normally does except when a specific window is active.
In the following example, NumpadEnter is made to perform normally except when a window titled "CAD Editor" is active. Note the use of the $ prefix in "$NumpadEnter", which is required to let the hotkey "send itself":
$NumpadEnter::
IfWinNotActive, CAD Editor
{
Send, {NumpadEnter}
return
}
; Otherwise, the desired application is active, so do a custom action:
Send, abc
return
This next example is more pure than the above, but it will only work if the "CAD Editor" application is designed to ignore the NumpadEnter key itself. The tilde prefix (~) makes NumpadEnter into a non-suppressed hotkey, meaning that the NumpadEnter keystroke itself is always sent to the active window, the only difference being that it triggers a hotkey action. The ~ feature requires Windows NT/2k/XP.
~NumpadEnter::
IfWinNotActive, CAD Editor
return
; Otherwise, the desired application is active, so do a custom action:
Send, abc
return
To quote from "MRCS" in this forum post, you may find the following useful:
The first one I named CapsLockR.ahk and contains the following script:
CapsLock UP::Run C:\Documents and Sett...[path to script]...\CapsLock.ahk
The second one is named CapsLock.ahk and has this script:
GetKeyState, state, CapsLock, T
if state = D
SetCapsLockState, off
else
SetCapsLockState, on
exit
Thus worse comes to worst, if you are having trouble modifying the "Behave like Foo if Active Window = Mathematica else behave like Bar" script, you can tack on this to manually toggle the CapsLock state I think. Googling will also reveal more results.
Linux
I know that on Linux, you can use the program called xbindkeys to bind the CapsLock to a script, from which you can in turn call xdo if you detect Mathematica is one of the topmost windows (e.g. via Getting pid and details for topmost window , or xdotool getwindowfocus) or worse-comes-to-worst, you can just have a script which toggles your configuration between CapsLock -> xdotool key Escape, xdotool type "whatever", xdotool key Escape ("Mathematica mode") and "normal mode"... though that may prevent you from YELLING AT MATHEMATICIANS OVER INSTANT MESSAGING WHILE DOING MATHEMATICS. Unless you You may need to find some way to programatically toggle CapsLock, perhaps by creating a dummy CapsLock key (though that's an extreme hack, it is likely one can find some kind of library; perhaps Anybody know how to toggle caps lock on/off in Python? may be useful). (This issue could be avoided by using a key besides CapsLock, or not caring that you want to keep your CapsLock functionality; you could also just turn another key you never use into CapsLock.)
Mac
Mac may have similar tools. For example, you can get xdotool like on Linux above via the MacPorts project. I hear the CapLock key cannot normally be rebound as easily on Mac, so if you can deal with another key it may be much easier. But theoretically it should be possible...
If you wish to use CapsLock, you can use PCKeyboardHack http://pqrs.org/macosx/keyremap4macbook/extra.html to remap the CapLock key to something which will tell OS X to let you remap the CapsLock. Then you remap it, then bind the key using Quicksilver to a script that makes calls xdotool to check if you're in Mathematica also also to issue the :esc:...:esc: if you are (see the Linux section of this answer). Otherwise you simulate a keypress on the CapsLock. But you remapped CapsLock! So you might need to make another dummy key you never use into the CapsLock key, and trigger a keypress on that using Cocoa libraries or a simple AppleScript. If you wish to pursue the CapsLock route, you might find Using Caps Lock as Esc in Mac OS X useful.
EventHandlerdoesn't seeCtrl,Opt/AltorCmdkeys either, yet Mma is aware of them. – Scullion