How does gdb work?

http://www.alexonlinux.com/how-debugger-works

 

SIGTRAP is used as a mechanism for a debugger to be notified when the
process it's debugging hits a breakpoint.

A typical way for something like GDB to use it would be something like
this:

- The user asks gdb to set a breakpoint at a certain address in the
target process. gdb uses ptrace to replace the instruction at that
address with the "int3" instruction, which generates a debug
exception. It also uses ptrace to ask that the process be stopped
when SIGTRAP is raised.
- When the target process hits that address, the exception is
generated. The kernel treats this as raising a SIGTRAP signal. The
process is stopped and gdb is notified.
- gdb lets the user examine the state of the target process. When the
user is ready to continue, gdb replaces the int3 with the instruction
that had originally been there, and uses ptrace to tell the kernel to
restart the target process from that instruction. AFAIK it would also
normally tell the kernel not to deliver the SIGTRAP signal to the
process, since by default that would kill it. So it would normally be
irrelevant how you are handling SIGTRAP (SIG_IGN or SIG_DFL or a
handler) because the target will never know it occurred.

It is also possible for a process to effectively set a breakpoint on
itself, which would also generate a debug exception and cause SIGTRAP to
be generated. On x86 this could be done by:
- executing an int3 instruction
- setting a hardware breakpoint or watchpoint using the CPU debug
registers (accessed via ptrace)
- setting the trace flag in EFLAGS

It's unlikely that you would do any of these things unintentionally,
although you are messing with the program pretty severely.

Here are a couple of ideas.

1. Does the same thing happen if you're not running under gdb?

2. siglongjmp restores the signal mask. Is it possible that the program
blocks SIGTRAP somewhere? If so, then maybe the SIGTRAP was actually
generated somewhere else (either by a breakpoint set by gdb, or by your
code doing something weird), and only raised when siglongjmp restored a
mask which didn't block SIGTRAP. This might have confused gdb into not
hiding the signal from the target process, since gdb didn't expect a
SIGTRAP to occur at that point because it never set a breakpoint there.