Comment by monk_the_dog

rr is my goto debugger. It opens up new super-powers for debugging scripts. Consider this gdb script:

  b broken_1_in_1000_times_at_random.cpp:1
  c
  b usually_runs_normally.cpp:1
  reverse-continue

And now step through forward to figure out what went wrong. This tool is worth your time to learn. There are a couple of minor annoyances (you can't re-use existing gdb scripts that contain the "run" command because the inferior is already running, but there are workarounds. Here's what I use:

  python
  import gdb
  class MyRunOrContinue (gdb.Command):
      """Run inferior if not already running, otherwise continue (useful for rr)."""

      def __init__ (self):
          super (MyRunOrContinue, self).__init__ ("just-go", gdb.COMMAND_USER)

      def invoke (self, arg, from_tty):
          if gdb.selected_inferior().pid != 0:
              gdb.execute('continue')
          else:
              gdb.execute('run')

  MyRunOrContinue()
  end

There are a few other minor annoyances, but all in all a great tool.