War Stories

Do you have any anecdotes of how you eventually tracked down a particularly gnarly bug? Or face-palm moments when you eventually did so? Here's a place you can share them, for adulation or sympathy.

  • 4 Topics
  • 3 Replies

4 Topics

E
Edward TaitNew Participant
posted in War Stories

Solving a heap corruption bug using reverse watchpoints

I recently added a function of our code whose purpose was to concatenate the contents of two memory regions. The basic scheme was:Ensure destination region is large enough Work out number of bytes that need copying (len) Work out offset to copy bytes to Do memcpy(dest + offset, src, len);It being early on a Monday and the coffee not yet having kicked in I erroneously determined that offset and len were the same quantity and ended up typing something like:memcpy(dest + offset, src, offset); This is broken as offset may be much larger than the number of extra bytes reserved at the end of dest. This defect manifested as seemingly unrelated failures a long time after the broken copy had taken place. These failures were due to corruption of data stored on the heap. I was able to obtain a recording. My normal work flow for debugging a recording of a session that terminated with an assertion failure (as this was) is something like:ugo end backtrace to get an impression of where I am reverse-f

3
E
1 year ago
G
gelNew Participant
posted in War Stories

Distributed debugging using wallclock

Although I’m supposed to be busy these days running the company I still like to write code when I can carve out some time. I tell myself it’s important to understand the world of our customers, but really that’s just an excuse - it’s still my happy place :) Forever we’ve wanted to make it easy to stitch together multiple processes. We have multi-process correlation for shared memory support and it’s super cool, but there’s nothing for processes communicating over the network. Imagine being able to reverse-step through a remote procedure call. It’s been on our todo list for ever, but so far it hasn’t quite made it to the top. Like most applications these days, UDB itself is composed of multiple processes. There is GDB, which talks to our own implementation of gdbserver (called, imaginatively enough, udbserver), which talks to a component we call nanny. All this happens over sockets. I was trying to diagnose a problem where our udbserver was receiving a bogus progress notification. As of

0
G
1 year ago
mhov
mhovNew Participant
posted in War Stories

Corruption of siginfo in sighandler when recorded

I've recently worked on a problem reported by a customer where our recording engine appeared to change the behaviour of signal handlers and I found the investigation and subsequent workaround interesting enough to share.First a bit on signal handlers. Processes can install their own functions to handle signals with the rt_sigaction syscall:struct sigaction sa;sa.sa_flags = SA_SIGINFO;sa.sa_sigaction = handler;sigemptyset(&sa.sa_mask);sigaction(SIGCHLD, &sa, NULL);When the kernel invokes this function to deliver a signal it passes a few parameters to the handler to provide information about the signal that is being delivered. Given that the SA_SIGINFO option was used to install the handler, one of those parameters will be a siginfo_t struct.typedef struct { int si_signo; int si_errno; int si_code; union __sifields _sifields;} siginfo_t;The most used fields in the siginfo_t struct are si_signo, which encodes the signal number, and si_code which encodes a reason for wh

0
mhov
9 months ago
G
Gareth ReesNew Participant
posted in War Stories

The missing XSAVE

1. IntroductionThis is a write-up of a bug in Undo’s software replay technology that I diagnosed and fixed. I’m writing it up while the details are still fairly clear in my memory and I have access to the relevant debugging transcripts. Hopefully this will be an entertaining story, and show off many useful features of UDB.2. The bugThis bug was provoked when I pushed a small commit removing some long-dead code. This caused a backwards-compatibility test to fail in the overnight run, and since my commit was “responsible” for the failure (as determined by git bisect), it fell to me to fix the problem. This put me in a bad mood, because I was certain that there was nothing wrong with my commit, and that what was going on was that removing the dead code had perturbed the code layout causing some previously latent bug to spring into life. There’s little that’s more annoying than to be blamed for something that isn’t your fault. But afterwards I was glad I worked on it.What was the failure?

0
G
6 months ago

gdbWatchPoint

Regular GDB tips from Undo CEO Greg Law.

Badge winners

Show all badges
Terms & ConditionsCookie settings