General bug hunting

This area is somewhere to discuss debugging in general, with or without the assistance of Undo.

7 Topics

E
Edward TaitNew Participant
posted in War Stories

Solving a heap corruption bug using reverse watchpoints

I recently added a function of our code whose purpose was to concatenate the contents of two memory regions. The basic scheme was:Ensure destination region is large enough Work out number of bytes that need copying (len) Work out offset to copy bytes to Do memcpy(dest + offset, src, len);It being early on a Monday and the coffee not yet having kicked in I erroneously determined that offset and len were the same quantity and ended up typing something like:memcpy(dest + offset, src, offset); This is broken as offset may be much larger than the number of extra bytes reserved at the end of dest. This defect manifested as seemingly unrelated failures a long time after the broken copy had taken place. These failures were due to corruption of data stored on the heap. I was able to obtain a recording. My normal work flow for debugging a recording of a session that terminated with an assertion failure (as this was) is something like:ugo end backtrace to get an impression of where I am reverse-f

3
E
1 year ago
Bruce
BruceBug Squasher
asked in How would you tackle...?

Debugging the wetware component of the protocol stack

As an example of how the Slack integration looks, how would anybody go about unblocking the wetware pipe to allow thoughts and conversations to flow into the system?

1
T
1 year ago
B
BDLongNew Participant
asked in How would you tackle...?

Problem doesn't occur under udb

Hi. I am working on a complex open-source project and I have a bug that occurs in about 2/3 of runs. There are at least 3 ways the problem expresses: segfault, map access out-of-range and another less common way. (So it could be more than one bug.) When the bug occurs there are usually 15-20 interacting threads running, some of which are state-machines.The problem is that when I run it under udb the problem never occurs. I’ve tried nearly 100 times.I’m looking for ideas to make the bug occur or at least increase the likelihood that the bug will occur under udb. Ideas? Any help is greatly appreciated!

1
G
8 months ago
G
gelNew Participant
posted in War Stories

Distributed debugging using wallclock

Although I’m supposed to be busy these days running the company I still like to write code when I can carve out some time. I tell myself it’s important to understand the world of our customers, but really that’s just an excuse - it’s still my happy place :) Forever we’ve wanted to make it easy to stitch together multiple processes. We have multi-process correlation for shared memory support and it’s super cool, but there’s nothing for processes communicating over the network. Imagine being able to reverse-step through a remote procedure call. It’s been on our todo list for ever, but so far it hasn’t quite made it to the top. Like most applications these days, UDB itself is composed of multiple processes. There is GDB, which talks to our own implementation of gdbserver (called, imaginatively enough, udbserver), which talks to a component we call nanny. All this happens over sockets. I was trying to diagnose a problem where our udbserver was receiving a bogus progress notification. As of

0
G
1 year ago
G
gelNew Participant
asked in How would you tackle...?

How would you debug a silent EINTR /without/ software failure replay?

I was asked to take a look at a failure of a test that I wrote many years ago. This failure was, as far as we could tell, new, but it didn’t look like a regression. The test was multi-threaded with asynchronous signals using setitimer() to fire SIGALRMs at it. It appeared to have completed successfully, but at the end it printed a “test succeeded” message which our test system was expecting to see, only it didn’t appear. Happily our test system had saved a recording of the debuggee. Turns out that the printf(“test succeeded”) had been interrupted by a SIGALRM and so returned -EINTR. Actually it was more complicated than that - the compiler had turned the printf() into a puts(), which returns zero if it fails. This took 10-20 mins of head scratching with the recording, a few minutes thinking I was looking at a libc bug, but it was pretty straightforward really. So my challenge is - how would you go about debugging that _without_ a recording?

0
G
1 year ago
mhov
mhovNew Participant
posted in War Stories

Corruption of siginfo in sighandler when recorded

I've recently worked on a problem reported by a customer where our recording engine appeared to change the behaviour of signal handlers and I found the investigation and subsequent workaround interesting enough to share.First a bit on signal handlers. Processes can install their own functions to handle signals with the rt_sigaction syscall:struct sigaction sa;sa.sa_flags = SA_SIGINFO;sa.sa_sigaction = handler;sigemptyset(&sa.sa_mask);sigaction(SIGCHLD, &sa, NULL);When the kernel invokes this function to deliver a signal it passes a few parameters to the handler to provide information about the signal that is being delivered. Given that the SA_SIGINFO option was used to install the handler, one of those parameters will be a siginfo_t struct.typedef struct { int si_signo; int si_errno; int si_code; union __sifields _sifields;} siginfo_t;The most used fields in the siginfo_t struct are si_signo, which encodes the signal number, and si_code which encodes a reason for wh

0
mhov
9 months ago
G
Gareth ReesNew Participant
posted in War Stories

The missing XSAVE

1. IntroductionThis is a write-up of a bug in Undo’s software replay technology that I diagnosed and fixed. I’m writing it up while the details are still fairly clear in my memory and I have access to the relevant debugging transcripts. Hopefully this will be an entertaining story, and show off many useful features of UDB.2. The bugThis bug was provoked when I pushed a small commit removing some long-dead code. This caused a backwards-compatibility test to fail in the overnight run, and since my commit was “responsible” for the failure (as determined by git bisect), it fell to me to fix the problem. This put me in a bad mood, because I was certain that there was nothing wrong with my commit, and that what was going on was that removing the dead code had perturbed the code layout causing some previously latent bug to spring into life. There’s little that’s more annoying than to be blamed for something that isn’t your fault. But afterwards I was glad I worked on it.What was the failure?

0
G
6 months ago

gdbWatchPoint

Regular GDB tips from Undo CEO Greg Law.

Badge winners

Show all badges
Terms & ConditionsCookie settings