OpenAI's journey to fixing a long-standing bug in GNU libunwind is a fascinating tale of how a seemingly impossible problem can be solved through innovative thinking and a shift in perspective. The story begins with a mysterious crash in Rockset, the C++ data infrastructure service powering ChatGPT's search and data plugins. The engineers initially struggled to pinpoint the issue, as every hypothesis they constructed seemed to be contradicted by the evidence.
What made this case particularly intriguing was the team's decision to approach it from a different angle. Instead of delving deeper into individual crashes, they embraced what they called 'epidemiological debugging'. This involved building a pipeline to automatically analyze every production core dump from the past year, seeking patterns at a population level rather than focusing on individual cases. This shift in strategy proved to be a turning point.
The team utilized ChatGPT to write a script that downloaded and analyzed core files, extracting registers and filtering known false positives. By doing so, they were able to identify two distinct crash populations with completely different signatures. One set of crashes, the 'misaligned-stack' crashes, was traced back to a single physical host in an Azure region. Interestingly, these crashes had a clear start date and never appeared on long-running nodes. Once the affected host was removed from service, the misaligned-stack crashes disappeared entirely, highlighting the importance of identifying and isolating specific sources of issues.
The remaining 'return-to-null' crashes, once separated from the hardware-corruption cluster, revealed a different story. The team had previously ruled out C++ exception unwinding as a cause, but this assumption was based on the assumption that there were counterexamples. However, these counterexamples were all from the hardware-corruption cluster. Once this contamination was removed, it became evident that every remaining crash was indeed happening during exception unwinding.
The root cause of the problem was a race condition in GNU libunwind's Ux8664_setcontext function, which had been present for 18 years. During C++ exception unwinding, this function updates the stack pointer (%rsp) before reading the instruction pointer from the old struct. This creates a window of vulnerability where a signal can arrive, corrupting the instruction pointer and leading to a crash. The race window is incredibly narrow, lasting only one instruction at modern clock speeds, roughly 100 picoseconds.
What makes this case particularly fascinating is the role of frequency in triggering the bug. OpenAI's Rockset uses timer_create to deliver a SIGUSR2 signal every few milliseconds of CPU time, creating a much higher frequency of signal delivery events than typical applications. This frequency turned a theoretically possible race into a production crash, highlighting the importance of considering the specific characteristics of the system under investigation.
The team's approach of building a high-quality dataset was instrumental in solving the problem. By having accurate and complete population data, they were able to see the structure of the problem more clearly. This data-driven approach allowed them to identify the two distinct crash populations and ultimately uncover the root cause of the issue. The lesson here is that sometimes, the most important step in debugging is not the clever assembly reading or deep knowledge of the details, but rather the ability to build a comprehensive dataset that provides a clear view of the problem at hand.
In conclusion, OpenAI's experience with this bug serves as a powerful reminder of the importance of innovative thinking and a shift in perspective when tackling complex problems. By embracing epidemiological debugging and focusing on building a high-quality dataset, they were able to solve a long-standing issue and gain valuable insights into the inner workings of their systems. This story is a testament to the power of creativity and a data-driven approach in engineering, and it inspires us to think outside the box when faced with seemingly impossible challenges.
