Cybersec Canon Ep 1. The Cuckoo’s Egg - Chapter 1 & 2. When Debugging Uncovers More Than Bugs
Debugging a production issue is quite rewarding and engaging.
Brief Note: I'm restarting this newsletter with a goal of reading books from the cybersecurity canon and posting some notes. Will plan to delete all the older posts to maintain consistency with the new theme. 
“The people I knew who called themselves hackers were software wizards who managed to creatively program their way out of tight corners. They knew all the nooks and crannies of the operating system. Not dull software engineers who put in forty hours a week, but creative programmers who can’t leave the computer until the machine’s satisfied. A hacker identifies with the computer, knowing it like a friend.”
Stoll, Clifford. The Cuckoo’s Egg (p. 10). Knopf Doubleday Publishing Group. Kindle Edition.
Cliff, an astronomer, is transferred from the Keck Observatory at LBL to the computer center at the same place. He is grateful that he has a job instead of being at the unemployment line. He takes an office instead of a cubicle. Near his office were the offices of a couple of experienced people - Dave & Wayne, who have differing opinions of their preferred systems. Dave supports DEC’s Vax computers, and Wayne is the Unix guru. Side note: I guess this was typical of the time. Twenty years ago, I had colleagues who were either Windows supporters or Java supporters, with Java supporters being seen as liberators and supporters of the “free” movement.
On the second day at his job, Dave asks Cliff to look at an issue with the accounting system which failed to account for 75 cents. Cliff takes it up, thinking that the 75-cent discrepancy might be due to a rounding error or some software flaw. There were two systems - one which was the ordinary Unix accounting software and another, built by Dave, that kept detailed records of who was using the computer. He tested both and found no issues with any of the programs. He finally figures out that the error was due to an unauthorized user named “Hunter” who used 75 cents of computer time without being billed. He asks around and finds that no one had added this user and that adding a user was automated and not manual. He deletes the user, thinking that someone set it up wrongly.
The next day, they get an email from a computer named Dockmaster, and the manager tells them that someone from the lab has tried to break into their system in Maryland. While investigating this, Cliff discovers inconsistencies in the accounting records for a user named Joe Sventek, an ex-Unix guru at the lab who was in England. He thinks maybe Joe is back, and the machines are showing different times because of time drift (clocks can go out of sync).
The next day, Cliff is still thinking about this problem even while he is attending a lecture about the structure of galaxies. He thinks about hackers and super-users and realizes that a super-user hacker might have infiltrated their system.
—
There are a lot of things to unpack in the first two chapters. It sets the stage for what is to come. It also gives a glimpse of life from a few decades ago. Even after all these decades, software professionals still do a lot of debugging. In fact, one could argue that most software development is about debugging issues and navigating through a codebase that is already built. In the past, it used to be mainly about debugging production issues, but in the past decade or so, especially after the DevOps practice took off, there has been a lot of debugging around the pipeline itself - debugging test failures during presubmit, integration and e2e test failures, canary failures, partial deployment failures, and so on. A lot of things that were supposed to make work easier, in fact, ended up adding a lot of work. For e.g., integration with static analysis tools showed issues in code, lack of coverage, etc. But it added a lot more work for people who were maintaining codebases already written. Some suggestions were useful, but most were ‘clean up’ stuff to make things look good. Some teams added all the issues into a task-tracking tool like Jira, making the backlog of things to be done huge. In the end, many teams decide to just use the tools to check newly written code. I think AI might help reduce this burden. It could help generate code and write tests that pass all the static analysis tools with flying grades. It could help create tests that are less flaky. So, in a way, people could go back to spending more time on debugging production issues, which is arguably more rewarding than pipeline issues. AI could help with parsing large log files and finding some information, but I think to really debug issues like discrepancies between systems, issues due to race conditions, etc., you need human expertise.
Also, I feel that debugging an issue in production is something that can get you into the ‘flow’ state easily, more than writing code, writing a document or other activities. Once you are in, once you’re unpacking layers and layers of complexity, you could easily be oblivious to lunchtime/friends/family, etc. Even when you come out of it, your mind could still be thinking about the issue. In Cliff’s case, he was thinking about it even as he was attending a lecture on a different topic.
With AI potentially helping with writing clean and more secure code right out of the box and also helping with syntax and some of the nuances of the programming language, people could now focus on building great software. In fact, this might be the golden age for software development. In a recent post by Dario Amodei, the CEO of Anthropic, he says that if there were more intelligent people, there would be more inventions, and one AI is adding to human intelligence at this point. So, we might be entering the world of even more complex software systems that might need even more expertise to debug.