fallenpegasus: (Default)
I was working on a highly constrained consumer electronics device, a little "satellite device" that spoke to the main device over a CATV RF coax cable and also received commands from an IR remote control. My code was failing in bizarre ways. I adopted an extremely paranoid defensive programming stance, filling my code with asserts and doing paranoid cross checking of all inputs. This didn't make the device work. Instead it consistently didn't work, instead of inconsistently, because the cross checks and asserts would usually (but not always) trip before it would crash. It also started to run out of memory because of the all the paranoia code I had added.

I asked for the source code for the driver for the IR receiver, and for the driver for the CATV RF digital transceiver, and for the peer code that was driving the cable digital that ran on the main device.

The driver for the CATF RF digital transceiver was handed to me the first time I asked. And by "handed to me" I mean that I was pointed to where it was sitting in the source repo.


The business partner / hardware supplier who was supplying the IR glue and drivers just , after giving me a runaround, finally just flat out refused, citing trade secrets, confidentiality, secret sauce, and similar bullshit.

So, I finally "stole" the source code with a disassembler. And found the sources of many of my problems. It was complete shit. "Unexpected" input from the silicon would cause wild random pointer writes. And random sunlight on the receiver optics would cause it. "Expected" input of undefined remote commands wasn't much better, generating and handing back blocks of garbage with incorrect block length headers.

I ended up writing, nearly from scratch, a replacement IR receiver driver.


The peer device driver code was written by a developer in a different group in my same company. I finally got the P4 ACLs to read it after loudly escalating, over the objections of it's developer and his group manager. It was also complete shit. I cannot even begin to remember everything that was wrong with it, but I not only figured out may of the sources of my own pain, I also found a significant source of crash and lockup bugs that afflicted the main device.

I was not allowed to rewrite the peer code, as it was not in my remit. However, I was able to sneak in and check in a large number of asserts, using the excuse that they were "inline documentation".


On, and the device driver for the CATF RF digital transceiver? The source code I got for the asking, without a fight? When I reviewed it was easy to understand, efficient, elegant, and as far as I could tell, bug free.


In the end, I made my part work. It just took over two months instead of the original guesstimate of less than two weeks. This caused a schedule slip in the release of the satellite box. Which would have been a more serious problem, except…


Except there was also major schedule slip for the main box. A significant reason for that slip was because the peer code that I had filled will asserts was now crashing with assertion failures instead of emitting garbage. I was lucky that I was not more officially "blamed" for that. The reason why I wasn't, was mainly because the people who understood what I did understood the problem, and the executives who didn't understand what the problem was were also too clueless to blame anyone, let alone me.


My lesson learned from this experience is: if someone is refusing to show the source to suspect driver code, citing trade secrets, confidentiality, secret sauce, partnership agreements, or similar excuses, it's not because they are protecting their magic. It's because they have screwed up, and they are trying to hide it.

A second rule of thumb I have is: source control systems that don't allow any developers to check out and review any arbitrary source code file are expressions of moral failure. It is unethical for an engineer, designer, or other technologist to ever sign off on a project that has been mutilated by such a broken tool.

Profile

fallenpegasus: (Default)
Mark Atwood

November 2016

S M T W T F S
  12345
6789101112
13141516171819
202122 23242526
27282930   

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Feb. 19th, 2017 11:30 pm
Powered by Dreamwidth Studios