How to impress me in a coding interview

At Square, candidates are expected to actually write code during the interview process – on a computer, not on a whiteboard. As a competent software developer, this kind of interview should feel like a gift. Your skills are going to be measured more or less directly by doing the very thing you’ve been doing all these years!

So how do you impress me – or any interviewer – in a coding interview? Here are a few things I consider when pairing with a candidate.

It’s about the process

First off, let’s deal with a myth: the interview is not about the answer to my question. Sure, I’d prefer that you get it right, but I’m far more interested in seeing how you work to get there. I want to see how you decompose problems, conceive solutions, and debug issues as they arise. Make sure you take the time to show me how you’re getting to the answer.

It’s about communication

A big part of the development process on a healthy team is communication. Ideally I’d like to leave the interview feeling like you’re someone who asks good questions and is receptive to feedback and external insights. A great way to blow this portion of the interview is to hunch over a notepad and scribble quietly while I wait. If you must do something like this, be prepared to explain your thoughts thoroughly when you’re done.

Don’t go it alone

My interview exercise is hard. I fully expect that you might need a hint at some point. I even enjoy working with you to debug the problems you hit as you work. But if you never ask, you’re not going to come out ahead. Coding on a real team often involves working on something that you don’t understand, but that the person sitting two seats down knows in detail. You should be self-sufficient only insofar as it makes you productive; spinning your wheels trying to go it alone is just indulgence.

You should run the interview

Don’t make me march you through all the steps. Once I’ve explained the exercise and you’ve started coding, it’s best if you drive. If you need help, ask. If you think you’re done, propose more test cases or ask for my thoughts. The best interviews I’ve had are more like working with the candidate than interviewing them. If you can do this in an interview setting, then I can believe you’ll work the same way when you’re on my team.

Use the tools

I am consistently impressed when a candidate is able to drop into a debugger or quickly access documentation in the IDE of their choice. It’s an aspect of professional mastery that will be extremely important to your day to day productivity. The inverse is also true – if you choose to write C++ in vim and don’t know how to save the file, it’s going to cost you.

No bonus points for doing things the hard way

One of the key attributes I select for in coworkers is pragmatism. Sometimes this means choosing an ugly but expedient solution instead of an elegant but expensive alternative. If you know when to make these tradeoffs, you’re someone I want to work with. A lot of candidates feel pressure to give me “perfect” solutions right away, but I’d rather see an answer than an incomplete, perfect one. Plus, I love it when we can iterate on the “quick and dirty” solution to build a refined version. This is how software works in the real world, and I love when candidates aren’t too self-aware to just go about it casually.

Moving on

In a turn that few will have seen coming, this coming Friday will be my last day at Rapleaf. After five years, I’ve decided that it’s time to move on and seek new challenges. To that end, after much consideration, I will be joining Square to help them scale up their analytics efforts. I’m incredibly excited to get to work with a new team on a totally new problem domain, and I believe I have a ton of value to add to the company.

This decision is exquisitely bittersweet, though. It’s difficult to even begin to describe how spectacular my time at Rapleaf has been. I found it telling that when I tried to sit down and write a summary of my experience for my resume, it seemed impossible to summarize all that I had learned and done. This is still totally insufficient, but I’ll boil it down to this: I was given the opportunity to make many amazing software systems; I was allowed to grow into a member of the broader engineering and open source community; I learned the meaning of scale; I got to be a real owner of the business’s direction and purpose; and I played a role in building a world-class team.
It is this last point in particular, the team, that puts the tears in my eyes. I have never before had the opportunity to work with so many brilliant, hard-working, interesting, and just generally nice people, and I am humbled and honored to have been counted among their number. My greatest worry – one which I consider not wholly irrational – is that the team I’m leaving behind is in fact of the rarest kind, and I’ll spend the rest of my career hoping to rebuild something as great.
To everyone I have worked with over these last five years, thank you for everything you have taught me. You are what has – and will continue to – make Rapleaf great. 
Let’s do this again some time.

Open Source, Forking, and Tech Bankruptcy

Open source software is a part of most of the things I do day-to-day. I use a ton of things made by others: Hadoop, Cascading, Apache, Jetty, Ivy, Ant – the list could literally go on for pages. But I also use and develop some things that I’ve built that have been released into the public. I contribute to Thrift frequently, and have released Hank, Jack, and other projects as part of my work at Rapleaf.

Working with so much open source software has given me lots of opportunity to develop perspective about how companies should engage with open source projects. In this day and age, nobody is going to counsel against using open source, since it’s an enormous productivity booster and it’s everywhere. However, there are some different schools of thought about how you should use and contribute to open source.

One way of using open source is to just use what’s released, never make any modifications, and never make any contributions. For some projects, this is perfectly fine. For instance, I find it hard to imagine making a contribution to Apache Commons. Everyone will take this approach on some projects, particularly the ones that are mature and useful but not mission critical: they’ll never produce enough pain to merit fixes nor produce enough value to merit enhancements.

However, the above model only works well on projects that are very stable. Other projects you’ll want to use while they are still immature, unstable, and actively developed. To reap the benefits, you might have to roll up your sleeves and fix bugs or add features, as well as dealing with the “features” introduced by other developers. This is where things get tricky.

There are two basic ways to deal with this scenario, which I think of as the “external” and “internal” approaches. The external approach involves your team becoming a part of the community of the project, contributing actively (or at least actively reporting bugs and requesting features), and doing your best to hang onto the bleeding edge or commit to using only public releases. The “internal” approach involves you picking an existing revision of the project, forking it into some internal repository, and then carefully selecting which upstream patches to accept into your private fork while mixing in your own custom patches.

Both of these options are imperfect, since either way you’re going to do a lot of work. A lot of companies see this as a simple pain now/pain later tradeoff and then choose accordingly. But I don’t think this is actually the case. What’s not easy to appreciate is that the pain later is usually much, much worse than the pain now.

Why is this the case? It comes down to tech debt. Choosing to create an internal fork of an open-source project is like taking out a massive loan: you get some time right now, but with every upstream patch you let go unmerged, you are multiplying the amount of effort you will ultimately need to get back in sync. And to make matters worse, people have a tendency to apply custom patches to their internal forks to get the features they need up and running quickly. This probably seems like a great idea at the time, but when it’s done carelessly, you can quickly get into a state where your systems depend on a feature that’s never going to make it into the upstream and might actually conflict with what the community decided to do.

When you get into the situation where your fork has diverged so much that you find yourself thinking, “I’ll never be able to switch to upstream,” then you’ve reached a state of tech bankruptcy – literally the only thing you can do is give up and stick with what you have or commit to an unbelievably expensive restructuring. At this point you cease to have a piece of open-source software: you have no external community, nobody outside to add features, fix bugs, and review your code, and you can lose compatibility with externals systems and tools.

Needless to say, the decision to make an internal fork should not be undertaken lightly. Weigh the perceived stability and flexibility benefits very carefully before starting down that road. If you must fork, make sure you understand the costs up front so that you can budget time to keep your fork in sync.

There’s a flip side to this. How often does a piece of internal code that “could be” an open source project go from closed to open? I know from my experience that it’s not easy to make the transition – you end up building in a feature that’s too domain-specific, or you tie it to your internal deploy system. I think that writing a decent piece of software that could be spun out as an open-source project and yet failing to do so is another case of accumulating tech debt. In this case, the bankruptcy state is a project that could have been open but never will be because of the time investment required.

The prescription in this case is easy: open source your project early, perhaps even before it’s “done,” continue to develop it in the open, and whatever you do, use the version you open sourced, not an internal fork.

Custom languages considered harmful

I use and contribute to a lot of open-source software projects, and one thing I see all over the place that drives me absolutely nuts is the prevalence of custom languages. In serialization frameworks, there are interface definition languages. In CAD tools, there are modeling languages. In the NoSQL and big data processing world, you’ll find my personal pet peeve, query languages.

Why are people making these custom languages? Abstractly, I get what they’re thinking: they have an application that would benefit from an expressive, terse, awesome set keywords and operators that will allow their users to be more productive. And this thinking isn’t fundamentally wrong. The productivity gain is real and definitely worth pursuing, and an intuitive language can go a long way towards making your product extremely accessible.

But here’s where I think these people go off the rails: aside from the very real NIH danger (Making your own language is cool, right? Guys?), there is a huge difference between a custom language – one in which you have to do the whole compiler contortion of lexing and grammars and whatnot – and a domain-specific language, which is more typically a set of functions and syntax that work from within another existing language to provide enhanced functionality. These two things seem a lot alike at first glance, but in my opinion, they couldn’t be more different. Why? Because when you decide to write a custom language – however simple it might seem to you right now – you are making the incredibly short-sighted statement that you know how to design a language better than all the people who have tackled this gargantuan problem before you. I don’t doubt that you know your domain better than anyone else, and that’s definitely what you need to get started, but what you probably don’t know is what you’re going to need later that will totally change your language. Or how best to implement recursion, scoped variables, memory management, or exception handling. The list could go on forever.

Let’s look at a concrete example. Lots of people in the 3D printing/maker community use OpenSCAD, an open-source 3D solid modeler with a decidedly programmer-tailored interface. Makers love it, because it’s powerful, fairly simple, and gives you really repeatable results. It uses a totally custom scripting language that is basically C-like: nesting, scope, function calls. But in combing through some of the syntax and reading the mailing list, I get the impression that a nontrivial amount of the developers’ time is spent reinventing features like variable assignment and for loops that could instead be spent actually developing the application itself.

To be totally clear, I love OpenSCAD, and I’m going to keep using it. That doesn’t mean that I don’t wish the developers had decided to write a DSL instead. If they had decided to embed their functions in a nice Ruby DSL, instead of having to figure out how to deal with the quixotic features of the native dynamic length lists, I could leverage all of the existing resources – community-vetted documentation, libraries, debugging and editing tools – and it would just work. As a user, I could just focus on learning the portion of the new syntax that applies to the specific domain, rather than also having to scramble to try and figure out how to do the things I already know how to do in other languages.

It’s clear that some folks out there get this right and some get it wrong. Thrift and Protobuf both have their own IDLs, which work well enough, though the code involved in analyzing IDL files is decidedly non-trivial. OpenSCAD and others like RapCAD have plowed tons of time on their own custom languages. Map/Reduce productivity tool Pig is a nightmare: since it’s not Turing complete, whenever you want to do something not imagined by the developers you have to make the significant development context switch into regular Java to make user-defined functions. Conversely, Cascading is a wonderfully usable and extensible pure-Java DSL for composing complex Map/Reduce workflows with arbitrary UDFs that don’t require you to context switch. (Even the splendid Clojure wrapper for Cascading, Cascalog, has the sense to use the existing Clojure language to give you an honest-to-god Turing complete language to work in.) Cassandra has CQL (which I admittedly haven’t used) while HBase has stuck with a solid functional API and has a JRuby REPL in which you can use it.

So next time you’re thinking about dusting off your context free grammar skills and busting out your favorite compiler compiler, just don’t do it. There’s an easier way.

Maker to Maker Business: Customer Service

When you start making something, there are a million things on your mind – minute design details, which parts to use, how to operate the tools, what it will take to assemble. All these problems are vital to the success of your product, and to a maker, they are inherently exciting: each decision is another tiny love letter to the unique awesomeness of your project, each an opportunity for error and complexity, but also for wonder and satisfaction.

Something you are probably not thinking about is customer service.

Ugh, customer service…

Let’s be very clear about this: selling your product involves a great deal of customer service. This might come as a shock to you, since so far you’ve been engrossed in the process of actually making your product, and it’s only when you list your first item for sale will it hit home just how much other work there is to do.

This realization can be a bit demoralizing initially, because in a lot of ways customer service isn’t actually part of making anything. It’s just a peripheral aspect of the process of selling your product, and as a maker, chances are that it’s the making part that got you started in the first place. But customer service is a crucial part of your maker business, and not something you can ignore.

This rest of this post is devoted to some practical tips we’ve learned in the process of operating our business. A lot of these concepts apply specifically to the DIY/maker business, but some of them are applicable to businesses in general.

Customer service is a product, too

It’s important to consider your customer service a product of your business. When your customers speak about their dealings with you, whether it’s on a feedback form or face-to-face with their friends, they’ll be rating both your product and their experience with you. 
Think of all the customer service interactions you’ve ever had. Chances are most of them were “good enough.” But those aren’t the ones you remember. It’s the amazing ones and the awful ones that stick with you. Don’t settle for a “good enough” customer service experience. After all, would you ship a “good enough” product, or do you want your business to be known for offering the best there is? If you try to go the extra mile to make your customer service excellent, it will pay for itself in a boosted reputation, happy customers, and additional sales.

Love your customers

It can be difficult to approach customer service with the right attitude. Every email you have to respond to takes time away from actually building your product.

The easiest way to get your head straight is to remember one simple thing: you love your customers. After all, they have a nigh-unlimited set of options, yet they eagerly forked over their hard-earned money to buy something you designed and manufactured. How cool is that? 

Once you’ve decided to love your customers, a lot of things get easier. In every interaction, you should always make it clear to them that you genuinely appreciate their business. When your customers have issues, you should always start off by apologizing, regardless of whether the problem is your fault. From their perspective, every problem they have with your product will be your fault, no matter what it is. Starting with an empathetic statement puts you on the same team as the customer. And then even if you have to give bad news later, you’ll at least have started out on a good foot.

Respond promptly

Responding to your customer service emails quickly is really important. A lot of times, surprising your customer by getting back to them within a few hours will really impress them and help you to reduce their irritation. When someone takes the time to send you an email with a complaint or question, it’s because of something that’s on their mind, and that’s going to stay with them for a while. The more quickly you can address it, the less of a strain it is for them.
When possible, try to reply to emails as they come in. If that’s too difficult, then you should at least answer all new emails every day, even if just to tell them you got the message and are working on a solution.

Take complaints to heart, but don’t take them personally

There is a 100% chance that you will get an irate email from a customer at some point. Your product will break in shipping, or get shipped to the wrong address, or be the wrong color, or one of a thousand other problems. They will be angry, they will be demanding, and sometimes they’ll even be downright rude. All of this can make you feel terrible, especially if what they are complaining about is actually your fault.
Feeling terrible is the last thing you want to do, though. This is something I learned back when I played a lot of online poker: it’s OK to feel bad intellectually about making a mistake, but you need to keep a fair amount of emotional detachment so you can make good decisions. Otherwise, you’ll find yourself making knee-jerk reactions that lose you both your customers and your hair.

When dealing with complaints, the first thing you should do is apologize and offer a remedy. Next, you should analyze the mistake and change your process so it doesn’t happen again next time. And then you should move on.

Stand up to your customers

Once in a while, you’re going to be up against a customer who is just plain being unreasonable. Some examples we’ve seen are customers asking for massive customization or complaining about our posted return policy after the fact. In our experience, these customers are really few and far between, but when you do run into them, they add a lot of stress.

Generally, I would say that you should go to lengths to provide a good customer experience, but sometimes that can go too far. In practice, only you will know the difference between making an exception to give someone a great experience and compromising your prior business decisions. But whatever you do decide, deliver the news clearly and calmly, and don’t let any irritation color your communication.

Laser cut hinge system – progress!

I wrote previously about the idea I had to make my wife Genevieve a jewelry box with the laser cutter at TechShop. In the last post, I showed a little physical mockup I made of the kind of hinges I wanted to use.  That proved it would work, but an hour with a pad an pencil and I still hadn’t unwound the math I needed to predict exactly where it would end up when it opened.

Luckily, the helpful commenters funneled through from the post on the Adafruit blog quickly answered my question. It’s called a four-bar linkage, and it’s pretty well studied in mechanical engineering. Still, I found the diagrams pretty baffling (lots of greek letters…), but ultimately I figured it out.

With the equations in hand, I used OpenSCAD to produce a set of drawings.

Clown colors for contrast only. 

I’m pretty pleased with the fidelity of the four-bar calculations. Once I became friends with the Law of Cosines (and corrected all my mistakes), the numbers started working out like I expected them to, and I had good positions for my pieces. I have an angle variable that I can tweak to see where the lid would be at various places in the full motion. And it proved valuable, too: after I saw the lid angle produced by my first random set of parameters, it was clear it would have to be adjusted.

The next step is to take the panelized version to TechShop and cut out a cardboard prototype. Here’s hoping my hinging system works out!

The Roominator – What I Learned

I’ve previously written about the Roominator, my project for combating conference room hijacking. In the end, it didn’t work 100%, which was disappointing. However, I learned a lot in the process, both about how to architect a system like this and about doing design projects like this one in general. I think that this project deserves a proper post-mortem so I can share some of the insights I gained.

What went well

Dumb displays, smart application server

Right from the start, we decided we were going to put most of the actual logic in an application server written in a high-level language like Java or Ruby. Even so, it took us a while to embrace this concept fully. As developers who mostly work in high-level languages, we went down the road of making lots of little classes and interfaces in the display code that proved to be distracting overkill.

Once we got past that problem, though, we ended up with our displays doing basically nothing but showing exactly what the app server told them to. This freed us from writing complicated Arduino code for things like what to print on the display and instead focus on the crucial components – communicating with the app server and making it easy to configure the display on startup.

DIY Power over Ethernet

Of all the things that could have gone wrong with this project, our power distribution strategy turned out not to be one of them. Admittedly, it’s a bit basic, but it turns out that sending 12V out over the unused pairs in an Ethernet cable and then knocking it down to 5V in the display unit worked fine. I would however probably add some fuses and protection diodes to each port so that it’s harder to accidentally fry some part of the system.

Professionally produced PCBs are a huge productivity booster

During the first week we worked on the Roominator, I spent at least 20 painstaking hours  hand-soldering the circuitry of two display units. Only two! There were just too many connections – and opportunities to make mistakes – for it to be sane to do any more like that.

Before the second hack week, I had the sense to order some professionally-produced PCBs of our circuit, and the difference in productivity was unbelievable. Two of us assembled the remaining 10 displays in one day! And only one or two of them required any rework.

Getting PCBs made can seem kind of daunting when you are starting out, but once you learn your way around Eagle, you can easily get boards produced by BatchPCB or Seeed Studio. Particularly with Seeed Studio’s pricing, I think the threshold of when you should opt for professional boards is pretty low.

What went poorly

I2C over long range

Early on in the planning for the project, I decided to use I2C for the communication between the display units and the central master. Despite the fact that this protocol is built into Arduino devices, this proved to be a bad choice for a number of reasons.

The first and perhaps most significant is that I2C is only designed to operate over a range of about one meter. Sure, there are extender and buffer ICs out there (which I ended up using), but this lead to the creation of an extremely complex master unit. And even with the extender ICs, there were some cable runs in our building that were too long. (My suspicion is that when cabling companies come in to wire up a building, they pre-cut lengths of Cat5 to the max ethernet spec and don’t trim the excess, since it doesn’t affect the performance of actual ethernet devices. But it does muck up funky extended I2C communication.)

The other problem with using I2C in this fashion is that it’s a truly shared bus, and if one of the nodes had weird behavior, it ends up blocking the whole bus. In practice this meant that the whole system kept working great until I plugged in the one unit that was too far away from the rest and the whole system crashed to a halt. For projects where all the parts connected by I2C are really one logical device, this is OK, since you can tightly control for issues like cable length. But for systems that are distributed over a wide physical range, there are much better tools for the job.

Focusing on the wrong costs

Aside from purely technical mistakes, the overall biggest mistake I made in developing the Roominator was being too focused on the wrong costs in the project. From the get-go, I was very interested in keeping the cost of parts for each individual display unit down as low as possible, since we were eventually planning to multiply the whole thing by 12. I gave myself a target price of $50 each, which I thought was achievable. And my initial design and research led me to believe this was possible.

But in reality, this whole line of thought was wrong. I thought I was designing an end-user system. In actuality, I was designing a one-off prototype system. What this means is that instead of being focused on how much it would cost to “manufacture”, I should have been focused on how much it would cost to develop
This change in mindset makes all the difference in the world. Yes, using Ethernet to connect the displays to the app server would have cost an additional $20-$50 per display. But if I hadn’t needed to spend so much time developing my super-complicated I2C hub, I would have saved hours and hours of development and fabrication time – time which could have been spent actually building the functionality of the system instead. And even if circumstances had lead to a desire to redesign the whole system for lower costs, I still would have had a fully functional prototype to work from.

This reasoning can be extended to a lot of the decisions in this project. I was concerned that display unit PCBs were too expensive (at around $25 for the two prototypes) and so I only bought them after I made the first two by hand. In retrospect, I could have ordered them early, learned firsthand whether they were flawed, and then ordered more, saving me hours and hours of work. Another thing I could have done would have been to use plain old 5-12V wall warts to power the display units instead of doing pseudo-PoE, obviating the master unit altogether.

As dismayed as I am that the system wasn’t a complete success at the end of the project, I am pretty pleased that there was such a general lesson to be learned from it. To restate it another way, when designing products at this stage, you should make design decisions that allow you to focus on the important parts first, and the unimportant parts later, if at all.