Speaking at Self.conference 2017

I will be speaking about home automation with Home Assistant at Self.conference in May! One of my goals this year is to do more public speaking, so I’m thrilled to get to be part of this event. There is a pretty amazing lineup of speakers so you should probably go ahead and register already.

Here’s the write-up of my talk:

A Smart Home To Call Your Own

The stuff in your home is getting smarter every day. It’s time to take control of it with Home Assistant, a Python open source app.

This talk will provide an overview of how Home Assistant can be set up and configured to make your home smarter. You’ll be able to control your smart home devices so that your home starts working for you. With presence detection, logging, and mobile access, you’ll be the master of your domicile.

Once we see what Home Assistant can do we will explore how to write a custom components to start automating our own devices. Welcome to the world of tomorrow!

My Favorite Bug

Years ago, at a financial services company…

It started with some mild grumbling. Some people were complaining about getting signed out on our customer portal. But it was probably just user error, right? The reports seemed to happen more and more though. Then, employees who worked on the product started to get signed out.

At this point we have to admit that there is a problem. Every so often someone is using the site and they are suddenly booted out.

It happens seemingly at random. If you have a bug that you cannot reliably reproduce, your goal is to gather enough information until you can. “Seemingly at random” is a euphemism for “caused by some piece that we aren’t considering.”

Being able to reproduce a bug is important. First, you want to be confident that you have fixed the issue. If you can’t be sure you’ll trigger the bug, you can’t be sure your fix worked. Second, the factors that go into triggering the bug give you insight into where the problem is. Working toward a consistent reproduction is helping you triangulate the problem. So we started gathering information in hopes that we could reproduce it.

One thing you can do is ask yourself “what changed?” One of the things that changed was that we were rolling out a new version of our in-house PHP framework. This felt important, but also not, since the affected site was on the old framework. What’s more, both frameworks used the same way to delete sessions. Something like:

sprintf('DELETE FROM `sessions` WHERE `expires_at` < %d', now());

Which is a pretty reasonable way to expire sessions. The session creation code was similarly reasonable

sprintf('INSERT INTO `sessions` (`cookie`, …, `expires_at`)  VALUES (:cookie, …, :expires )', $cookie, …, now() + SESSION_LENGTH);

Jason, one of our analytics folks, was able to add HTTP monitoring. We figured out what the problem looked like from the browser side (a request with a session id setting a different session id in less than n seconds) and started logging how often it happened.

What we learned from that the frequency was accelerating. The longer it took us to fix, the worse it was going to get.

We still cannot reliably reproduce this at this point, so we have to keep gathering information. I was tempted to deploy changes out to see if they worked. This is dangerous. If you don’t understand the problem, you don’t know if a fix is going to inadvertently make things worse. Remember: if it seems random, you don’t understand it.

Adam, one of the devs on the team, added logging to our session delete code. Before it ran the DELETE statement, it did something like

debug_log(query(sprintf('SELECT cookie, expires_at FROM `sessions` WHERE `expires_at` < %d', now())));

so we could see exactly which sessions were being deleted. That’s when things started to get weird.

We would compare that to our HTTP monitoring and find the session and see that it definitely should not have expired based on the timing. But the session logging says that it definitely should be deleted based on the time it was logged.

At this point we reach the “Eureka” moment. For context, “Eureka” is what old Archimedes shouted as he jumped out of the bathtub. While the bathtub is clearly useful in realizing that water displacement can measure volume, I don’t think enough credit is given to the bathtub as a place of thought in that story. I consider showering (along with dog walking, driving, and sleeping) to be an important part of debugging.

And so I was, when I was trying to figure out how to reconcile the two facts:

  1. Sessions were expiring before they should be
  2. Sessions were expiring on time

I realized the disconnect, hopped out of the shower and sent an email to our Unix admins asking them to check the clocks on our production servers. Sure enough, one of the load-balanced servers wasn’t running ntp and it’s clock had drifted out of sync.

What was happening was if the session code was called on the rogue server then now() was wrong. That meant that sessions created on it would live extra long, but sessions deleted on it would die young.

Why was it accelerating? It turns out that in V1 of our framework the session-delete code was mistakenly not hooked up, but that was fixed in V2. The way PHP works is that every n requests it calls the cleanup function. We were moving more and more traffic to V2, which meant that the cleanup function was being called more often.

I like debugging. Well, I like having debugged. It can be maddening in the middle of it but there is a lot of satisfaction in solving a puzzle. A bug is a puzzle with stakes. If you are spending your time on a bug, that means its important. There’s also satisfaction in finally arriving at a simple, testable, theory of a bug.

I like this bug because it required us to understand so many pieces that make up the system: HTTP requests, session persistence, load balancing, and server clocks. It’s also nice when the fix is to adjust the clock instead of recode your entire app.

Image credit: Public Domain image NH 96566-KN The First “Computer Bug”

Well… how did I get here?

Johns Malkovich

Employee referral programs are killing diversity in the tech industry

Roughly 70% of tech companies have programs to encourage referrals and most importantly, referrals account for up to 50% of new hires in the US.
…
Unfortunately, an employee being asked to refer someone for a given position will almost always go for somebody who shares the same social capital: skills, tastes, posture, clothing, mannerisms, material belongings, credentials, degree etc.

I’ve had seven tech jobs. I got three of them as a referral from an existing employee. Importantly, those three jobs also accounted for the bulk of my professional growth. Furthermore, for the second and third jobs in that list I was referred by my network from my first referred job.

I got the best jobs of my career because I knew someone who worked at the company. Someone who was like me.

That doesn’t mean I didn’t deserve the job, wasn’t qualified for it or anything. But it’s useful to remember that someone without the same networks as me would find it difficult to compete with me for those jobs. At a macro level it means access consolidates around similar people. That’s a problem for companies, since diversity is good for business.

Referrals are cheaper and more successful than less targeted funnels. Referral bonuses seem like a decent thing too. A referral bonus is also a good answer to the question “if you’re paying that recruiter to hire people, why should I do their job for free?” I’ve earned a few referral bonuses over the years and I’ve known a couple of people who turned referral bonuses into a second income stream.

That post I linked glosses over another piece of referrals. When you are interviewing for a job, you are also interviewing the company to see if you want to work there. Tech workers are in a tight labor market (for now). Employers have a tough time finding talent. A large part of the reason I switched to the jobs at which I knew people is because I had social proof that they were good jobs.

I only switched jobs because people I trust told me it was a good place to work.

If referral programs are successful, if they make hiring more efficient, and if they benefit workers, then what is the problem? This could be a reminder that “what you measure, you improve” is both a promise and a warning. Changing the metrics of success changes all of that. However, employee retention and success are easier to track than the innovation benefits of diversity. One of my takeaways from Deep Work was “The Principle of Least Resistance:”

In a business setting, without clear feedback on the impact of various behaviors to the bottom line, we will tend toward behaviors that are easiest in the moment.

So what is the answer for companies?  How do they fill that funnel while still convincing experienced talent to work there? I don’t really know, sorry. I know there are people who think about this deeply. If you know who they are please point me in their direction.

I do know that diverse workforces attract diversity. Knowing that there are people like you is reassuring for people, including me. Getting to a workforce where there are “people like you” (along whatever axis you’re considering) for candidates is the sticky wicket.

It’s not like there were no hurdles in getting those jobs, but it’s valuable to remember that a hurdle doesn’t prove an even playing field. What I can say is that when you are unpacking things like privilege and diversity, I think it’s important to ask “Well… how did I get here?“

Deep Work

I recently finished Deep Work by Cal Newport.

To build your working life around the experience of flow produced by deep work is a proven path to deep satisfaction.

Deep Work feels like a continuation of Newport’s previous book So Good They Can’t Ignore You. That book argues that your mastery of your craft is a larger factor in your fulfillment than the choice of work. It then goes on to prescribe mastery through “deliberate practice” to improve. Once you have leveled up, Deep Work offers a next step: a path to productivity.

The ability to concentrate intensely is a skill that must be trained.

Newport spends a good amount of pages making the case that blocks of focus and solitude are vital for a knowledge worker to produce the best output. His strategies should not be surprising. “Don’t goof off on the internet” seems like “eat right and exercise” for productivity – advice that’s easy to say but hard to live. Hopefully, by arguing the value of deep work and the need to train your brain, readers will have success with his methods.
When I get the urge to check Twitter during a build/deploy, it’s helpful to have a reminder that small choices matter. Reducing my time on social media – the internet’s Skinner box – has helped me focus. It’s also been helpful to also consider the schedules he describes. Blocking off time to be unreachable has been a boon. So much so, that I’m revisiting RescueTime to try creating focus blocks.

The book offers working templates for different personalities. While I am productive in the solitude of my office, I feel that I am most productive when I am pairing with someone (this may be some selection bias – only certain work lends itself to pairing). In his section on work styles, I’m glad that he steps back from the isolation he describes elsewhere and offers support for working in partnership.

I enjoyed this book and am glad to have found it. The most valuable piece for me has been the reminder that my minute-by-minute decisions become powerful habits. I recommend this to any knowledge worker. I’d also be extremely interested to find out if it benefits other professions.

See also: There’s a good hacker news discussion

Brewing on a school night

One thing I tell friends about parenthood is that the important stuff sticks around. You may not be able to spend a weekend binge-watching shows anymore, but you’ll find ways to keep the hobbies that are important to you.

Jenny got me Adventures In Homebrewing’s Vanilla Porter recipe for my birthday. I’ve made this kit before and it’s tasty. So, last night after we got the girls in bed, I was able to brew a batch of beer.

I don’t have 6 hours to spare for an all-grain batch. It was satisfying to knock out an extract beer in 3 hours (including clean up). Sure it was a bit cold in the garage, but the nice thing about brewing in the winter is that the wort chills much faster.

Recipe Details

Batch Size Boil Time IBU SRM Est. OG Est. FG ABV
5 gal 60 min 25.0 IBUs 22.8 SRM 1.056 1.015 5.4 %

Style Details

Name Cat. OG Range FG Range IBU SRM Carb ABV
American Porter 20 A 1.05 - 1.07 1.012 - 1.018 25 - 50 22 - 40 2.3 - 2.9 4.8 - 6.5 %

Fermentables

Name Amount %
Caramel/Crystal Malt - 60L 8 oz 5.56
Black (Patent) Malt 4 oz 2.78
Chocolate Malt 4 oz 2.78
Pale Liquid Extract 8 lbs 88.89

Hops

Name Amount Time Use Form Alpha %
Cluster 1 oz 60 min Boil Pellet 7
Willamette 1 oz 5 min Boil Pellet 5.5

Miscs

Name Amount Time Use Type
Vanilla Bean 2.00 Items 0 min Secondary Spice

Yeast

Name Lab Attenuation Temperature
SafAle English Ale (S-04) DCL/Fermentis 73% 59°F - 75.2°F

Download

Download this recipe's BeerXML file

If Your Boss Could Do Your Job, You’re More Likely to Be Happy at Work

If Your Boss Could Do Your Job, You’re More Likely to Be Happy at Work

Studies of leaders often focus on their style or charisma, but we wanted to look at how workers are affected by their boss’s technical competence. That is, is the boss is a real expert in the core business of the organization? How much expertise does he or she have? Boss competence is, admittedly, a multifaceted concept. Hence we measured it in three different ways:

  • Whether the supervisor could, if necessary, do the employee’s job.
  • Whether the supervisor worked his or her way up inside the company.
  • The supervisor’s level of technical competence as assessed by a worker.

Using these three measures of supervisor competence, we found that employees are far happier when they are led by people with deep expertise in the core activity of the business. This suggests that received wisdom about what makes a good boss may need some rethinking. It’s not uncommon to hear people assert that it’s a bad idea to promote an engineer to lead other engineers, or an editor to lead other editors. A good manager doesn’t need technical expertise, this argument goes, but rather, a mix of qualities like charisma, organizational skills, and emotional intelligence. Those qualities do matter, but what our research suggests is that the oft-overlooked quality of having technical expertise also matters enormously.

I have mostly worked with technical leaders. I’ve found that the more aligned my job is with their experience and the core business, the happier I am.

The lesson seems to be to actively grow leaders in your core business area. And, if you want to change your core business area, to devote time towards growing leaders in that new area.

Mythical Man Month draws an interesting lesson from this idea:

I have earlier argued that the sheer number of minds to be coordinated affects the cost of the effort, for a major part of the cost is communication and correcting the ill effects of miscommunication (system debugging). This, too, suggests that one wants the system to be built by as few minds as possible.
…
The conclusion is simple: if a 200-man project has 25 managers who are the most competent and experienced programmers, fire the 175 troops and put the managers back to programming.

Brooks goes on to conclude that is not realistic, and the real solution is to break up systems to reduce communication overhead. But, when they read “put the managers back to programming,” I know some pretty great leaders who get excited at the prospect.

Hike

I’m trying on a new birthday tradition. Walk in nature, alone, for a couple hours. This is my first year in but so far I recommend it. Even if your birthday is in January.

Panoramas and [vr]

Automattic, the company I work for, has released Jetpack 4.5. It includes the [vr] shortcode that my team mates worked on. The extent of my effort was modeling for some 360º shots so here is my contribution: panoramas!

First we have my favorite panorama. It’s a little lower res – shot with Dermandar on an iPhone 4 – but it’s the subject matter I love. The Paul A. Johnson Pencil Sharpener Museum:

Jenny and I stopped at a rest stop on our drive home from North Carolina and found this magnificent display purely by chance.

Next is an abandoned-ish building on the Belle Isle disc golf course:

The Belle Isle Conservancy. Also on Belle Isle but less abandoned and less tolerant of flying discs:

Finally, a panoselfie with my brother on Sugarloaf Mountain in Marquette:

Ultimately this post has been an excuse to brag to everyone. Yes, I’ve visited the Paul A. Johnson Pencil Sharpener Museum. Keep sharp, be sharp, act sharp, stay sharp, look sharp.

Detroit Tourism Roundup

2017 seems to be the year to visit Detroit.

New York Times puts Detroit as #9 on its list of places to visit. The QLine streetcar seems to have been part of that decision, so maybe there’s hope yet for a regional transit authority. Maybe…

National Geographic made Detroit their North American pick for Unexpected Cities for the Food Lover. They pick up on a few of the standard restaurants (I’m post-post-Slows hype) but miss out on a gaggle of favorites. I don’t miss the commute downtown but I do miss the options at lunch time.

There’s so much good food downtown, I wish I had more opportunities to explore them. My personal list of Detroit dining this weekend consists of Buddy’s Pizza (no, Little Caesar’s doesn’t count as Detroit-style), Tubby’s Submarines and maybe Gold Cash Gold. I should figure out a way to sneak a coney dog in there.

One final word of warning. If you do come to visit Detroit, don’t come here during the winter…

Emergency Fun

…d. Sorry, that title is supposed to be “Emergency Fund”. Now that I reread that headline, it is a little misleading.

In addition to the advice in my last post, here’s something think about: 63% of Americans Can’t Afford $500 Car Repair or $1,000 Emergency Room Visit.

I used to think that paying off debt was more important than establishing savings. After seeing people using emergency funds to smooth out rough spots in life, I’m convinced that having a small emergency fund is a top priority.

One of the easiest ways to get savings started is to automate it. Set a recurring transfer at your bank to put 10% of every paycheck into savings. 10% is usually small enough to stomach, but you’ll have deposited 3 months of paychecks in 2 1/2 years. After that, you just need to make sure it stays topped off.

An emergency fund is a kind of self-care. You can rest a bit easier and you can have money set aside for when disaster strikes. It’s a lot easier to live life in the moment when you have that little bit of piece of mind.

If you still need some emergency fun, check out the best media corrections of 2016.

Photo Credit: (CC-BY) Kevin Dooley