Diversified Investments–Software style

I worked for Intel during the .com bust era.  It was interesting to watch as we came up on quarterly deadlines where it was going to be tight.  In order to make sure that we made money travel was frozen, discretionary spending was frozen, and the worst was the constant freezing and unfreezing of hiring.  All to make sure we hit the quarterly targets.  When the hiring doors were unlocked there was a sense that we had to get someone in fast before they closed again.  You can guess how effectively that works as a hiring strategy…

CEOs these days have it tough (and have for a while) with the short termism that reigns supreme in the investing industry.  Very few are able to keep a long term vision on course when it comes at the cost of short term profits (Jeff Bezos is a great example of a CEO who has been able to do this IMO). 

But it isn’t just CEOs everyday product/software teams have this same problem.  The short term focus for those teams is the looming product deadline, the current support crisis, or the feature that sales or marketing is clamoring for (and in some cases selling already).  Those concerns can easily consume 100% of a team’s bandwidth.  When that happens it is easy to put all your eggs in the here and now.  That often leads to the cannibalism of the long term strategy for your product.

Software and product leaders would be wise to practice the principles of diversified investing.  Even though I am a long ways from retirement my retirement funds have a blend from conversative to risky in them.  Likewise as you consider the velocity/capacity of your team and where you should invest you need to make sure that you have a blend of short, medium, and long term investments.  Engineering enhancements (like a new build or version control system), experimenting with technologies that you will want to be integrating into your product in the next one to two years, etc… need to have some capacity reserved for them amongst all the tactical requests coming from users and the business. 


Posted in Technology | Leave a comment

A day with Azure SQL, Azure Data Factory, and ETL in the cloud

  I have a goal of writing a tech blog Entry a week.  I am terrible at that goal.  Even now with a better plan for when I will write I am still terrible at it.  With the acknowledgement that I am terrible I am going to get this written even if I don’t have as much time as I would like to write it.

My day job is leading up the Platform Product line at Health Catalyst so I have a good feel for ETL, moving data, data warehousing, etc…  This past weekend I was working on a side project that involved moving data to the cloud.  I was working with Azure, Azure SQL specifically.  I was needing to take a collection of CSV files that had days as columns and rows that reflected a specific entity and the value on that day for that entity populated.  I wanted to take the values for each day and pivot the data so that it was row based instead of columnar.  That meant for most files I was transforming 12 thousand records or so to 1 or 2 million.  The biggest file turned into 62 million records (and a couple of Gigabytes).

I tried the following items

  • PolyBase with Azure SQL

Surprise this doesn’t work/isn’t enabled.  I was surprised by that as MSFT has gotten into a cadence where they generally ship new features into the Cloud first and then into their on-prem “box” products.  PolyBase is a new feature that integrates HDFS distributed tables into SQL Server and enables them to be configured for querying.  Since it is being included in SQL 2016 as I understand I figured it would show up in Azure SQL – no dice.  Not available and no communication that I could find if it every would be.  It is available to Azure SQL Data Warehouse, but that is a different product with a much higher cost.

  • Azure Data Factory (ADF)

I have used SSIS for years and have a love/hate relationship with it.  It does some things really well and a lot of things not well at all (version control, upgrades between SQL Server versions, etc…).  It has been interesting to see that SSIS has not been ported to the cloud like other products, but that ADF which is also a platform for data movement has been built as a cloud first product seemingly in place of SSIS.  I don’t have time in this post to go through my complaints, but I have thought for a long time that ADF is the answer to SSIS and that is definitely not the case.  I was surprised by how ADF was built and the weird combination of scheduling integrated with ETL that it mixes.  The strangest issue I found is that if I wanted to run a data pipeline in ADF just one time (not on a schedule) then I had to redefine it if I wanted to rerun it.  Very strange.  Would love to talk to the Program Managers for the product to see what direction they are going because it certainly didn’t match what I expected for an SSIS (but better) in the cloud – probably I am thinking about what it is trying to solve wrong.

  • R with BCP

The solution I settled on that worked for me was using R to transform the data into the row centric format instead of the column centric format (a pivot like operation) and then taking the output files and loading them up through BCP.  It took a little bit of time to produce the files, but seeing as how I had to learn a little R to figure out how to do it I was pleased with how quickly it went.

Posted in Azure, Azure Data Factory, Cloud, Data Warehousing, ETL | Tagged , , , , | 2 Comments

Are you overrating your experience?

I love this quote from Stephen Covey,

“Covey taught me a priceless principle that would forever change my outlook on the nature of education and experience.  He said, ‘Richie, experience is overrated.  Some people say they have twenty years, when in reality, they only have one year’s experience, repeated twenty times.’


Are we growing in experience or just growing in years?  That is a question we should answer each and every day. 

Liz Wiseman takes this same approach in her book Rookie Smarts (high on my to read list, but haven’t read yet).  She makes the statement that “For today’s knowledge workers, constant learning is more valuable than mastery.”

Posted in Leadership, Technology | Tagged , , | Leave a comment

V Shaped Engineers: The evolution of T-shaped

The concept of T-shaped individuals goes back 25 years apparently, but I became aware of the term probably 7 or 8 years ago. A T-shaped individual has depth in a given area, but also has the ability and passion to be involved in a wide variety of other areas.  I appreciated the contrast shared on the Code Renaissance blog between a T-Shaped Individual and an I-Shaped individual

T-Shaped people are like the traditional I-Shaped person, a specialist, but they also branch out wider, where an I-Shaped person will primarily go deeper and deeper

These days a rough synonym for T-shaped individuals is that of Full Stack Developer

Benefits of T-Shaped

I appreciate teams built with T-shaped individuals because they tend to adapt to the challenges ahead of them easier.  Because of the breadth of the team members if there is a bottleneck in one area of the team’s workflow (be that software development or something entirely different) the team can adjust to address the constraint and move forward. 

Additionally T-shaped individuals tend to be passionate about many things which creates a very positive team dynamic.  The variety of discussions and the insights from those discussions are better and create a learning environment (and generally an open minded one as well).  Perhaps this also explains my preference/advice to those considering going back for a second degree to look at a different field rather than doing a graduate degree in the field their previous degree was in. 

I-shaped specialists if not careful approach every problem with the same solution because it is the solution they are passionate about.  The well-used everything is a nail when all I have is a hammer analogy could be applied here.  That isn’t to say specialists don’t have a place, but the number of places they can be utilized is limited.  If you have more than enough work in that space than this isn’t a problem unless they become the constraint (how often in a software engineers career has the DBA been the constraint….)

Not only does the breadth of a T-shaped/full stack engineer help them do work in different areas it also increases their troubleshooting/problem solving capabilities.  They are better able to connect the dots and trace the challenges across different domains.  The relationships between different fields/concepts are more readily seen as well which can be a catalyst for greater insight.

For those technical

Arguments against

One argument against is that as applications and systems grow you need more depth and that those that spread themselves out as much as down will not be able to address the technical challenges that emerge.  I think there is a truth to that.  I am not yet settled but I believe given the choice to be either deep or broad and not very deep that deep is the better choice because the solutions you propose will generally be based on facts about the technologies that you truly understand.

Another argument against is that in the new application world that we live in where machine learning, mobile platforms, and persistence platforms beyond relational have emerged that no one can have advanced knowledge in all.

To this point I would simply say that the T-shaped concept and by association the full stack engineer does not purport that a single individual is advanced in all the areas above.  A team of T-shaped individuals would ideally have the appropriate balance of depth in the required areas. 


The concept of T-shaped individuals applies to not only software architectures and layers, but also languages and frameworks, business domains, etc.…  Additionally when using the term T-shaped the perception is perhaps given that there is one area of depth. 

Much more frequent (and more desirable) in my mind is more of a V-shape.  As you expand beyond your areas of greater depth you inherently are a little deeper in adjacent areas because of their relationship with your areas of specialty. 

Regardless of the term used

  • T-shaped
  • Full Stack
  • Generalizing Specialists
  • V-shaped

the more you want to lead the teams or lead the engineering of software applications the more true this concept becomes.  An I-shaped individual generally doesn’t have the relevant skills to function on a more broad scale (thus the visualization of an I).  Unfortunately often T or V shaped individuals asked to lead become — shaped individuals because they fail to continue to invest in their depth as their leadership requires them to broaden out.  All too common in our industry is the software engineering manager who is so out of touch that they no longer could function in a technical capacity (or have any technical street cred left).  That is why I so enjoyed the recent interview that Shawn Wildermuth did on his Hello World podcast with Scott Guthrie who heads up Microsoft Azure.  He talks about his journey from a deep technical role and how he balances it.  He includes a fun story about a recent incident where he inadvertently showed his continued technical street cred to a team of engineers who were troubleshooting a production Azure incident.

Here is to the continued development of V-shaped individuals and to the encouragement of leaders in all disciplines and domains to continue to invest in their depth as well as their breadth.  We are all the better for it.


Posted in Technology | Leave a comment

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 5,400 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 9 years to get that many views.

Click here to see the complete report.

Posted in Technology | 1 Comment

Launching a Debugger on application start

I have been working a lot with Entity Framework Migrations lately and needed to debug the migrate.exe utility, but to do so I needed the debugger to start when the process tried to start (as opposed to downloading the now open source code and trying it that way which I could have done).  I found this link on MSDN that addressed the situation perfectly.  Now everytime I launch migrate.exe it attempts to connect to a debugger prior to executing.  Lovely!


Posted in Technology | Tagged , | 1 Comment

Installing Windows 8 RTM on a Retina Macbook Pro

I thought I would capture my experience installing Windows 8 RTM on my Macbook Pro (the new Retina version with SSD).  First I must say that I am enjoying my Macbook much better now that I have an SSD in it.  Computers without SSDs are just painful to use these days especially as a developer. 

For the most part the upgrade went exactly like any other upgrade.  That was good.  When I booted up the first problem I ran in to was that the Trackpad drivers for the Mac wouldn’t load and I couldn’t load the Boot Camp Control Panel (weird error about not being able to access the startup disk).  I looked in Device Manager and found that there were two Trackpad devices registered, but they couldn’t start.  So I decided to start with those first two items and resolve them.  Thankfully the Internet is a wonderful place, full of all sorts of answers!

Issue #1 – Not being able to access the Boot Camp Control Panel

The essence of the solution was to run the Boot Camp Control Panel in a different UAC security context


Issue #2 – Trackpad wasn’t working

The solution is simply to map the drivers to USB Input Devices and then back and magically it works (weird I know!)


Issue #3 – Windows 8 Enterprise kept complaining about activating

From the command-line you access a UI that allows you to enter in a key that is valid which is what was causing this to complain in the first place.  This may be because I used my MSDN account to install Windows 8 Enterprise (I am doing that to test it out at my employer for those wondering about the license concerns there).


Posted in Technology | Tagged , | 1 Comment