All Posts in “DevOps”

What does pardoning a turkey have to do with Post Mortems anyway?

One turkey every year, typically named Tom, get’s a pardon from the President of the United States every Thanksgiving. That’s a pretty stupid tradition, but politics has a litany of action for the sake of public approval over actual substance scenarios that play out even dumber than this one. The thing that strikes me about this though is that the turkey is being forgiven of something that he never did, essentially a pardon is a waiving of the burden of guilt and the subsequent sentence associated with it.

Think about every post mortem or crisis room you have been in, how often is the fault of the issue acknowledged and forgiven so completely? Or even better how often is the problem not a direct result of anyone in the crisis room? It doesn’t happen is most likely the answer unless you are running blameless post mortems.

Now here is the thing, blameless post mortems are AWESOME in theory. No finger pointing, no screaming, just a calm discussion of what happened, how it was diagnosed and how it was solved who wouldn’t want to be a part of that after a crisis? In practice though, this requires a level of maturity with
everyone in the room, a room full of mature IT people is like a porcupine with a good hair day, not nonexistent just a rarity.

What if though we all take a deep breath in the room with all the stress around us and realize that we are all passionate professionals? What if we are able to separate our individualism for the sake of the team?

Bare with me as I have a back in my day moment ….

Back when I was in the operations business sitting in the data center every silo’d team had friends on other teams. We laughed a luncheons together, we had inside jokes about late night patching or outages or pranks we pulled. When it came to troubleshooting a serious issue we banded together and fixed it as as a team no matter what the root cause ended up being. But at the after action we were all so proud we would try to deflect root cause away from our team. It was not the right way to deal with it.

I have been fortunate enough to have seen blameless post mortem meetings since leaving full time operations. Wow what a difference. The meetings are swift and on target with determining root cause and sharing information and results. The efficiency is striking because no one feels like they have to explain their actions away, instead they are just explaining how things occurred and were fixed. Crisis meetings in these groups are equally impressive, with teams coming together and everyone helping with the troubleshooting process.

If you are interested, I would highly recommend checking out the folks over at Etsy who seem to do this better than anyone and are open enough to write about it. Here are some links:

https://codeascraft.com/2012/05/22/blameless-postmortems/

https://www.pagerduty.com/blog/blameless-post-mortems-strategies-for-success/

https://www.etsy.com/teams/7716/announcements/discuss/10641726/

What are your experiences with this?

Oh and Happy Thanksgiving everyone!!

DevOps more than a buzzword

DevOps isn’t just a buzz word. Ok it’s a little buzz wordy, DevOpsCatbut the movement is real and Enterprises need to look at it. That’s a bit of a stretch many enterprises are looking at DevOps already. It’s a part of a strategy which is key, you have to have a strategy to be successful otherwise it’s just dumb luck and guessing. Those will only take you so far.

Enterprises need to avoid lock-in, that means having multiple concurrent solutions, with contingencies. A VMware strategy, an OpenStack strategy, a PaaS and an IaaS, various public clouds, etc; all of that is necessary as businesses grow and expand. Beyond the infrastructure or cloud play there is a need to determine how application development should be performed, waterfall and agile approaches are each needed depending on the application you are trying to deliver.

Yet, there are more than just technical drivers for incorporating DevOps into your environment. From a business perspective, DevOps have proven to provide faster time to market, higher new customer capture, larger revenue gains, and better user adoption. These are key metrics to make the C levels happy and to help drive IT as a critical cog in the gears of the business.

So while as engineers we too often hear buzzwords bounce around conference rooms, social networks, and corporate emails, take the time to look into DevOps. It’s worth your time, even if it doesn’t apply to every project you are working on.

Bi-Modal Schmi-modal

Everyone does work that they try to make the best they can because most of us take pride in our work. It’s that pride that instills passion in our debates when we defend what we have done or our concept of what we have done against criticism. While I am as guilty of this pride in my work as anyone else there are times where a debate is justified.

This is one of those times.

bimodaldareThe concept of bi-modal isn’t a new one, and in fact has been around since before I was working in IT. However, books like the Lean Enterprise and others have lauded bi-modal as the defacto method for running an IT shop to enable Developers and Operations to each be efficient. This concept has matriculated in recent years to the analyst community to the point where even Gartner has adopted the conceptual notion of bi-modal, and if Gartner says it, it must be true.

Now I am not trying to punch up at the bigger folks here, just merely my opinions and views based on what I have read and experienced.

Let’s first dive into what bi-modal is, “bi” means 2 and modal means modes (you expected that sarcasm didn’t you?). So if you have 2 modes one would be operations and one would be developers. Each mode has it’s own unique set of requirements, not to pick on Gartner here but they use the analogy of Sprinters and Marathon runners.

bimodalSprinters are agile, fast, and their races are short, this translates to developers nicely even to the point that agile development use the term sprint for it’s development cycles.

Marathon runners on the other hand are more methodical; they play a long game, and worry about endurance. This sound familiar? Yeah, that’s the Operations side, concerns about maintaining the environment they are charged with, and the longevity of success for application and hardware therein. Availability, reliability and a solid foundational plan are key. It can take months or more to fully implement a project for enterprise operations folks.

Now you may be saying to yourself sure this makes perfect sense what’s wrong with that? Well bi-modal goes a step further, because these two modes have different requirements, the belief than is that they need to different infrastructures to support each. Yeah I get I work for one of those infrastructure companies, but I don’t necessarily support this view. Because what this means is new gear hits the floor and say it’s for Mode 2 the Sprinter’s who is going to get strapped with doing that management? Yeah Mode 1, because Developers can’t be bothered to maintain their own hardware or if they do have to they get dinged on security compliance issues (been there done that).

It just doesn’t make sense it’s like how Twix builds the left and right Twix in different factories. I mean sure Right Twix has that cool steampunkesque packing tape dispenser but it seems like a huge waste of resources and how did it ever get that far? Was there not someone overseeing manufacturing expenditures?

Hey that’s a great Segway and no I am not complimenting Woz on his localized form of mobility. If bi-modal isn’t right what is? Let me answer that question that you didn’t ask with a question that you won’t really answer, what is missing in bi-modal?

I will give you a second.

That’s right you are smart, we are missing an architectural planning element. You see while bi-modal is conceptually right about the needs of these two groups, what they miss is that if you let either go off on their own never the two shall meet. That misses a huge mark when it comes to trying to converge Devs and Ops to ensure the goals of the organization are achieved don’tchathink?

Tri-modalEnter someone who I tend to agree a great deal with in Simon Wardley, who presents the case that Bi-modal is more of the same archaic silo’d approach to meeting business objectives which have stagnated and caused discourse in IT organizations for decades. He poses that tri-modal IT would be far more effective as an approach. Laying it out in the analogy of Pioneers, Settlers and Town Planners. This got a chuckle from the NMVMUG crowd the other day when I brought it up, as most East Coasters don’t really understand the concept of how vast the homesteading territory actually was. But I digress. In a tri-modal approach Pioneers are the developers they are constantly in search of what is new and never sitting still. Meanwhile Settlers are the Ops folks who stake roots in the datacenter and ensure that it thrives as an ecosystem. Town planners are the glue here, they leverage strategy to ensure that the Pioneers are getting what they need, and the Settlers are getting support from the Pioneers and that the goals are being met. Back into the analogy the Town Planner would ensure that a pioneer wouldn’t set up a temporary cattle paddock next to the drinking well because that’s how you get **it in the water.

The concept of tri-modal speaks to something else though, which is the maturity of an organization. While developers tend to be the founders of software or companies, once established their findings and creations become the baseline which operations have to maintain. Once we hit an operational maintenance mode for anything we start to look at efficiencies in how we productize it (that’s business). Some of the time that will mean changing the way we position or license the product set or the way we manage the infrastructure and move towards the elasticity of cloud, that is commoditization. Look at the path storage has taken, at once it was all built to suite specific needs, then productized and marketed and startups jumped in the ring, now it’s commoditized as we look to moving our data to the cloud. That is the natural progression of most successful solutions.

Bi-modal isn’t wrong but I think a disservice is being done to those who buy into it as the only or best method to combat the ever-changing ecosystem. Business and organizations today are concerned with being out paced or becoming obsolete. That means competitive advantages what better advantage is there than being able to reach full evolutional maturity of products faster than your competitors? Proper planning and execution is how you do that and breaking the silos through team alignment towards business objectives is the key. That’s why tri-modal makes more sense.

/Rant