powdermonkey

For Want of a Patch (& a Supply Chain)
My new post from CTOvision: https://ctovision.com/2014/12/want-patch-supply-chain/

For Want of a Patch

For want of a patch the component was lost.

For want of a component the stack was lost.

For want of a stack the system was lost.

For want of a system the message was lost.

For want of a message the cyberbattle was lost.

For want of a battle the enterprise was lost.

And all for the want of a software patch.

As the old proverb reminds us, logistics is important: most battles are over before they’ve begun due to having or not having a solid logistics tail. During WW2 the Allies found out the hard way with the invasions of Africa: ships loaded incorrectly led to delays in material onto the beaches and towns, things like ammunition, fuel and medical supplies are needed before typewriters and tents. As subsequent amphibious invasions progressed (North Africa, Sicily, Italy) the military learned how to coordinate better the planning and ultimate loading and unloading of material and manpower to have the largest effect in the fight. These processes ultimately culminated with the successful massive invasion of Normandy to end the 3rd Riech’s hold on Europe.

The key lesson was to view logistics in war as a continuous process that feeds a fast and continuously maneuvering Army.

Cyberwar is no different and more closely follows the proverb: one unpatched line of code can leave an entire enterprise open to assault. Why? Accelerated use of software, more software dependencies on other pieces of software AND all that software is constantly in need of being updated. Current organizational processes to keep software updated can’t keep up with the change being generated by the outside world.

Example: Amazon software deployments for May 2014 for production hosts and environments: 11.6 seconds is the mean time for deployments and 1,079 max deployment in one hour: how many military systems can claim that many deployed changes in a month? (Ref: Gene Kim, slide 23 http://www.slideshare.net/realgenekim/why-everyone-needs-devops-now ) I doubt any, but this is what the military (and modern enterprises like Sony) must prepare for: never ending change and updating on near random cycles.

More to the point: continual and unscheduled software patches are the landscape in this new maneuver environment. And since they can’t be planned for, organizations need to learn to evolve for change and deploy software and new capabilities continually.

Software supply chain planning is no longer something that can be starved of funds. Malware, continuous monitoring, and network scanners can tell you which barn doors are open and that the horses are leaving, but leave enterprises with a massive punch list of fix it items. Funding, time and effort need to spent on the supply chain. It is the first true line of cyber-defense.

Parting shot, question for CIOs/CTOs: Can you patch all of your systems in the next hour, using existing processes and not bypassing things? For most organizations the answer is no, OpenSSL patches (seriously!) get emailed around from dubious sources is akin to Mom mailing ammo to her son in a care box in Afghanistan.

For want of a message the cyberbattle was lost.

For want of a battle the enterprise was lost.

And all for the want of a software patch.

Government get your Sh** Together

Commenting on the ‘new’ government digital service and TechFAR. 

Great article here:  New US Digital Service Looks to Avoid IT Catastrophes 

Discussion by Gunner: US Digital Service is Born

Steve Kelman, FCW:  The REAL regulatory challenge of agile development

 This simply isn't bold enough. This effort simply try’s to fix whats broken instead of looking more deeply at why its broken and how to ‘reformat’ the way IT services are built and most importantly used by the government and citizens.

The Service and TechFAR is akin to what the military seems to plan for: planning and building systems to fix yesterdays battles instead of really and deeply thunking for the future. 

UK: the UK gov digital service worked because its much, much smaller than our government (think California), two the UK gov is setup very differently than our government bureaucracy. The UK digital service reports directly to the PM and as I understand it, really could walk into any UK Agency (save MoD) and demand changes + take over projects. Also in a Parliamentary system, what the PM says goes, no discussing with Parliament, arguing over budgets, etc. a number of projects get killed pretty fast when there is a gov changeover. 

The US has neither. 

GSA is a pretty widely ignored Agency (for a number of historical reasons), maybe this time it will be different, I’m sure they can help web pages load faster. But the Digital service is going to report a few levels down from POTUS and the head of GSA, both of which have many other things on their plate. 

Also US Agencies have two masters they play pretty well off each other, Congress and the Pres.  Scale: the gov is friggin huge. Also some government systems have very unique and multiple functions for only one customer (the Gov + citizens).  Fixing FAA and SSA isn’t a few agile sprints over pizza and Dew.

 Instead of fixing past problems, we need deeper thinking about HOW and WHY the government should provide services. 

Examples: 

- The government no longer runs motor pools, it outsources the entire job to companies with specific service levels and agreements (SLA).

- Failed example: SABRE (the airline reservation company) came to DoD and pitched the idea that they would take care of all military travel for something like $60 a ticket. Some govvies pitched they could do it cheaper, they tried and build a disaster of a service ++ Defense Travel continues to eat funds way above and beyond. And continues to frustrate and strand military travelers overseas. 

 There are many more, but the basic take away is this: the government must not recreate services the private sector does cheaper, better and faster unless its part of its' core mission. 

• Bombs, check - part of core military mission. 
• HR? (outside of the military and CIA), the government should license or buy as a service HR capabilities. 
• Websites? I’m having a tough time with this. If the government could come up with a list of requirements and define a serious set of SLAs, they are any number of companies who would gladly offer website as a service, with the government managing the input.
• IRS systems: parts of it could defiantly be outsourced, especially all of fraud monitoring. 

One last point: 

YAR - yet another review, I don’t see how another YAR by the Digital Service is going to add to government agility and flexibility. Make no mistake, the gov is setup with a number of very expensive and time consuming YARs already, each which must be planned for and dumbed down to senior management. 

TechFAR inculcates YAR + yet another ref doc to read, that  won’t apply to any Agency that doesn’t adopt it. 

Many of these suggestions sound good for a few projects, but crumple and slow down system creation when scaled to 1000s of project in an $80 billion portfolio. 

Some question to ask as the Gov builds systems: 

1. Is the thing your Agency needs to develop a core competency? 

2. if not, define how to buy it as a fixed price service offering w/ a tight SLA

3. if yes, first start developing small + draft off of any existing efforts (open source software, or other state, local, international gov’s) ++ be open to the outside

Errata: 

• stop fixing current problems with past thinking 
• stop telling industry how to suck the egg (i.e., stop dictating to industry what methods to use to develop technologies, CMMI was great for adding bodies - thanks for the revenue, is Agile really the endpoint of development methods? Lets not hard code something (again) into how the gov does procurement)
• automate eixsting jobs, rethink how the work gets done (IRS a body shop)
• automate existing process and rethink if you need them at all 
• Start to collapse Agencies and processes. Does every Dept and Agency really need a CIO and associated staff? 
• What things could Agencies outsource to each other (like paychecks that Dept of Ag does for smaller Agencies)

 

Learning to Breathe Fire

Book to buy! Learning to Breathe Fire: The Rise of CrossFit and the Primal Future of Fitness

http://goo.gl/IWTJaT

“Learning to Breathe Fire is one of those books that come along every generation or so that brings to life a sub-culture so vividly, so deliciously that it makes you want to run out now and become a part of it! Written with great verve, comprehensive research and a novelist’s knack for deft characterizations, this is CrossFit’s War and Peace.”
--Charles Gaines, New York Times bestselling coauthor of Pumping Iron

“A beautifully written mix of evocative vignettes and lucid explanations that shows us what we’re capable of when we train hard and connect with our instinctive nature. This is a book about digging deep, about kindling a spirit that allows us to push past our wildest expectations. Whether your fitness habit involves going it alone or tunneling through an extreme workout as part of a group, you’ll find this CrossFit journey thoroughly immersive.”
--Marshall Ulrich, Badwater-146 Record Holder and author of Running on Empty

“CrossFit is a phenomenon, both as a radical way to confer fitness and as a virally successful business. Herz tells both stories with exceptional insight---plus the inside lore of a dedicated CrossFitter.”
--Stewart Brand, creator of the Whole Earth Catalog (and CrossFitter at age 75)

“The remarkable rise of the CrossFit movement is grounded and propelled by a great moral truth. Effort alone is all we may bring to life. Everything else -- our genes, our talents, and our teachers -- are gifts. J.C. Herz has written a compelling book around this truth as embodied in the CrossFit culture. As a society, we forget that we are evolved to realize our greatest strength when we are truly tested. The originators of CrossFit have rediscovered that, and proved it by becoming the fittest humans on the planet. In the view of CrossFit athletes ‘the only possible sin is slacking off.’ Herz writes with sweep and depth about great characters, often racked with doubt, finding their limits and surpassing them. This is the ultimate chronicle of how they created a training method, a championship, a corporation and a loving community devoted to the sacred tenant of effort.”
--Kenny Moore, award-winning writer for Sports Illustrated, former American record holder in the marathon, author of Bowerman and the Men of Oregon, and co-screenwriter of “Without Limits”

“J.C. Herz expertly debunks many longstanding fitness beliefs and shows how high-intensity exercise can yield the greatest return on your workout investment. Her Learning to Breathe Fire chronicles the rise of CrossFit, showing – in a way that is always interesting and insightful -- how ordinary people have achieved extraordinary results following this program. A must read for anyone looking to maximize his or her potential.”
--Dean Karnazes, ultra-endurance star, New York Times bestselling author, and one of Time Magazine’s “100 Most Influential People”

“I couldn’t put this book down. J.C. spares no detail in helping us see into into the heart and soul of a CrossFitter. Her description of what CrossFit athletes overcome is truly unbelievable. She captures the essence of the sport and what it represents, most especially the ability to push through barriers, whether physical or mental. This is about the gut-busting journey to the last rep, but it’s also about life. Whether you’ve tried CrossFit or just thought about trying it, Learning to Breathe Fire is a must read.”
--Chrisanna Northrup, New York Times bestselling author, CrossFit Level 1 Coach, and former CrossFit Box owner

About the Author
J.C. Herz is a Harvard-educated former New York Times columnist as well as a former rock critic and tech writer for Rolling Stone and Wired. A two-time author and technology entrepreneur, she started doing CrossFit in a gym where white-collar professionals and new moms cranked through pull-ups and Olympic lifts next to active duty military and members of the presidential Secret Service detail. Her favorite CrossFit workout is “Cindy.”

Cost Overruns in Large Systems (HHS take note)

From my Masters Thesis, wonder how much planning went into Healthcare.gov... so if NASA who has the brains about how to build build systems has a tough time getting it right, HHS has no hope. There were also time over runs as well. 

... design decisions made early in the life cycle can have a large impact later when the technology is in operation.  An example of this in the graph by Werner M. Gruhl (Cost and Analysis Branch, NASA) presented at a INCOSE (International Council of System Engineering) System Engineering Seminar in 1998.

In the Figure 1-1, Phase A & B Costs are the costs associated with early stage conceptual planning and design of a technology. One early stage decision in the life cycle of a technology is how much money to allocate to the early stages of development and design. The chart shows that (for systems at NASA) if less than ten percent of the total cost is allocated to these early stages there can be an expectation of cost overruns. Making the decision to allocate more money to these earlier stages allows for the use of more resources to make sure the technology being developed closely matches what was envisioned.

 

 

Healthcare.gov is a 4-Star Mess

 More of a mess in the Healthcare.gov website, Now with copyright violations!!!

Obama Administration Uses Pirated Code on Healthcare.gov 

per http://torrentfreak.com/ 

"The new Obamacare website Healthcare.gov has had its fair share of problems over the past weeks, and the trouble continues. As it turns out, the Government website uses the open source software DataTables, which is a plug-in for the jQuery Javascript library. While using open-source software is fine, the makers of Healthcare.gov decided to blatantly remove all references to its owners or the original copyright license." 

Why this matters: 

 What they have basically done is strip away the marking that lable this as someone’s work. Its akin to taking a friend’s homework and erasing their name at the top of the page.  

This (unfortunatly)  happens all the time wrt open source software.. 

I hope whoever owns the copyright lodges a complaint and maybe a lawsuit. 

Fire the Head of HHS: Gov, tech, accountability and acquisitions

The current failure of the HHS 'Obamacare' website and healthcare portal heathcate.gov and extreme cost of the failure (some site list $634 million) has pointed out once again the government has no business in specifying, building or developing large and complex technology projects. 

Why? 

1. Gov internals 

The folks inside (and helping) the government oftentimes aren't clued in with what is 'state of the art' in commercial industry. Not their fault, skills degrade as you work in a large bureaucracy. 

Worse: the gov also tells the industry not just what it wants (bad specifying usually) it also tells industry how to build tech, which is really bad because the gov doesn't know how modern tech is built (see DevOps). Adding extra processes that aren't used in commercial practice simply creates a sub-class of specialized companies hose only competitive advantage is understanding gov-speak (CMMI, CONC, CNA, etc.), to say nothing of actually gov contracting. 

 Also as skills degrade (and this is especially important for higher tech goods that are assembled with other tools) the knowledge of the tools around the tech also degrade. Example: in the lead up to WW2 the US Army was still bolting tanks together using railroad chassis, where by that time private industry had moved to building beefed up truck chassis with newer and faster welding for large load trucks. The US Army's knowledge of the process for how to build large vehicles was way behind industry, because the new, fun, interesting and hard problems to solve were in the private sector. This leads to point 2.

2. Smart and innovative tech people do not want to work for the government, except is very short doses or on really hard and unique problems. No extremely talented tech person wants to work at HHS, HUD, etc.. Reasons: they don't pay well and the problems they have aren't hard and/or require unique skills. The only exception to this are areas where the government is still the leader in a tech: crypto and comm's (NSA), cancer research (NIH), nuclear (DOE) and weapons (DoD). Agencies like NRO and NGA and their contractors  (satellites and maps) can't hold onto top tech talent anymore as the commercial industry is more fun and less burdened by regulations and bureaucracy. 

3. The acquisitions process. 

The process for how the gov acquires tech is slow per 1, but why? Because it grew up to buy hardware (chairs, desks, tanks and planes), not tech that rapidly changes like the more malleable soft tech's (computer hardware / software). In soft-tech since the technologies are so malleable, evolve and morph over time, the tooling and process for how these systems get built is more important than the resulting product. Its the process that matters more for soft-tech than the actually product (because the product is simply a finite snapshot of what the process can deliver at that time). 

So having an acquisitions process that is historically geared to buying stuff versus process leads to failure. 

Also, hardware buying front-ends design optimization to build many things cheaply per unit (in gov speak 'measure a xxx times, and cut once') whereas in soft-tech the end product is constantly optimized throughout its lifecycle (design, test: repeat). 

So where does that lead gov tech and acquiring soft-tech?

Rule 1: The gov should not acquire or pay to build a system that is not core to the Agency mission. 

Unless an Agency truly can say it has the best and brightest in the industry working for it (all the way up and down the chain AND is driving that industry) to help it specify what it wants (and can throw the BS flag) it must not be in a role to specify a system to be built. The knowledge just isn't there, its not a fair fight. How can a blind man ask for red hat?

Examples of things that aren't core: travel systems, motor-pools, HR (unless you run under-cover operatives), tax-return software...

Things that are core: see above, but crypto, weapons, tools associated with espionage, research, others…. 

An example of acquiring non-core: years ago the DoD decided it needed a new travel system. The options were build or buy the service. The travel industry came back and said: we will build you a system and charge you per ticket, just give us the high level needs (security, etc.) and we will charge you per ticket issued (i think it was $64). Morons in DoD said 'no, we can do it more cheaply' and then set out to blow through allot of money (see: http://www.gao.gov/products/GAO-09-577). $ 2 billion? to say nothing of the hassle of actually using it. 

Sound a lot like healthcare.gov?

Even the CIA and NS have realized that building cloud computing technologies aren't core to its mission. The CIA is outsourcing its cloud to Amazon (probably) and the NSA has adopted other cloud tech for its use, versus developing it on their own. 

Rule 2: If a desired capability isn't part of the Agency core mission, outsource the need. 

Example: if industry can meet the high level requirements at a fraction of the cost that the gov can, why wouldn't you outsource it?

In the above Defense Travel Example: setting high requirements service level agreement around security, uptime, response time for buying tickets, help desk, website usage, etc. would be so much better than trying to specify a capability based on old outdated knowledge of tech and tooling. Also in this manner industry can more rapidly upgrade and evolve the capability to take advantage of new tech and ideas. Like adding mobile smartphones to travel, etc. 

Whats to be done with Helathcare.gov/ObamaCare portals?

Monday morning QB in full effect…

HHS should put out high level req's around the service it wants for citizens (its customers) to have: response times, help desk, etc. and then put a bounty or fee for any company who could meet it. Some companies might do it stickily over the phone out of Indian call centers, some might use tech. we just don't know since HHS preconfigured and (worse) assumed a solution. 

A fee for every person successfully signed up would be better than to blow $634 million in a few years with nothing, but a 404 to show for it (I got 404 problems, but a plan ain't one). $1k per person initially and then declining over time as the tech becoming cheaper, more reliable and more companies enter the marketplace. 

If the goal was to sign up as many people as possible for healthcare, why would you assume you know what the answer should be? Why not unleash American ingenuity and business on that problem to solve it in anyway possible. 

/rant on/

Frankly: Kathleen Sebelius (the head of HHS) should be held accountable and fired, weaseling out and saying we didn't know, etc. its the process that sucks is a cop out we should not expect as taxpayers. Nothing will change until someone takes responsibility for this, she's in charge otherwise we let mediocrity rule. 

Expect more from your Gov, it should suck less. 

/close