Exceeding the Benefits of Complexity? A Fractal Model for the Social Business Era

Over the weekend my friend and industry colleague JP Rangaswami wrote an insightful post that pondered how we have gone about delivering on customer experiences as connected to our back-end capabilities. Specifically, he explored an issue that is increasingly challenging many of the large-company CIOs I speak with these days: That the present rates of change demanded of the accumulation of 20-30 years of legacy business systems is greatly exceeding the ability of our enterprises and associated software “stacks” to deliver on them, particularly as cloud, social, and mobile dramatically transform computing today.

The problem lies in our classical views of enterprise architecture and business architecture both. But JP puts it more poetically:

Development backlogs are endemic, as the sheer complexity of the grown-like-Topsy stack slows the process of change and makes it considerably more expensive to change. The stack has begun to fossilise, just at the time when businesses are hungrier for growth, when the need to deliver customer-facing, often customer-touching, applications is an imperative.

Which makes me wonder. What Tainter wrote about societies, what Shirky wrote about companies, are we about to witness something analogous in the systems world? A collapse of a monolith, consumed by its own growth and complexity? As against the simpler, fractal approach of ecosystems?

This fractal aspect of user systems, Web 2.0, and SOA is one that I deeply explored in the 2005-2007 timeframe, and my ideas on this even made the cover story of the SOA/Web Services Journal at one point. This is when we began to see successful, composite systems sprouting up “in the wild” that were eminently natural and highly successful, such as Amazon’s, Flickr’s, and many others. Innovative new open API-based businesses had definitively emerged and shown us — in fact, virtually proved to anyone willing to observe — that business/technology ecosystems could be routinely produced that were far more successful than the ones virtually any traditional business had managed to produce for itself internally with methods like SOA. It was clear something important and new was happening and we tried to learn from it.

Traditional Closed Business Stack vs. Open Networked Supply Chain API

The signature lesson in this time period was that being “in the business” of ecosystems, at multiple levels, was the key to resilience, growth, and sustainability. It’s still a far cry from how most organizations are structured today, but I do see the first hints that this is starting to change. Companies like Best Buy and Sears have been getting the message among the old guard firms and looking at integration, partnership, and engagement as an open network activity.

The Web has also shown us that complexity, as important as it is to address and resolve the many inherent vagaries of dealing with the real world, is largely the enemy, whenever it exists needlessly. Our assumptions of where the complexity should be, in the transports for example, was wrong. It was in the ecosystem itself and we paid dearly for half a decade (and running) based on that misapprehension. But we’ve learned. SOAP usage has almost completely shifted to REST, complex WS-* stacks have largely moved on to simple standards and methods like Web-Oriented Architecture.

But technical discussions obscure the very important truths about what we’ve learned in the interim between the “aha” moment when we realized that what was happening so successfully “out there” was something that would make us just as successful inside our organizations, for internal and customer-facing needs both. After all, this holistic ‘integratedness’ — usually triggered via our connections to the Internet — was a part of our businesses increasingly and we simply had to realize this and engage as Internet natives. But back then, Andrew McAfee worried that focusing on the plumbing was the wrong emphasis to have on the exciting changes taking place, and he was largely right from a communication perspective.

However, building fractal architectures that addresses the old and the new isn’t as easy as just deciding to cast off the old, less effecive ways, as JP notes:

What I’d established in my own mind was a growing belief that the issue was to do with rates of change and costs of change. Vertical integration paid off when the rate of change was low. Networked small-pieces approaches paid off when the rate of change was high.

We are clearly living in times of much greater rates of change. Unfortunately, the old systems, the old architectures, the old business “stack” is often running the core of the business. It’s usually deeply vertically integrated, and not made of small networked pieces, loosely joined that are truly agile, deeply harness innovation on the edge, and turn IT into a profit center instead of a cost center.

So, while I have a lot more to say about this, because I believe that social business context will be at the root of the future of our how we design our products, services, processes, and workplaces, below are a few basic rules you can take away that if you stay true to, can indeed help you make the transition to the future. Note that customer and worker experiences are outside the scope of this, this has more to do with the new business stack below the social layer:

4 Ways to Create Sustainable Business Ecosystems

If resilient, networked, recomposable, open business capabilities are the future of business and IT architecture, getting there then requires a substantial change in stance:

  • Open Your Business Stack. To everyone, internally and externally, generally in the order of the systems and data that are most often requested for integration. Make it easy for anyone to onboard themselves and use the simplest and most egalitarian technology possible to deeply connect the world to your business. Self-onboarding is crucial because it enables the killer asymmetric business advantage of systems that respond to external engagement best: Export most of the change effort to outside your organization and impose it on those that wish to participate and integrate with you. Enterprises that close the “clue gap” with Big Data will have an enormous advantage.
  • Go Into the Business of Ecosystems. Stand behind your open supply chain: Invest in it for the long term, market it, evangelize it, support it, and stand behind it like your customers — again, internally or externally — will be building world changing products and services from them (if you do this, the track record is that they will.) Be fair and generous with your IP and enterprise data; you are under-utilizing it by several orders of magnitude until you do this, leaving much or even most of the potential of your business on the floor. Worse, until you treat it like a business, you won’t get external contributions to enrich your ecosystem that will drive down the cost of change while greatly increasing the velocity of new solutions and local adaptations.
  • Begin Fractal Reconciliation of Your Classical Systems. While hanging a trendy Web API off an ancient COBOL accounting system might be a step in the right direction, long-term success will mean going well behind the API. The existing business/IT landscape must be converged and rationalized so they themselves are made of “networked small pieces” that are faster to change as well as more resilient and responsive to being so much more connected (and therefore useful and valuable) to the world.
  • Spin Up New 21st Century Business Models. All new possibilities, many of which will be foreign to non-digital native firms, are reachable once you have created a fractal enterprise stack. Dominating classes of data, metered business knowledge, all new monetization methods, and more become possible once the three transformations above have started.

Unfortunately, after having advised and directly assisted many large companies in doing much of what I’m describing here over the years, it’s clear that most organizations are at a major disadvantage in making this transformation. The organizational will required is actually quite low, since most of this is fairly easy to do now with cloud technologies, increasingly capable technology partners, and the latest development stacks. Instead, it’s the understanding of how important it is to kill excessive complexity and focus on ways of thinking about business and technical architecture that will propel companies into bright futures.

We must leave behind vertical integration and radically collapse the size of our seams (APIs) in our businesses, while radically increasing the number of useful endpoints such that we look at them as the very fundament of our enterprises. The benchmark for success is how many others are depending on your ecosystem for their own success. Starting now, the activity in your organization must be focused on optimizing for growth of this number. That is the only way we’ll ensure we’re building businesses that are the most relevant, meaningful, and valuable to us going forward. The enterprises that don’t simply won’t survive in their present form. I believe that much is at stake here. Unfortunately, the “cultural gap” John Hagel wrote of back when this conversation started between the enterprise crowd and the Internet crowd is still much too large and is still hindering the move forward for many organizations.

I look forward to continuing this discussion and exploring just how much we’ve recently learned about the future of business and enterprise architecture.

Advertisements

A Timeless Way of Building Software

Most of my readers know that I’m a software architect by trade.  I’ve been creating software large and small for over twenty years.  And I’ve experienced movement after movement in software design from object-orientation in the 1980s and early 90s to component-based design, distributed objects, Web-based software, service-oriented architecture and too many others to even mention.  I’m pretty jaded at this point because I’ve learned, in general, the problems that you couldn’t solve in the previous generation of technique are often only marginally more solvable in the next generation (which is invariably invented to "fix" the previous problems.)

Alas, a genuinely better mousetrap is really hard to find.

So in the end, if you couldn’t do whatever it is you wanted to do with the previous generation of technique, it’s actually not that likely you’ll succeed in the next.  Certain software problems remain hard, and in general, it mysteriously happens to involve the juncture between technology and people in some way.  To paraphrase this, I could say that the software and techniques get better fairly constantly, but people remain the same.

And please, bear with me because I’m going to try out a real zinger on you in a minute. 

Because every once in a long while, something new and big actually does come along.  Or at least something that looks new and big.  One of the new and big things that came along about ten years ago was the concept of design patterns.  It was pretty neat stuff.  It said that despite the current technology we have, the processes that continue to evolve, there are certain timeless solutions to certain software design problems.  It was a revelation at the time.  And the writers of the book that explained this got both famous and very successful.  Why? Because these design patterns really worked is why.  And anyone who has read the books and has ever really built software recognizes these patterns.  And what was strange was that no one really expected it. One day, we just had them.  And the kicker was, they were always there, but now they were in our conscious thought and we had real names for them.  My point: They were in our face all the time but most of us couldn’t see them.

We are in a similar place with the Web right now.  We’ve done this Web stuff enough now that we are just beginning to see the design patterns.  What works, and why, in a specific situations, bounded by forces.  Some folks have had the hubris to give this next generation a name and to tease out these patterns.  Some people are just now going aha, and some people haven’t got it yet, and most of the rest of us either aren’t ready for it or just haven’t heard of it.  But, I will tell you this.  It’s quite real.  The best practices and design patterns of Web software are just starting to become understood.  The strange part is, we’re discovering the same things over again.  What’s old is new again.

Now, before you get all worked up or worse, I bore you and you stop reading, I will give you a nice list of the the forces behind these patterns.  If you recall, design patterns are a solution to a problem in context.  We are starting to get the context and even the outlines of the patterns of this "new" generation of software.  But we have a long way to go still.  The Web is a monstrously big space with big problems, and it’s not getting better.  There are one billion of us out here now.  Clearly understanding what it takes to create great software on the Web that is successful, useful, and vibrant will be an ongoing challenge for a long time.  But it will get easier because we are codifying our knowledge of this exciting and huge place where we now find ourselves.

Comparing SOA, Web 2.0, and a Timeless Way of Building Software
Figure 1:  The driving forces in modern software.
With a rough comparison between SOA
and The Timeless Way (Web 2.0 by any other name).


Now is where I’m going to hit you with a flight of fancy.  I’m going to use Christopher Alexander’s opening chapter of a Timeless Way of Building and tailor it to describe this old-but-new way of building the Web and software for it.  We are lacking for a little inspiration and this book in particular continues to sell upwards of 10,000 copies a year, 25 years after it was frst published.  And Christopher Alexander, for those of you who may not know, was the person that originally discovered the design pattern.  But it wasn’t for software.  It was for creating great, timeless buildings.  He was one of the first that realized that his field of endeavor has certain elemental, timeless cores, no matter the technique, building material, or the people.  It was an amazing discovery that poured over into the software world with considerable success. 

My assertion is that nothing has really changed in software, we might understand the forces better but they are almost always the same.  People want software that does what they want, is available when they need it.  They want software that grows with them, helps them, teaches them, and lets them do the same with others.  They want software that gets out of their way, disappears, and is more convenient by far than inconvenient.  And they want to pay as little as possible for it, but enough so that it’s worth it.  They are willing to have software get right into the middle of their lives.  If it’s the right software.  And as long as we’ve had software, they’ve always wanted this. But now they might actually start getting it.

In any case, I don’t literally believe every phrase in this take-off, but I do believe the overall concept deeply and profoundly as a software professional.  And I will continue to update the diagram above (clearly marked beta 1) until we have more of the forces in it. And some are definitely missing.  Please, as always, leave any comments and suggestions for improvement below.

And now, without further ado, here is the The Timeless Way of Building Software, with sincere apologies to Christopher Alexander:

The Timeless Way of Building Software
Inspiration For The Next Generation of Web Software


There is one timeless way of building software.  It is decades old and is the same today as it’s always been.  And because it is timeless, it will always remain this way.

The great software of our time has always been created by people who were close to this way.  It isn’t possible to create great software – software that is satisfying, and useful, and makes itself a natural extension of life – except by following this way.  And as you will see, this way will lead anyone who looks for it to elegant, vibrant software which is itself timeless in its form.

It is the process by which the function of a piece of software grows directly from the inner nature of people and naturally out of the raw bits, the otherwise meaningless digital medium, of which it is made.

It is a process which allows the life inside a person, or a group of people, or a community to flourish, openly, in freedom, so vividly that it gives rise, of its own accord, to the natural order which is needed to be contained within it.

It is so powerful and fundamental that with its help you can create software that is as beautiful and enriching as anything else you have ever seen.

Once you understand this way, you yourself will be able to create software that is alive, that is intertwined comfortably with your life and the lives of others. You will design worlds where you and others will want to work, play, and co-exist together; beautiful places where you can sit and dream comfortably.

This way is so powerful, that with its help hundreds or thousands, or even hundreds of thousands of people, can come together together to create software and community which is as alive and vibrant, peaceful and relaxed, as any living experience has ever been.

Without the central control of authorities and experts, if you are working in this timeless way, a genuine place will grow right from underneath your fingertips, as steady as the grass in the fields or the trees in your backyard.

And there is no other way in which a software which is fundamentally good can possibly be made.

That doesn’t mean that all ways of making software are identical.  Quite the contrary. It means that at the core of all successful software and at the core of all successful processes of creation and evolution, there is one fundamental invariant feature which is responsible for their success.  Although this way has taken on a thousand different forms at different times, in different places, still, there is an unavoidable, invariant core to all of them.

Take a look at the some of the great Web software like Google’s search page, Flickr or del.icio.us.  They all have that unique, yet unhurried, grace which comes from perfect ease and natural balance.  But what is it they have in common exactly?  They are beautiful, ordered, harmonious – yes, all of these things.  But especially, and what strikes to the heart, they live.

Each one of us yearns to be able to bring something to life like this. Or just be a part of it somehow.

It is a fundamental human instinct, as much a part of our desire as the desire to be part of something greater than ourselves.  It is, quite simply, the desire to make a part of nature, to complete a world which is already made of mountains, streams, stones, buildings, ourselves, our living systems, and our increasing connectedness together.

Each one of us has, somewhere in our heart, the dream to make a living world, a universe, and place of our own for us to share with others.

Those of us who have trained as software designers have this desire perhaps at the very center of our lives; that one day, somewhere, somehow, we shall build a software experience which is wonderful, beautiful, and breathtaking; a place where people can go and live their dreams.

In some form, every person has some version of this dream; whoever you are, you may have the dream of one day creating a most beautiful place, virtual or otherwise, where you can come together with others and freely share your knowledge, learn, participate in your community or government, and otherwise conduct your daily interaction with the rest of the world.

In some less clear fashion, anyone who is concerned with communities and other large group efforts has this same dream, perhaps for the entire world.

And there is a way that software can actually be brought to life like this.

There is a definable sequence of activities which are the heart of all acts of software design, and it is possible to specify, precisely, under way conditions these activities will generate software which is alive.  All this can be made so explicit that anyone can do it.

And just so, the process by which a group of independent people can make software become alive and create a place as real as any other can equally be made precise.  Again, there is a definable sequence of activities, more complex in this case, which are the heart of all collective processes of software creation.  And it is also possible to specify exactly when these processes will bring things to life.  And once again, these processes can be made so explicit, and so clear, that any group of people can make use of them.

This process is behind the design of community built software like Linux, Apache, Wikipedia, and many others.  It was behind the design of the great virtual places for people to live and work: the Internet, Usenet, and the World Wide Web.  It was behind the creation of simple, satisfying software of the kind that powers the iPod, the Blackberry, and Firefox; of SourceForge, Wikipedia, and BitTorrent.  In an unconscious form, this way has been behind almost all ways of creating software since the beginning.

But it has become possible to identify it, only now, by going to a level of analysis which is deep enough to show what is invariant in all of the different versions of this way.

This hinges on a form of representation which reveals all possible design processes, as versions of one most fundamental set of patterns.

First, we have a way of looking at the ultimate constituents of the environment: the ultimate "things" which a piece of software is made of.  As we shall see, every piece of software is made of certain fundamental entities known as design patterns; and once we understand software in terms of its patterns, we have a way of looking at them, which makes all software, all of their parts and function, all members of the same class of thing.

Second, we have a way of understanding the generative processes which give rise to these patterns: in short, the source from which the ultimate constituents of software come.  These patterns tend to come from certain combinatory processes, which are different in the specific patterns that they generate, but always similar in their overall structure, and in the way they work.  They are essentially like languages.  And again, in terms of these pattern languages, all the different way of building software, although different in detail, become similar in general outline.

At this level of analysis, we can compare many different software creation processes.

Then, once we see their differences clearly, it becomes possible to define the difference between those processes which make software vibrant, alive, and useful, and those which make them the opposite.

And it turns out that, invariant, behind all processes which allow us to make great software, there is a single common process.

This single idea is operational and precise.  It is not merely a vague idea, or a class of processes which we can understand: it is concrete enough and specific enough, so that it functions practically.  It gives us the power to make software and virtual communities live, as concrete as a match gives us the power to make flame.  It is a method of a discipline, which teaches us precisely what we have to do make our software what we want it to be.

But though this method is precise, it cannot be used mechanically.

The fact is, that even when we have seen deep into the processes by which it is possible to make software alive, in the end, it turns out this knowledge only brings us back to that part of ourselves which is forgotten.  Although the process is precise, and can be defined in exact scientific terms, finally it becomes valuable, not so much because it shows us things which we don’t know (though it may do that as well), but instead, because it shows us what we know already.

Of course, this way of building software has never be named.  It’s not service-oriented architecture, or the personal software process, or agile methodology, or the unified process, or CMM, or any of the others.  It’s the actual things that are conceived and done and worried about when software is created and used.  For now, because all software is quickly becoming connected to all other software, and because the Web is becoming the place where more and more of the relevant software is, and finally because it is a more complete reconception of what we thought we knew, we’ll give it a name temporarily.  An unsatisfying name, but one that we can remember for now.

We will call it Web 2.0.

What do you think?  Are we at a place where we can really identify the design patterns in Web 2.0?

Is Web 2.0 The Global SOA?

Are we heading towards an architectural singularity in the software industry? Sometimes it looks that way. If you do a superficial comparison at least, Web 2.0 is all about autonomous, distributed services, remixability, and is fraught with ownership and boundary/control issues. And yet, Service-Oriented Architecture (SOA) is all about, you guessed it, autonomous, distributed services, composite functionality, and is fraught with ownership and boundary/control issues. Sound similar, no?

It does seem that we have a classic case of fractal architecture on our hands. Is Web 2.0 actually the most massive instance possible of service-oriented architecture, realized on a worldwide scale and sprawling across the Web? The answer folks is, apparently so.

I’ve been thinking about this carefully for several weeks now as the similarities seemed to inexorably call to each other as I worked with each of them in turn (disclaimer: I’m a SOA architect by trade). Both Web 2.0 and SOA are already slippery, nebulous concepts yet there are unmistakable patterns within each that actually are very tightly related, though wrapped in slightly different cloth. Each encourages the liberation of the underlying functionality of software systems by providing open access to everyone that needs it. Both warmly embrace Web services and the aggregation of existing functionality into new solutions. And Web 2.0, according to O’Reilly, looks at Data as the Next Intel Inside, making large, back-end database driven functionality a core competency. SOA totally gets this as well. And both Web 2.0 and SOA provide the building blocks for creating people-centric processes starting at the scale of an organization and going up.



Granted, most SOAs are conceptually trapped inside an organization’s firewall or VPN. And Web 2.0 envisions the global Web as the stage writ large upon which to act out your grand visions of building collective intelligence and mashed up functionality. But scale is only one of the minor differences really, and not a genuine discriminator at all.

Do the linkages go deeper, to a more fundamental level? Are Web 2.0 and SOA different at their core and if not, how exactly do they relate?

I believe that there are at least two significant connection points. One is that Web 2.0 can be conceptualized as a global SOA. Two, that many traditional brick-and-mortar business that are currently using SOA as their architectural model will want to connect their Web/Web 2.0 faces up to their SOA. This makes Web 2.0 not just being the Global SOA but makes meeting smaller SOAs everywhere not just likely, but inevitable. Note that the respected industry analysis firm, Gartner, recently said that by 2008 that 80% of all software development will be based on SOA. And interestingly by 2006, Gartner believes that 60% of the $527 billion IT professional services industry will be based on exploiting Web services and technology. We’re talking serious convergence of focus here folks. If true, this means that more than half of all software development, SOA and otherwise, will revolve around the Web, inside or outside organization boundaries. All of this means Web 2.0 and SOA will meet each other both coming and going, and begin to become each other as well.

Web 2.0 = Global SOA? Why Should We Care?

But the real question really is does the relation of the two give us an advantage as we design and build Web 2.0 applications, services, and SOAs? One problem could be that many folks outside the IT industry just haven’t heard of SOA. And even then, there are vociferous arguments about what an SOA really is, just like there are endless debates about what Web 2.0 is. But in the end, there are best practices that need to cross pollinate here and SOA’s IT-bound sphere of influence isn’t really a factor. In fact, really only Web 2.0 designers (yes, you) will have to understand these techniques and their connections. Web 2.0 users themselves will generally be blissfully unaware of Web 2.0 as the global SOA.

Now, don’t get me wrong. Web 2.0 and SOA also have significantly divergent elements too. Web 2.0 emphasizes a social aspect that SOA is completely missing. And probably to its lasting detriment. SOA has much more central control, management, and governance while Web 2.0 is free wheeling, decentralized, grassroots, and with absolutely no command and control structure. Web 2.0 also talks about presentation, the front-end displayed to the user. SOA is largely silent on the issue of presentation though it certainly admits it exists. So SOA tends to be generic and faceless where Web 2.0 makes much of human/service interaction. They seem to need each other to be whole. Finally, Web 2.0 is almost too informal and practically calls out for discipline while SOA is mute and autistic in comparison, a technical virtuosity that wants to be social but doesn’t know how.

All of this makes me want to view one through the other to check basic principles. For example, SOA has best practices for building business processes into vast supply chains (so too does Web 2.0). SOA is also a mature view of software that eschews a technical view of information and data. And it identifies a motive force for these processes via something called orchestration. This is a concept that Web 2.0 does not have explicitly and could certainly use (an orchestration mash-up anyone?) though it is provided in some degree by its users. SOA principles also encourage creation of a common vocabulary across systems that is in the language of the domain (common definitions of customers, order, channel, product, for example). So it gets very close to addressing a big issue in software development today: That too many IT systems today tend to have technology myopia and ignore their most important elements… the people that use them and the way that they work. Web 2.0 gets this part even more right in all the significant ways. Web 2.0 embraces people, collaboration, architectures of participation, social mechanisms, folksonomies, real-time feedback, etc. All things that SOA, in its grey, dull, corporate clothes, does not, at least not explicitly. The complementary nature between the two seems clear.

So, I believe there are complementary synergies between these two powerful software approaches. One can very much be used to check and finish the other. SOA is both the "Mini-Me" of Web 2.0 (identical in almost every way but 1/8 its size) and a key archetype for it as well. Though admittedly one that lacks a few important ingredients. What is compelling, and I’ll talk about this in detail in future articles, is that Web 2.0 actually has powerful mechanisms that "complete" SOA (if you’ll allow one last Austin Powers reference.) Web 2.0 offers a face to SOA with numerous best practices for presentation, has emerging technical innovation like radical decentralization that is necessary for stability and scalability which SOA too often doesn’t address, and Web 2.0 identifies important techniques to immerse users into social processes that can make SOA data and services vastly more valuable.

Yes, so Web 2.0 is a global SOA, done right for the whole world. It’s big, it’s everywhere, and it’s here today.

Do you agree that Web 2.0 is the Global SOA? Post your thoughts below…

Update: Early stage VC investor Peter Rip had some interesting things to say about this article, including "Web 2.0 is a lighter weight version of SOA."
Update 2: Both Richard Monson-Haefel and ZDNet’s Joe McKendrick weighed in on this topic with good observations and commentary.
Update 3: This eventually turned into the SOA Web Services Journal cover story in Dec. 2005: Web 2.0: The Global SOA.  This in turn led to March’s SPARK event with Microsoft on the convergence of Web 2.0, SOA, and SaaS.
Update 4: This topic (and related issues for Web 2.0 in the enterprise) has turned into a regular blog on ZDNet.
Update 5 in May, 2006: Om Malik writes a detailed piece on the future of Web 2.0 and is most sanguine about it for the enterprise, interestingly enough, and links to this post.


Technorati:
web2.0, soa