Social Media Goes Mainstream

While some will dispute what mainstream is defined as exactly — with my own personal favorite being when my grandparents and their grandchildren both are doing whatever is under discussion — the rise of consumer-powered media platforms has all the

While some will dispute what mainstream is defined as exactly — with my own personal favorite being when my grandparents and their grandchildren both are doing whatever is under discussion — the rise of consumer-powered media platforms has all the hallmarks of being something that’s not only here to stay, but something that’s increasingly pushing everything else off the stage.  Yes, I’m talking about blogs, but also wikis and every other kind of two-way, user controlled participation tool that is currently proliferating on the Internet in every country and almost all demographics.

Now before I present my case for the mainstreaming of shared, collaborative media, we should more carefully define the term that captures this best: social media.  Wikipedia of course has the most easily accessible definition of social media, describing it as “online tools and platforms that people use to share opinions, insights, experiences, and perspectives with each other. Social media can take many different forms, including text, images, audio, and video. Popular social mediums include blogs, message boards, podcasts, wikis, and vlogs.”  The key here is that people are the ones that use and control these tools and platforms instead of organizations and large institutions.  Further, I would add to this that social media platforms tend to work best in networked environments , particularly on the Web, but also behind firewalls though to a lesser degree.  Why is the networked aspect so important?  Primarily because it’s a powerful democratizing force due to its pervasive, low cost nature; anyone can get in the conversation with only a small investment of their personal time and access to a network.  And since communication is essentially free over computer networks today, combining an architecture of participation powered by network effects makes social media platforms almost certainly the most powerful form of media yet created.

The Emergence and Rise of Mass Social Media in the Web 2.0 Era

These todays anyone posting anything on a simple blog lets them automatically reach the 1.1 billion users on the Web today.  And with syndication, social media content is picked up and spread throughout Internet via feed engines and the entire syndication ecosystem and can be found by anyone looking for information via Technorati, Google Blog Search, TechMeme or dozens of other innovative discovery mechanisms. At long last, hundreds of years after the invention of the printig press, anyone can truly reach a global audience by spending a couple of minutes of their time creating a blog on one of the hundreds of free blog sites.  I’ve highlighted in the past how social media has been used in both emergent and deliberate fashion to do everything from locating the survivors of natural disasters to motivating end-users en masse to create online video advertisements for a major corporation.

Of course, any effective technique or phenomenon has those who attempt to co-opt it or copy it, the latter which is the most sincerest form of flattery.  The recent Public Relations 2.0 flap, which ostensibly boiled down to whether or not traditional organizations can even conceive of how these new freeform platforms work, was a good example of how institutions firmly grounded in the 20th century struggle to understand the power shift under way.  Because these platforms are no longer under anyone’s control for the very reason that the Web is a system without an owner, except all of us together.

Bounding the Social Media phenomenon 

But how significant is this really?  What are the compelling datapoints that tell use that social media is changing the landscape of communication, collaboration, and personal interaction?  David Sifry’s quarterly State of the Blogosphere, most recently updated in October, is an excellent place to start. Taking a look at this, we can tracking over 57 million blogs, with over 900,000 blog posts a day on just about any conceivable subject.  3 million new non-spam blogs were created in just the most recent 3 months of tracking.  But blogs are primarily text and there are many other forms of social media and so it’s worth looking at podcasting and video, two important types of social media that are growing rapidly with the spread of high quality, fast Internet connections.  Fortunately or unfortunately, unlike blogs, podcasts or video sharing do not have their own syndication system and for the most part they just ride inside the existing RSS/ATOM feed systems.  This makes it hard to discern what is really happening and so we can only pull on some individual data points such as Google Trends data showing the rapid rise of podcasting as a search term.

The video side of social media is a bit easier, which Hitwise and YouTube providing enough hard data on the most recent version of the YouTube Fact Sheet to get a general though unscientific impression of what’s happening there.  According to this, YouTube has 60% of all online video viewers with up to 70 million viewers in an evening and over 65,000 videos uploaded every day, making it both the #1 online video site and #1 social video sharing site online.  This implies that most video consumption on the Web is already based on social media, and that there are over 115 million online viewers of video overall.  At least for video, social media is not an edge case and is they dominant model overall. Note: Yes, one can quibble about whether YouTube is truly a social media site and certainly it skirts the concept but in my book it makes the list.

Why is YouTube considered Social Media though?  What aspects does it — any many of the most successful media sites — have that make it social and non-coincidentally so popular?  To understand this best, it’s worth creating a list of what exactly must an aspiring social media platform actually have in order to be considered such.  Here is my take, culling the capabilities and features of the most popular social media sites as well as the consensus of leading thinkiners in this space such as Stowe Boyd, Tina Sharkey, and others.

Defining Social Media: Some Ground Rules
(as we understand them circa January 2007)

  1. Communication in the form of conversation, not monologue.  This implies that social media must facilitate two-way discussion, discourse, and debate with little or no moderation or censorship.  In other words, the increasingly ubiquitious comments section of your local blog or media sharing site is NOT optional and must be open to everyone.
  2. Participants in social media are people, not organizations.  Third-person voice is discouraged and the source of ideas and participation is clearly identified and associated with the individuals that contributed them.  Anonymity is also discouraged but permissible in some very limited situations.
  3. Honesty and transparency are core values.  Spin and attempting to control, manipulate, or even spam the conversation are thoroughly discouraged.  Social media is an often painfully candid forum and traditional organizations — which aren’t part of the conversation other than through their people — will often have a hard time adjusting to this.
  4. It’s all about pull, not push.  Like John Hagel and John Seely Brown observed in the McKinsey Quarterly a year ago or so, push-based systems, of which one-way marketing and advertising and command-and-control management are typical examples are nowhere near as efficient as pull systems.  Pull systems let people bring to them the content and relationships that they want, instead of having it forced upon them by an external entity.  Far from being a management theory, much of what we see in Web 2.0 shows the power of pull-based systems with extremely large audiences.  As you shape a social media community, understanding how to make embrace pull instead of push is one of the core techniques.  In social media, people are in control of their conversations, not the pushers.
  5. Distribution instead of centralization.  One often overlooked aspect of social media is the fact that the interlocutors are so many and varied.  Gone are the biases that inevitably creep into information when only a few organizations control the creation and distribution of information.  Social media is highly distributed and made up of tens of millions of voices making it far more textured, rich, and heterogeneous than old media could ever be (or want to be).  Encouraging conversations on the vast edges of our networks, rather than in the middle, is what this point is all about.

The rise of social media platforms within businesses, often dubbed Enterprise 2.0 , will place a significant challenge on organizations as they try to grapple with the ground rules above.  That’s because not following them will tend to reduce the long-term success and effectiveness of social media in business.  Also, increasingly, as more and more time and world-wide attention is given to social media, who really owns the discussions online will become a bigger and bigger deal.  YouTube recently announced they will begin paying their users for their video contributions (which are the seeds for often virulent conversation on that site), but they still place far too many restrictions on the content that is uploaded including making it belong to YouTube.

Both of these trends show that when users are in control via the highly democratizing tools of the Web, the fundamental ground rules change.  Understand them, follow them, and embrace them, this is the pre-eminent media model for the 21st century.

These aren’t the only rules for social software however, just social media in particular.  Be sure to check out my Notes on Making Good Social Software for more good ideas.

What else did I miss? What makes social media uniquely what it is?

Product Development 2.0

While the window on using the “2.0” suffix is probably closing, I thought it would be worthwhile to explore an especially significant trend in 2006 that will likely see much more widespread uptake in 2007.  Specifically, I’m talking about building highly competitive online products by turning over non-essential control to users directly via the Web.  For now, I’m calling this online business trend “Product Development 2.0”, a concept that embodies the use of Web 2.0 concepts such as harnessing collective intelligence, users as co-creators, and turning applications into platforms, three of the most powerful techniques in the Web 2.0 arsenal.

What is Product Development 2.0 exactly?  It’s an informal term I’m applying to something that online startups and traditional businesses both are increasingly doing: leveraging of mass user contributions, providing open architectures for others to build on as they like, and even handing control over key product decisions directly to users.  The reasoning behind doing this is simple:  Satisfied customers have always been essential to having the most successful business, both online and offline.  But how best can you ensure that they get exactly what they want from you, as customized and quickly as possible?  This is where the scale, new tools, and business models of Web 2.0 have stepped in, giving us the potential to provide our customers with better, rich products, much more quickly, and with more of what they want.  Taken as a whole, it’s increasingly clear that there are new business models afoot that are just now being well understood.

Product Development 2.0: Apply Web 2.0 to Product Creation and Development

Given that any business typically is vastly outnumbered by its customers and potential customers, and that putting a bureaucratic, centralized product development team into the critical path of product creation and ongoing maintenance highlights how little we can actually serve them, especially in an individualized way. And with everyone online, it’s increasingly obvious where the biggest source of talent, engagement, innovation, agility, and worker bandwidth really lies: with your customers.  Using the techniques and technologies that have emerged in just the last few years, you can now finally give them the tools and motivation to tweak, tune, refine, and contribute to your products and services.  And increasingly, they’ll probably do it.  YouTube is still currently one of the best examples of user co-development of a world-class product in its pure form (65,000+ videos uploaded by users per day), but sites like eBay, Slashdot, and many others have been leveraging their users in product development for a long time now.  And as it turns out, Product Development 2.0 is not a small topic and starts off at collecting explicit user contributions, leveraging the Database of Intentions, and putting in automated real-time feedback loops to identify the best or most popular new content or capabilities for other users that come along later.

It’s important to note that it’s a fundamental shift for a business to turn over a large part of its product development to its users, becoming more of a mediator and facilitator than a product creator or owner.  This is the shift of control from institutions to individuals that the apparently relentlessly democratizing force of the Web has begun exerting on the business models of organizations of every description around the world.  As more organizations figure out how to apply Product Development 2.0 to their individual offerings, they will reap significant competitive advantage over those not harnessing the Web to directly connect to customers and begin a rapid and never-ending innovation cycle.  This is another aspect of the perpetual beta concept that reflects the fact that increasingly, products and services online are never finished, and indeed, can’t ever be finished as changes and additions seamlessly pour in over thousands of millions of Internet connections.

But enough about the possibilities.  Let’s talk some examples, both in terms of what older style product development did vs. what this new style is doing.  Finally, let’s talk about some companies actually doing this successfully.  Note: Incidentally, though I normally write about services in terms of Software as a Service (SaaS) or Web Services, for the purposes of this discussion I’m talking about non-physical business processes for sale, such as car or medical insurance, tax preparation, etc. and not software.

Like the recently discussed Programming 2.0 concept — a set of software development tools, techniques, and attitudes that is, not incidentally, enabling much of this — and the original Web 2.0 definition, it is examples in lieu of principles that’s one of the best ways to paint a picture of what appears to be happening in the evolution of product development:

The Move to Product Development 2.0

  Product Development 1.0  Product Development 2.0 
Primary Customer Interaction Channel:  Telephone, Mail, Face-to-Face, One Way Media (Print, TV, Radio, etc.), e-mail
World Wide Web, e-mail, IM
Source of Innovation: Organizations Customers
Innovation Cycle: Months, Years
Minutes, Hours, Days, Weeks
Content Creators:
Internal Producers  External Producers
Feedback Mechanisms: Market research, satisfaction surveys, complaints, focus groups Analytics, online requests, user contributed changes
Customer Engagement Style: Controlled, well-defined process Spontaneous and chaotic
Product Development Process: Upfront design
Less upfront, much more emergent
Product Architecture: Closed, not designed for easy extension or reuse by others; walled garden
Open, very easy to extend, refine, change and add on to, ecosystem friendly, designed (and legal) for widespread remixing and mashups
Product Development Culture: Hierarchical, centralized, Not Invented Here, somewhat collaborative, expert-driven Egalitarian, decentralized, remix instead of reinvent, highly collaborative, Wisdom of Crowds
Product Testing: Internal, dedicated test groups, hand-picked select customers Users as testers
Customer Support: Customer Service
User Community
Product Promotion: One-Way Marketing and Advertising
Viral propagation, explicit leveraging of network effects, word of mouth, user generated and other two-way advertising
Business Model: Product Sales, Customer Service and Support Fees, Service Access Charges, Servicing High Demand Products
Advertising, Subscriptions, Product Sales, Servicing All Product Niches (The Long Tail), Unintended Uses
Customer Relationship:
External Buyer (Consumer) Partner and — increasingly remunerated — Supplier (Consumers as Producers )
Product Ownership: Institution, particularly executive management and shareholders Entire User Community
Partnering Process: Formal, explicit, infrequent, mediated Ad hoc, thousands of partners online, disintermediated
Product Development and Integration Tools: Heavyweight, formal, complex, expensive, time-consuming, enterprise-oriented
Lightweight, informal, simple, free, fast, consumer-oriented
Competitive Advantage: Superior products, legal barriers to entry (IP protections), brand name advantage, price, popularity, distribution channel agreements #1 or #2 market leader, leveraging crowdsourcing effectively, mass customization, control over hard-to-create data, end-user sense of ownership, popularity, cost-effective customer self-service, audience size, best-of-breed architectures of participation

It’s worth noting a couple of key points about the table above.  One is that the Web makes the shift of control possible by putting every business in direct contact with every one of its customers.  No small system can remain unchanged by sustained contact with a much larger system, and this means that any business (which is the small system in this scenario) which embraces its customers over the Web in a two-way fashion will likely undergo a move fairly quickly from the first column to the second.  The fact is, if you have loyal customers who like the products and services that you offer online, you’re going to have a hard time avoiding the shift of control and opening up of your product designs and architecture.

The second is that those that play to the strengths of the Web as platform, instead of trying to fight it, can exploit the most powerful software platform, or indeed, platform of any kind, that has been created to date.  Triggering network effects, building an extensible platform out of our product offerings (whether it’s an online software application or if you’re an insurance company, doesn’t matter), and you can see the advantage to be had in the assyemtric model of business on the Web; all of the potential is on the edge of our networks now (where the users are) instead of the middle.  And waiting too long to enter the Product Development 2.0 arena potentially means waiting for your competitors to get their ahead of you.  And the longer you wait to get the clock started on collected the Database of Intentions (continuously turning 100% of all customer interaction into enriching your product dynamically), the more likely you will face competitive dislocation and even lock-out.  Amazon is famous for collecting user contributions to enrich their product database and they are about a decade ahead of potential competitors of in terms of the enriched, hard-to-recreate database they have built.

Now on to a few examples to highlight what companies are actually doing that has many of the elements of Product Development 2.0.  First, the usual preamble about checklists of features; just like Web 2.0, one doesn’t have to implement every one of these in order to deliver better results, just the ones that apply in your situation.  So let’s look at a couple of stories of companies — and I have many others I’ll be sharing as soon as I can — that are going part of the way down the Product Development 2.0 path and getting valuable early experience.  I selected real-world companies since that’s the majority of companies that have to figure out whether they’re going to play in this space or let others do it for them.

Product Development 2.0 Examples 

XM RadioXM Radio is a satellite radio provider that has recently embraced some of the tenets of Product Development 2.0.  Compellingly, the Top 20 on 20 channel is one of the most popular channels XM has yet created.  Why? Because control of it has been entirely handed over to its users.  Says the Wikipedia entry on Top 20 on 20: “The channel plays everything new from rock to rap, with the songs chosen by online votes to the XM website. One can also vote their favorite songs by calling the station number, or text messaging. The channel is completely automated by listener voting with no DJ interruption. [DH- My emphasis] Top 20 on 20 is also one of the most popular music channels on XM. According to XM’s internal research, the channel achieves 1.8 million listeners a week.”  And though the channel was relaunched with some changes in December that have proven unpopular to many (less music, live DJs), it presents the cautionary tale of what happens when you assert bureaucratic authority over something that you’ve co-developed with your users; the possibility that you’ll kill the goose that lays the golden eggs of user contribution and engagement.

General MotorsGeneral Motors conducted its highly innovative Chevy Apprentice campaign early last year and made quite a demonstration of convincing users by the thousands to generate online video commercials for its new Chevy Tahoe SUV. By opening up the contest to anyone on the Web and only screening submissions for truly objectionable content they were able to elicit a stunning 22,000 user generated commercials exhibiting an impressive variety of creativity with both positive and negative messages.  From the beginning of the effort, they realized that in a freeform environment created by Web 2.0 tools, that they would only be able to respond to criticism and not control the message.  As expected, environmentalists famously picked up the tools to create ads savaging SUVs in general but GM’s Ed Peper understood that only by engaging in conversation instead of censoring dissent could they gain trust and get more information into people’s hands than they could otherwise.  Ultimately, GM created its own ads that highlighted the high amount of recycled parts and the best fuel-efficiency in its class of the Tahoe.  A brave piece of Product Development 2.0 for sure and one that many traditional business followers probably viewed incredulously as GM truly let their customers and potential customers co-create their advertising campaign with them on the world stage.  For the curious: You can see the many Chevy Apprentice commercials still up on YouTube.

The Potential for Disruption and Opportunity

The Web is a fundamentally different platform from any platform we’ve seen before. Unlike previous general-purpose platforms, the Web is fundamentally communications-oriented instead of computing-oriented.  Sure, computing still happens but what the Web does that’s so important is its ability to connect information and people together.  The hyperlink is the intrinsic unit of thought on the Web .  So, it’s information connected by links instead of programs that operate on data, that’s the basic difference.  But why does this hold the potential to put traditional product development on its head and usher in Product Development 2.0?  1) Because the aforementioned information can now truly be generated by anyone.  And 2) because we’re all nearly universally connected to this new medium by the devices on our desktops, in our briefcases, and in our pockets.  All of us can now be directly and continuously connected to the products and services which we need, which increasingly, is the rest of us and not a handful of large companies.  The very best companies in the future are likely ones that will create innovative new ways to facilitate innovation and collaboration by the hundreds of millions of us that can be reached and embraced by effective architectures of participation.  The big winners will enable us and encourage us to take control, contribute, shape, and direct the designs of the products and services that we in turn consume. 

The good news: Only a few industry leaders and early adopters fully appreciate the significance of these trends as yet or even how to fully exploit and monetize them.  There’s still enormous opportunity, and for existing businesses with large investments in existing business models, blowing your business model up before someone else does will be the order of the day.  This will prove though very hard for most to do successfully.  And therein lies the potential for significant industry disruption in the next 5 years as new players with core competency in Product Development 2.0 push older, slow-to-adapt businesses off the stage. 

While this is far-fetched for some, effectively embracing the Web is key to business success today.  Why do you think this will or won’t be the ultimate future of how we do business?

All We Got Was Web 1.0, When Tim Berners-Lee Actually Gave Us Web 2.0

The blogosphere flew into its usual uproar a few days ago when the inventor of the World Wide Web himself, the venerated Tim Berners-Lee, was recently recorded in a podcast calling Web 2.0 nothing more than a piece of jargon.  There is little love and plenty of misunderstanding for this term in many quarters of the industry, despite the fact it has been painstakingly described by those that identified it to the world.  For all the folks tired of hearing about Web 2.0 and very often not knowing what it means, there nevertheless remains the underlying reason for coining it: clearly apparent, widespread new trends in the way the Web is being used.

Tim Berners-Lee Gave us Web 2.0 and All We Got Was Web 1.0

Of all the analysis I’ve read of the Berners-Lee podcast (and there’s a bunch, read Dana Gardner, John Furrier, even Dead 2.0), it’s Jeremy Geelan who has captured the real insight here with his post, "The Perfect Storm of Web 2.0 Disruption", where he brilliantly explains what is probably the key to the real significance of the Web 2.0 phenomenon as a portentous crossroads between the old and the new:

Web 2.0 is an example of what the historian Daniel Boorstin would have called "the Fertile Verge" – "a place of encounter between something and something else." Boorstin (and here I am wholly indebted to Virginia Postrel) pinpointed such "verges" as being nothing short of the secret to American creativity.

Postrel sums up what Boorstin was saying as follows:

"A verge is not a sharp border but a frontier region: where the forest meets the prairie or the mountains meet the flatlands, where ecosystems or ideas mingle. Verges between land and sea, between civilization and wilderness, between black and white, between immigrants and natives…between state and national governments, between city and countryside – all mark the American experience."

— Jeremy Geelan

Web 2.0 Is Much More About A Change In People and Society Than Technology

But is Web 2.0 really about the Web, or us? The rise of architectures of participation, which make it easy for users to contribute content, share it —  and then let other users easily discover and enrich it, is central to Web 2.0 sites like MySpace, YouTube, Digg, and Flickr. But this is still just another aspect in the way that we, ourselves, have changed the way we use the Web.  Not only have we gained 950 million new Internet users in the last ten years, but a great many of them use the Internet differently now too, with a hundred million of them or more directly shaping the Web by building their own places on the Web with blogs and "spaces", or by contributing content of virtually infinite variety.

Let’s not forget that there were important issues that really held back the early Web and prevented the widespread flourishing of the collaboration and connecting of people that Tim Berners-Lee originally intended.  This included privacy concerns, almost entirely one-way Web sites, lack of skills using the Internet, and even slow connections.  But these have now continued to drop away rapidly in recent years, with many younger people in particular not hindered by these issues at all (rightly or wrongly.)

And for sure, let’s not forget that the Web has changed over the years.  There have been countless technological refinements and even improvements to the physics of the Internet itself. These range from the adoption of broadband, improved browsers, and Ajax, to the rise of Flash application platforms and the mass development of widgetization such as Flickr and YouTube badges.  But the trend to watch is the change in the behavior of people on the Internet.  Because much of this Web 2.0 phenomenon comes from mass innovation flowing in from the edge of our networks; that’s millions of people blogging, hundreds of thousands more producing video and audio, hundreds of Web 2.0 startups creating hugely addictive social experiences, sites that aggregate all the contributed content that one billion Internet users can create and more.

Yes, the original vision of Berners-Lee is now apparently happening, so he’s right in a sense there while glossing over the reality of the early Web.  But though his vision was largely possible since the advent of the first forms-capable browser, at first we only got what we could call "Web 1.0"; simple Web sites that were largely read-only or at least would only take your credit card. The essential draw of mountains of valuable user generated content just wasn’t there.  And the millions of people with the skills and attitudes weren’t there either.  Even the techniques for making good emergent, self-organizing communities and two-way software were in their very infancy or were misunderstood.  An example: How long did it take the lowly editable Web page (aka wikis) to be popular and widespread?  Nearly a decade.  The fact is, most of us know that innovation is all too likely to race ahead of where society is.  I run into folks from Web 1.0 startups fairly often that bitterly complain about how they were building Web 2.0 software in 2000, but nobody came.

What Exactly Is Special About The Web 2.0 Era?

I write frequently that we as an industry rediscover over and over again the same classic design issues right at the juncture of people and software, just repackaged enough so we don’t recognize them until it’s too late.  This time around the sheer numbers and scale of the Internet have distorted our traditional, more parochial views of what we thought networked software was and online communities were.  One outcome is the illusion that we had large degree of control over what happens when large groups of networked people can join together collaborate and innovate.  We don’t.  It’s like a large door has been opened behind us and everyone is now just getting a sense of that it’s there and where it leads.

But what exact is new here?  I mentioned a few things, but a more complete list is better:

  • There are over a billion Internet users now. Network effects can quickly climb in even small corners of the Internet, since small can now mean just a hundred thousand users.
  • Many of these users have become profoundly Web-fluent.  Robert Scoble observed recently that many users are still casual and non-expert, but almost all can search, they can post, they can edit a Wiki, and a lot of of them are now comfortable cutting and pasting Javascript snippets, and frequently much more.  They are in control of the vehicle now.
  • Powerful practices in Web site design are becoming widely known.  The best and often most successful sites are finding out that carefully designing what Tim O’Reilly calls harnessing collective intelligence deep into the design of their sites can provide truly amazing results.  And this aspect of how we use the Web is actually far behind the first two trends.  It’s still surprising to me that many people cite Ajax as the exemplar of Web 2.0 and not building networked applications that leverage user contributions and trigger network effects.  There’s quite a bit of headroom here in fact, and I expect to continue to see compelling advances in Web 2.0 software design.

  • Thus, power and control is shifting to the new creators.  As the users of the Web produce the vast majority of content (and soon, even software), they are therefore in control of it.  This shift of control has enormous long-term consequences since the Internet tends to route right around whatever central controls try to be applied.  The implications for traditional organizations are fascinating and will only increase as the MySpace generation heads into the workplace in large numbers.

There are more secondary trends related to Web 2.0 but the first three are the key, without all three, I would assert we would not be seeing some of the truly amazing things out on the Web that we see today.  Is all of this "frothy", as Robert Scoble recently claimed.  Not in the slightest.  Are people excited about it?  Yes, and they should be.  And while I don’t find the term itself to be particularly important — it’s the ideas behind it that are so interesting — the fact that so many people feel so strongy about the term Web 2.0 tells us that it’s something we should understand better.

BTW, in that last link Scoble was talking about The New New Internet, a Washington DC-based Web 2.0 conference I’m involved in.  I can assure you it’ll be as far from content free as you can get and I do hope to see you there.

Web 2.0’s Real Secret Sauce: Network Effects

A lot of the early descriptions of and speculation around Web 2.0 last year were either chock full of examples or long lists of interesting new phenomena that seemed to be emerging on the Web.  My list of Web 2.0 explanations last year is a good survey of these.  This year however we seem to be zeroing in on the core phenomena underlying the sea changes of the Web and even in society itself.  Sure, some fundamental upgrades to the Web’s infrastructure – including steady and significant improvements to the physics of the Web – have also helped things along (i.e. those great Ajax/RIA applications we love so much like Gmail and Gliffy, download in seconds now.)  But it’s the widespread use of the two-way Web that’s really the harbringer of an increasing sense of disruption.

Watch a TV panel that I was on with Google’s Adam Bosworth on how the improvements to the physics of the Internet have enabled Google’s newer products, particularly around SaaS and Ajax.

I wrote recently about the trend of Web 2.0 reductionism that is helping us get to Web 2.0 fundamentals and assisting us in understanding why it’s such a game changer.  The upshot is that the Web is rapidly evolving and is increasingly being shaped by its users.  If you’re not entirely sure that this is really a big deal though, you have only to look at the example of MySpace.  MySpace is as pure a Web 2.0 play as you’re likely to find and they’ve exploited user generated content and network effects to become the #1 visited site on the Web (as of last week), from out of practically nowhere two years ago.  This is an extremely impressive achievement and shows the power of Web 2.0 techniques to quickly best even the very largest and most established industry leaders, including Google and Yahoo!  This demonstrates that the successful exploitation of what has been labelled "Web 2.0" can be an extremely disruptive force.  People are sitting up and taking notice.

Inducing Network Effects with Web 2.0

If only the keys to doing this were in everyone’s hands, the thinking goes, then disruption could be and would be widespread, even rampant, across business and society.   And certainly, a lot of folks are very interested in taking advantage these techniques (such as maximizing your users’ social surface areas) both out on the Web, and within their own organizations.  This further explains the interest by venture capital investors, startups, and enterprises to get into the game before others use these techniques first to upend the traditional industry leaders in their areas of interest.  Mark my words, this year is only the beginning and those in less regulated industries that aren’t constrained from using Web-based network effects effectively are going to have a very tough time of it.  This will prove to be true particularly in the next 24 months as everyone scrambles to try to figure out where this is all headed.

Fantastically and ironically, the keys to doing this actually are in everyone’s hands today, right now.  Web 2.0 is a truly egalitarian force and though certain core competencies are required (being able to keep up with your own growth being one of them), just about anyone with a innovative idea and a good formulation of the techniques can access and tap into the existing audience of over 1 billion Web users.

Triggering Network Effects: As Simple As Establishing an Effective Viral Feedback Loop?  Surprisingly, yes.

The Web is the fabric upon which an ever increasing amount of our lives is woven into.  This is now most media including newspapers, TV, radio, entertainment, music, arts, etc. as well as what I call lifestyle logistics; e-mailing, IMs, calendaring, travel planning, time/task management, and more.  They are all moving or have
already moved to the Web.  This very habit most of us have of being on the Web so much of the time, along with the easy lure of the hyperlink, which can redirect anyone via any of these ‘channels’ into a new Web 2.0 experience or site.  Thus, if someone loves your new site, they send their friends the link, they send their friends, and so on.  Instant pile-ons involving tens or hundreds of thousands of users overnight are now common.  And good viral feedback loops keep them there and keep them coming back, and bringing their friends with them.

Of course, in a few years the exact design patterns for triggering a new MySpace, Facebook, or similar social juggernaut will become common.  Then most likely balance will be reachieved in the industry and there will be less disruption.  But for now the secret balance of Web 2.0 techniques that powers growth through efficient access to network effects is still an art (read my Notes on Making Good Social Software for an idea of how these design patterns might look however).  The bottom line: the upside and downside potential of Web 2.0 is truly significant.  And it means that in most industries doing nothing is really no longer an option.

But let’s talk about network effects for a minute.  What are they?  I’ve written before that the core description of a network effect  is when a good or service has more value the more that other people have it too.  Easy so far right?  Examples include e-mail, IMing, the blogosphere, and even the Web itself.  But what’s not clear from this description is the raw power that is caught up in and represented by network effects.  Most rigorous studies and mathematical formulations reveal that there is tremendous geometric power in network effects.  Though Robert Metcalfe originally coined Metcalfe’s Law to describe the raw potential of network effects in computer networks, most recent formulations have attempted to capture the exact value of them more precisely.  Most notable has been David Reed with his relatively well-thought out and profound  Reed’s Law, as well as Odlyzko and Tilly with their work.  However, whichever formulation you believe is right (and Reed’s Law, if true, has staggering implications in this regard), the result is clear: At even an early point, the cumulative value of a large number of connected users goes exponentially off the charts.

The Potency of Network Effects and Thus Web 2.0

The end result: If you have a million people visiting your Web site but you’re not leveraging network effects with them (such as by letting them contribute and letting others see and respond to those contributions), then you’re probably squandering the greater part of the value of that million person audience.  Along comes someone else who does exploit the network effects of their users.  They can quickly leapfrog you if they figure out a good way to establish and harness the connections between those same users, again in some kind of viral feedback loop.  The new site will have the combined aggregated output of those users as part of its value proposition to new and existing users and it’s clear which will win.  Examples: Digg bested Slashdot in just such a way.  YouTube is set to do something very similar to network and cable television.  Google is staging itself to best the #1 software company in the world.  There are many others.

So what do you do?  Most of us will soon face situations where the person or organization we’re competing against will be trying to use these same techniques to gain advantage, marketshare, or whatever.  Agile development processes have taught us in recent years that that the tightest closed feedback loops resulted in better products, faster.  The Web is now allowing feedback loops of this kind between staggering numbers of non-centrally coordinated people from a ready -to-tap worldwide audience.  Emergent content, communities, and unintended results are all side effects of systems (and yes, your enterprise) with rampant network effects.  Learning how to channel these feedback loops and the resulting network effects constructively will become a significant competitive advantage and a required core competency across the majority of professional disciplines in the very near future.

Read David Berlind’s excellent overview of Eventful and how they are using Web 2.0 techniques to be successful

But is this really happening?  Is the competitive landscape about to be torn up?  Are companies starting to respond?  It does appear so.  For one thing, I’m increasingly seeing traditional organizations trying to do things like "MySpacing" their current product offerings.  Everything from Starwood Resorts to major national interest groups are looking at creating business communities that are really viral social networks powered by their users.  It is a fascinating time to be building or reinventing a business. And if you haven’t started, now is probably the time to begin the effort.  And worry not if you can’t figure it all out right now, the entire world is still trying to figure out how it applies to them too.

How will you apply Web 2.0 to your life, business, or organization?

RSS is the Web 2.0 “Pipe”

Problematically, RSS is still not quite a household word yet, and even the software industry is just beginning to realize the importance of this workhorse syndication format. Though at this point it’s already clear RSS will be the key enabler of Web 2.0 and Software As a Service. It will do this both as a notification system and as the actual glue that will eventually hold many Web 2.0 services and mash-ups together.

Dave Winer famously created the current incarnation of RSS but its implications are still rippling through the industry. The folks that can fully appreciate RSS will reap corresponding rewards. Microsoft CTO’s Ray Ozzie is a good example of the folks that "get" the significance of RSS. I love the quote from his much-discussed leaked memo this week and I haven’t been able to stop using it for the last day: "[RSS] is filling a role as the UNIX pipe of the internet as people use it to connect data and systems in unanticipated ways."

And we can’t forget that RSS feeds, storage, synchronization will be a central new feature of the next version of Windows.

So expect to see RSS on every blog, every Web 2.0 service, web site, and piece of desktop software going forward. If you can’t find the feed, you can be sure whatever it is won’t last long.

And for the fully buzzword compliant and for the record, I do fundamentally believe REST/RSS is the new EAI. And the glue of first choice for lightweight SOA as well. And I’m actively starting to see some folks drop their traditional Web services and go RSS wholesale.

Of course, the real problem right now is that most people on the Web still don’t have any idea what RSS is. At best, the average Web user might understand that RSS forms some kind of information "feed". More sophisticated users notice that if they can find an RSS link somewhere (a blog or news site for example) that they can use it like a URL to get updates of information within services like My Yahoo, Bloglines, or something called an RSS reader.

Murkiness and partial examples are the enemy here. Raising awareness and finding clear examples that fully express the potential and power of RSS should be the goal.

Here are some clear, canonical examples that I think convey the full scope of what RSS does for us in a Web 2.0 world:

RSS Use Cases

Notification: Need to inform a lot of people about changes to information? Don’t want central control? Want to enable self-service? Use RSS.

Syndication: Publishing new information regularly? Put it into an RSS feed. This flows out to the world your blog entries, news articles, podcasts, videos, job posts, weather reports, financial updates, bug reports, etc. The software you use should be able to take your information and make it into an RSS feed. If your current software can’t, find new software. It’s that important.

Glue: Need to connect any service to another service on the Web (or anywhere else)? Trying to mash together data? Building supply chains? There is generally no need to ever ask anyone to stand up a new web service. Pull everything you need via its RSS feed. Some software developers will disagree with this and say there are better methods, but to this I point out: 1) RSS is robust enough that it’s all you’ll ever need nine times out of ten and 2) it’s what you’re going to offered automatically anyway, take it and get something done.

RSS creates the Web 2.0 information ecosystem by enabling interconnectedness, network effects, emergent behavior, and much more as well. And RSS doesn’t demand control of the other end of the conversation. This is a big enabler all by itself and is a classic Web 2.0 force. By letting consumers of RSS use any tool or service they want on their side, barriers are eliminated and connectivity is encouraged.

That doesn’t mean that RSS doesn’t have its weaknesses either and certainly there are other ways to create feeds, but RSS has the mindshare, support, and the goods right now. So though it’s not perfect, it’s more than up to the job. Let’s spread the word…

What did I miss? What other canonical examples are there?

Technorati: web2.0, rss

Is Web 2.0 The Global SOA?

Are we heading towards an architectural singularity in the software industry? Sometimes it looks that way. If you do a superficial comparison at least, Web 2.0 is all about autonomous, distributed services, remixability, and is fraught with ownership and boundary/control issues. And yet, Service-Oriented Architecture (SOA) is all about, you guessed it, autonomous, distributed services, composite functionality, and is fraught with ownership and boundary/control issues. Sound similar, no?

It does seem that we have a classic case of fractal architecture on our hands. Is Web 2.0 actually the most massive instance possible of service-oriented architecture, realized on a worldwide scale and sprawling across the Web? The answer folks is, apparently so.

I’ve been thinking about this carefully for several weeks now as the similarities seemed to inexorably call to each other as I worked with each of them in turn (disclaimer: I’m a SOA architect by trade). Both Web 2.0 and SOA are already slippery, nebulous concepts yet there are unmistakable patterns within each that actually are very tightly related, though wrapped in slightly different cloth. Each encourages the liberation of the underlying functionality of software systems by providing open access to everyone that needs it. Both warmly embrace Web services and the aggregation of existing functionality into new solutions. And Web 2.0, according to O’Reilly, looks at Data as the Next Intel Inside, making large, back-end database driven functionality a core competency. SOA totally gets this as well. And both Web 2.0 and SOA provide the building blocks for creating people-centric processes starting at the scale of an organization and going up.

Granted, most SOAs are conceptually trapped inside an organization’s firewall or VPN. And Web 2.0 envisions the global Web as the stage writ large upon which to act out your grand visions of building collective intelligence and mashed up functionality. But scale is only one of the minor differences really, and not a genuine discriminator at all.

Do the linkages go deeper, to a more fundamental level? Are Web 2.0 and SOA different at their core and if not, how exactly do they relate?

I believe that there are at least two significant connection points. One is that Web 2.0 can be conceptualized as a global SOA. Two, that many traditional brick-and-mortar business that are currently using SOA as their architectural model will want to connect their Web/Web 2.0 faces up to their SOA. This makes Web 2.0 not just being the Global SOA but makes meeting smaller SOAs everywhere not just likely, but inevitable. Note that the respected industry analysis firm, Gartner, recently said that by 2008 that 80% of all software development will be based on SOA. And interestingly by 2006, Gartner believes that 60% of the $527 billion IT professional services industry will be based on exploiting Web services and technology. We’re talking serious convergence of focus here folks. If true, this means that more than half of all software development, SOA and otherwise, will revolve around the Web, inside or outside organization boundaries. All of this means Web 2.0 and SOA will meet each other both coming and going, and begin to become each other as well.

Web 2.0 = Global SOA? Why Should We Care?

But the real question really is does the relation of the two give us an advantage as we design and build Web 2.0 applications, services, and SOAs? One problem could be that many folks outside the IT industry just haven’t heard of SOA. And even then, there are vociferous arguments about what an SOA really is, just like there are endless debates about what Web 2.0 is. But in the end, there are best practices that need to cross pollinate here and SOA’s IT-bound sphere of influence isn’t really a factor. In fact, really only Web 2.0 designers (yes, you) will have to understand these techniques and their connections. Web 2.0 users themselves will generally be blissfully unaware of Web 2.0 as the global SOA.

Now, don’t get me wrong. Web 2.0 and SOA also have significantly divergent elements too. Web 2.0 emphasizes a social aspect that SOA is completely missing. And probably to its lasting detriment. SOA has much more central control, management, and governance while Web 2.0 is free wheeling, decentralized, grassroots, and with absolutely no command and control structure. Web 2.0 also talks about presentation, the front-end displayed to the user. SOA is largely silent on the issue of presentation though it certainly admits it exists. So SOA tends to be generic and faceless where Web 2.0 makes much of human/service interaction. They seem to need each other to be whole. Finally, Web 2.0 is almost too informal and practically calls out for discipline while SOA is mute and autistic in comparison, a technical virtuosity that wants to be social but doesn’t know how.

All of this makes me want to view one through the other to check basic principles. For example, SOA has best practices for building business processes into vast supply chains (so too does Web 2.0). SOA is also a mature view of software that eschews a technical view of information and data. And it identifies a motive force for these processes via something called orchestration. This is a concept that Web 2.0 does not have explicitly and could certainly use (an orchestration mash-up anyone?) though it is provided in some degree by its users. SOA principles also encourage creation of a common vocabulary across systems that is in the language of the domain (common definitions of customers, order, channel, product, for example). So it gets very close to addressing a big issue in software development today: That too many IT systems today tend to have technology myopia and ignore their most important elements… the people that use them and the way that they work. Web 2.0 gets this part even more right in all the significant ways. Web 2.0 embraces people, collaboration, architectures of participation, social mechanisms, folksonomies, real-time feedback, etc. All things that SOA, in its grey, dull, corporate clothes, does not, at least not explicitly. The complementary nature between the two seems clear.

So, I believe there are complementary synergies between these two powerful software approaches. One can very much be used to check and finish the other. SOA is both the "Mini-Me" of Web 2.0 (identical in almost every way but 1/8 its size) and a key archetype for it as well. Though admittedly one that lacks a few important ingredients. What is compelling, and I’ll talk about this in detail in future articles, is that Web 2.0 actually has powerful mechanisms that "complete" SOA (if you’ll allow one last Austin Powers reference.) Web 2.0 offers a face to SOA with numerous best practices for presentation, has emerging technical innovation like radical decentralization that is necessary for stability and scalability which SOA too often doesn’t address, and Web 2.0 identifies important techniques to immerse users into social processes that can make SOA data and services vastly more valuable.

Yes, so Web 2.0 is a global SOA, done right for the whole world. It’s big, it’s everywhere, and it’s here today.

Do you agree that Web 2.0 is the Global SOA? Post your thoughts below…

Update: Early stage VC investor Peter Rip had some interesting things to say about this article, including "Web 2.0 is a lighter weight version of SOA."
Update 2: Both Richard Monson-Haefel and ZDNet’s Joe McKendrick weighed in on this topic with good observations and commentary.
Update 3: This eventually turned into the SOA Web Services Journal cover story in Dec. 2005: Web 2.0: The Global SOA.  This in turn led to March’s SPARK event with Microsoft on the convergence of Web 2.0, SOA, and SaaS.
Update 4: This topic (and related issues for Web 2.0 in the enterprise) has turned into a regular blog on ZDNet.
Update 5 in May, 2006: Om Malik writes a detailed piece on the future of Web 2.0 and is most sanguine about it for the enterprise, interestingly enough, and links to this post.

web2.0, soa