Building Modern Web Apps? Better Have A Deep Competency in Web 2.0, Open APIs, Widgets, Social Apps, and More

The Web has an interesting property that those building Web applications and online businesses usually encounter soon after they first launch: It has its own unique and unforgiving rules for success and failure.  Appreciating them requires a certain level of understanding of the intrinsic nature of the Web and how it works.  Actually leveraging those rules requires an even deeper and more profound understanding of the Web. The challenge these days? The Web competency bar is climbing fast.

To drive the right decisions in what they do product designers, marketing teams, software architects, developers, strategy officers, and other key roles in today's generation of online businesses need to have a solid handle on an extensive array of Web topics.  This ranges from appreciating why plain old HTTP is so good at underpinning the Web to more sophisticated topics like modern application architecture, the latest in online user experiences, next generation computing models (grid/cloud/utility/SaaS/PaaS), cost-effective scalability, user identity, network effects, Jakob's Law, analytics, operations, user community, as well as the many compelling new distribution models that are nearly mandatory in the first release of most products. 

This extensive set of competencies is what's required nowadays to deliver a credible online product to a receptive user base and it has dramatic implications for both uptake and overall cost/time-to-market.  Worse, this body of knowledge has become extensive enough that many Web startups frequently fall far short of what they need to know in order to be successful with these far flung practice areas. 

Web Product Distribution Models - Web 2.0, Widgets, Social Apps, Open APIs

Does this complex body of knowledge mean the era of the two-to-five person Web startup is coming to a close? Not at all, at least not yet. The productivity level of the latest tools and techniques remains almost astonishing though the level of knowledge required of these teams is creeping up and up.  And as we'll see, new models for product distribution are pushing the capability envelope of the typical Internet startup team to the point we may very well see the day soon that they won't have all the skills necessary to deliver a fully-scoped modern Web application.  It is also one reason why fewer and fewer Web startups have the goods to be all around hits out of the gate.

Certainly, varying depths in subject matter are required depending one's exact role in a Web business, but Web-oriented products are fundamentally shaped the vagaries of the network itself.  Tim O'Reilly himself still has the best quote on the subject: "Winners and losers will be designated by who figures out how to use the network." And as we'll see, the Web is driving the evolution of a major new generation of online distribution models.

Why Adopting New Distribution Models Is Crucial 

As an example of this, I've been tracking some of the latest discussions around the hot topic du jour in the Web world: Social networking applications.  Specifically, it's been interesting to watch the surprisingly low level of industry attention around the titanic competition brewing between social networking application formats from Web giants Facebook and Google.  Why is this?  Some might say it's because these applications still have largely unproven business models.  Others, like Nick O'Neill at the Social Times recently observed (rightly in my opinion) that the struggle may have to do with a deficit in understanding why these new types of Web applications are so important. Nick notes that these widget and social networking style models for packaging and distributing Web apps often "have more eyeballs looking at their products than television channels have" and the challenge is that too many people just "don’t know what any of this means", despite the major players divvying up the online pie for themselves.  With the size of these next generation distribution audiences, ignorance has an extremely painful price: failure to produce results and growth, poor engagement with the marketplace, and loss of market share.

An excellent summary of the truly massive, but largely underappreciated scale of these new Web application models was last week's TechCrunch piece on the progress of Google's OpenSocial, an increasingly successful model for creating portable social networking applications that will run on any OpenSocial-compliant site.  Erick Schonfeld reported that OpenSocial now has a total reach of an astonishing 350 million users and it will soon be 500 million.  There are over 4,500 OpenSocial apps today, a healthy number for the application format but a small drop in the bucket compared to the number of Web sites in the world. But the key is that these applications are integrated much deeper into the social fabric of an engaged audience, interjecting themselves into the daily personal and work habits of the "captive" users of social sites and even have access to the personal habits and data of users of these sites.  Facebook's story is impressive as well with over 37,000 applications that have been installed over 700 million times.

And social networking applications are just one of many news ways that applications have to be packaged and distributed, yet far too many organizations persist in a very 1990s view of Web experiences, namely that Web sites themselves are the center of online product design.  Many even think that some of these other new distribution models are interesting but not part of their core online product.  Unfortunately, that's very much a parochial view in the present era.  Federated applications, atomized content and functionality, 3rd party product ecosystems through open APIs, and much more are required to establish a strong and resilient network effect which fends off competitors that are themselves bringing these potent new competencies to bear. 

 In fact, one of the things we emphasize over and over again in our conference workshops and in Web 2.0 University is that having a Web site is usually the least interesting things about new products.  Worse, it makes the customer have to find you amongst tens of millions of other sites.  Instead, these new models tend to focus on going to the customer, instead of making them come to you which is a much harder proposition. This can instantly give you the ability to reach millions of potential people with dramatically lower effort and cost, as long as you have something interesting to offer.

Unfortunately, the number of capable practitioners of these new distribution models remains relatively small compared to the large body of experts in traditional Web product development.  Demand is also low for these new skills as most organizations have been painfully slow to appreciate how much online product development has changed.  A quick search of the job aggregator SimplyHired tells the tale: Nearly a thousand Web designer positions are available while only 36 OpenSocial and 40 open API positions are open, for example.  This despite the the latter skills being able to project a product across the Web into hundreds of social sites or create an API that allows the product to be incorporated into countless other products for far less cost per customer than traditional methods.

The lesson here is that these new models still have a lot of fertile, unclaimed territory and many otherwise fierce competitors have not yet become fully aware of these new opportunities.  Get your piece of the pie while there's still time. 

The new Web 2.0 era distribution models remain largely untapped

I also find that the Web development industry has been slow to change, particularly outside the valley, and there is depressingly scarce information on how to deliver well on things like widgets, open APIs, social networking applications, and even syndication.  To help with this, I've put together a short primer and some good references for those that want to get started.

Because the good news is that there remains tremendous opportunity for growth and success — for both startups and traditional businesses — if they will actively begin incorporating these new product delivery models into their own online capabilities.

Overview of Online Product Delivery Models 

  1. Web sites.  This the classic model for Web presence.  During the early Web, creating a Web site was just about the only option for engaging with those online (e-mail being the other.)  Most early Web sites were used for publishing and not for user participation or peer production.  These days, Web sites are still important, though by no means mandatory, and have their content syndicated via RSS and ATOM (pushing the content to where it's wanted), provide an access point to obtain widgets, and maintain user identity, and create communities of users.  Upshot: They've evolved a lot but Web sites are only part of an extensive set of capabilities that must be brought to bear in the Web 2.0 era.
  2. Syndication. It took ten years for the Web community to figure out a workable syndication model.  Now RSS and ATOM are now the expected models used to distribute content off a single site and across the Web. Countless aggregation services now exist that make a site's information embedded in their services as well as a way to offer users a method for pulling information from a site and experienced in a means of their choosing, from Google Reader and Newsgator to the innovative Yahoo! Pipes.  Most sites still heavily underutilize syndication even for notifications and pushing out frequently changed information to draw attention to it much less the strategic ecosystem and integration opportunities it affords.
  3. Web 2.0 applications.  You might argue that Web 2.0 itself is not a product distribution model but a set of design patterns and business models and that would be a true statement. However, in this context we're referring to the fact that Web 2.0 apps package up the 3rd major type of networked value: user participation.  Before then, Web sites and syndication primarily had only centrally produced content or functionality that they could expose over the network and offer to the marketplace.  In other words, user participation its purest form — sometimes known as peer production —  ultimately results in products like Mechanical Turk and Predictify that provide direct networked access to user participation, but there are many fine gradations to this.  The bottom line, Web 2.0 applications plug the user into the network like never before and are a critical rung in the distribution ladder since it offers access to the largest set of content and information by harnessing collective intelligence.
  4. Open APIs and Web services.  This is one of the most important long-term decisions most online businesses can make.  Offering an open API lets anyone take the online components of a business, from its data and functional capabilities to the users themselves, and makes them open and accessible over the Web to be incorporated into other products and services, sometimes in the form of mashups and sometimes in the form of entire online products.  Amazon, one of the first Web companies in existence and is hence far downrange in terms of the experience curve, has been using this distribution model with notable success recently.  So have hundreds of others.  The real challenge has been how foreign this model is to the original Web model and thus to the various management and development competencies in most organizations.  It's much more an a way to OEM a product and leverage the customers and investments of hundreds of other partners.  However, overall, it affords the potential for much larger business outcomes than could ever be created with point Web presence.  It's now considered a significant oversight not to have an open API available for the typical online product.
  5. Web widgets.  Selecting parts of a Web site and it's data and packaging it up to make it run inside a portable, user distributable widget has been growing more and more popular over the last few years. For example, WidgetBox currently distributes 74,000 different kinds of Web widgets from its partners to over 1.2 million other sites.  Widgets lets users distribute a Web site to other places on the Web at no extra cost and it also creates an ecosystem effect, where other Web sites users become the users of the new site.  The YouTube badge is a notoriously well-known example of this that also helped drive the extraordinarily fast growth of the site.  Like APIs, widgets are now considered a mandatory must-have for new and existing online products. But unlike APIs where it's up to the API users, figuring out users want out of your site's widgets is still an art form.
  6. The Plaxo Pulse Story with OpenSocialSocial networking applications.  Sometimes viewed as an extension of the Web widget model, social networking applications are applications designed to run inside of popular social networking environments and usually have capabilities that tap into and make use of the social graph information resident in a user's social network account.  This is an amazingly fast moving field as you can see from a recent post on the latest happenings on the OpenSocial blog, to the extent it's hard even for well-funded companies to keep up.  However, despite skepticism that large businesses can be built exclusively through a social networking application, it's become ever more essential for a site to make its capabilities accessible usable in these environments.  Not only will users help distribute online products in these formats to their contacts but it also increases the overall usage of the your application including participation and its consequently growth of a site's network effect.  While not yet considered mandatory for online products, the ease with which these social network applications can be created and the large numbers of users they make available makes it a smart distribution option for most Web businesses.  Like widgets, however, figuring out what users will find engaging in a social networking application featuring your online product takes some research and experimentation.  However, the results can be very rewarding and some social networking applications have millions of daily users.  See the Plaxo Pulse story on Mashable for the details of how OpenSocial drove a 5x improvement in traffic in only 3 weeks.
  7. Semantic Web and Web 3.0. The Semantic Web, one of the original visions for the World Wide Web, has taken a while to arrive but it's beginning to look like it may hit critical mass in the next 12-24 months.  Combined with Web 3.0, which takes the architectures of participation at the core of Web 2.0 and drives it through a lens of Semantic Web capabilities.  The benefits can be profound and can greatly increase the value and leverage of information on the Web.  While this is very much not prime time yet, unlike #1-#6 above, it likely will be and smart organizations can get ahead of the learning curve and get an early market lead using these techniques.  For now, however, I recommend that most organizations focus on executing well on the first six items before tackling this and waiting for the technologies to finish emerging and maturing.

The list above should provide good guidance for starting move into the potent new models for distribution on the Web.  I'm seeing, however, that because of the major shifts in strategy and product design emphasis these techniques demand, most organizations take an inordinately long amount of time to become effective with them.  The lesson here: Start small now and build core competency.  Small investments now can pay off later in terms of valuable experience made from early experiments and pilots.  When done right,
these new distribution models can become the dominant channels that the world uses to interact with your business, like they already have with Amazon and Twitter.

I'll be talking about these and other strategic online product design topics in my upcoming Building Next Generation Web Apps Workshop at the inaugural Web 2.0 Expo 2008 NYC next month.  I'll have more details about this deep-dive session in an upcoming post.

Web 2.0’s Real Secret Sauce: Network Effects

A lot of the early descriptions of and speculation around Web 2.0 last year were either chock full of examples or long lists of interesting new phenomena that seemed to be emerging on the Web.  My list of Web 2.0 explanations last year is a good survey of these.  This year however we seem to be zeroing in on the core phenomena underlying the sea changes of the Web and even in society itself.  Sure, some fundamental upgrades to the Web’s infrastructure – including steady and significant improvements to the physics of the Web – have also helped things along (i.e. those great Ajax/RIA applications we love so much like Gmail and Gliffy, download in seconds now.)  But it’s the widespread use of the two-way Web that’s really the harbringer of an increasing sense of disruption.

Watch a TV panel that I was on with Google’s Adam Bosworth on how the improvements to the physics of the Internet have enabled Google’s newer products, particularly around SaaS and Ajax.

I wrote recently about the trend of Web 2.0 reductionism that is helping us get to Web 2.0 fundamentals and assisting us in understanding why it’s such a game changer.  The upshot is that the Web is rapidly evolving and is increasingly being shaped by its users.  If you’re not entirely sure that this is really a big deal though, you have only to look at the example of MySpace.  MySpace is as pure a Web 2.0 play as you’re likely to find and they’ve exploited user generated content and network effects to become the #1 visited site on the Web (as of last week), from out of practically nowhere two years ago.  This is an extremely impressive achievement and shows the power of Web 2.0 techniques to quickly best even the very largest and most established industry leaders, including Google and Yahoo!  This demonstrates that the successful exploitation of what has been labelled "Web 2.0" can be an extremely disruptive force.  People are sitting up and taking notice.

Inducing Network Effects with Web 2.0


If only the keys to doing this were in everyone’s hands, the thinking goes, then disruption could be and would be widespread, even rampant, across business and society.   And certainly, a lot of folks are very interested in taking advantage these techniques (such as maximizing your users’ social surface areas) both out on the Web, and within their own organizations.  This further explains the interest by venture capital investors, startups, and enterprises to get into the game before others use these techniques first to upend the traditional industry leaders in their areas of interest.  Mark my words, this year is only the beginning and those in less regulated industries that aren’t constrained from using Web-based network effects effectively are going to have a very tough time of it.  This will prove to be true particularly in the next 24 months as everyone scrambles to try to figure out where this is all headed.

Fantastically and ironically, the keys to doing this actually are in everyone’s hands today, right now.  Web 2.0 is a truly egalitarian force and though certain core competencies are required (being able to keep up with your own growth being one of them), just about anyone with a innovative idea and a good formulation of the techniques can access and tap into the existing audience of over 1 billion Web users.

Triggering Network Effects: As Simple As Establishing an Effective Viral Feedback Loop?  Surprisingly, yes.

The Web is the fabric upon which an ever increasing amount of our lives is woven into.  This is now most media including newspapers, TV, radio, entertainment, music, arts, etc. as well as what I call lifestyle logistics; e-mailing, IMs, calendaring, travel planning, time/task management, and more.  They are all moving or have
already moved to the Web.  This very habit most of us have of being on the Web so much of the time, along with the easy lure of the hyperlink, which can redirect anyone via any of these ‘channels’ into a new Web 2.0 experience or site.  Thus, if someone loves your new site, they send their friends the link, they send their friends, and so on.  Instant pile-ons involving tens or hundreds of thousands of users overnight are now common.  And good viral feedback loops keep them there and keep them coming back, and bringing their friends with them.

Of course, in a few years the exact design patterns for triggering a new MySpace, Facebook, or similar social juggernaut will become common.  Then most likely balance will be reachieved in the industry and there will be less disruption.  But for now the secret balance of Web 2.0 techniques that powers growth through efficient access to network effects is still an art (read my Notes on Making Good Social Software for an idea of how these design patterns might look however).  The bottom line: the upside and downside potential of Web 2.0 is truly significant.  And it means that in most industries doing nothing is really no longer an option.

But let’s talk about network effects for a minute.  What are they?  I’ve written before that the core description of a network effect  is when a good or service has more value the more that other people have it too.  Easy so far right?  Examples include e-mail, IMing, the blogosphere, and even the Web itself.  But what’s not clear from this description is the raw power that is caught up in and represented by network effects.  Most rigorous studies and mathematical formulations reveal that there is tremendous geometric power in network effects.  Though Robert Metcalfe originally coined Metcalfe’s Law to describe the raw potential of network effects in computer networks, most recent formulations have attempted to capture the exact value of them more precisely.  Most notable has been David Reed with his relatively well-thought out and profound  Reed’s Law, as well as Odlyzko and Tilly with their work.  However, whichever formulation you believe is right (and Reed’s Law, if true, has staggering implications in this regard), the result is clear: At even an early point, the cumulative value of a large number of connected users goes exponentially off the charts.

The Potency of Network Effects and Thus Web 2.0


The end result: If you have a million people visiting your Web site but you’re not leveraging network effects with them (such as by letting them contribute and letting others see and respond to those contributions), then you’re probably squandering the greater part of the value of that million person audience.  Along comes someone else who does exploit the network effects of their users.  They can quickly leapfrog you if they figure out a good way to establish and harness the connections between those same users, again in some kind of viral feedback loop.  The new site will have the combined aggregated output of those users as part of its value proposition to new and existing users and it’s clear which will win.  Examples: Digg bested Slashdot in just such a way.  YouTube is set to do something very similar to network and cable television.  Google is staging itself to best the #1 software company in the world.  There are many others.

So what do you do?  Most of us will soon face situations where the person or organization we’re competing against will be trying to use these same techniques to gain advantage, marketshare, or whatever.  Agile development processes have taught us in recent years that that the tightest closed feedback loops resulted in better products, faster.  The Web is now allowing feedback loops of this kind between staggering numbers of non-centrally coordinated people from a ready -to-tap worldwide audience.  Emergent content, communities, and unintended results are all side effects of systems (and yes, your enterprise) with rampant network effects.  Learning how to channel these feedback loops and the resulting network effects constructively will become a significant competitive advantage and a required core competency across the majority of professional disciplines in the very near future.

Read David Berlind’s excellent overview of Eventful and how they are using Web 2.0 techniques to be successful

But is this really happening?  Is the competitive landscape about to be torn up?  Are companies starting to respond?  It does appear so.  For one thing, I’m increasingly seeing traditional organizations trying to do things like "MySpacing" their current product offerings.  Everything from Starwood Resorts to major national interest groups are looking at creating business communities that are really viral social networks powered by their users.  It is a fascinating time to be building or reinventing a business. And if you haven’t started, now is probably the time to begin the effort.  And worry not if you can’t figure it all out right now, the entire world is still trying to figure out how it applies to them too.

How will you apply Web 2.0 to your life, business, or organization?