So far, I’ve found two really good use cases for Twitter and the real-time web based on the way I like to consume information:

-Following Events – Twitter hashtags are a great way to keep up with the pulse of what’s happening at tech conferences that I can’t attend. It’s way easier for me to follow than liveblogging as it gives me both shorter updates about what’s going on and allows me to easily hear from a wider variety of voices than a single blog can provide.
-Quick pulse of what’s on my friends’ minds – About once or twice a day I’ll see a link that gets retweeted by lots and lots of people I know. That tells me that they’re all reading or at least expressing interest in the content. That’s a good signal to me that I should probably check out whatever it is that they’re passing around.

Aside from those two use cases, I see a few key areas where the latest incarnation of the real-time web isn’t really making my life better, making me more productive, or where I’m having a hard time understanding all of the euphoria:

While the speed with which people can create and publish information continues to increase, I’m gated by my own ability to process information. I only have so many cycles to process new and existing information. While I’m always trying to get better at processing more information more quickly, there’s an upper bound on how fast I can get. Also, I am frequently connected to the Internet but I do log off from time to time. There are lots of real-time web tools that work great if you sit there and babysit them all day, but what about once you’re gone? Or what if something interesting or important happens while you’re away? I’m looking for the real-time web to help me make more sense of the vast streams of information floating around me – increasing the speed of the information coming out of the firehose is interesting but not nearly as useful as helping me prioritize what to read and evaluate first. I don’t necessarily need real-time filters to prioritize information – I’d trade a slower, stronger signal for a weak, fast signal at this point of my usage of the web.

Lots of information is delivered to me in real-time but isn’t really time-sensitive – This is particularly true of the real-time link sharing / re-tweeting use case. Seeing that all of my friends are re-tweeting or sharing a piece of content is a strong signal that it’s important. But that’s not the same as important right this very second. I find that most of the articles I see getting shared on Twitter either started up or end up on Techmeme or some other aggregation service. Seeing them a bit earlier than when they hit my Google Reader is cool but not critical in my information consumption patterns.

I’m not really sure that search is important for ephemeral information – A decent chunk (I’m not willing to hazard a guess) of the information I encounter on the real-time web is ephemeral. Status updates, thoughts, feelings, etc are all really interesting in the moment but can lose relevance over time. Even “breaking news” loses relevance once it’s broken – once a story is broken, I’m more interested in finding the source with the best coverage than I am sticking with the person who broke the story. This makes me wonder whether search is really a big deal for the real-time web. I must be in the minority here because both Google and Twitter seem convinced that search for real-time information is a big opportunity. But I’m having a hard time seeing it. For things that aren’t ephemeral, they can’t hide – they’ll make their way into other places on the web (blogs, websites, structured content that’s easily indexed by search engines, etc). For fluid real-time situations (natural disasters, crises, breaking news, sports, etc), I can see a ton of value in having a real-time window into what’s happening as a story is unfolding.

The one thing about the real-time web that does make sense to me is the disintermediation use case for brands and celebrities. If you’re a celebrity, brand (personal or corporate), or other public figure, there are a lot of people standing between you and the people you’re trying to reach – your own handlers, the media, technology platforms, etc. A lot of real-time web tools allow you to break this dependency in two ways. You don’t need to navigate the maze of people above to reach your constituents or fans (just get them to opt into hearing from you via a less noisy channel than email / print / radio – you know, something like Facebook or Twitter) and you’re not gated by communication caps. If you want to share everything you’re doing, there’s nothing stopping you. The time required to produce and distribute an update is very small. So it’s much easier for brands and celebrities to create a stronger bond with their fans / constituents by cutting out all of the people who stand between them and by sharing the kinds of details and information that has been difficult to surface until now. That’s not a use case that strikes me as being dependent on search. That’s a use case that’s about noiseless, frictionless distribution and communication.

Anyway, that’s a set of fairly disjoint thoughts I have about the real-time web and Twitter. Feel free to leave comments as I love them.

It would be cool to have a simple service that did two things for my friends’ status updates:

Status Update Keyword Cloud – Just like a tag cloud, display the most common words / themes found in my friends’ status messages in the last few days. Like any good tag cloud, I’d be able to click on the keyword and see whose status messages reference that keyword. It would be a cool way to visualize what your friends are talking about or referencing via their status messages.

Search – I’d like a keyword search tool that would let me search my friends’ status updates for key terms or keywords that I’d like to find. This would be helpful as a simple way to figure out which of my friends might know something about a given keyword or topic.

I have been doing a lot of thinking about vertical search engines lately, especially because I find them to be very useful in getting information in categories where general search doesn’t always give me the right answer right away. As useful as vertical search engines can be, I often wonder whether any of these vertical search engines are garnering enough traffic to be viable standalone properties.

Why does this question matter? Well, from my point of view it looks like vertical search sites who can’t generate enough organic traffic of their own through SEO, other techniques that will drive higher organic search results in Google or Yahoo, or having a great destination site that brings users directly to the front door will be destined to be in the “powered by” category offering white label syndicated solutions to partners. I don’t want to denegrate the “powered by” model — it has made a ton of money for Google and its partners and is an effective way to distribute great technology. However, given the amount of venture capital invested in this sector, my hunch is that investors are looking for ways to build meaningful, standalone companies who can be destinations of their own given that very few vertical search engines have been acquired in the web 2.0 world (I am excluding all of the web 1.0 comparison shopping and job sites that got acquired).

In order to answer this question of whether vertical search sites are stalling out in their eventual aim to become destination sites of their own, I took a look at some Alexa data on some of the most prominent vertical search sites. Yes, I realize that Alexa data is imperfect, skewed, etc. That being said, it’s publicly available and easy to access. I took a look at the Alexa ranking for as many vertical search sites as I could think of and took note of their current Alexa ranking. The next step was to then bucket them into categories (top 500, top 1000, and top 10000). The results of this mini-exercise can be seen below.

If you don’t want to scroll down, here’s the punchline — only two “web 2.0″ vertical search properties cracked the Alexa Top 1,000. Yes 1,000 is an arbitrary number to choose. Yes Alexa data is iffy. But with all of those things in mind, we are only talking about 2 companies (of the 12 I found in the top 1,000) who made that cutoff.

Looking at this data tells me two really important things. First, it is really hard to build a vertical search engine that gets significant amounts of traffic. Looking at the clustering of sites that managed to crack the top 10,000 shows a serious weighting toward sites where the underlying topic of interest (travel, product comparison, jobs and recruiting) is a market where significant amounts of money change hands.

The other major thing I take away from this data is that more and more vertical search engines ought to be looking to “power” the search of other folks unless they have some really clever ways to generate organic traffic — it’s really hard to become a highly ranked vertical search property.

Feel free to leave me a comment or point me toward any search property I might have missed.

Raw Data Appendix Below (Global Alexa Rank as of March 19, 2007)

In the Alexa Top 500

IMDB (#35)
LinkedIn (#178)
Technorati (#215)
Monster.com (#327)
Careerbuilder.com (#375)
Shopping.com (#440)
Nextag (#471)
TripAdvisor (#494)

In the Alexa Top 1,000

Expedia (#400)
Travelocity (#574)
Orbitz (#597)
Citysearch (#734)

In the Alexa Top 10,000

Shopzilla (#1,403)
Zillow (#1,542)
Kayak.com (#1,561)
Yelp (#1,738)
ZoomInfo (#2,364)
Indeed.com (#2,386)
Trulia.com (#4,995)
Oodle (#8,056)
Blinkx (#8,250)
Become.com (#8,335)
SimplyHired (#9,054)

1. How do you aggregate enough questions to have meaningful coverage?
2. How do you convince users to invest in posting and answering questions on your platform?

I don’t have a great answer to the second question yet, but I do have some thoughts on the first question. I am subscribed to a lot of mailing lists and other distribution groups for alumni organizations, interest groups, or other organizations that regularly publish content or solicit input. By and large, message board or mailing list traffic falls into three major categories – rants/flame wars, legitimate announcements, or requests for information. A perfect example is the general alumni mailing list operated by my alma mater, Stanford. At least once per month there is a flame war or thread that spawns tens of messages and generally decreases the utility of the list. Some other fraction is general announcements (job postings, events, etc). The most interesting, traffic, however, are what I call requests for information. There is a lot of this type of traffic on the mailing list. People are always asking questions about mortgages, looking for service providers, asking for input or advice on some topic or decision they need to make, etc.

Sadly, this information stays locked up in the list archive. There is no good way to publish that information out to an external Q&A system. I am surprised that there isn’t a Q&A system out there who has come up with a way to have interested users feed questions into the system from message board traffic or other systems.

The other interesting wrinkle to this approach is that many message lists are thematic in nature — they attract people with an interest or expertise in a particular topic. The targeted nature of most mailing lists and other distribution lists ought to make classifying questions and answers easier and frankly more useful to the end user.

I can’t imagine that it would be terribly difficult to pull this off from a technical perspective. Yahoo, as one of the largest groups operators and Q&A operators, is in a good position to do this. There are also a lot of non-Y! groups who have great content as well and who could enhance the value of any Q&A system.

As long as vertical search supports or enables an economic transaction, there is a chance to create a viable business model. Taking a small cut of an existing transaction (or replacing the existing intermediary) seems to be much easier than getting people to pay for something where they are not accustomed to paying.

What are some examples? Trulia in real estate (connecting home buyers and agents), Indeed and SimplyHired for jobs (connecting job seekers and employers), and all of the travel vertical search sites (connecting travel seekers with travel options). Just because there is a business model that might work does not mean that any or all of these companies will be successful in achieving their aims.

For some reason I find the jobs vertical to be particularly interesting, so I will focus on that one for the time being. There are lots of questions as to how vertical job search sites will generate revenue. The challenge, as I see it, is as follows:

1. You can’t charge the employers whose content you are scraping/aggregating as they didn’t ask you to do it.
2. Charging job seekers for access to this content would be hard; it is contrary to most successful aggregation models. Plus, this is not proprietary content — it is all information that can be found publicly.
3. Revenues from participating in an ad network won’t be enough to really grow the business.

My suggestion is that these vertical job search sites look at the advantages that they have as traffic aggregators. They know which jobs, companies, and geographies (and keywords) are most popular across a wide number of users. Why not create something like AdSense for Jobs (we will call it JobSense)? Why not offer employers the opportunity to bid on the right to be a sponsored result when certain keywords are entered? For example, when I checked SimplyHired and ran a query for “product manager” in Sunnyvale, CA I got 639 results. That’s a lot, no? How far would a job seeker be willing to wade before he/she either narrowed the search or just got bored? It turns out that a lot of these jobs are at Yahoo. Would Yahoo! (or a startup company for that matter) be willing to bid to be a sponsored result at the top of the page (or on the side) whenever someone typed in a search for product manager in Sunnyvale? Why not? Presumably the jobs available on company websites are ones that need to be filled. Assuming that filling these jobs (or at least generating more candidate inquiries) is important, perhaps employers would be willing to pay for that privilege.

I can see a model where employers pay for the right to be listed as sponsored results for keyword searches based on geography, job title, etc. There are even ways to do more interesting wrinkles. For example, if Yahoo had 5-10 jobs that they really wanted to fill, they could show those as sponsored results anytime someone searched for Yahoo as a keyword regardless of the specific job title. Or, if you were a start-up looking to get web 2.0 types, you could bid on the Yahoo! keyword and hopefully redirect job seekers looking to join the big exclamation point. There are lots of other things that could be done along these lines – it could be executed as CPC, CPM, or through some other mechanism.

Thoughts? You can always email me.

Technorati Tags:verticalsearch

I worked at Excite for a brief spell and am glad to see some former Excite people up to interesting stuff.

If you are going to the event and would like to meet up, email me. I will probably post something about it after the fact.

Technorati Tags: searchsig, oodle

1. The content you are trying to locate is something of interest to the current universe of taggers (breaking news, technology topics, poltiics, etc.)

2. The data sources that Wink culls include the services most likely to have what you are looking to find (dgg, de.li.cio.us, furl, etc)

I have found that Wink is hugely useful for searching for breaking news, technology topics, or very geeky web 2.0 technology implementation advice. The supplemental results I get from Google are great and give me some sense for what I am finding on Wink that is unique and different from what I would get from Google. I will probably post a more complete review of Wink later.

The thing I have realized, however, is that I still like Technorati the most for searching for blog content. Why? Well, most of the blogs I read do a really good job of tagging their content (whether they are using explicit Technorati tags or category tags using their blog publishing tool) in such a way that it is easy to retrieve in Technorati. After sleeping on what I heard at tonight’s Search SIG event, I am beginnig to appreciate the importance of tag accuracy (or at least a shared understanding of what a given tag means) over tag breadth or tag volume.

The ultimate utility of a tool like Wink will be in its ability to get access to high quality sources of tagged data. If companies who are collecting lots of tag data (such as Technorati) decide not to share such data, the overall utility of Wink will be limited. It’s not just about getting tag breadth but about getting tag quality.

Technorait tag: searchsig
Technorati tag: wink

Observation #1: Either these panelists know a whole lot more than they admitted about what motivates taggers, who is actually tagging, and what the future holds or they are learning about it at the same time as we are, albeit with a better view as to what’s going on. One of the most interesting themes of the talk was what I interpreted to be the relatively nascent understanding of who taggers are (besides the “early adopter” crowd generically defined), the balance between those who tag to help others versus those who tag to create knowledge for others, the privacy concerns around tagged data, and what it will take to make tagging more interesting and relevant to a larger audience.

Observation #2: The folks on the panel (Digg and De.li.cio.us in particular) are really thinking about the social etiquette and social interactions associated with what they are developing. I was impressed with the amount of thinking that Joshua and Kevin expressed about the social implications for how people find information, share information, make it known that they are experts, and utilize/form social networks in determining how to advance their products. The most interesting point of this discussion was about how to take lessons that others have learned (LinkedIn, MySpace, etc) from their usage models and apply to features to include and features to avoid. I was pleased to hear how much thought they were putting into this area and the way that they seemed to want to watch and take cues from emerging behavior rather than shove features at users.

Observation #3: Like all useful communication media, pollution is on the horizon. It is clear that splagging is on the mind of all of these folks, Digg in particular. One of the more interesting audience comments came from a courageous search engine marketer who admitted that she saw the opportunity to whack these services to improve search visibility. Evidence of attempts to game some of these systems is already evident. These arms races are always difficult to win (see email spam and blog spam).

How large does the universe of tag contributors need to be before tagging has a more meaningful impact on the Internet as a whole? I still maintain that it is an enthusiast activity and that there are a lot free riders benefitting from the work of those who tag/classify news and websites in high technology. I am curious to see how tagging will spread to new markets.

Will the collaborative spirit that exists among many companies in this space continue for much longer? There were hnts that not everybody on the panel was in favor of sharing the information they had collected, be it due to privacy or business concerns. There are some obvious gains to standardization and collaboration at this point in the tag market’s development.

searchsig

If you plan on coming, drop me a line and we can meet up during the networking break.