Sunday, February 17, 2008

Semantic Web Should Improve Search

So I got the flu the other day. It was probably passed to me via my elementary school-age kids who also had a mild case. The weird thing is, three of us each acquired an unusual eye twitch, all starting within a day or so of each other. I was curious if this might be a common symptom of the flu, maybe related to body aches or something. So I did some Internet searching e.g. using 'flu "eye twitch"' and did not find anything in the first page of results that associated the two conditions. I did find lots of pages where eye twitch was discussed, and flu was mentioned elsewhere in the page (often in an advertisement). I pondered my search failure, and accepted that popular search engines using term co-occurrence always run the risk of returning irrelevant content. What would really improve search is the clue that flu and eye twitch are both medical conditions, and my query is whether one condition is related to another, e.g. via a symptom-of relationship. On subsequent searching I did find two blog posts (Eyelid Twitching and Eye Twitch) that each discuss associations between flu viruses and eye twitches. While I've still read nothing official, there is apparently some anecdotal evidence out there to support the hypothesis that a flu virus may cause eye twitches. I look forward to the day when search engines will recognize mentions of e.g. medical conditions and domain-specific relations in text, and when the corresponding search interfaces offer users ways to specify semantic queries.

2 comments:

Harry Chen said...

Joe,

You are right that if we can teach machines to understand semantic relationships between concepts, we can build better search engines.

The question is how do we go about doing that. I don't know the answer...

Semantic Web "purists" may argue that the development of an intelligent search engine will depend on a wide adoption of OWL and RDF for describing web content.

Information Retrieval folks may suggest the need to create new search algorithms that go beyond PageRank. Researchers at Ask, PowerSet and Hakia are taking a similar approach.

Have you tried feeding your query into Hakia and Ask? When I tried Ask, in addition to the matching search results, it also recommends alternative search terms that could narrow my search.

Median Joe said...

I did try a few search engines including Ask. In this case, Ask's search term narrowing and expanding was not appropriate- it simply added words that occur with some of my search terms, or suggested slightly different forms of the words. E.g. recommending "eyelid spasm" is somewhat relevant, but in the end not helpful.

I'm sure it's not going to be easy, but I believe there is definitely an opportunity to go beyond term co-occurrence and term distances on a page. I recall now some interesting news last fall about Powerset's NLP approach to enabling the semantic web.