A debate rippled across a few tech blog sites following Erick Schonfeld’s reiteration, a few weeks ago, of some claims made by Nova Spivack concerning the fate of traditional keyword search. As Schonfeld explains, Spivack is of the opinion that as the number of web pages a search engine has to sift through explodes exponentially, the efficacy of a simple keyword search will drop off. Spivack himself explains the problem as follows:

“Keyword search engines return haystacks, but what we really are looking for are the needles. The problem with keyword search such as Google’s approach is that only highly cited pages make it into the top results. You get a huge pile of results, but the page you want — the ‘needle’ you are looking for — may not be highly cited by other pages and so it does not appear on the first page. This is because keyword search engines don’t understand your question, they just find pages that match the words in your question.”