So far we've looked at why we need to organize documents, at some of the fundamental problems in the process of organization and at how to decide on the subjects of documents. In the next part of the book we shall look at the tools which we need to bring those two operations together in the formal processes of classification and subject cataloguing.

Natural language indexing and searching

Indexing, classifying and the subject description of documents are on the whole very labour-intensive (and intellectually arduous) processes. This is one of the reasons why abstracting and indexing services, and bibliographic databases, are so very expensive to subscribe to. An obvious question is whether we really need to use complicated artificial systems of classification and indexing to describe documents. Why not use the language of our everyday speech, since surely this would make the assignment of the keywords much quicker and easier? And the titles of documents or the text itself could provide us with the terms that are needed for indexing. In any event, whatever cataloguers do, the endusers will choose search terms out of their own heads, and not from a classification scheme which they have never seen or heard of.

On the face of it, it would appear to be much easier to use this natural language approach to index documents, both for the initial indexing, and also for the retrieval process, particularly where machines can help with the process. When automation began to be used more widely in libraries in the 1970s it was generally thought that the immense power of the computer to scan the whole of a large store of information in a few seconds would obviate the need for all this expensive and exhausting brain work. Free-text searching and natural language indexing became the norm, classification and indexing were seen as moribund and pointless, and the study of classification disappeared from many library school curricula during the 1970s and 1980s.

Initially, the advent of the world wide web reinforced this attitude, since even the information illiterate can search and retrieve masses of information using only natural language. On the other hand, any acquaintance with web searching will have shown you how inefficient natural language searching can be, because the web is an information store so large that the juggernaut techniques of the early automated systems cannot solve the problems that searchers encounter.