paris5.htm: How to search the web, by fravia+ (¯`·.¸(¯`·.¸ Paris, ecole polytechnique ¸.·´¯)¸.·´¯)

fravia @ Paris
Ecole polytechnique, 6 February 2001

The importance of subsidiarity and 'micro-searching' for seekers

where else to look: local / regional / usenet / webrings, netcraft and other search_tricks

Where else to look: usenet / local / regional

The importance of subsidiarity and 'micro-searching' for seekers
Webrings, homepages clusters, maillists, messageboards, national engines, usenet
So-called "local" search engines are extremely important to fetch info. Freepages providers, counters, stats applications and all other finalized search engines... that you can use at leisure... for fun, searching purposes, combing and/or knowledge profit.

There's noway I can give you in the short time of my slot a complete description of the huge plethora of tools available.

Usenet
Among the different 'local' areas of the web, usenet is without any doubt extremely important for searchers, and Dejanews (was one of the few remaining usenet repositories, has been now bought by google) has proved to be an unconturnable service for seekers.
To 'correctly' access usenet you should use specific news reader software, but to peruse Usenet, you just access a news server, which is nothing else than a computer system which provides shared storage for Usenet articles. Usenet must use "shared storage" because its total volume is HUGE and Transient. Usenet articles are kept in central locations and thus can be "shared" by many readers. This is a typical feature of the 'web of old' that still survives, all commercial diseases notwithstanding.
Each server has ONLY A PART of the total articles of Usenet. Your news reader (or your browser) can only display articles which are present on the news server you access. If an article is not present on the server, it is because either that article has never arrived at your server, or it has been cancelled, or else it has expired.

Because there is a limited amount of storage on a local news server, articles "must" expire. But fortunately, the accumulated body of information of Usenet is not lost: There are a number of WWW sites which archive and index Usenet articles. Thus you can retrieve posts which have expired (or perhaps had not even arrived) at your local news server.
Instead of asking a question in a news group, you can use the Usenet archives to retrieve articles which discussed your question long ago. By searching first and posting questions only if you find no answers, everyone will be much happier (this is valid for messageboards as well, btw :-)
Searching usenet first you will get an answer faster. You don't have to wait for your message to reach the far corners of the world. You don't have to rely on someone nice enough to write a reply. Perhaps those in the know aren't listening right now, even if they answered the same question in depth in the past. Searching offers an added advantage: you will also find the groups which have an interest in your topic. Do not underestimate it. So if you cannot find the answer in the archived messages, you will have a good starting place to ask questions.

Note that most Internet service subscriptions include access to a news server which you should be able to use for free. In order to make a profit, many news servers are not open to the general public (ahem... in theory :-) in fact in recent years they have been more and more commercially "bastardised" for exclusive use by "customers".
Keep in mind that usenet searching can be VERY time-consuming, and that some special skills are needed to perform it with a relative degree of effectiveness.
A final word of WARNING: there are a series of anonymity concerns that you should consider when posting on usenet. You may consider visiting sites with a section giving advices about anonymity, like mine :-)

Local search engines

Microsearching (local searching) means taking advantage of a plethora of tools that the web offers, from the specific search engines that most homepages clusters have, to the webrings search engines, the maillists search engines and the 'local' search engines like AOL and hitbox. Public 'free' counters are for instance a very useful resources when searching specific material, especially if they are categorized by the users themselves, like hitbox. Micro/local searching is very valuable in many fields: try your own "hobbyes" searches on any personal homepage cluster à la Geocities or xoom (or fortunecity, or thousand other "free" page providers), compare with the results you would get through the main search engines and you'll immediately understand what I mean.

Regional search engines

Regional search engines are even more important in a more and more international web. Take note that among the future www developments cross language information retrieval, or CLIR (which 'grossomodo' means querying in one language documents in many languages and implies a lot of semanthic algos) will become more and more important.

The searching value of "Subsidiarity" (a term which per se is nothing else than a synonym for feudalisation in the European Union's jargon) is extremely high. So-called "regional" search engines are incredibly important for fetching 'less biased' info. As you may expect, russian, chinese, indian, japanese and 'you-name-them' collections of datas have a completely different 'cut' vis-à-vis the stuff you can find in the euroamerican world. (This is of course true for all sort of data: files, images, sounds, films, software... the well-known motto being "always go ftp-fishing far from the copyright-obsessed euroamericans).
Anyway there are huge differences (in quality, 'cut' and bias) between 'european' and 'american' data colections as well.

The value of critical information is increasing, and such information is now - for the first time in the history of our race - available for everybody (once he knows how to search: unfortunately the volume of commercial rubbish you'll have to wade through is increasing exponentially).

The web is an Ocean of knowledge... about two centimeters deep. Finding rare snippets of information among tons of crap is getting more and more difficoult because on a more and more 'commercial' web you are compelled to wade through a lot of noise to get to the signal.
Most search engines DO NOT index the most interesting parts of the web: they index commercial over educational sites, 'popular' sites (read sites loved by the zombies) over relatively unknown sites and US sites over European sites... Well you should at least try to compensate this last disadvantage using search engines located outside the States.

Do not underestimate the task of a seeker, though: even mastering the 'subsidiariety searching' lore, and knowing some elementary microsearching techniques is not enough to be a good seeker: in order to search effectively you must also know some more 'broad' 'reversing techniques', else you'll never be able to build your own bots.

««««« Back to paris4.htm ««««« »»»»» Forward to paris6.htm »»»»»