http://www.open-search.net/Blog The open-search blog en-us Copyright © 2010 by contributing authors TWiki Administrator [webmaster@open-search.net] TWiki Administrator [webmaster@open-search.net] TWiki http://www.open-search.net/Blog http://www.open-search.net/Blog/BlogEntry25 <a class="natExternalLink" target="_blank" href="http://www.open-search.org">Open-Search</a> was invited to participate in the <a class="natExternalLink" target="_blank" href="http://www.janvaneyck.nl/0_2_3_events_info/arc_07_quaero_thinktank.html">Forum on Quaero</a> at the Jan van Eyck Acadamie in Maastricht, September 29 and 30, 2007. The purpose of the forum was to question and investigate the European intentions to build a search engine and, broader, to investigate the cultural, political, and philosophical issues related to information search and access. It turned out to be a critique on centralized search engines and a plea for systems like Open-Search: decentralized, open and privacy respecting. Following is an elaborate report and my impression of the two day forum on Quaero.<p /><div class="twikiToc"> <ul> <li> <a href="#Quaero"> Quaero</a> </li> <li> <a href="#Eight_privacy_protecting_demands"> Eight privacy protecting demands</a> </li> <li> <a href="#The_absence_of_Quaero"> The absence of Quaero</a> </li> <li> <a href="#Search_engines_their_borders_and"> Search engines, their borders, and appropriation of the image</a> </li> <li> <a href="#Quaero_Uncorporate_Virtual_Terri"> Quaero Uncorporate &amp; Virtual Territories, Real Borders</a> </li> <li> <a href="#Political_Algorithms_Value_Sensi"> Political Algorithms: Value-Sensitive Design</a> </li> <li> <a href="#So_what_is_a_crawl_scrape_index"> So what is a crawl / scrape / index?</a> </li> <li> <a href="#Maps"> Maps</a> </li> <li> <a href="#Blogging_and_searching_as_Self_M"> Blogging and searching as Self-Management under Communicative Capitalism</a> </li> <li> <a href="#Do_engines_have_politics_Do_poli"> Do engines have politics? Do politics have engines?</a> </li> <li> <a href="#The_semantic_web_with_the_use_of"> The semantic web with the use of a universal ontology or a folksonomy?</a> </li> <li> <a href="#Open_Search"> Open-Search</a> </li> <li> <a href="#Freedom_of_Expression_and_Search"> Freedom of Expression and Search Engines</a> </li> <li> <a href="#The_final_discussion"> The final discussion</a> </li></ul> </div> <p /> <h2><a name="Quaero"></a> Quaero </h2> <p /> Quaero is a [...] research and development program which has the goal of developing multimedia and multilingual indexing and management tools for professional and general public applications (such as search engines) (<a class="natExternalLink" target="_blank" href="http://www.aii.fr/upload/flb/90/Dossier_de_presse_1146047080747.pdf">1</a>). Initiated by the French ,Quaero aimed to be a European backed project but in 2006 Hartmut Schauerte, a state secretary within the [German] Ministry for Economics and Labour, announced during the IT-Gipfel summit in Potsdam that a German consortium has put together a semantic search project called Theseus that will be distinct from Quaero (<a class="natExternalLink" target="_blank" href="http://www.heise.de/newsticker/meldung/82708/from/rss09">2</a>). The main source of disagreement was the format of the search engine, with German engineers favoring a text-based search engine and the French engineers favoring a multimedia search engine. Many German engineers also balked at what they thought was becoming too much of an anti-Google project, rather than a project driven by its own ideals (<a class="natExternalLink" target="_blank" href="http://www.iht.com/articles/2007/01/02/business/search.php">3</a>). <p /> <h2><a name="Eight_privacy_protecting_demands"></a> Eight privacy protecting demands </h2> <p /> The first presentation at the public forum on Quaero was by Michael Zimmer, who did his Phd on <a class="natExternalLink" target="_blank" href="http://www.michaelzimmer.org/dissertation">The Quest for the Perfect Search Engine: Values, Technical Design, and the Flow of Personal Information in Spheres of Mobility</a>. In his talk Michael explained the goals of Google by using some of their quotes: "process and understand all the information in the world" and "understand exactly what you mean and give back what exactly what you want". In search engine talk this would be called 'perfect reach' for the former and 'perfect recall' for the latter. Google summarizes this when they state that they want to be "like the mind of God". Michael went on to say that personal data which was previously obscure is now searchable, and that Google collects as much information as possible about the searcher through server logs, cookies and user accounts. However, Google does not only have a search engine but also a plenitude of other product and services, of which the information is of course combinable with that gathered by offering search. The perfect search thus promises breadth, depth, efficiency, and relevancy but you lose 'security via obscurity' and it involves the widespread capture of personal and intellectual information. Michael Zimmer calls this the 'Faustian Bargain'. Quaero's perfect search would involve and combine sound, image and video search. To show one of the dangers of perfect multimedia search, Michael showed a series of pictures with automagically identified faces. The most salient of those was the one where protesters of a surveillance protest were algorithmically identified. After showing us a lot of threats of perfect search, Michael Zimmer argued for value-conscious design and stated <a class="natExternalLink" target="_blank" href="http://michaelzimmer.org/2007/09/29/privacy-and-quaeros-quest-for-the-perfect-search-engine-threats-and-opportunities/">eight privacy protecting demands</a>. (<a class="natExternalLink" target="_blank" href="http://michaelzimmer.org/files/Zimmer%20Quaero%20slides.pdf">4</a>) He called upon all people from Quaero in the audience to really think about this. Unfortunately nobody from Quaero attended the conference … <p /> <h2><a name="The_absence_of_Quaero"></a> The absence of Quaero </h2> <p /> In the conference booklet was an email discussion between the organization and the director of the French Quaero® program. The director tried to convince the organization not to use the name Quaero because 'The event announced by Jan van Eyck mentions the Quaero program and uses the Quaero name without endorsement by the Quaero program partners'. 'Quaero aims – just – at a series of industrial and technological objectives', and therefor he was also not interested in the forum because he has 'no authority on cultural, political and philosophical aspects of search engines'. He would 'rather leave these aspects to the people with strong convictions in these areas'. And that was a pity because according to the organization 'One could see this Forum as the democratic counterpart to a strategy session' and for a democracy to work you have to hear different opinions. <p /> <h2><a name="Search_engines_their_borders_and"></a> Search engines, their borders, and appropriation of the image </h2> <p /> Although the industrial and technological actors in Quaero were absent, we continued our democratic obligation to investigate the politics of search engines with a presentation by <a class="natExternalLink" target="_blank" href="http://www.debalie.nl/persoon.jsp?personid=17941">Florian Schneider</a> (no, not the guy from Kraftwerk). Florian had a very interesting though difficult talk. In the following paragraphs I tried to reproduce Florian's arguments from my notes. I hope the idea gets through <p /> Florian investigated the circularity between knowledge and power and analyzed the correspondences between migration machines (border control or management), as an ongoing attempt to control freedom of movement on the one hand, and how it corresponds to management and emergence of new information regimes on the other. <p /> Florian sees search engine technology as an attempt to image knowledge. In the digital world an image is understood as an exact replication of any kind of information. A search engine however is not a backup but a snapshot / image of some of the information. The question then is who decides when, how, and what kind of information is stored. <p /> When using a search engine we experience it as something that produces an image. The result of a query is a meta design. The results are reduced to a matter of graphical design; it needs to be functional and aesthetic at the same time. The search engine has a clearly defined purpose with a manufacturing process that is hidden to the user. The crucial point is the algorithm (the manufacturing), which is hidden to user and consists of the management of the algorithm. <p /> The search engine is sort of an interface for what one would call a mirror by e.g. googling yourself or search engine optimization. This constitutes the paradigm of imaginary order, a subject is permanently captured and formed by its own image. Search engines thus deal in imaginary property. But what does it mean to 'own' such an image? The question is not so much who is owning the image but what it 'means' to own an image. This actually addresses privacy to theoretical reconsiderations of what an image is in an area of digital reproduction. In a purely Kantian sense: it's a matter of imagination. <p /> The manufacturing (the algorithm) is involved in an act of determining space and time, as a rule of production. It is defined by the way in which the search engine is built and installed, and forms the objects that are supposed to be spidered, crawled, and stored. Search engines then appropriate this image as their property. The conservation of this image is subject to invariant operations – of the algorithm. The images might be results of structures for imaginations. Images then manage the violations rather than prevent them from the environment. The question of ownership has turned into executions in real-time. It's not only facial recognition – solutions are executed in real time. <p /> A question which pops up is then: Who has the right to read write and access this image / property. Google claims ownership but is not owning up. Now there is a quite dramatic shift in the concept of property. <p /> As the migrant new economy was build on illegalized migrants, according to Florian Schneider, web 2.0 is based on the imaginary revenue. This also means legalizing content. Valorization is built on immediate appropriation of content and building imaginary content from it. The arrival of search engines clearly corresponds with the migrant wave in the 20th century. The post-modern border machine appears as a system of control with constant modulation of rights that are transformed in a set of permissions. The border thus becomes a matter of performance. <p /> The function of post-modern border management is to illegalize migration. Migrants are allowed in as long as it is 'useful labor force'. When translated to search engines, they operate in corresponding vain. The question is about imaginary property and it's valorization. It is important to see their borders, e.g. Google only indexes 10 to 15% of the 'surface web'. But how can you border the WWW? The question what is included / excluded remains. Is the WWW in principle limitless or do you want to border the territory so you can assure information is found? Behind the surface web there is a deep web, and before a query happens there is a certain quadrage of imaginary that defines territory or maps it out. Every search engine has to manage what is on and what is of. Censorship appears as a constant modulation in real time; this ranges from China to ignoring everything that is not text. <p /> Searching thus is an act of imagination. Finding is not the opposite of missing. We rewrite imagination in the same way as we rewrite the future – when you actually find something. There is a delicate relationship between searching and measuring. Searching needs a measurement on what is accessible and therefor <em>could</em> be found. "I am not searching what i know that i need to find. I am searching for what i do not yet know that i want to find." The success of a search engine can be explained as an abstract machine which processes and colonizes (our power to) imagination. <p /> We need to form new forms of autonomy in new forms of society of control. Search engines are misfunctional, success is build on misuse, search is appropriating the right to misuse. p2p access to cultural heritage is already available and re-appropriated by the mass. The problem of Europe and the European search engine is controlling and managing of what can be imagined as European. <p /> Clearly, Florian Schneider's notion of border, the indexable, and the imaginary are very important concepts in an attempt to understand the politics of search engines. <p /> Richard Rogers asked how this image could be described. You got the original, the copy, and … the third object? How does one think about and deals with this 'third object'? How can you justify the ownership of it? <p /> <h2><a name="Quaero_Uncorporate_Virtual_Terri"></a> Quaero Uncorporate &amp; Virtual Territories, Real Borders </h2> <p /> <a class="natExternalLink" target="_blank" href="http://www.metahaven.net">Metahaven</a>, designers by origin, showed various newly designed logos for Quaero, thereby wondering what the characteristics of a good search engine should be. The second part of the presentation was about a sketch of a layered 3D interface that visualized the isolation within search results. Unfortunately I was very tired and I did not pick up much from their presentation. My apologies. <p /> <h2><a name="Political_Algorithms_Value_Sensi"></a> Political Algorithms: Value-Sensitive Design </h2> <p /> Tsila Hassine showed us the Image Tracer and Schmoogle. <p /> The <a class="natExternalLink" target="_blank" href="http://www.geuzen.org/tracer/">Image Tracer</a> is a collaborative project between Tsila Hassine and <a class="natExternalLink" target="_blank" href="http://www.geuzen.org">De Geuzen</a>. It evolved out of their interests in media images and the way their significance and presence fluctuate in the ecology of the world wide web. It is built on top of Google Images and grabs images each day for a particular query. Image density builds and wanes through time, reflecting how long or little, a jpeg, gif or png has been online and how its rank shifts over time. Opacity denotes the consistent presence and position of an image through multiple searches, while more transparent or murky images have appeared for less time or moved in rank. (<a class="natExternalLink" target="_blank" href="http://www.geuzen.org/?page_id=89">5</a>) <p /> <a class="natExternalLink" target="_blank" href="http://www.geuzen.org/cgi-bin/shmoogle/shmoogle_form.cgi">Schmoogle</a> also lives on top of Google. For each query it fetches all 1000 results which are returned by Google. Instead of displaying it in the order (with the rank) Google gives them, they randomly reorder the results. This way they question and critique the ranking algorithm of Google. What 'if the result you were looking for was hiding in page 53?' <p /> <h2><a name="So_what_is_a_crawl_scrape_index"></a><a name="So_what_is_a_crawl_scrape_index_"></a> So what is a crawl / scrape / index? </h2> <p /> At the end of the first day Ingmar Weber did some hands on demonstrations of various aspects of building a search engine. By using the free software program <a class="natExternalLink" target="_blank" href="http://www.httrack.com/">HTTrack</a> Ingmar started to crawl some websites, excluding the tld .org. This way he showed us the importance of starting-points, the design decisions of how and what to crawl (depth, outdegree, inclusion/exclusion), the resulting found websites and how a reversed index is built – everybody has encountered it as the word/concept index in a book. Ingmar then went on to demonstrate other kinds of search engines such as the p2p search engine yacy and the human search engine chacha.com. The latter was very funny, as Joris van Hoboken told us that most of the operators actually use Google to find information. <p /> <h2><a name="Maps"></a> Maps </h2> <p /> The next day started of with <a class="natExternalLink" target="_blank" href="http://utangente.free.fr/index2.html">Bureau d'etudes</a> who presented a beautiful map on Quaero. Unfortunately the map is not online (yet?) so I will not discuss it now, as it would not make any sense. <p /> <h2><a name="Blogging_and_searching_as_Self_M"></a> Blogging and searching as Self-Management under Communicative Capitalism </h2> <p /> On <a class="natExternalLink" target="_blank" href="http://netzmedium.de/2007/10/02/questioning-quaero/">netzmedium.de</a> a good summary of Jodi Deans' presentation has been written, so I will quote it here: <p /> <blockquote> <a class="natExternalLink" target="_blank" href="http://jdeanicite.typepad.com/">Jodi Dean</a> presented an interesting comparison between blogs and search engines. From a psychoanalytical perspective, she interpreted both phenomena as an answer to anxieties associated with a chaotic and unstructured information space. Based on algorithmic ranking, search engines promise an objective ordering of this space and introduce an element of purity and immaculacy. Blogs, on the other hand, act as guides in the information space by presenting a strictly individual view. Dean portrays them as &#8220;technologies for managing distributed subjectivities&#8221;. </blockquote> <p /> <h2><a name="Do_engines_have_politics_Do_poli"></a> Do engines have politics? Do politics have engines? </h2> <p /> Next up was <a class="natExternalLink" target="_blank" href="http://www.govcom.org">Richard Rogers</a>. He had 4 case studies at hand. <p /> First was the demise of the expert libraries. Using the <a class="natExternalLink" target="_blank" href="http://www.waybackmachine.org">wayback machine</a> Richard found out that the directory disappeared from the web. Google, in its early days, had one accessible on the front page. Through the years however you had to click through more and more until it finally disappeared. <p /> Second was the <a class="natExternalLink" target="_blank" href="http://mastersofmedia.hum.uva.nl/2007/09/28/911truthorg-dissappears-from-google/">disappearance of 911truth.org from Google's result page for the query '9/11'</a>, while it used to have a prominent place in the top 5 in the months before. Although there is no clear reason about the disappearance (is it a result of manual censorship? Is it a violation of Google's term and conditions by e.g. SEO?), it does not make it less disturbing. Because Google's ranking is a secret, there is no way of finding out why it happened – which is one of the most problematic aspects of the current search engines in an era where access to information is largely controlled by one company. <p /> After giving us an insight into the politics of search engines, Richard Rogers turned to the question whether politics have engines. To illustrate this he showed two examples, researched by the <a class="natExternalLink" target="_blank" href="http://dmi.mediastudies.nl">Digital Methods Initiative</a>. Quoting from their <a class="natExternalLink" target="_blank" href="http://wiki2.issuecrawler.net/twiki/bin/view/Dmi/MethodsByTheme#Source_Distance">source distance project</a>: <blockquote>The focus here is on the prominence of particular sources in different spheres (e.g. blogosphere, news sphere, images), according to different devices (e.g. Google, Technorati, del.icio.us). For example, how far are climate change skeptics from the top of the news? For comparison sake, how far are they from the top of search engine returns? The answer to this and similar 'cross-spherical' inquiries goes a way towards answering the question about the quality of old versus new media.</blockquote> <p /> By searching climate change skeptics in the top 100 Google sites for the query 'Climate Change', Richard Rogers noted that "There is distance between the skeptics and the top of the search engine returns. [...] few skeptics appear on the Websites of the top ten results in Google. When they do appear (Patrick Michaels, Steven Milloy) their resonance is not particularly resounding. [...] From the visualization one is able to see the 'skeptic-friendly' sources, realclimate.org and, to a lesser extent, climatescience.gov stand out as skeptic-friendly. Sourcewatch also is prominent, albeit as a progressive watchdog group 'exposing' the skeptics. Remarkably, news sites, generally speaking, do not mention the climate change skeptics by name. Whilst news watchers and listeners may have the impression that 'uncertainty' in the climate change 'debate' continues in a general sense (as opposed to, say, in more specific, scientific sub-discussions), 'uncertainty' appears to be discussed without resort to the well-known, or identified, skeptics." (<a class="natExternalLink" target="_blank" href="http://wiki2.issuecrawler.net/twiki/bin/view/Dmi/ClimateChangeSkeptics#Findings">6</a>) <p /> His last example concerned which animals associated with climate change, <a class="natExternalLink" target="_blank" href="http://wiki2.issuecrawler.net/twiki/bin/view/Dmi/IssueImageAnalysis">issue animals</a>, were represented in different spheres. It turned out Google, Google News, and Technorati present a whole different 'image' about which animals are referred to a lot in the climate change debate. <p /> <h2><a name="The_semantic_web_with_the_use_of"></a> The semantic web with the use of a universal ontology or a folksonomy? </h2> <p /> Florian Cramer declared all attempts at creating universal classification schemes (ontologies) futile. To say rephrase his arguments in the words of Theodor Nelson, who coined the term Hypertext: <blockquote>Last week's categories, perhaps last night's field, may be gone today. [...] <em>The</em> categories are chimerical (or temporal) and our categorization systems must evolve as they do. Information systems must have <em>built in</em> the capacity to accept the new categorization systems as they evolve from, or outside, the framework of the old. (<a class="natExternalLink" target="_blank" href="http://portal.acm.org/citation.cfm?id=800197.806036">7</a>), emphasis by the author. </blockquote> <p /> Florian illustrated this with historical material, but his real target was the German <a class="natExternalLink" target="_blank" href="http://theseus-programm.de">Theseus initiative</a>, which seeks to develop AI-like tools for &#8220;automated logical deduction&#8221;. Florian sees folksonomies as more promising then ontologies. <p /> <h2><a name="Open_Search"></a> Open-Search </h2> <p /> My presentation can be found on <a href="http://www.open-search.net/Opensearch/QuaeroTalk" target="_top">http://www.open-search.net/Opensearch/QuaeroTalk</a> In this talk I explained what the problem with the current search engines is, according to Open-Search: because the control over information is central and nobody can look into their algorithms or decisions, it is susceptible to censorship, manipulation and profiling. Getting out the central part and making it open would solve a lot of problems. We believe this is possible through an open source p2p project. Of course there are a lot of challenges as well: spam, efficient query propagation, etcetera, but most of all we haven't had time to build a community and we do not have the time nor money to put a lot of continued effort in the development of Open-Search. <p /> Richard Rogers had some constructive comments for Open-Search. <p /> <strong>Manipulation vs. spam</strong> <p /> <em>Normally search engine companies argue that their logics are closed and not transparent, because if they were open, the search would be open to spamming and manipulation. In the presentation, you have that you strive to be open about the logics, but have no answer yet for spam. search engine companies already have an answer. Yours?</em> <p /> My first answer would be the one from <a class="natExternalLink" target="_blank" href="http://wiki.apache.org/nutch/FAQ#head-58c08bdc65a929eb4ba2688827938685963a817c">Nutch</a>: <blockquote>Search engines work hard to construct ranking algorithms that are immune to manipulation. Search engine optimizers still manage to reverse-engineer the ranking algorithms used by search engines, and improve the ranking of their pages. For example, many sites use link farms to manipulate search engines' link-based ranking algorithms, and search engines retaliate by improving their link-based algorithms to neutralize the effect of link farms. <p /> With an open-source search engine, this will still happen, just out in the open. This is analagous to encryption and virus protection software. In the long term, making such algorithms open source makes them stronger, as more people can examine the source code to find flaws and suggest improvements. Thus we believe that an open source search engine has the potential to better resist manipulation of its rankings.</blockquote> <p /> A second answer would be that because people can devise their own ranking plug-in's with different ranking schemes, it would be harder to manipulate and spam them. <p /> <strong>Personalization vs. tribalism</strong> <p /> <em>With plug-in's people can choose their own ranking algorithm, and thus can personalize results. Classically, there is the idea that shared media experiences (people seeing similar things) make society — i.e., commonality in exposure but different views about what it all means (cf. community, which is different). Personalization, however, creates tribes, hate groups privileging other hate sites and their results, dictatorial regimes privileging 'official' sites. Search engines claim a kind of egalitarianism (indexing the 'whole web'), but then ranking according to authority. Yours?</em> <p /> Search engines might claim a kind of egalitarianism but this is not because of indexing the whole web. <a href="http://www.open-search.net/Blog/BlogEntry25#Search_engines_their_borders_and" class="twikiAnchorLink">Florian Schneider's presentation</a> for example has questioned the 'borders' of the database and index of a search engine. There is also the tendency of search engines to localize and personalize, e.g. iGoogle serving results which are specifically tailered to you (recall also perfect recall from <a href="http://www.open-search.net/Blog/BlogEntry25#Eight_privacy_protecting_demands" class="twikiAnchorLink">Michael Zimmer's talk</a>), or a specific country version of a particular search engine. One might also be tempted to talk about an 'objective' ordering but there are enough <a href="http://www.open-search.net/Opensearch/WhyOpenSearch" target="_top">examples</a> that prove the current ranking schemes of the big search engines are not that objective at all. <p /> The shared media experience that makes society is already diffusing by the broading offer of information channels via internet and interactive television. Of course personalization of the current engines also adds to the decline of the shared media experience. <p /> One might think a p2p engine like Open-Search has the danger of tribalism and segregation because your peers might not 'know' as much as somebody else's peers. At the level of the search engine however, there are no peers. To the open-search engine, the p2p layer is an abstraction of a generic storage device, a database. It is assumed that each peer will have access to the entire database at all times, regardless of which peers connect to which peers. We are not using a flooding model, as does e.g. Soulseek. <p /> <h2><a name="Freedom_of_Expression_and_Search"></a> Freedom of Expression and Search Engines </h2> <p /> <a class="natExternalLink" target="_blank" href="http://www.jorisvanhoboken.nl/">Joris van Hoboken</a> has been interested in Open-Search from the beginning. I invited him to do a presentation with me. Joris is a Phd student whose research concerns regulation problems in search engines. On the public forum on Quaero he gave a short presentation on the implications for freedom of expression for search engine law and government involvement in particular. There are three actors on which you can focus regarding freedom of expression and search engines: the search engine, the information provider, and the user. From the user's perspective "Freedom of expression in the context of Internet search implies that a search engine has to make its index and ranking machinery openly available for its end users of various kinds to support free (not as in gratis) access to information. (user becomes the real search engine editor)." <p /> <h2><a name="The_final_discussion"></a> The final discussion </h2> <p /> Two days of conference raised a lot of critique on privately owned centralized search engines. Questions about the borders of indexed information, access to and control over information, objectivity in ranking, and concerns about privacy led the participants to favor a system like Open-Search. Open-Search and similar systems however, are in its infancy – a lot of problems still need to be solved technically and politically. <p /> Jodi dean has a <a class="natExternalLink" target="_blank" href="http://jdeanicite.typepad.com/i_cite/2007/09/still-more-quae.html">great post</a> about the final discussion, which I quote here: <p /> <blockquote> The conference closed on a shockingly optimistic and unified note [...] Why such a great note? Likely because of three crucial interventions that resulted in the sense of a politics around P2P/open search, a politics that would assert the failures and limits of a search (foregoing the knowledge claims of a god/subject supposed to know and thus attempting to divert transferential investments into authority) engine. This could seem counter-intuitive. Who wants a search engine that doesn't claim to be reliable, thorough, and objective? Perhaps those who recognize that there is no such search engine and take responsibility for this limited, partial, and shared knowledge. The three interventions: <p /> Florian Schneider directly politicized open search. It had been implicit in the discussion, but he made it explicit and political. He also used the term exodus as a kind of movement constitutive of the political. <p /> Daniel van der Velden rendered exodus as more of a decision, and thus as requiring a kind of awareness or even consciousness (which makes the projects demonstrating the failures and interventions of google--which doesn't live up to its anti-evil ideals--all the more important). <p /> For me, these two ideas seemed to conflict. There is hardly an exodus from google, rather the opposite--the problem is the way people flock to it, rely on it--like Wal-Mart and McDonalds. <p /> But Florian Cramer traversed this dilemma, refused the false choice--and gave a rousing speech that all agreed marked an appropriate end point for this phase of the conversation. So, he said that exodus is a metaphor, with limits, and that exodus can't mean here anything like a kind of neo-luddite movement/moment. And, he refused the demand for an image. More specifically, he said that the very question of 'what would a European search engine look like' should be eliminated (for good techie reasons involved API, available public interface). There isn't one answer, one image, one model. <p /> This fits well with the theme of the imaginary that I took from the conference. It accepts neither the imaginary, nor calls for a symbolic (name, authority, law). It traverses these with a different kind of accountability (clearly not quite ready for release, but maybe soon in beta). Maybe this is something like an act in information politics. <p /> And, if information is value and search engines add, create, and arrange value, what sort of value would P2P search engines create? </blockquote> <p /> To me it was a great forum. A lot of interesting intellectuals shared their thoughts and theories about search engines. It gave me a lot of inspiration and deepened my conceptualization about the politics of search engines. I hope that this group of researchers goes on with the problematique and that follow-up conferences are organized. Further discussion of the pitfalls and benefits of a system like Open-Search is encouraged to take place in the comments and on the <a href="http://www.open-search.net/Opensearch/MailingLists" target="_top">mailing lists</a>. <p /> <p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=quaero, information politics, search, google" rel="tag">quaero, information politics, search, google</a> ErikBorra 2007-10-03T13:38:00Z UsersChat ErikBorra http://www.open-search.net/Blog/BlogEntry24 In a well documented case of censorship, google again shows its true meaning of "don't be evil".<p />In a university blog, open-search suspect Erik Borra <a class="natExternalLink" target="_blank" href="http://mastersofmedia.hum.uva.nl/2007/09/28/911truthorg-dissappears-from-google/">writes about the decline and final demise of 911truth.org</a>. in google search results. On 911truth.org, it has become apparent that <a class="natExternalLink" target="_blank" href="http://www.911truth.org/article.php?story=2007092200814732">several readers have noticed the disappearance of 911truth.org from google</a>. <p /> Google attracted more bad press, with <a class="natExternalLink" target="_blank" href="http://it.slashdot.org/it/07/09/27/167249.shtml">the recent gmail vulnerabilities</a> also posing a serious threat to users of google services.<p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=google,news" rel="tag">google,news</a> KoenMartens 2007-09-28T12:58:00Z UsersChat ErikBorra http://www.open-search.net/Blog/BlogEntry23 Open-search is asked to participate in a public think tank on the politics of the search engine. The forum is brought together by Quaero, a consortium of technology firms and research labs working together on multimedia and web search projects.<p />From the <a class="natExternalLink" target="_blank" href="http://www.janvaneyck.nl/0_2_3_events_info/arc_07_quaero_thinktank.html">flyer</a>: <p /> "QUAERO: isn't that the search engine that former French president Jacques CHIRAC declared to be the EUROPEAN challenge to Google? A pub-lic alternative to Silicon Valley-born commercial search engines, funded by the French state, in service of the PUBLIC GOOD, in the true tradition of the GRAND PROJET? An INFORMATION MACHINE capable of reclaiming European LANGUAGE and intellectual HERITAGE in the age of GLOBALIZATION? <p /> NO. Quaero is the name of a CONSORTIUM of technology firms and RESEARCH LABS working together on multimedia and WEB SEARCH PROJECTS. It is a STATE-SPONSORED effort to stimulate PRIVATE French technological competitiveness. <p /> According to Franc,ois BOURDONCLE, one of its participating developers, `Quaero is definitely not a project to build a web search engine, it is a project to make significant advances on the handling and indexing of multimedia content. It is completely out of the question to build a new, state-owned, state-operated, or even state-funded search engine. Only the R&amp;D around these cutting-edge multimedia indexing technologies are in the scope of the project.' <p /> But still, the issues that the idea of Quaero has raised – since its public launch by the former French president – constitute a formidable challenge. Internet search engines are political projects proper if only because they give and take power; they represent science, technology, (trans)national politics, private enterprise, culture, territoriality and language in ever different combinations. They are also social spaces. Internet search, the indispensable public tool that allows one to survey the ever increasing web, is currently in the hands of only a few global players, to whose private interests its setup corresponds." KoenMartens 2007-09-14T10:26:00Z Project KoenMartens http://www.open-search.net/Blog/BlogEntry22 Koen's has written an article about Google as the Microsoft of the 21st century. It can be found <a class="natExternalLink" target="_blank" href="http://www.livre.nl/achtergronden/internet/google-is-de-microsoft-van-de-21ste-eeuw-24052007.html">here</a> (in Dutch). Another interesting (Dutch) article about Google, by Michiel Leenaars from gridnet.nl is <a class="natExternalLink" target="_blank" href="http://www.webwereld.nl/articles/46452/google-s-verklikkernetwerk.html">Google's verklikkernetwerk</a>. <p /> Apart from developing (see our previous post) we are also very busy giving presentations about open-search. Last week at <a class="natExternalLink" target="_blank" href="http://www.versgeperst.info">versgeperst.info</a> Tomorrow at <a class="natExternalLink" target="_blank" href="http://www.isoc.nl/activ/2007-ALV-juni.htm">ISOC.nl</a> and Monday at the <a class="natExternalLink" target="_blank" href="http://www.hosc.nl/">Holland Open Source Conference</a>. <p /> ErikBorra 2007-06-05T16:00:00Z UsersChat ErikBorra http://www.open-search.net/Blog/BlogEntry21 closing in on the holland open.<p />Here's some technical details that we're gonna reveal in a prototype at the open-holland conference: <p /> <ul> <li> the <em>oscar</em> peer to peer layer uses a <em>gossip-engine</em> to multiplex traffic over a single TCP port. – it also allows to tunnel traffic eg. via HTTP to circumvent firewall, without much performance penalty. </li></ul> <p /> <ul> <li> in this prototype the data-signing is not part of open-search, but the P2P layer: Every p2p-packet sent or received contains a RSA-SHA; (yielding ~35% overhead at the moment) – the signature will identify the host responsible for providing the information (not the ID of the Source of the Information ie. crawler or user) </li></ul> <p /> <ul> <li> only "bad clients" not "annoying users" can be <em>banned</em> and the ban affect the local open-search-client only, not the whole network. </li></ul> <p /> <p /> <ul> <li> we're testing &amp; debugging a crawler with a local perl::DBD database (currently sqlite) – The SQL interface can be replaced with a p2p layer and is currently used to develop and debug the latter at the moment. </li></ul> <p /> <p /> <ul> <li> privacy can be archived as an additional Feature. A user can select </li></ul> any content of the p2p-network using it's identifier and checksum to sign it's content with a private key. A user's signed content can be referenced ,like any other content in the p2p-network (you can search for it) and it's up to the end-user-application to make use of it. <p /> <p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=Development" rel="tag">Development</a> RobinGareus 2007-06-05T13:50:00Z Development RobinGareus http://www.open-search.net/Blog/BlogEntry20 <p />with the current endeavor to plug a distribted hash table below the open-search-client, here is the protocol used to for communication between the open-search core and the p2p-layer. <p /> the p2p engine will run as separate process, and communication is done via a local TCP socket. for testing and develpment we use two simple perl-scripts (contrib/hatsim) and it will act as a p2p-db simulator. <p /> the protocol is yet rather simple: <p /> we use "++" as separator for ASCII-text encoded arguments. the format is <code>COMMAND</code> ++ <code>opt-args</code> ++ <code>HASHED-KEY</code> ++ <code>XML-VALUE</code> ++ <p /> example perl-printf's: <pre> "STORE"."++"."&lt;p2p-opt-args&gt;"."++"."&lt;HASH_FOR_KEYWORD:&gt;"."++"."&lt;XML-LIST-OF-URL_HASH&gt;"."++"; "STORE"."++"."&lt;p2p-opt-args&gt;"."++"."&lt;HASH_FOR_URL&gt;"."++"."&lt;XML-DATA_FOR-URLS&gt;"."++"; "LOOKUP"."++"."&lt;p2p-opt-args&gt;"."++"."&lt;HASH&gt;" # &rarr; returns either &lt;XML-LIST-OF-URL_HASH&gt; or &lt;XML-DATA_FOR-URLS&gt; </pre> <p /> this very simple protocol will be enough for a protptype test implementation. – the &lt;p2p-opt-args&gt; are yet unused but we forsee a couple of features that would need to pass extra information to the p2p-engine.. <p /> currently packets are gpg signed when they pass tru the p2p layer, using a host-key rather than individual crawler signature. – also the format of the xml-data is yet unknown to the the p2p layer. once search-filtering is optimized and parts of it can be done at network-layer, there'll be a new command <code>STORE_IDX</code> and a separate LOOKUP functions for different return types. – it remains to be discussed whether priotity-tags are implemented as optinal-argument or separate command. <p /> <p /> RobinGareus 2007-05-23T13:36:00Z BlogDev, Development RobinGareus http://www.open-search.net/Blog/BlogEntry19 peer to peer to peer<p />Teased by our 2nd Workshop, Marco Fiscato has jumped aboard our team of developers. His expertise in peer to peer networks and his implementation of a chord based p2p layer will be of great help for prototyping p2p open-search. <p /> This weeks Todo includes a simulator that implements the distributed-database query interface: the simulator (dubbed hatsim) allows to "replay/simulate" crawler &amp; user activity for the P2P software and emulates a dummy p2p to be used for client development. <p /> <p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=blog development news" rel="tag">blog development news</a> RobinGareus 2007-05-18T11:50:00Z BlogDev, Project RobinGareus http://www.open-search.net/Blog/BlogEntry18 So, last friday I had to visit google NL hq for business related matters. It is located near station Amsterdam Zuid, in one of the new buildings in the area that is to become the 'south axis'. On ground level, shadow reigned. High-rise buildings obscure the sky, and i doubt the sun will ever reach down to the entrance of the building. At the 15th floor, i was greeted by a NDA that wanted to be signed. After doing so i was allowed to enter the realm of google NL.<p />It was not very impressing, although the facilities sure are nice and the skippy balls looked inviting. However, as I later learned there are no engineers in this branch of google, just sales and marketing people. <p /> Putting the google bashing aside for a moment, their motto does sound noble: “organize the worlds' information and make it universally available''. And when they say information they mean <strong>all</strong> information. They started out with web indexing, but have moved far beyond that. Google Earth, although scarry, also is a wealth of geographical information. For example, I can just look up the layout of tramway depots in my city, or check out railyard layouts around the world. With just a few clicks and drags. <p /> Their move into the audio-visual world is also an example of making information available to the world. The internet was once, long before youtube, touted as the replacement for traditional broadcasting models. Everyone could now send out their information. But it wasn't before sites like youtube that indeed everyone did publish their videos. Now, with google video, youtube and the likes, putting your own video online has become so easy that the masses are jumping on it. And google stores and streams this information, free of charge, no strings attached (or are there?). <p /> The problem with all this is that not only do they collect this information, and make it available. They also collect information about who uses what information when (and if possible, with what purpose). Basically, they do this to to improve their services. Let us assume that this is indeed the case. Then there still is a big privacy issue here. Once google has these databases that they can use to build detailed profiles of your interests and behavioural traits, its use can be extended. For example, the US government could claim this database for 'homeland security'. Or (remember the AOL debacle), the database or a part thereof could accidentally leak and become available on the internet. Or hackers could gain access to the system and extract information about you. Perhaps to create a false identity, abusing your profile. <p /> Their move into online applications, such as <a class="natExternalLink" target="_blank" href="http://docs.google.com">google docs &amp; spreadsheet</a> is especially notable in this respect. Although at first sight, it sounds nice. You don't need openoffice.org or microsoft office on your computer, you just use your browser to do all your document editting. Store your documents on googles servers and be able to access them from any computer where you can browse the net. <p /> But several problems are apparent: first of all, what happens when your data gets lost? That this is not far-fetched has become apparent recently with the iGoogle mess, where users data was lost. Apparently, they were able to recover the data after-all. But all the same, it could have been lost forever. Imagine this happening to your documents! <p /> But of course, once your documents are on googles servers, they can use them to build an even more detailed profile on you. They can scan these documents, maybe these are letters you write to your insurance company, maybe these are letters to friends or your financial administration in a spreadsheet. All this is then open to the same dangers mentioned before: abuse by governments, identity-theft by criminals and what more. <p /> And if you are unfazed by all this, because you think the government wouldn't do such things, or that you have nothing to hide, consider what would happen when in 10 years there is a different government, with different priorities. Then you might all of a sudden be the 'criminal' they are looking for, and your online profile comes back to bite you!<p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=google" rel="tag">google</a> KoenMartens 2007-05-06T19:45:00Z KoenMartens http://www.open-search.net/Blog/BlogEntry17 <p />first of all: thanks to everyone participating in the open-search workshop! <p /> it's been less fruitful than expected but a very blooming workshop! Though we did produce some code fragments the major contributions were in discussion and the community building process. <p /> We made contacts with coders outside open-search, in particular we've found a candidate from the vrije universiteit to plug a P2P layer in the open-search client. We compiled and tested a first windows version at the workshop, though there is no package yet. <p /> We've received comments from designers &amp; layouters, and also some hands-on design experiments and brainstorm about the user-interface that are to be followed up… <p /> the road is still long, but the open-search path is on the verge to become a street! stay tuned. <p /> RobinGareus 2007-05-01T13:08:00Z Project RobinGareus http://www.open-search.net/Blog/BlogEntry12 With the second workshop, we released the alpha version of <a href="http://www.open-search.net/Software/WebHome" class="twikiLink">our search client</a>. Not much of the projected functionality is present, but it does provide the general framework that will be extended with various plugins. The <a href="http://www.open-search.net/Opensearch/SecondWorkshop" class="twikiLink">second workshop</a>, next to being the official release event of the client, is also the starting point of more community involvement.<p />Because that is what we need right now: people who care about the project and want to help out in making it a reality. Our paid developer Robin Gareus will be leaving in one or two months (unless we manage to secure further funding before that time), so the project will have to be able to sustain itself. <p /> Although we do need programmers, there are a lot of other tasks where we could use your help. If you are wondering where you can help out, check out the <a href="http://www.open-search.net/Opensearch/TechnicalTodo" class="twikiLink">TechnicalTodo</a> and <a href="http://www.open-search.net/Opensearch/TheoreticalTodo" class="twikiLink">TheoreticalTodo</a>. Also, make sure to subscribe to our discussion mailing list to keep up to date with current efforts. <p /> All in all, the workshop was a succes, despite the low attendance. It was almost a 30 degrees celsius in Amsterdam, not the best weather to sit inside behind a laptop screen. Thanks to all who did come regardless! KoenMartens 2007-04-28T14:55:00Z Project KoenMartens http://www.open-search.net/Blog/BlogEntry16 <p />we're currently preparing the talk and topic &amp; slides for the workshop… <p /> after the keynote it's gonna be a 3track workshop: <ul> <li> open software philosophy (legal issues; end-user participation; editing) </li> <li> web-development (open-search look &amp; feel and desktop integration) </li> <li> core-development (p2p bindings, search&amp;page ranking) </li></ul> <p /> we'll provide a framework to "plug in" code-prototypes that runs on osX and Linux. <p /> It should compile under windows with CYGWin but we do not have access to a MS operating system. Maybe also sth to try at/after the workshop. – we'll provide bootable gnu/Linux CDs for participants with a PC. <p /> more later..<p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=news workshop" rel="tag">news workshop</a> Main.RobinGareus 2007-04-17T18:00:00Z Project ErikBorra http://www.open-search.net/Blog/BlogEntry7 <p />In the wake of easter we are heading towards the 2nd workshop.. with no public software client it's gonna be a close call, but after the attached screen-shot we hope to announce an alpha version a week before the workshop. probably already April 19 2007. <p /> Last week has been rather distracting (installing python on our web-server, re-furbishing php and installing mantis on bug.open-search.net) – and the easter weekend turned out rather sunny which did not aid open-search development after all. – so without furter ado.. <p /> <div id="" class="imageFloat imageFloat_right " ><a href="/pub/Blog/BlogImages/os-sshot1.png" class="imageHref" title="os-sshot1.png"><img border="0" align="middle" src="/pub/Blog/BlogImages/_250___os-sshot1.png" alt="os-sshot1.png" width="250" height="192" longdesc="os-sshot1.png" /></a></div> <p /> <ul> <li> local-webserver: check </li> <li> cgi-interface: check </li> <li> AJAX-engine: partly </li> <li> design, layout &amp; language support: preliminary. </li> <li> serach-backend: faked local. </li> <li> crawler: independant (preliminary perl AND/OR python) </li> <li> P2P-lib: none – exeriments and simulations. (planning to use gnutella for the workshop) </li></ul> <p /> <p /> current ToDo: debug and document AJAX api (javascript, xslt); code a P2P API-wrapper (c, py-bindings) , renice twiki-blog (hammer). <p /> <p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=blog news" rel="tag">blog news</a> RobinGareus 2007-04-10T16:00:00Z BlogDev RobinGareus http://www.open-search.net/Blog/BlogEntry4 It has been about four months ago that we started this project off with the support of digital pioneers. So it is time to look back a bit, and look forward.<p />To start with, we underestimated the amount of work, as our current developer Robin already mentioned in <a href="http://www.open-search.net/Blog/BlogEntry3" class="twikiLink">his blogpost</a>. We planned on having a beta product ready by the end of april, but that seems unrealistic now. The next days a new planning will be determined, but the goal will still be to present a working product (albeit beta) by the end of our digital pioneers funding. <p /> Another thing we underestimated was the amount of time the volunteers (mainly Erik and me) have to put in to this project to make it a succes. We now finally have something of a project architecture coming together though, as this weblog exemplifies. This weblog also marks the shift of focus from setting up the internal structures to facing outward and getting the community interested. <p /> Another important step is the registration of the <a class="natExternalLink" target="_blank" href="http://sourceforge.net/projects/open-search/">sourceforge project space</a>, where we will shortly be putting online the first snippets of code. <p /> Financially, things are still on budget. That is, if we only look at the original budget. It has become clear that more paid development might very well be necessary. There is not enough of a community yet to take over the project after Robin's term ends. We therefore need to look into further funding (suggestions are welcome of course). <p /> So, summarizing. We have done a great deal of research and have invested in a solid project infrastructure. Next up is community building and securing continued funding.<p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=progress finance board" rel="tag">progress finance board</a> KoenMartens 2007-04-05T14:01:00Z Project KoenMartens http://www.open-search.net/Blog/BlogEntry5 We'll have a blog with more info soon.<p /> ErikBorra 2007-04-01T12:00:00Z Project ErikBorra http://www.open-search.net/Blog/BlogEntry8 On March 31 2007, we were part of a panel on (cyber)dissidents and restrictions at the <a href="http://www.open-search.net/Opensearch/ImpaktPanel" target="_top">impakt festival</a>.<p />You can find our presentation and extra links and maps in <a href="http://www.open-search.net/Opensearch/ImpaktPanel" target="_top">ImpaktPanel</a>. Also Nicholas Dearden's presentation from irrepressible.info can be found there. ErikBorra 2007-04-01T09:47:00Z Project ErikBorra http://www.open-search.net/Blog/BlogEntry3 <p /> Besides our enthusiastic kick-off, we slowly gain a more realistic but not pessimistic view on peer-to-peer web-indexing and searching: The upcoming year(s) will bring various PhD theses on that subject, addressing shortcomings of pioneer work, and providing a fundament for next-generation P2P applications. <p /> open-search.net is aiming towards a tightrope walk between academic and real-world appliance. [NPO, open-source, free -&gt; distinct from software-industry !] – For the time being, we feel that we can best provide valuable solutions for issues that are not addressed at University, while refraining from actual research on Peer-to-Peer and resort to existing P2P-implementations. <p /> open-search.net will wrap one or more p2p lib/app and provide a open-source framework to aid development, prototyping, testing and distribution a p2p search-engine. <p /> We split the project into 2 major task-forces: <p /> <ul> <li> (A) API + back-end devel </li> <li> (B) Community + front-end </li></ul> <p /> <p /> <h1><a name="A"></a><a name="_A_"></a> (A) </h1> It is still one of our goals to povide the core in POSIX-C but it feels much more realistic to glue P2P libraries and search cores via local sockets and provide an API to bind crawlers, P2P-storage and data-processing to other programming languages. <p /> (using java itself introduces latencies which are an order of magnitude larger that a local-socket API. Python, perl or haskell bindings might come in handy.) <p /> The framework comprises a simple HTTP server, open-search configuration and session management, and API abstration to manage search-requests/results wrapped in a installable package. <p /> The database to store the index, will evolve from static-file, via a dynamic query interface (forward queries to other search-engines) to a P2P example implementation (eg. using gnutella &amp; existing torrents). well use the opensearch and other XML based import/export mechanisms. <p /> p2p is not yet ready for a commercial grade search engine: thus flexibility and rapid prototyping are to be focused on! <p /> <p /> <h1><a name="B"></a><a name="_B_"></a> (B) </h1> different audiences: <ul> <li> devels (core + framework) </li> <li> devels (serach + p2p) </li> <li> desigeners + front-end devels </li> <li> testers and users </li> <li> privacy – part of the open-search frameworks API. </li> <li> anonymity – separate layer (eg. TOR) – check p2p lib. support </li></ul> <p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=blog development news" rel="tag">blog development news</a> RobinGareus 2007-03-13T10:54:00Z RobinGareus http://www.open-search.net/Blog/BlogEntry2 open-search development is slightly behind schedule!<p />I eventually received the book "Peer-to-Peer-Systems and Applications" and there's no doubt that priorities will shift a little over the next days. <p /> we should have read that before writing our proposal, but it was worth the learning experience… <p /> good thing is: we made our March 1st deadline to install the blog plugin! <p /> cheers, robin<p /><b>Tags</b>: <a href="http://www.open-search.net/Blog/BlogArchive?mode=tag&amp;search=news, development blog" rel="tag">news, development blog</a> RobinGareus 2007-02-28T16:19:00Z Development, UsersChat RobinGareus http://www.open-search.net/Blog/BlogEntry9 The credits and idea can be found <a href="http://www.open-search.net/Opensearch/OpensearchDesign" target="_top">here</a><p /> ErikBorra 2007-02-14T10:19:00Z Project ErikBorra http://www.open-search.net/Blog/BlogEntry10 You can read and comment on it @ <a class="natExternalLink" target="_blank" href="http://doc.open-search.net">http://doc.open-search.net</a><p /> ErikBorra 2007-02-04T10:30:00Z Development ErikBorra http://www.open-search.net/Blog/BlogEntry11 We are currently visiting congresses and workshops like <a class="natExternalLink" target="_blank" href="http://events.ccc.de/congress/2006/Fahrplan/events/1752.en.html">23c3</a> and <a class="natExternalLink" target="_blank" href="http://www.fosdem.org/2007/">FOSDEM '07</a> to gather extra input and build a community of volunteers.<p /> ErikBorra 2007-01-05T10:33:00Z ErikBorra