with the current endeavor to plug a distribted hash table below the open-search-client, here is the protocol used to for communication between the open-search core and the p2p-layer.

the p2p engine will run as separate process, and communication is done via a local TCP socket. for testing and develpment we use two simple perl-scripts (contrib/hatsim) and it will act as a p2p-db simulator.

the protocol is yet rather simple:

we use "++" as separator for ASCII-text encoded arguments. the format is COMMAND ++ opt-args ++ HASHED-KEY ++ XML-VALUE ++

example perl-printf's:

"STORE"."++"."<p2p-opt-args>"."++"."<HASH_FOR_KEYWORD:>"."++"."<XML-LIST-OF-URL_HASH>"."++";
"STORE"."++"."<p2p-opt-args>"."++"."<HASH_FOR_URL>"."++"."<XML-DATA_FOR-URLS>"."++";
"LOOKUP"."++"."<p2p-opt-args>"."++"."<HASH>"  # → returns either <XML-LIST-OF-URL_HASH> or <XML-DATA_FOR-URLS> 

this very simple protocol will be enough for a protptype test implementation. – the <p2p-opt-args> are yet unused but we forsee a couple of features that would need to pass extra information to the p2p-engine..

currently packets are gpg signed when they pass tru the p2p layer, using a host-key rather than individual crawler signature. – also the format of the xml-data is yet unknown to the the p2p layer. once search-filtering is optimized and parts of it can be done at network-layer, there'll be a new command STORE_IDX and a separate LOOKUP functions for different return types. – it remains to be discussed whether priotity-tags are implemented as optinal-argument or separate command.

Teased by our 2nd Workshop, Marco Fiscato has jumped aboard our team of developers. His expertise in peer to peer networks and his implementation of a chord based p2p layer will be of great help for prototyping p2p open-search.

This weeks Todo includes a simulator that implements the distributed-database query interface: the simulator (dubbed hatsim) allows to "replay/simulate" crawler & user activity for the P2P software and emulates a dummy p2p to be used for client development.

In the wake of easter we are heading towards the 2nd workshop.. with no public software client it's gonna be a close call, but after the attached screen-shot we hope to announce an alpha version a week before the workshop. probably already April 19 2007.

Last week has been rather distracting (installing python on our web-server, re-furbishing php and installing mantis on bug.open-search.net) – and the easter weekend turned out rather sunny which did not aid open-search development after all. – so without furter ado..

• local-webserver: check
• cgi-interface: check
• AJAX-engine: partly
• design, layout & language support: preliminary.
• serach-backend: faked local.
• crawler: independant (preliminary perl AND/OR python)
• P2P-lib: none – exeriments and simulations. (planning to use gnutella for the workshop)

current ToDo: debug and document AJAX api (javascript, xslt); code a P2P API-wrapper (c, py-bindings) , renice twiki-blog (hammer).