Here's some technical details that we're gonna reveal in a prototype at the open-holland conference:

• the oscar peer to peer layer uses a gossip-engine to multiplex traffic over a single TCP port. – it also allows to tunnel traffic eg. via HTTP to circumvent firewall, without much performance penalty.

• in this prototype the data-signing is not part of open-search, but the P2P layer: Every p2p-packet sent or received contains a RSA-SHA; (yielding ~35% overhead at the moment) – the signature will identify the host responsible for providing the information (not the ID of the Source of the Information ie. crawler or user)

• only "bad clients" not "annoying users" can be banned and the ban affect the local open-search-client only, not the whole network.

• we're testing & debugging a crawler with a local perl::DBD database (currently sqlite) – The SQL interface can be replaced with a p2p layer and is currently used to develop and debug the latter at the moment.

• privacy can be archived as an additional Feature. A user can select

any content of the p2p-network using it's identifier and checksum to sign it's content with a private key. A user's signed content can be referenced ,like any other content in the p2p-network (you can search for it) and it's up to the end-user-application to make use of it.

with the current endeavor to plug a distribted hash table below the open-search-client, here is the protocol used to for communication between the open-search core and the p2p-layer.

the p2p engine will run as separate process, and communication is done via a local TCP socket. for testing and develpment we use two simple perl-scripts (contrib/hatsim) and it will act as a p2p-db simulator.

the protocol is yet rather simple:

we use "++" as separator for ASCII-text encoded arguments. the format is COMMAND ++ opt-args ++ HASHED-KEY ++ XML-VALUE ++

example perl-printf's:

"STORE"."++"."<p2p-opt-args>"."++"."<HASH_FOR_KEYWORD:>"."++"."<XML-LIST-OF-URL_HASH>"."++";
"STORE"."++"."<p2p-opt-args>"."++"."<HASH_FOR_URL>"."++"."<XML-DATA_FOR-URLS>"."++";
"LOOKUP"."++"."<p2p-opt-args>"."++"."<HASH>"  # → returns either <XML-LIST-OF-URL_HASH> or <XML-DATA_FOR-URLS> 

this very simple protocol will be enough for a protptype test implementation. – the <p2p-opt-args> are yet unused but we forsee a couple of features that would need to pass extra information to the p2p-engine..

currently packets are gpg signed when they pass tru the p2p layer, using a host-key rather than individual crawler signature. – also the format of the xml-data is yet unknown to the the p2p layer. once search-filtering is optimized and parts of it can be done at network-layer, there'll be a new command STORE_IDX and a separate LOOKUP functions for different return types. – it remains to be discussed whether priotity-tags are implemented as optinal-argument or separate command.

I eventually received the book "Peer-to-Peer-Systems and Applications" and there's no doubt that priorities will shift a little over the next days.

we should have read that before writing our proposal, but it was worth the learning experience…

good thing is: we made our March 1st deadline to install the blog plugin!

cheers, robin