Relay servers for public posts: Difference between revisions

From diaspora* project wiki
(Fix Phases of post delivery numbered list)
(→‎Concepts of the solution: a code tag was leaking out due to typo :))
 
(12 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Work in progress}}
{{Note|Code enabling relays to work with diaspora* has been merged into 0.6 version.}}
{{Note|This is a proposal, not an accepted specification.}}
{{Note|Discussion regarding this on [https://discourse.diasporafoundation.org/t/public-post-federation/264 Discourse].}}
{{Note|Discussion regarding this on [https://www.loomio.org/discussions/7963 Loomio].}}


== The problem ==
== The problem ==
Line 37: Line 36:
* Tags
* Tags


=== Pod configuration and statistics.json / NodeInfo ===
=== Pod configuration and .well-known/x-social-relay ===


A pod can publish information about itself in the current ''statistics.json'' route, which in the future will be replaced by ''NodeInfo''. The attributes related to relays are as follows, in JSON format (with the first possible value as default):
A pod can publish information about itself using the <code>.well-known/x-social-relay</code> ([http://the-federation.info/social-relay/well-known-schema-v1.json schema]). An example:


<pre>
<pre>
"relay": {
{
    "subscribe": (false/true),
  "subscribe": true,
    "scope": "(all/tags)",
  "scope": "tags",
    "tags": []
  "tags": ["foo", "bar"]
}
}
</pre>
</pre>
Line 53: Line 52:
<pre>
<pre>
relay:
relay:
   send_outbound: false/true
   outbound:
  outbound_handle: relay@defaultrelay.tld
    send: false/true
   subscribe: false/true
    handle: relay@defaultrelay.tld
  scope: all/tags
   inbound:
  tags_source: users/podmin
    subscribe: false/true
    scope: all/tags
    include_user_tags: false/true
    pod_tags: foo,bar,etc
</pre>
</pre>


The <code>tags_source</code> setting governs how <code>tags</code> is populated in to the ''statistics.json / NodeInfo'' output.
The <code>relay.inbound</code> configuration controls what the <code>.well-known/x-social-relay</code> file will contain. If <code>scope</code> is <code>tags</code> then the <code>tags</code> list for the well-known will be populated as follows;


* ''users'' -- All tags that active users (6mo) follow will be added to the list
* Any tags in <code>pod_tags</code> setting
* ''podmin'' -- List of tags is maintained by podmin in the admin menu (needs db table). In the admin menu to not require constant restarts.
* If <code>include_user_tags</code> is true, then all tags that active users (6mo) follow will be added to the list


Rationale is that podmins can control whether they want the pod to focus on certain topics or let the users influence the topics.
Rationale is that podmins can control whether they want the pod to focus on certain topics and also let the users influence the topics.


=== List of active pods ===
=== List of active pods ===
Line 83: Line 85:
==== Core code changes required for outbound sending ====
==== Core code changes required for outbound sending ====


Diaspora already includes a kind of "carbon copy" possibility in Postzord. If outbound relay has been activated, a code change will add to each public post the relay handle as recipient. POC working code already exists [https://github.com/jaywink/diaspora/commit/77aa340cdc0da7372facada3674b9918437debba here] (with a hard coded locally created relay Person object id).
Any pod wanting to send public posts out to a relay should carbon copy them to a relay. This is the only change that is required. Support in diaspora* for this is in the develop branch, coming out in the 0.6 release.


==== Phases of post delivery ====
==== Phases of post delivery ====
Line 100: Line 102:
== What about comments, likes, reshares and other participations? ==
== What about comments, likes, reshares and other participations? ==


Spec to make sure everything flows to pods who have no other contact to originating post pod needs to be thought through. If necessary, relays should cache locally the following data, ''for each post it delivers'':
TBD
 
<pre>
[
    ....
    { "post_guid": [
        "list", "of", "pods"
    },
    ....
]
</pre>
 
Then pods should always deliver comments and other participations for public posts '''always''' to the relay also, which checks against its database who, if any, to follow it to.
 
The other option would be to have receiving pods ping back when they receive a public post from a relay. This would be to create the necessary participation on the originating pod to allow comments etc to federate.
 
Either of these options would increase the weight of the relay system to all pods. The flow of federation needs to be first checked what would be the smallest possible change that could be made to ensure delivery of participations to all participants.


== Other considerations ==
== Other considerations ==
Line 122: Line 108:
=== Pod database size ===
=== Pod database size ===


Most of the pod database size is actually caused by participations, not posts. So unless the participations problem above is solved, pod databases shouldn't be affected that much, most of the additional weight will be on Sidekiq, and that weight can be partly controlled by pod subscription settings.
Most of the pod database size is actually caused by participations, not posts. Before participations are distributed with the relay system, the diaspora* database size needs to be fixed.


However, some solution to the way participations bloat the database should be thought about to make sure the amount of participations can be increased through the relay system, to make information flow where it needs to go.
The problem with participations bloating the database is highlighted [https://github.com/diaspora/diaspora/issues/4920 in this GitHub issue].


== Show me the code! ==
== Show me the code! ==
Line 132: Line 118:
* [https://github.com/jaywink/social-federation Social-Federation] is a Python library that aims to abstract multiple federated social networking protocols under it. Currently it only supports the Diaspora protocol and only relevant parts of it. Fully implemented is parsing a <code>status_message</code> message which covers public posts.
* [https://github.com/jaywink/social-federation Social-Federation] is a Python library that aims to abstract multiple federated social networking protocols under it. Currently it only supports the Diaspora protocol and only relevant parts of it. Fully implemented is parsing a <code>status_message</code> message which covers public posts.
* [https://github.com/jaywink/social-relay Social-Relay] is a Python Flask application that will aim to cover this proposal, using the ''social-federation'' library. It handles public post receiving and storing to Redis. It is currently receiving all public posts from the pod iliketoast.net.
* [https://github.com/jaywink/social-relay Social-Relay] is a Python Flask application that will aim to cover this proposal, using the ''social-federation'' library. It handles public post receiving and storing to Redis. It is currently receiving all public posts from the pod iliketoast.net.
* [https://github.com/jaywink/diaspora/commit/77aa340cdc0da7372facada3674b9918437debba Change to Diaspora ''deferred_dispatch'' worker to make public posts be sent out to a relay]
* [https://github.com/jaywink/diaspora/commit/77aa340cdc0da7372facada3674b9918437debba Change to Diaspora ''deferred_dispatch'' worker to make public posts be sent out to a relay] [merged
 
There is an example relay running at https://relay.iliketoast.net.


This relay is currently running at http://relay.iliketoast.net.
A relay could be written with any programming language - the only constraint is talking the Diaspora protocol.


== The way forward ==
== The way forward ==


* Implement missing parts to ''social-federation'' library, meaning webfinger etc
Blockers for 0.6:
* Finish up PoC ''social-relay'' server to include exposing relay handle, read pod lists, query remote pods and deliver posts
* <strike>Implement missing parts to ''social-federation'' library, meaning webfinger etc</strike>
* <strike>Finish up PoC ''social-relay'' server to include exposing relay handle, read pod lists, query remote pods and deliver posts</strike>
* <strike>Submit PR to Diaspora core to include configuration and post mirroring to relays</strike>
* <strike>Investigate federation flow regarding participations</strike>
* <strike>Lots of testing before 0.6 release.</strike> (months of testing has shown relay system works and causes no problems to the core)
 
Non-blockers for 0.6:
* Document suggestions regarding participations flow
* Document suggestions regarding relay decentralization
* Implement participations flow
* Implement relay decentralization


[[Category: Proposals]]
[[Category: Proposals]]
[[Category: Federation]]
[[Category: Federation]]

Latest revision as of 13:25, 4 April 2019

NoteNote:Code enabling relays to work with diaspora* has been merged into 0.6 version.
NoteNote:Discussion regarding this on Discourse.

The problem

Diaspora is an excellent communication platform for sharing ideas and discussing them. Decentralization however, while bringing benefits also causes some issues. One of those "make or break" issues is the lack of federation for public posts. Setting up your own pod at the moment doesn't really make sense since you will instantly lose one of the biggest features - tag following. Since posts are generally only delivered to pods where there are participants for that post, lonely pods will not generally get a bulk of the public posts going around.

This creates a broken network and this proposal aims to fix that in a way that in a more lighter way than just pushing all public posts to all pods (which technically is not a good thing to do).

In fact, this proposal doesn't limit itself to diaspora*, but would also serve other federated social networks using the same protocol (ie RedMatrix and Friendica).

Concepts of the solution

Relay server

A relay server is a lightweight server app that has only one function - passing of messages from one place to another. In terms of federation message support, the server needs to implement:

  • /.well-known/host-meta to provide discovery for the special relay handle
  • WebFinger or parts of it, to allow querying of the special relay handle
  • hCard for the special relay handle
  • /receive/public to receive public posts. The relay can and should ignore signature verification of authors which will be done by receiving pods anyway
  • POST of received messages to /receive/public of other pods

There should not be only one relay server. There should be many relay servers and anyone in the community should be able to run one. There should be a default relay server specified (in the format of a diaspora handle ie relay@defaultrelay.tld) the core code base so new pods can easily enable the public posts outbound relay feature.

In the future the relay network could be load balanced, if necessary, by having pods exchange lists of working relays between themselves.

Storage

The relay is not required to store messages for any longer than it needs to send them out. The storage should preferable work in a "first in first out" principle, so something like a list would do well.

Relay should store the pod list locally for active pods. Relay should add to this pod list details retrieved from the pods themselves, including the following:

  • Subscription status
  • Scope
  • Tags

Pod configuration and .well-known/x-social-relay

A pod can publish information about itself using the .well-known/x-social-relay (schema). An example:

{
  "subscribe": true,
  "scope": "tags",
  "tags": ["foo", "bar"]
}

The pod configuration file should have the following available (defaults first):

relay:
  outbound:
    send: false/true
    handle: relay@defaultrelay.tld
  inbound:
    subscribe: false/true
    scope: all/tags
    include_user_tags: false/true
    pod_tags: foo,bar,etc

The relay.inbound configuration controls what the .well-known/x-social-relay file will contain. If scope is tags then the tags list for the well-known will be populated as follows;

  • Any tags in pod_tags setting
  • If include_user_tags is true, then all tags that active users (6mo) follow will be added to the list

Rationale is that podmins can control whether they want the pod to focus on certain topics and also let the users influence the topics.

List of active pods

Where the relay gets the active pods list doesn't matter, there are several possibilities already, including http://the-federation.info, https://diasp.net/active and http://podupti.me. A relay should be flexible in falling back to other lists when needed. A relay should also cache the list of pods internally. A relay should query the podlist on a regular basis, for example daily, so newly registered pods don't have to wait for a long time.

Diagrams

Data flow

PublicRelays2 1.png

Process of post delivery

PublicRelays2 2.png

Core code changes required for outbound sending

Any pod wanting to send public posts out to a relay should carbon copy them to a relay. This is the only change that is required. Support in diaspora* for this is in the develop branch, coming out in the 0.6 release.

Phases of post delivery

As per federation requirements, the sending pod needs to first fetch via normal methods the contact of the relay handle, if it hasn't been retrieved yet. This doesn't require any additional code changes.

  1. Relay contact discovery
  2. Delivery as normal to recipients
  3. Relay saves message to its outbound queue
  4. Relay analyses received posts fifo
  5. Relay opens message, ignoring verifying author signatures and parses tags from the message
  6. Post delivered to remote pods according to their subscription settings

Note, the relay opens the message only to read the tags from it. It should store the actual message just as a dump of the whole XML payload received, and send that out exactly as received. Tampering of the payload would invalidate the message signatures.

What about comments, likes, reshares and other participations?

TBD

Other considerations

Pod database size

Most of the pod database size is actually caused by participations, not posts. Before participations are distributed with the relay system, the diaspora* database size needs to be fixed.

The problem with participations bloating the database is highlighted in this GitHub issue.

Show me the code!

There is some PoC code to back up this proposal.

  • Social-Federation is a Python library that aims to abstract multiple federated social networking protocols under it. Currently it only supports the Diaspora protocol and only relevant parts of it. Fully implemented is parsing a status_message message which covers public posts.
  • Social-Relay is a Python Flask application that will aim to cover this proposal, using the social-federation library. It handles public post receiving and storing to Redis. It is currently receiving all public posts from the pod iliketoast.net.
  • Change to Diaspora deferred_dispatch worker to make public posts be sent out to a relay [merged

There is an example relay running at https://relay.iliketoast.net.

A relay could be written with any programming language - the only constraint is talking the Diaspora protocol.

The way forward

Blockers for 0.6:

  • Implement missing parts to social-federation library, meaning webfinger etc
  • Finish up PoC social-relay server to include exposing relay handle, read pod lists, query remote pods and deliver posts
  • Submit PR to Diaspora core to include configuration and post mirroring to relays
  • Investigate federation flow regarding participations
  • Lots of testing before 0.6 release. (months of testing has shown relay system works and causes no problems to the core)

Non-blockers for 0.6:

  • Document suggestions regarding participations flow
  • Document suggestions regarding relay decentralization
  • Implement participations flow
  • Implement relay decentralization