Relay servers for public posts

Note:

Code enabling relays to work with diaspora* has been merged into 0.6 version.

Note:

Discussion regarding this on Discourse.

The problem

Diaspora is an excellent communication platform for sharing ideas and discussing them. Decentralization however, while bringing benefits also causes some issues. One of those "make or break" issues is the lack of federation for public posts. Setting up your own pod at the moment doesn't really make sense since you will instantly lose one of the biggest features - tag following. Since posts are generally only delivered to pods where there are participants for that post, lonely pods will not generally get a bulk of the public posts going around.

This creates a broken network and this proposal aims to fix that in a way that in a more lighter way than just pushing all public posts to all pods (which technically is not a good thing to do).

In fact, this proposal doesn't limit itself to diaspora*, but would also serve other federated social networks using the same protocol (ie RedMatrix and Friendica).

Concepts of the solution

Relay server

A relay server is a lightweight server app that has only one function - passing of messages from one place to another. In terms of federation message support, the server needs to implement:

• /.well-known/host-meta to provide discovery for the special relay handle
• WebFinger or parts of it, to allow querying of the special relay handle
• hCard for the special relay handle
• /receive/public to receive public posts. The relay can and should ignore signature verification of authors which will be done by receiving pods anyway
• POST of received messages to /receive/public of other pods

There should not be only one relay server. There should be many relay servers and anyone in the community should be able to run one. There should be a default relay server specified (in the format of a diaspora handle ie relay@defaultrelay.tld) the core code base so new pods can easily enable the public posts outbound relay feature.

In the future the relay network could be load balanced, if necessary, by having pods exchange lists of working relays between themselves.

Storage

The relay is not required to store messages for any longer than it needs to send them out. The storage should preferable work in a "first in first out" principle, so something like a list would do well.

Relay should store the pod list locally for active pods. Relay should add to this pod list details retrieved from the pods themselves, including the following:

• Subscription status
• Scope
• Tags

Pod configuration and .well-known/x-social-relay

A pod can publish information about itself using the .well-known/x-social-relay (schema). An example:

{
  "subscribe": true,
  "scope": "tags",
  "tags": ["foo", "bar"]
}

The pod configuration file should have the following available (defaults first):

relay:
  outbound:
    send: false/true
    handle: relay@defaultrelay.tld
  inbound:
    subscribe: false/true
    scope: all/tags
    include_user_tags: false/true
    pod_tags: foo,bar,etc

The relay.inbound configuration controls what the .well-known/x-social-relay file will contain. If scope is tags then the tags list for the well-known will be populated as follows;

• Any tags in pod_tags setting
• If include_user_tags is true, then all tags that active users (6mo) follow will be added to the list

Rationale is that podmins can control whether they want the pod to focus on certain topics and also let the users influence the topics.

List of active pods

Where the relay gets the active pods list doesn't matter, there are several possibilities already, including http://the-federation.info, https://diasp.net/active and http://podupti.me. A relay should be flexible in falling back to other lists when needed. A relay should also cache the list of pods internally. A relay should query the podlist on a regular basis, for example daily, so newly registered pods don't have to wait for a long time.

Diagrams

Data flow

Process of post delivery

Core code changes required for outbound sending

Any pod wanting to send public posts out to a relay should carbon copy them to a relay. This is the only change that is required. Support in diaspora* for this is in the develop branch, coming out in the 0.6 release.

Phases of post delivery

As per federation requirements, the sending pod needs to first fetch via normal methods the contact of the relay handle, if it hasn't been retrieved yet. This doesn't require any additional code changes.

1. Relay contact discovery
2. Delivery as normal to recipients
3. Relay saves message to its outbound queue
4. Relay analyses received posts fifo
5. Relay opens message, ignoring verifying author signatures and parses tags from the message
6. Post delivered to remote pods according to their subscription settings

Note, the relay opens the message only to read the tags from it. It should store the actual message just as a dump of the whole XML payload received, and send that out exactly as received. Tampering of the payload would invalidate the message signatures.

What about comments, likes, reshares and other participations?

TBD

Other considerations

Pod database size

Most of the pod database size is actually caused by participations, not posts. Before participations are distributed with the relay system, the diaspora* database size needs to be fixed.

The problem with participations bloating the database is highlighted in this GitHub issue.

Show me the code!

There is some PoC code to back up this proposal.

• Social-Federation is a Python library that aims to abstract multiple federated social networking protocols under it. Currently it only supports the Diaspora protocol and only relevant parts of it. Fully implemented is parsing a status_message message which covers public posts.
• Social-Relay is a Python Flask application that will aim to cover this proposal, using the social-federation library. It handles public post receiving and storing to Redis. It is currently receiving all public posts from the pod iliketoast.net.
• Change to Diaspora deferred_dispatch worker to make public posts be sent out to a relay [merged

There is an example relay running at https://relay.iliketoast.net.

A relay could be written with any programming language - the only constraint is talking the Diaspora protocol.

The way forward

Blockers for 0.6:

• Implement missing parts to social-federation library, meaning webfinger etc
• Finish up PoC social-relay server to include exposing relay handle, read pod lists, query remote pods and deliver posts
• Submit PR to Diaspora core to include configuration and post mirroring to relays
• Investigate federation flow regarding participations
• Lots of testing before 0.6 release. (months of testing has shown relay system works and causes no problems to the core)

Non-blockers for 0.6:

• Document suggestions regarding participations flow
• Document suggestions regarding relay decentralization
• Implement participations flow
• Implement relay decentralization