Relay servers for public posts: Difference between revisions

Revision as of 14:38, 11 July 2015

Work in progress:

This article is a work in progress, it may contain incomplete or inaccurate information. There may be additional information on the talk page.

Note:

This is a proposal, not an accepted specification.

Note:

Discussion regarding this on Loomio.

The problem

Diaspora is an excellent communication platform for sharing ideas and discussing them. Decentralization however, while bringing benefits also causes some issues. One of those "make or break" issues is the lack of federation for public posts. Setting up your own pod at the moment doesn't really make sense since you will instantly lose one of the biggest features - tag following. Since posts are generally only delivered to pods where there are participants for that post, lonely pods will not generally get a bulk of the public posts going around.

This creates a broken network and this proposal aims to fix that in a way that in a more lighter way than just pushing all public posts to all pods (which technically is not a good thing to do).

In fact, this proposal doesn't limit itself to diaspora*, but would also serve other federated social networks using the same protocol (ie RedMatrix and Friendica).

Concepts of the solution

Relay server

A relay server is a lightweight server app that has only one function - passing of messages from one place to another. In terms of federation message support, the server needs to implement:

/.well-known/host-meta to provide discovery for the special relay handle
WebFinger or parts of it, to allow querying of the special relay handle
hCard for the special relay handle
/receive/public to receive public posts. The relay can and should ignore signature verification of authors which will be done by receiving pods anyway
POST of received messages to /receive/public of other pods

There should not be only one relay server. There should be many relay servers and anyone in the community should be able to run one.

Storage

The relay is not required to store messages for any longer than it needs to send them out. The storage should preferable work in a "first in first out" principle, so something like a list would do well.

Pod configuration and statistics.json / NodeInfo

A pod can publish information about itself in the current statistics.json route, which in the future will be replaced by NodeInfo. The attributes related to relays are as follows, in JSON format (with the first possible value as default):

"relay": {
    "subscribe": (false/true),
    "scope": "(all/tags)",
    "tags": []
}

The pod configuration file should have the following available (defaults):

relay:
  outbound_handle: relay@defaultrelay.tld
  subscribe: false
  scope: all
  tags_source: users/podmin

The tags_source setting governs how tags is populated in to the statistics.json / NodeInfo output.

users -- All tags that active users (6mo) follow will be added to the list
podmin -- List of tags is maintained by podmin in the admin menu (needs db table). In the admin menu to not require constant restarts.

Rationale is that podmins can control whether they want the pod to focus on certain topics or let the users influence the topics.

List of active pods

Where the relay gets the active pods list doesn't matter, there are several possibilities already, including http://the-federation.info, https://diasp.net/active and http://podupti.me. A relay should be flexible in falling back to other lists when needed. A relay should also cache the list of pods internally. A relay should query the podlist on a regular basis, for example daily, so newly registered pods don't have to wait for a long time.

Diagrams

Data flow

Process of post delivery

@@ Line 1: / Line 1: @@
+{{Work in progress}}
 {{Note|This is a proposal, not an accepted specification.}}
 {{Note|Discussion regarding this on [https://www.loomio.org/discussions/7963 Loomio].}}
@@ Line 7: / Line 8: @@
 This creates a broken network and this proposal aims to fix that in a way that in a more lighter way than just pushing all public posts to all pods (which technically is not a good thing to do).
+In fact, this proposal doesn't limit itself to diaspora*, but would also serve other federated social networks using the same protocol (ie ''RedMatrix'' and ''Friendica'').
 == Concepts of the solution ==
-=== The central hub ===
-We need a central hub. This central hub should be maintained by appointed community members until proper foundation resources can be used. For transparency ALL the data stored on the central hub should be public. The functionality of the central hub would be to hold an official pod list (opt in for podmins of course) and relating to this relay server functionality proposal hold a list of tags each participating pod is subscribing to.
-Summa summarum;
-* Central hub that is maintained by appointed community members
-* ALL data on central hub should be open (REST API)
-* Pods add themselves and maintain their data (opt in)
-* Pods can also list tags they are interested in
 === Relay server ===
-A relay server is a lightweight server app that has only one function - passing of messages from one place to another. Technical specs could be, for example (just to not be too Ruby specific :));
+A relay server is a lightweight server app that has only one function - passing of messages from one place to another. In terms of federation message support, the server needs to implement:
-* NodeJS
+* <code>/.well-known/host-meta</code> to provide discovery for the special relay handle
-* Redis
+* ''WebFinger'' or parts of it, to allow querying of the special relay handle
+* ''hCard'' for the special relay handle
+* <code>/receive/public</code> to receive public posts. The relay can and should ignore signature verification of authors which will be done by receiving pods anyway
+* <code>POST</code> of received messages to <code>/receive/public</code> of other pods
-There should not be only one relay server. There should be many relay servers and anyone in the community should be able to run one. When someone runs a relay server, that relay will notify the central hub that they are open for business. The central hub in turn notifies pods (when they ask for this) about a list of relays available.
+There should not be only one relay server. There should be many relay servers and anyone in the community should be able to run one.
-If the relay app could use the normal Diaspora protocol for communication it should.
+=== Storage ===
-== Events ==
+The relay is not required to store messages for any longer than it needs to send them out. The storage should preferable work in a "first in first out" principle, so something like a list would do well.
-=== Someone on a pod adds a hashtag to his/her followed hashtag lists ===
+=== Pod configuration and statistics.json / NodeInfo ===
-If pod is not already subscribing to this hashtag, it will notify the central hub that it wants these posts.
+A pod can publish information about itself in the current ''statistics.json'' route, which in the future will be replaced by ''NodeInfo''. The attributes related to relays are as follows, in JSON format (with the first possible value as default):
-=== Someone on a pod unfollows a hashtag ===
+<pre>
+"relay": {
+    "subscribe": (false/true),
+    "scope": "(all/tags)",
+    "tags": []
+}
+</pre>
-If no one else on the pod follows this hashtag, pod will notify central hub that it doesn't want these posts any more.
+The pod configuration file should have the following available (defaults):
-=== Someone writes a public post with one or more hashtags ===
+<pre>
+relay:
+  outbound_handle: relay@defaultrelay.tld
+  subscribe: false
+  scope: all
+  tags_source: users/podmin
+</pre>
-If podmin has opted in to relay functionality, pod will randomize one of the cached relays and delivers that post to the designated relay.
+The <code>tags_source</code> setting governs how <code>tags</code> is populated in to the ''statistics.json / NodeInfo'' output.
-=== Relay receives a post from a pod ===
+* ''users'' -- All tags that active users (6mo) follow will be added to the list
+* ''podmin'' -- List of tags is maintained by podmin in the admin menu (needs db table). In the admin menu to not require constant restarts.
-Relay already has a cached list of pods and what hashtags they want so relay will deliver post to pods that are interested in one or more of the hashtags in this post. Relay is not for public message keeping - it will delete any posts as soon as they have been pushed out.
+Rationale is that podmins can control whether they want the pod to focus on certain topics or let the users influence the topics.
-=== Pod receives post from relay ===
+=== List of active pods ===
-Pod checks from originating pod that the post hash matches the one from the relay server. If hash matches, post is added to pod.
+Where the relay gets the active pods list doesn't matter, there are several possibilities already, including http://the-federation.info, https://diasp.net/active and http://podupti.me. A relay should be flexible in falling back to other lists when needed. A relay should also cache the list of pods internally. A relay should query the podlist on a regular basis, for example daily, so newly registered pods don't have to wait for a long time.
 == Diagrams ==
@@ Line 58: / Line 66: @@
 === Data flow ===
-[[File:Relays1.png]]
+[[File:PublicRelays2_1.png]]
 === Process of post delivery ===
-[[File:Relays2.png]]
+[[File:PublicRelays2_2.png]]
-== Alternative solution that doesn't require core code changes ==
-This solution is based on using the current federation protocol and manual user action from users to subscribe to tags. This solution would not require core code changes and could be used as an initial stage. This would not transparently make users receive public posts from the network, it would require each user sending out ```subscribe/unbsubscribe``` messages.
-The downside of this solution vs the original one is that each relay would need to keep track of all users subscribing, instead of relays just needing to know which pods subscribe. For privacy, this solution is much worse.
-=== Relay server ===
-==== Federation ====
-In this solution, the relay server would talk like a normal diaspora* pod - in terms of federation it would look to other pods just like a normal pod. However, it doesn't need to support the whole protocol, only a subset. These parts include;
-* Share / stop sharing
-* Receive private and public status messages
-* Push out private and public status messages
-Everything else can be ignored.
-==== Tech stack ====
-The server could be built taking the current diaspora* server federation parts or using some other solution, for example Pyaspora.
-==== Sharing of information between relay servers ====
-Initially, one server could be good enough for POC. The relays should however also exchange information between them via the normal protocol status messages. Each relay would keep track of other relays out there and notify them of changes regarding subscribers. If a relay goes down for too long a time, it is forgotten by all the other relays.
-=== Subscribing and unsubscribing to tags ===
-When a user wants to subscribe to a tag, they first start sharing with a relay of choice. Then once the sharing is complete, they write mention the relay in a status message, listing which tags they would like to subscribe or unsubscribe to. This message is sent as normal to the relay server, which then notes down the changes in its db (and notifies other servers too). For the relay, it really only cares about which pod is interested in which tags - but the way to keep track of this is to work with users directly.
-=== Delivery of public messages ===
-Only messages from users who are sharing with a relay will be delivered to relays, due to the fact how the current code works. Once the relay receives the message, it will look at its db and see which pods are interested in the message, by analysing the tags it contains. It will then deliver the message to the pods who are interested (minus the sending pod of course).
-After delivery, it will delete the message. Any additional copies of it will be redelivered (for example a possible future post edit feature).
 [[Category: Proposals]]
 [[Category: Federation]]