Diaspora powered by Tent

The General Idea

This document describes a high level outline of what is required to get Diaspora running on Tent. The Tent guys are willing to work with Diaspora community devs to make this a reality. The desired outcome is to maintain the look of the existing app but build a completely new backend using the Tent JavaScript app framework. The benefits of doing so include:

• Diaspora would become a Tent app, and would use the Tent API. This would allow other Tent applications to interact with data created in Diaspora and vice versa.
• Federation compatibility with other Tent severs, such as TentStatus
• Upstream development of our federation stack would be based on the work being done by Tent.
• User data import, so that users could freely migrate from one Diaspora-Tent pod / Tent server to another. The data import would be into a Tent server, it wouldn't cover migration between old Diaspora pods. Tent provides its own migration features that allow moving between servers easily.
• A solid foundation for Diaspora to build app-specific features on top of in the future. (Groups, Albums, etc)
• An API for third-party clients to make use of. (Gwibber, Choqok, Dianara, as well as numerous mobile clients)
• Authentication with Tent apps using Diaspora

The current front-end is built on top of jQuery, Backbone, and Bootstrap; it uses Handlebars templates which shouldn't be too hard to update to use Mustache. Handlebars is a fork of the Mustache template language with more features but slightly worse performance. This very minor change will increase the speed of the app a little bit.

What is Tent?

Tent is an open protocol for personal evented data vaults and decentralized real-time communication. Tent has two APIs: one lets Tent apps talk to Tent servers, the other lets Tent servers talk to each other. A Tent server can have one or many users, and anyone can run their own Tent server. Users can change their Tent address instantly and move easily between Tent servers as part of the protocol, address books are updated automatically. When a user changes Tent servers, she takes her data and relationships with her. Users follow one another asymmetrically, like on Twitter, but choose what kinds of posts they want to receive from each other. For example Alice follows Bob, but only subscribes to his status posts. Bob follows Eve's photo posts, and so on. Every Tent post has permissions. Posts can be public, private, or shared with only specific users and groups. Tent groups are similar to Diaspora aspects.

Tent apps can send and fetch posts from Tent servers. Tent servers send posts to each other as soon as they are created. Posts are grouped by type. For example, essay posts have a title and large block of text, while photo posts contain images. Developers can create new post types and posts created by one app can be read by another. For example, short posts created by Diaspora (powered by Tent) would show up in Tent status. Tent apps can do much more than social. While it is possible to build Twitter Facebook, and Instagram type apps that use Tent, apps like Dropbox or Google Docs could also be created for Tent. This means users can store all their data and relationships in one place and use it in many different applications.

Search engines find Tent servers and allow anyone to search their public posts. Unlike the web Tent works in real time, so Tent search engines include new posts immediately.

Things Podmins Should Know

Scalability

Amount of Users

Tentd is designed to be single-tenant, and does not currently have a multi-user pod architecture. It is not currently multi-user at all. With simple modifications it could support ~100 users, but would require substantial work to support the pod sizes Diaspora is familiar with. There are also no multi-user administrative tools at present. Users can host their own tentd instance easily on a PaaS like Heroku or their own hardware or choose an existing Tent host like http://tent.is, https://campr.me/, http://lavv.us/. Self hosting or a hosting service are the only immediately available choices.

1. Given time, the Diaspora dev community and other Tent devs could add multi-tenant administration and functionality to tentd for small groups of users or substantially rewrite (or start anew) to accomodate current pod sizes. Individual users are free to host it wherever they want, and it works well on Heroku. While this is acceptable for single-user pods, most existing pods have more than one user. We have two strategies to choose between, depending on amount of available development time and resources.

2. Allow users and podmins alike a window of time to let users migrate to their own Diaspora-Tent pods, or to smaller community Diaspora-Tent pods. Technically speaking, smaller pods aren't necessarily a bad thing. The argument could be made that smaller pods have less user data on them, and thus less incentive for the podmin to attempt to try to profit from user data.

Apps and Server Load

Tent enables many more interactions and use cases than Diaspora, and is designed to be a personal data vault; the one place where users store all of their data. This means that the load on servers and the amount of storage required is several orders of magnitude higher per user. Potential apps include everything from file synchronization (think Dropbox) to storing health/genome data to logging physical location history and other data for personal analytics.

Compatibility

Diaspora-Tent will not be backwards-compatible with the current implementation of Diaspora, due to the fact that running multiple federation protocols in parallel can be a significant drain on the server. We want to make converting over to Tent infrastructure easy for podmins on Diaspora-Current to do. Some pods, such as Calispora, are currently running the Diaspora-Pistos fork. In an effort to keep such pods from becoming dead-in-the-water, we'd like to reach out to these podmins as well. Podmins could agree on a specific date to switch over by, so that users on pods will still be able to talk to each other.

Diaspora-Pistos is realtively popular due to the fact that it has more than a few features that make it unique. It is not out of the question that these features could be ported to Backbone in the future and be made part of the upstream development being done on D*.

Development

Due to significant differences with Diaspora's codebase, initial development will be done on the side by the Tent devs, rather than directly in our upstream development branch. When Diaspora-Tent is matured enough to have feature parity and the ability to scale, we can gradually move it over to Diaspora's GitHub repo and base our master and development branches around the new work being done. This will also allow us to easily add in any features that the Tent devs may not have implemented themselves.

Development on making Tentd scale up for multi-user capabilities will require an outreach to projects in the Tent community, such as [Lavv.us lavv.us] in an attempt to maintain a cross-community consolidated effort.

App Overview

Create Post

• Limit based on one or many groups
  • "All Aspects" (all groups) can be achieved by selecting all groups currently in existence and limiting the post to them.
• Posts are not limited by size but are plain text. Therefore posts <= 256 characters should be created as a status post and larger posts as essays.
• Posts can contain many images of seemingly any size. These can be created as individual photo posts and mention the original post or as attachments to the original post.
• [Hash]Tags are inline and not parsed.

Posts Feed

• Links and hashtags are auto-linked.
• (Public|Limited), Like, Reshare (repost), and Comment actions.
• Visibility on the number of likes a post has (and who they're from).
• Comments are truncated at 1943 characters with a "show more" button to display the full text.

Email Notifications

An app backend will need to be setup to handle incoming web-hooks and send out email notifications.

The existing app sends email notifications for the following actions based on email preferences:

• someone comments on a post you've commented on (if you're following the author)
• you are mentioned in a post
• someone comments on your post
• you receive a private message
• someone 'starts sharing with you' (which just means 'someone followed you')
• someone likes your post
• someone reposts your post

Make followings in a group visible to each other

This can be easily done by adding the group to each followings' permissions within the group.

Community Spotlight

This feature will require an API powered by Skate.

People Search

This feature will require an API powered by friends.tent.is.

Upload Profile Photo

This can be handled via S3.

Tags / Tag Following

Tent posts already support tags, an API powered by Skate will be required to search them. Skate is a Tent search engine that we run. Tent search works more like the web, in that search is not handled by the protocol, but anyone can crawl posts. As a result search is realtime and global, rather than pod specific. Anyone could implement an alternative search engine and use their own API if they were so inclined.

Comments

Comments are really just a reply that is text only (e.g. status or essay post depending on the length.)

[private] Messages

These can be handled as status and essay posts (depending on size).

Bookmarklet

Ideally this would use a link scheme and be handled by any Tent app. Not a high priority and may be dropped, although it is noted that the bookmarklet could be kept if enough people wanted it.

Cross posting (Twitter/Tumblr/Facebook)

This feature may be dropped. In the case of some networks such as Twitter, this feature is an explicit violation of the Terms of Service. We're not interested in exposing ourselves to the legal ramifications of this. Due to the way Tent works, this could be a completely separate application that someone else could write easily, and it could also support other protocols like OStatus.

Note: This is still a feature that many users on Diaspora want and have come to enjoy having. I would propose at the very least to leave crossposting functionality in, and we can just write and use Service connectors to networks that are fine with cross-posting (StatusNet instances, App.net, Tumblr, Posterous, for starters) --DeadSuperHero (talk) 17:21, 13 December 2012 (CST)

XML Import

The XML dump contains:

Namespace | Description
--------- | -----------
`export.user` | `username`, `serialized_private_key`, `person`
`*.person` | `guid`, `diaspora_handle`, `url`, `profile`, `exported_key`
`*.profile` | `diaspora_handle`, `first_name`, `last_name`, `image_url`, `image_url_small`, `image_url_medium`, `gender`, `bio`, `location`, `searchable`,  `nsfw`, `tag_string`
`*.profile.searchable` | Boolean describing if user is searchable
`*.profile.nsfw` | Boolean describing if user is "Not safe for work"
`*.person.exported_key` | Public Key
`export.aspects` | `name`, `post_ids`
`export.contacts` | `contact`
`*.contact` | `user_id`, `person_id`, `person_guid`, `aspects`
`*.contact.aspects` | `aspect`
`*.contact.aspects.aspect` | `name`
`export.posts` | `status_message`
`*.status_message` | `raw_message`, `guid`, `diaspora_handle`, `public`, `created_at`
`export.people` | `person`

and contains enough information to recreate posts with permissions, assign entities to groups, and determine if Skate follow them or if they should be added to friend finder automatically.

Friend finder could be given enough information to automatically add entities into groups upon following them.

There is also an option to download photos, but clicking it opens an alert "Photo exporting currently unavailable".

It would be helpful if the export data also included the users email, but that is already asked for when setting up the user in friend finder.

App Changes

Convert JavaScript Templates to Mustache

Most of the templates use basic Handlebars conditionals and variable interpolation which is very close to Mustache syntax. There are a few macros being used, but can easily be moved out of the template and into the context.

Convert Haml Templates to Mustache

A fair portion of the site is rendered server-side using Haml. As the app is being converted to primarily use JavaScript, these need to be converted into Mustache templates.

Compress images into sprites

There are many images that appear in the same context but are fragmented into separate assets. These need to be converted into CSS sprites or rendered as SVGs. The `graphics` directory in the repository contains some SVGs and an Adobe Illustrator file.

Missing Tent Features

These features currently don't exist in the Tent stack, and will have to be done on the app side of Diaspora for now.

Likes

Diaspora lets users "like" posts. This feature is currently scheduled for Tent 0.4 in March.

Ignore entity

Diaspora allows users 'ignore' other users. The Tent equivalent would be dropping notifications from a blacklist of entities. The app can handle filtering the content until this feature is implemented. (Not currently scheduled for any release.)

Why not XMPP / OStatus / ActivityStreams / etc?

While it's true that Tent could eventually be built on top of XMPP and remove the dependency on HTTP, it will take time to get Tent into a position to do that. Much of Tent's architecture is based on ideas from existing standards, such as OStatus and ActivityStreams. This discussion and this discussion give a good overview of where existing standards and implementations fall short, and why an in-house solution is preferable.