Testing workflow

In order to assure code quality, ease refactoring and to make sure only the code that is really necessary gets written we strive to encourage TDD (Test Driven Development) or BDD (Behaviour Driven Development) when possible. There is a huge list of benefits that result from this style of development and even some research on it referenced from the Wikipedia articles, if you are interested in reading up on that topic.

test driven development flow

In essence, TDD means that before you write some actual code, you are supposed to write a testcase firsts, using the feature or function the way it should represent itself to the outside. Of course, because no actual code has been written yet, the testcase will fail if being run. Now the idea is to write only the minimal amount of code that gets the testcase to pass and as soon as it does so, refactor the code until you are left with perfectly structured and modular code, that even passes your super-awesome testsuite.

Now, this can be done on several layers, starting at single functions, methods and classes up to what the user gets presented with in the UI. There are different tools for each one of these.

General preparations

Some tests require the static error pages to exist, generate them with

bin/rake assets:generate_error_pages

Rspec

Since Diaspora is based on Ruby on Rails, we get the rspec testing environment practically for free. Our rspec tests are located in the spec/ directory and that directory is split up in subdirectories differentiating which parts of the code are being tested with the containing files (e.g. models in spec/models/, controllers in spec/controllers/ or the Diaspora lib in spec/lib/).

Lets look at an example: If we were to test the validation and cleanup rules of the Profile model, the place to look for them is spec/models/profile_spec.rb. Inside that file you will see that the tests are written in plain ruby code structured by a DSL (Domain Specific Language), in this case a few helpful ruby methods that let us nest the tests logically and check the results more easily.

describe Profile do
  describe 'validation' do
    describe "of last_name" do
      it "strips leading and trailing whitespace" do
        profile = Factory.build(:profile, :last_name => " Ohba ")
        profile.should be_valid
        profile.last_name.should == "Ohba"
      end
    end
  end
end

This shows us that the description of the tests is supposed to be easily human-readable as this is also the part that gets shown, when the test fails ("validation" "of last_name" "strips leading and trailing whitespace"). Another thing that should catch your attention in this example is the use of a factory for creating mock objects to test with. This greatly reduces the amount of work that goes into creating somewhat realistic tests.

Before running any tests for the first time, make sure you have created the database schema for the `test` environment by doing;

RAILS_ENV="test" bin/rake db:create db:migrate

If you have too much time or you want to make sure you didn't break anything apart from the stuff you were working on, you can run all rspec tasks

bin/rake spec

To run just our single profile spec file from the example

bin/rspec spec/models/profile_spec.rb

If your test is failing, the command output will present you this

=> Building fixtures
=> Built aspect_memberships.yml, aspects.yml, contacts.yml, people.yml, profiles.yml, and users.yml
Run options: exclude {:performance=>true}

  1) Profile validation of first_name fails on purpose
     Failure/Error: false.should == true
     expected: true
     got: false (using ==)
     # ./spec/models/profile_spec.rb:11:in `block (4 levels) in <top (required)>'

  47/47:       100% |==========================================================================| Time: 00:00:00

Finished in 0.95665 seconds
47 examples, 1 failure

Failed examples:

rspec ./spec/models/profile_spec.rb:10 # Profile validation of first_name fails on purpose

Randomized with seed 28937

If all of your tests are fine, it will show you a summary and some statistics similar to the following

=> Building fixtures
=> Built aspect_memberships.yml, aspects.yml, contacts.yml, people.yml, profiles.yml, and users.yml
Run options: exclude {:performance=>true}
  46/46:       100% |==========================================================================| Time: 00:00:00

Top 10 slowest examples:
  Profile tags strips more than 5 tags
    0.14156 seconds ./spec/models/profile_spec.rb:258
  Profile validation of first_name can be 32 characters long
    0.07987 seconds ./spec/models/profile_spec.rb:16
  Profile serialization includes tags
    0.05348 seconds ./spec/models/profile_spec.rb:157
  Profile tags it should behave like it is taggable.format_tags doesn't mangle text when tags are involved
    0.05271 seconds ./spec/shared_behaviors/taggable.rb:42
  Profile tags it should behave like it is taggable.format_tags supports non-ascii characters
    0.05078 seconds ./spec/shared_behaviors/taggable.rb:27
  Profile tags it should behave like it is taggable.format_tags responds to plain_text
    0.04862 seconds ./spec/shared_behaviors/taggable.rb:38
  Profile tags it should behave like it is taggable.format_tags links each tag
    0.04851 seconds ./spec/shared_behaviors/taggable.rb:31
  Profile tags it should behave like it is taggable#build_tags builds the tags
    0.04171 seconds ./spec/shared_behaviors/taggable.rb:91
  Profile serialization includes location
    0.02894 seconds ./spec/models/profile_spec.rb:165
  Profile validation#contruct_full_name generates a full name given only first name
    0.02722 seconds ./spec/models/profile_spec.rb:56

Finished in 0.94219 seconds
46 examples, 0 failures

Randomized with seed 16731

It may be important to know to always run the rspec tests first if you are planning on using any of the other tests, because there are rspec cases that generate fixtures (hmtl/json snippets) that are needed for the other tests to work.

Jasmine

The Jasmine tests for JavaScript are located inside the spec/javascripts/ directory. They are split up in a similar manner to how the JavaScript files are structured in app/assets/javascripts/.

Jasmine tests try to resemble rspec tests in both structure and nomenclature of available helper methods, and we are using factories here, too. You can read this nice introduction if you never worked with it before to get a general idea of what it's all about and what it should be used for.

Note! Jasmine tests use some fixtures generated by the rspec tests, so if you haven't run those yet, generate the fixtures with the following command first:

bin/rake tests:generate_fixtures

In case the command above fails, try to recreate the test-database: RAILS_ENV=test bin/rake db:drop db:create db:migrate

The Diaspora client side code makes use of Backbone.js and the Handlebars templating system. This gives us a model/view based approach to rendering many items of the same type in the browser, while at the same time relieving the burden on the server since it doesn't have to do any significant rendering anymore. Let's take a look at a Jasmine test for the JavaScript User model, located at spec/javascripts/app/models/user_spec.js.

describe("app.models.User", function(){
  beforeEach(function(){
    this.user = new app.models.User({});
  }); 

  describe("authenticated", function(){
    it("should be true if ID is nil", function(){
      expect(this.user.authenticated()).toBeFalsy();
    }); 

    it('should be true if ID is set', function(){
      this.user.set({id : 1});
      expect(this.user.authenticated()).toBeTruthy();
    }); 
  }); 

  describe("isServiceConnected", function(){
    it("checks to see if the sent provider name is a configured service", function(){
      this.user.set({configured_services : ["facebook"]});
      expect(this.user.isServiceConfigured("facebook")).toBeTruthy();
      expect(this.user.isServiceConfigured("tumblr")).toBeFalsy();
    }); 
  }); 
});

You can see, this is pretty similar to rspec, the only major difference being the JavaScript syntax. It is easy to recognize the human-readable naming of the testcase and the special functions that evaluate the outcome of the test (expect(...)).

Jasmine tests are run in the browser, since they require a JavaScript environment, although for CI (Continuous Integration) they can also run on the command line which requires virtual framebuffer as substitue for an actual screen. The webserver for Jasmine is started by

RAILS_ENV=test bin/rake jasmine

Which, when finished, will tell you, that it is reachable on http://localhost:8888/

your server is running here: http://localhost:8888/
[...]

[2017-09-03 17:35:29] INFO  WEBrick 1.3.1
[2017-09-03 17:35:29] INFO  ruby 2.4.1 (2017-03-22) [x86_64-linux]
[2017-09-03 17:35:29] INFO  WEBrick::HTTPServer#start: pid=16270 port=8888

If you visit that page in your browser, and all the tests are passing, it should look like this

In case you want to show only the tests of one file you don't specify the path on the command line as it was with rspec. Instead you supply an additional parameter to the URL in your browser. E.g. http://localhost:8888/?spec=app.models.User
Add a failing test and you will be presented with the following

You can leave the webserver started by Jasmine running in the background while working on either your code or the test, and hit refresh in the browser to see if something changed after you are finished editing.

To run all jasmine tests from the command line, run the following command:

bin/rake jasmine:ci

Cucumber

Warning:

Before running any cucumber tests make sure that your PhantomJS version is at least 2.1.

Cucumber tests or cukes are the farthest from implementaion details and actual code problems. They really just describe a behavioural aspect of the system and test that. This fact becomes very clear if you have a look at one of the files inside the features/ folder. The test cases are written in plain english in a way even the person most averse to technology in the whole world could understand them.

Lets look at the Photos feature in features/desktop/photos.feature

@javascript
Feature: photos
    In order to enlighten humanity for the good of society
    As a rock star
    I want to post pictures of a button

  Background:
      Given a user with username "bob"
      When I sign in as "bob@bob.bob"

      And I am on the home page

  Scenario: deleting a photo will delete a photo-only post if the photo was the last image
    Given I expand the publisher
    And I attach the file "spec/fixtures/button.png" to hidden element "file" within "#file-upload"
    And I wait for the ajax to finish
    And I press "Share"
    And I wait for the ajax to finish

    When I go to the photo page for "bob@bob.bob"'s latest post
    And I follow "edit_photo_toggle"
    And I preemptively confirm the alert
    And I press "Delete Photo"
    And I go to the home page

    Then I should not see any posts in my stream

Of course, all this is not something a computer program can understand on its own. That is why there needs to be a bunch of code behind that in order for it to actually work. This is done in Ruby with the help of regular expressions. E.g. in features/step_definitions/posts_steps.rb it says

Then /^I should not see an uploaded image within the photo drop zone$/ do
  all("#photodropzone img").should be_empty
end

Then /^I should not see any posts in my stream$/ do
  all(".stream_element").should be_empty
end

Given /^"([^"]*)" has a public post with text "([^"]*)"$/ do |email, text|
  user = User.find_by_email(email)
  user.post(:status_message, :text => text, :public => true, :to => user.aspects)
end

Given /^"([^"]*)" has a non public post with text "([^"]*)"$/ do |email, text|
  user = User.find_by_email(email)
  user.post(:status_message, :text => text, :public => false, :to => user.aspects)
end

When /^The user deletes their first post$/ do
  @me.posts.first.destroy
end

When /^I click on the first block button/ do
  find(".block_user").click
end

Probably the best way to start with cukes - as with every other test system - would be to copy and paste existing tests that do something similar to what you are trying to achieve and adapt them to your specific case. Cucumber has an informative wiki, that will help getting you started.

Running all cucumber features is initiated by

bin/rake cucumber

This will take a little time to start up, but then you can watch your browser open and the terminal logging everything that is run, which takes about 7-21 minutes (depending on your machine). If you just want to run a single feature you can run

bin/cucumber features/desktop/photos.feature

This will output something similar to

Using the default profile...
...................................................

3 scenarios (3 passed)
48 steps (48 passed)
0m30.300s

Debugging

Since cucumber tests are run in phantomJS so without a real browser opening, you can't see them running. This makes tests harder to debug. That's why we introduced the SCREENSHOT_PATH environment variable. Launch tests setting this variable to have a screenshot of all the failing scenario at the moment they failed. So for example, the command line above will save screenshots of failing scenario in the diaspora installation folder.

SCREENSHOT_PATH="." bin/cucumber features/desktop/change_settings.feature

Guard

You can automatically run the affected tests as you edit the files, simply let guard running in the background:

bin/guard

Troubleshooting testsuites launch

I get tons of errors from PostgreSQL when running the specs

If you get errors that look like this:

...
  4) PollParticipation it is relayable it should behave like it is relayable validations on :author_id the author is on the parent object author's ignore list when object is created works if the object has no parent
     Failure/Error: Unable to find matching line from backtrace
     ActiveRecord::StatementInvalid:
       PG::ConnectionBad: PQsocket() can't get socket descriptor: BEGIN
     Shared Example Group: "it is relayable" called from ./spec/models/poll_participation_spec.rb:125
     # /home/user/.rvm/gems/ruby-2.2.1@diaspora/gems/activerecord-4.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:155:in `async_exec'
     # /home/user/.rvm/gems/ruby-2.2.1@diaspora/gems/activerecord-4.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:155:in `block in execute'
     # /home/user/.rvm/gems/ruby-2.2.1@diaspora/gems/activerecord-4.2.3/lib/active_record/connection_adapters/abstract_adapter.rb:473:in `block in log'
...

then you can, in development configurations, try to turn off SSL for PostgreSQL in its configuration file. E.g. in /etc/postgresql/9.4/main/postgresql.conf change line ssl = true to ssl = false and restart PostgreSQL.

That's it

In case you still have questions on how to write or use tests, join us on IRC or post your problem on the discourse.