Helper methods after asserts end assert testing in block

[danielpclark]

Using minitest (5.11.3)

I've been noticing issues with some tests not running with the rest of my test suite for some time now and I think I've discovered the why. Here's an example:

require 'test_helper'
class RelativePathFromTest < Minitest::Test
  def test_relative_path_from
    assert_equal FasterPath.relative_path_from("a", "b").to_s, "../a"
  end

  def relative_path_str(dest, base)
    FasterPath.relative_path_from(dest, base).to_s
  end 

  def test_returns_a_path_relative_from_root
    assert_equal relative_path_str('/usr', '/'), 'usr'
  end
end

All of the tests before my relative_path_str method here will run. All of the tests after it won't. If I move my helper method to the very top of the class block then all the tests and assertions will run as I would expect them to. This sneaky bug has been egging me on for a while now.

[utilum]

Kindly help me understand, you define a method and test it both inside the same test?

[danielpclark]

Let me clarify the test.

require 'test_helper'
class RelativePathFromTest < Minitest::Test
  def test_relative_path_from
    assert_equal relative_path_str("a", "b"), "../a"
  end

  def relative_path_str(dest, base)
    FasterPath.relative_path_from(dest, base).to_s
  end 

  def test_returns_a_path_relative_from_root
    assert_equal relative_path_str('/usr', '/'), 'usr'
  end
end

Only the first test_ method runs and not the last one.

These two next examples all the tests will work.

require 'test_helper'
class RelativePathFromTest < Minitest::Test
  def relative_path_str(dest, base)
    FasterPath.relative_path_from(dest, base).to_s
  end 

  def test_relative_path_from
    assert_equal relative_path_str("a", "b"), "../a"
  end

  def test_returns_a_path_relative_from_root
    assert_equal relative_path_str('/usr', '/'), 'usr'
  end
end

and

require 'test_helper'
class RelativePathFromTest < Minitest::Test
  def test_relative_path_from
    assert_equal relative_path_str("a", "b"), "../a"
  end

  def test_returns_a_path_relative_from_root
    assert_equal relative_path_str('/usr', '/'), 'usr'
  end

  def relative_path_str(dest, base)
    FasterPath.relative_path_from(dest, base).to_s
  end 
end

[utilum]

Thank you, but what is it you are trying to do by testing a method defined inside the test?

[danielpclark]

what is it you are trying to do

Brevity. Having a method wrap around complex reused behavior makes the tests that use it shorter.

by testing a method defined inside the test?

Not sure we're on the same page here. This is standard practice in Ruby to write helper methods even within the context of the same test class. The Ruby language has many such methods written. The method I'm using above is taken from Ruby itself: see source.

The behavior in Minitest that is peculiar is that only consecutive tests will be tested.

I believe these work

TEST
TEST
TEST
HELPER

HELPER
TEST
TEST
HELPER

HELPER
TEST
TEST
TEST

This doesn't work

TEST
HELPER
TEST
TEST

Only the first test will run.

[zenspider]

I can't repro this and have hard time believing your claim without a repro given that you're saying that plain ruby method definition order is affecting test run outcome. (Did I understand that correctly?)

OK. Poking further:

Without the FasterPath code/assertions in place, I always get 2 test runs with your original code. Converting the code from FasterPath to Pathname also results in 2 test runs:

require "minitest/autorun"
require "pathname"

class RelativePathFromTest < Minitest::Test
  def test_relative_path_from
    assert_equal relative_path_str("a", "b").to_s, "../a"
  end

  def relative_path_str(dest, base)
    Pathname.new(dest).relative_path_from(Pathname.new(base)).to_s
  end

  def test_returns_a_path_relative_from_root
    assert_equal relative_path_str('/usr', '/'), 'usr'
  end
end

(ETA: is there something here I missed?)

Minitest just uses ruby methods and doesn't do anything special around them. Anything odd about how minitest finds/runs tests is going to be on ruby itself.

OR... I just looked... it will be on your extension. I'm not in a position to be able to repro w/ your rust extension. I gave up on rust years ago and don't have a dev environment for it anymore.

I think I've discovered the why.

Where is the "why"? Either I'm missing/misunderstanding it or you never got to it.

P.S. Those assertions are all backwards. assert_equal(exp, act)

[danielpclark]

I can't repro this and have hard time believing your claim without a repro given that you're saying that plain ruby method definition order is affecting test run outcome. (Did I understand that correctly?)

Yes

P.S. Those assertions are all backwards. assert_equal(exp, act)

Thanks.

Sadly I didn't quarantine a git commit with when this happened. I just repeated the same thing I did before by copying Ruby Spec code in and converting it from spec style to Minitest standard and I myself was unable to replicate the behavior.

This was what the test file looked like at the time: https://gist.github.com/danielpclark/fb0e794b10e7f63f0010c82407b6653e

The manner in which I tested it was through my VIM plugin which grabs the context of the current line I'm on and runs all the tests within scope.

This is the command it ran at the time:

bundle exec rake test TEST="test/relative_path_from_test.rb" TESTOPTS="--name='/RelativePathFromTest/'"

The output would only produce one dot indicating only the first test before my method helper was run. When I moved the method to the top the rest of the dots showed up.

Since I can't reproduce it I'm going to close this issue. It looks like more of a fluke than a Minitest issue. It remains within the realm of possibility that I ran my minitest specific line test on the wrong line which would have made the first method the only scope to test with:

bundle exec rake test TEST="test/relative_path_from_test.rb" TESTOPTS="--name='/RelativePathFromTest#test_relative_path_from$/'"

But I know I attempted the test twice to double check and I'm in the habit of looking at the command line argument generated so I was pretty sure at the time that I had done what I meant to do in testing the whole test class. Now I'm not so sure.

Thanks you both for taking the time to check with me on this. I really appreciate it.

I gave up on rust years ago and don't have a dev environment for it anymore.

If you ever want to get into it I'd be more than glad to help. I'm preparing several resources on teaching it including more specific Ruby & Rust integration.

[zenspider]

Sadly I didn't quarantine a git commit with when this happened. I just repeated the same thing I did before by copying Ruby Spec code in and converting it from spec style to Minitest standard and I myself was unable to replicate the behavior.

1. Are you working on a mac?
2. Do you have time machine on?
3. Do you have /Volumes/MobileBackups/?

If yes to all 3, then you might have 3 days of local backups that you can pull from to replicate your exact setup. Try:

% ls /Volumes/MobileBackups/Backups.backupdb/*/Latest/*/$PWD

[zenspider]

I added the following to your gist:

FasterPath = Pathname

class FasterPath
  def self.relative_path_from a, b
    new(a).relative_path_from(Pathname.new(b)).to_s
  end
end

and get 11 tests run via ntimes 20 ruby bug748.rb.

[danielpclark]

Are you working on a mac?

No… Ubuntu Linux.

I added the following to your gist:

FasterPath = Pathname

Thanks for reminding me. I forgot to update the helper method:

  def relative_path_str(dest, base)
    FasterPath.relative_path_from(dest, base).to_s
  end 

I still can't reproduce the issue.

you can pull from to replicate your exact setup

This was some code I didn't use git commit on because I realized I took it from ruby/spec and my project already runs those tests as a submodule. I didn't leave anything in git stash and didn't accidentally leave the VIM swap file behind to restore from.

Although seeing all the effort you're going to make me think I should really setup a staging backup like Mac has for 3 days of revisions. I can make a wrapper around the rake command using my shell (fish shell) and have it stage copies of the current work directory every time rake is run. Then I just need to have a cron job clean up anything that's out-dated … maybe I'll try 7 days.

Reproducible states of each step through development is a really great idea.

[danielpclark]

As for you being able to run my gem on your local machine I now have Github Releases holding builds of the library so you don't need Rust for any of the recent Ruby versions. During gem install it will simply download the library from Github and work. Right now I've only tested that feature from Linux and need some one with a Mac to try it danielpclark/faster_path#143 .

You don't have to test it for me, but I just wanted you to know that you can actually try my project without Rust.

[danielpclark]

Thank you for your backup remedy suggestion. I've now implemented project logging and backups for every time the rake command is run on my system. I've written a short blog on how I did it as well. Next time I ever come to you with an issue I'll have the exact state in which it occurred available.