Skip to content

2.12.0
Compare
Choose a tag to compare
@qameta-ci qameta-ci released this 02 Apr 10:08
· 21 commits to refs/heads/main since this release
2.12.0
11d695e

What's Changed

🚀 New adapter for Reqnroll
⚠️ Namespace changes in Allure.NUnit and Allure.Xunit
🎯 New target frameworks of Allure.Xunit
🔬 Improvements
    New implementation of AddScreenDiff (#435)
    Standard failed/broken classification
    Other improvements
🐞 Bug fixes
Deprecations and removals
Other changes
    ⬆️ Dependency Updates
    📦 Packaging changes
    👻 Internal changes
Pull requests and contributors
New contributors

🚀 New adapter for Reqnroll

Meet Allure.Reqnroll - a new Allure adapter for Reqnroll.

Give it a try by installing the Allure.Reqnroll package to your Reqnroll project. Once it's done, you're all set! Run your tests and create the report from the allure-results in the project output directory.

The documentation is currently in progress. Once it's done, the link will be added to this description and the root README file of the repo.

Notable differences with Allure.SpecFlow

Since Reqnroll is a fork of SpecFlow (it was forked back by Gáspár Nagy (gasparnagy), the original creator of SpecFlow), adapters for Reqnroll and SpecFlow have some similarities. However, some notable changes are listed below.

Support for xUnit.net

Allure.Reqnroll correctly works with all unit test frameworks supported by Reqnroll out-of-the-box: NUnit, xUnit.net, and MSTest. In contrast, Allure.SpecFlow currently can't work with xUnit.net due to problems with Allure context isolation (see #431).

Zero configuration

Allure.Reqnroll integrates with Reqnroll in a way that differs from how Allure.SpecFlow integrates with SpecFlow. Because of that, users are not required to alter the Reqnroll configuration to enable the adapter.

SpecFlow, on the other hand, requires users to add an entry to the stepAssemblies list in specflow.json. A missing entry leads to an error and is a common source of problems.

allureConfig.json format changes

Allure.SpecFlow features the specflow section of allureConfig.json where you may define Gherkin tag patterns and configure the table-to-parameter conversion.

In Allure.Reqnroll that section was moved to allure/gherkinPatterns. Some properties were renamed. The packages sub-section was removed (the package label is automatically filled by Allure.Reqnroll).

An example: 
{
  "allure": {
    "gherkinPatterns": {
      "stepArguments": {
        "createFromDataTables": false,
        "nameColumn": "ParameterName",
        "valueColumn": "ParameterValue"
      },
      "grouping": {
        "suites": {
          "parentSuite": "allure\\.parentSuite:(.+)",
          "suite": "allure\\.suite:(.+)",
          "subSuite": "allure\\.subSuite:(.+)"
        },
        "behaviors": {
          "epic": "allure\\.epic:(.+)",
          "story": "allure\\.story:(.+)"
        }
      },
      "metadata": {
        "owner": "allure\\.owner:(.+)",
        "severity": "(normal|blocker|critical|minor|trivial)",
        "label": "allure\\.label\\.([^:]+):(.+)"
      },
      "links": {
        "link": "allure\\.link:(.+)",
        "issue": "allure\\.issue:(.+)",
        "tms": "allure\\.tms:(.+)"
      }
    }
  }
}

The schema can be accessed at https://raw.githubusercontent.com/allure-framework/allure-csharp/2.12.0/Allure.Reqnroll/Schemas/allureConfig.schema.json.

Gherkin pattern defaults

Allure.SpecFlow allows users to define patterns to convert Gherkin tags to Allure data. Allure.Reqnroll provides reasonable defaults for such patterns. That also contributes to the zero-configuration experience of Allure Reqnroll (see #439).

The list of the default patterns

The default patterns used by Allure Reqnroll are equivalent to having the following allureConfig.json in place:

{
  "allure": {
    "gherkinPatterns": {
      "stepArguments": {
        "createFromDataTables": false
      },
      "grouping": {
        "suites": {
          "parentSuite": "allure\\.parentSuite:(.+)",
          "suite": "allure\\.suite:(.+)",
          "subSuite": "allure\\.subSuite:(.+)"
        },
        "behaviors": {
          "epic": "allure\\.epic:(.+)",
          "story": "allure\\.story:(.+)"
        }
      },
      "metadata": {
        "owner": "allure\\.owner:(.+)",
        "severity": "(normal|blocker|critical|minor|trivial)",
        "label": "allure\\.label\\.([^:]+):(.+)"
      },
      "links": {
        "link": "allure\\.link:(.+)",
        "issue": "allure\\.issue:(.+)",
        "tms": "allure\\.tms:(.+)"
      }
    }
  }
}

Reliable test plans

Unlike Allure SpecFlow, Allure.Reqnroll doesn't utilize runtime patching to implement test plans. That means features that depend on test plans (e.g., selective run) will work with Allure.Reqnroll in areas where they might not work with Allure.SpecFlow. That includes:

  • ARM machines
  • Release builds
  • New versions of .NET

⚠️ Namespace changes in Allure.NUnit and Allure.Xunit

In this release, we target some historical discrepancies between package names and namespaces of Allure.NUnit and Allure.Xunit.

Old namespaces

Allure.NUnit

The root namespace of Allure.NUnit has been NUnit.Allure up until now. That's the namespace Nick Chursin (unickq) used in his original allure-nunit project until the project was transferred to allure-framework. It matched the package name at the time (NUnit.Allure; it is still available at nuget.org, although deprecated).

Once we changed the package name according to our general convention Allure.<Framework>, the namespace stopped matching the package name. That has created an extra barrier for users who usually expect the root namespace to be the same as the package name. That may also cause users to mix the code of Allure.NUnit and the code of NUnit.

Allure.Xunit

In Allure.XUnit, there are two namespaces: Allure.Xunit and Allure.XUnit (note the casing of u). The public API has been split between both of them at the same time. The attributes have been contained in Allure.Xunit while some other APIs, like AllureStepAttribute - are in Allure.XUnit.

New namespaces

Now, the root namespace of Allure.NUnit is Allure.NUnit. The root namespace of Allure.Xunit is Allure.Xunit.

Additionally, the NUnit.Allure.Core.AllureNUnitAttribute should now be accessed via the root namespace instead of the Core sub-namespace, i.e., as Allure.NUnit.AllureNUnitAttribute.

We advise you to transition accordingly.

Examples

For example, given the following code:

using NUnit.Framework;
using NUnit.Allure.Attributes;
using NUnit.Allure.Core;

namespace MyNamespace;

[AllureNUnit]
class MyTestClass
{
    [Test]
    public void MyTest()
    {
        /* ... */
    }

    [AllureStep]
    void MyStep()
    {
        /* ... */
    }
}

It should be change to the following one:

using NUnit.Framework;
using Allure.NUnit.Attributes;
using Allure.NUnit;

namespace MyNamespace;

[AllureNUnit]
class MyTestClass
{
    [Test]
    public void MyTest()
    {
        /* ... */
    }

    [AllureStep]
    void MyStep()
    {
        /* ... */
    }
}

IDE features like "find & replace all" might assist you in the transition.

The transition becomes trivial if you've added Allure namespaces to the list of global usings. Given the following part of the .csproj file:

<Project Sdk="Microsoft.NET.Sdk">
  <!-- ... -->

  <ItemGroup>
    <Using Include="Allure.Xunit.Attributes">
    <Using Include="Allure.XUnit.Attributes.Steps">

    <!-- Other usings -->
    <!-- ... -->
  </ItemGroup>

</Project>

Just a single character should be changed:

<Project Sdk="Microsoft.NET.Sdk">
  <!-- ... -->

  <ItemGroup>
    <Using Include="Allure.Xunit.Attributes">
    <Using Include="Allure.Xunit.Attributes.Steps"> <!-- Note the casing here -->

    <!-- Other usings -->
    <!-- ... -->
  </ItemGroup>

</Project>

Backward compatibility

The old namespaces remain in place for both Allure.NUnit and Allure.Xunit. Accessing the API through them now triggers the CS0618 warning. While we advise you to transition to new namespaces, you may ignore the warnings and continue using the old ones.

We will remove the public API access from the old namespaces in the next major release.

🎯 New target frameworks of Allure.Xunit

Allure.Xunit now targets netstandard2.0 (down from netstandard2.1) and netcoreapp3.1 (increased from netcoreapp2.0 because of security issues; for example: CVE-2021-26701).

Due to how xUnit.net dependencies are managed, that means that the package works in the following runtimes:

  • .NET Core 3.1
  • .NET 5.0 or greater

Other runtimes are excepted to work except the following:

  • .NET Framework (any version)
  • .NET Core before 3.1

The general rule is this:

The runtime is supported by Allure.Xunit if, given that runtime, Nuget selects an assembly of xunit.runner.utilities that is compatible with the chosen assembly of Allure.Xunit. I.e.:

  • If the netcoreapp3.1 version of Allure.Xunit is selected, the netcoreapp1.0(*) version of xunit.runner.utilities is also selected, OR
  • If the netstandard2.0 version of Allure.Xunit is selected, the netstandard1.5(*) version of xunit.runner.utilities is also selected.

(*) Specific versions of xunit.runner.utilities's target frameworks are subject to change by the xUnit.net team in the future.

🔬 Improvements

New implementation of AddScreenDiff (#435)

AllureApi.AddScreenDiff now aligns with allure-framework/allure2#1145. The new implementation allows multiple screen diffs to be attached to a single test result.

Standard failed/broken classification

Previously, each adapter has used its own rules to distinguish between failed and broken tests/steps/fixtures:

  • Allure.NUnit has checked if the error message provided by NUnit contains at least one of the magic words that makes the test broken. If no match is found, the test is considered failed. The words had to be specified via the brokenTestData configuration property. Usually, those were type names of exceptions.
  • Allure.Xunit has interpreted a test that resulted in an unhandled exception of a type from the Xunit.Sdk namespace as failed; all other exceptions have corresponded to the broken status.
  • Allure.SpecFlow has resolved tests/steps as failed, regardless of the exception type.

Now, a general algorithm is implemented in Allure.Net.Commons. It checks the exception's type against a pre-defined list of type names associated with assertion errors. Each adapter provides its own list of such exception types. Users may redefine the list via the failExceptions property of allureConfig.json:

{
  "allure": {
    "failExceptions": [
      "NUnit.Framework.AssertionException",
      "MyCustomAssertionException"
    ]
  }
}

If an exception matches an entry from the list, the status is resolved as failed. Otherwise, it's resolved as broken. An entry is considered a match if it equals to the full name of one of the following types:

  • the type of the exception,
  • one of the exception's base classes,
  • one of the interfaces implemented by the exception.

Currently, if a type denoted by failExceptions is generic, it must be a closed constructed class.

Steps and fixtures added via AllureApi, ExtendedApi, [AllureStep], [AllureBefore], or [AllureAfter] use this algorithm automatically.

Allure.Xunit also uses that algorithm to resolve a test's status.

Allure.Reqnroll and Allure.SpecFlow both use it when resolving statuses of steps and features created from bindings, as well as statuses of test results created from scenarios.

Due to the limitations of NUnit, Allure.NUnit uses a different algorithm that analyzes assertion results provided by NUnit to distinguish between tests with unhandled exceptions and tests with a failed assertion. The new algorithm is more reliable than the old one and requires no extra configuration.

Other improvements

  • AllureLifecycle.ScheduleTestCase(TestResult testResult) was added to schedule a test case. The test case should then be started via AllureLifecycle.StartTestCase().
  • Allure.NUnit now doesn't include empty console logs and status details in the report (#363).
  • Allure.NUnit now doesn't emit empty containers for ignored tests.
  • Allure.SpecFlow now includes scenario descriptions in the report as test result descriptions (#429).

🐞 Bug fixes

  • A parameter serialization algorithm now falls back to ToString() in case serialization fails (#438, #448).
  • dotnet test now doesn't return exit code 1 if Harmony fails to patch xUnit.net, but all tests pass (#441).
  • Now assemblies and Nuget packages depend on the same version of AspectInjector.
  • Now Allure.NUnit correctly reports fixtures if some but not all [OneTimeTearDown] methods of a test class are annotated with [AllureAfter].
  • [AllureAfter] now doesn't throw IndexOutOfRangeException if applied to a method that has parameters (#419).
  • Allure.NUnit now doesn't report ignored tests if they aren't included in the test plan, even if [AllureDisplayIgnored] is applied to the class.
  • Allure.Xunit now only scans assemblies matching the *reporters* pattern when trying to find a second reporter. That improves the startup time and prevents assembly loading exceptions in many scenarios (#412).
  • Now Allure.Xunit doesn't warn about missing parameters of theories skipped with [Theory(Skip = "...")] (#421).

Deprecations and removals

Deprecations and removals in Allure.Net.Commons

The following previously deprecated API is removed from Allure.Net.Commons:

  • AllureLifecycle:
    • CurrentTestIdGetter (it's not currently in use)
    • Old UUID-based lifecycle management methods (their UUID-less counterparts should be used instead):
      • StartTestContainer(string parentUuid, TestResultContainer container)
      • UpdateTestContainer(string uuid, Action&lt;TestResultContainer> update)
      • StopTestContainer(string uuid)
      • WriteTestContainer(string uuid)
      • StartBeforeFixture(FixtureResult result, out string uuid)
      • StartBeforeFixture(string uuid, FixtureResult result)
      • StartBeforeFixture(string parentUuid, FixtureResult result, out string uuid)
      • StartBeforeFixture(string parentUuid, string uuid, FixtureResult result)
      • StartAfterFixture(string parentUuid, FixtureResult result, out string uuid)
      • StartAfterFixture(string parentUuid, string uuid, FixtureResult result)
      • UpdateFixture(string uuid, Action&lt;FixtureResult> update)
      • StopFixture(string uuid)
      • StartTestCase(string containerUuid, TestResult testResult)
      • UpdateTestCase(string uuid, Action&lt;TestResult> update)
      • StopTestCase(string uuid)
      • WriteTestCase(string uuid)
      • StartStep(StepResult result, out string uuid)
      • StartStep(string uuid, StepResult result)
      • StartStep(string parentUuid, string uuid, StepResult result)
      • UpdateStep(string uuid, Action&lt;TestResult> update)
      • StopStep(string uuid)
    • User API methods (AllureApi/ExtendedApi should be used instead):
      • AddAttachment (all overloads)
      • AddScreenDiff (all overloads)
  • Old step-related classes (they currently do nothing and shouldn't be used):
    • StepFailedException
    • IStepLogger
    • IStepActionLogger
  • CoreStepsHelper (AllureApi/ExtendedApi should be used instead)

Deprecations and removals in Allure.NUnit

The following API is deprecated now in Allure.NUnit:

  • AllureExtensions:
    • WrapSetUpTearDownParams - it has no effect.
    • WrapInStep (all overloads) - use AllureApi.Step instead.

The following previously deprecated API is removed from Allure.NUnit:

  • AllureNUnitHelper(*):
    • WrapInStep - use AllureApi.Step or [AllureStep] instead.
    • SaveOneTimeResultToContext - it wasn't meant to be public and shouldn't be used.
    • AddOneTimeSetupResult - it wasn't meant to be public and shouldn't be used.
  • StepsHelper - [AllureStep], [AllureBefore], [AllureAfter], AllureApi, or ExtendedApi instead.
  • AllureNUnitAttribute:
    • AllureNUnitAttribute(bool wrapIntoStep) - the wrapIntoStep parameter has no effect and can safely be removed.
  • AllureExtensions:
    • AddScreenDiff - use AllureApi.AddScreenDiff instead.

(*) The AllureNUnitHelper and AllureExtensions classes are now internal.

Deprecations and removals in Allure.Xunit

The following previously deprecated API is removed from Allure.Xunit:

  • Using-style steps/fixtures:
    • Allure.Xunit.AllureBefore
    • Allure.Xunit.AllureAfter
    • Allure.Xunit.AllureStep
    • Allure.Xunit.AllureStepBase
      Use [AllureStep], [AllureBefore], [AllureAfter], AllureApi, or ExtendedApi instead.
  • AllureAttachments - use AllureApi.AddAttachment or Allure.Xunit.Attachments instead.
  • AllureXunitHelper:
    • StartTestContainer
    • StartTestCase
    • MarkTestCaseAsFailedOrBroken
    • MarkTestCaseAsPassed
    • MarkTestCaseAsSkipped
    • FinishTestCase
      Those methods weren't meant to be public and shouldn't be used at all.
  • Steps - [AllureStep], [AllureBefore], [AllureAfter], AllureApi, or ExtendedApi instead.
  • AllureXunitAttribute and AllureXunitTheoryAttribute - use [Fact] and [Theory] instead.

The following APIs are internal now (those wasn't meant to be public in the first place):

  • AllureXunitFacade
  • AllureXunitHelper
  • AllureMessageSink

Other changes

⬆️ Dependency Updates

  • Bump Lib.Harmony from 2.3.0-prerelease.2 to 2.3.3 for Allure.Xunit and Allure.SpecFlow.
  • Direct dependency on AspectInjector was removed from Allure.Xunit (it's still a transitive dependency).
  • Bump Microsoft.SourceLink.GitHub from 1.1.1 to 8.0.0.

📦 Packaging changes

  • Links to release notes are now included in all packages.
  • Allure.Xunit was renamed from Allure.XUnit to match the name used by xUnit.net. The only difference is in casing which shouldn't be a problem because Nuget operates in a case insensitive manner even on case sensitive operating systems.

👻 Internal changes

  • Stale unused projects, files and solutions are removed from the repo
  • CI workflows are updated to comply with the Node.js 16 deprecation by GitHub. See here for more details.
  • The issue and PR templates were updated/fixes.
  • Nuget.config was updated to clear any existing sources.
  • Nuget restore now uses default packages location to benefit more from caching.
  • The Commons and Adapters solution configuration were replaced with Publish. The Publish configuration includes all package projects in Release mode.
  • The test and examples projects now target .NET 6.0. Their dependencies were bumped to the most recent versions.
  • The Allure.XUnit.Reporters project was removed. Allure.Xunit now produces Allure.Xunit.reporters.dll.
  • The Allure.SpecFlowPlugin project was renamed to Allure.SpecFlow.
  • The Allure.SpecFlowPlugin.Tests project was renamed to Allure.SpecFlow.Tests.
  • The Allure.Features project was renamed to Allure.SpecFlow.Tests.Samples.
  • Allure.SpecFlow.Tests now uses correct configuration when running dotnet test against the samples project.
  • Increase reliability of Allure.SpecFlow tests.

Pull requests and contributors

The full list of pull requests of the release:

New contributors

Full Changelog: 2.11.0...2.12.0

Contributors

delatrie, TonEnfer, and dependabot
Assets 2
Loading