Test running spins indefinitely when exception thrown from resilience pipeline

[Vijay-Braidable]

In VS I'm in debug mode, when I debug the test it seems to execute just fine, 100% of the time, when I run the test and the resiliancepipeline throws a couple of exceptions (for the multiple failures) it just spins forever (I left a 1 second test for many minutes to check this) unwinding a stacktrace. This did not happen when I was using the non-preview version of VS and XUnit, only when I ported to Preview with TUnit.

Running via dotnet test shows the stacktrace being written to the console for over 5 minutes.

I've speculatively included information I think may be helpful, let me know if I can provide something else.

TUnit Version: 0.2.26
Operating System: Windows 10 Pro
Visual Studio Version: 2022 (640bit) - Preview (17.12.0 Preview 4.0)
.NET Version: net8.0
Extensions: The only installed extension is ReSharper, I disabled this and the test still failed in the same way.

The Test:

    [Test]
    [MethodDataSource(typeof(TestBase), nameof(TestData))]
    public async Task TestSomething(SomeType someType)
    {
...

TestData in base class

    public static IEnumerable<SomeType > TestData()
    {
        yield return ManyTypes.SomeType;
    }

Generated Test Hooks:

// <auto-generated/>
#pragma warning disable
using global::System.Linq;
using global::System.Reflection;
using global::System.Runtime.CompilerServices;
using global::TUnit.Core;
using global::TUnit.Core.Interfaces;

namespace TUnit.SourceGenerated;

[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
file partial class Hooks_REDACTED_Tests : TUnit.Core.Interfaces.SourceGenerator.ITestHookSource
{
    [global::System.Runtime.CompilerServices.ModuleInitializer]
    public static void Initialise()
    {
        var instance = new Hooks_REDACTED_Tests();
        SourceRegistrar.RegisterTestHookSource(instance);
    }
    public global::System.Collections.Generic.IReadOnlyList<StaticHookMethod<TestContext>> CollectBeforeEveryTestHooks()
    {
        return
        [
        ];
    }
    public global::System.Collections.Generic.IReadOnlyList<StaticHookMethod<TestContext>> CollectAfterEveryTestHooks()
    {
        return
        [
        ];
    }
    public global::System.Collections.Generic.IReadOnlyList<InstanceHookMethod> CollectBeforeTestHooks()
    {
        return
        [
        ];
    }
    public global::System.Collections.Generic.IReadOnlyList<InstanceHookMethod> CollectAfterTestHooks()
    {
        return
        [
            new InstanceHookMethod<global::REDACTED.Tests>
            {
                 MethodInfo = typeof(global::REDACTED.Tests).GetMethod("REDACTED", 0, []),
                 Body = (classInstance, context, cancellationToken) => AsyncConvert.Convert(() => classInstance.REDACTED()),
                 HookExecutor = DefaultExecutor.Instance,
                 Order = 0,
            },
        ];
    }
}

Generated Tests:

// <auto-generated/>
#pragma warning disable
using global::TUnit.Core;
using global::System.Reflection;
using global::System.Linq;

namespace TUnit.SourceGenerated;

[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
file partial class REDACTED_Tests : TUnit.Core.Interfaces.SourceGenerator.ITestSource
{
    [global::System.Runtime.CompilerServices.ModuleInitializer]
    public static void Initialise()
    {
        SourceRegistrar.Register(new REDACTED_Tests());
    }
    public global::System.Collections.Generic.IReadOnlyList<SourceGeneratedTestNode> CollectTests()
    {
        return
        [
            ..Tests0(),
        ];
    }
    private global::System.Collections.Generic.List<SourceGeneratedTestNode> Tests0()
    {
        global::System.Collections.Generic.List<SourceGeneratedTestNode> nodes = [];
        var classDataIndex = 0;
        var testMethodDataIndex = 0;
        try
        {
            var testClassType = typeof(global::REDACTED.Tests);
            var methodInfo = typeof(global::REDACTED.Tests).GetMethod("REDACTED_MethodName", 0, [typeof(global::REDACTED.TestData)]);

            var objectBag = new global::System.Collections.Generic.Dictionary<string, object>();

            var resettableClassFactoryDelegate = () => new ResettableLazy<global::REDACTED.Tests>(() => 
                new global::REDACTED.Tests()
            );

            var resettableClassFactory = resettableClassFactoryDelegate();

            foreach (var methodData in global::REDACTED.Tests.SomethingTest())
            {
                testMethodDataIndex++;

                nodes.Add(new TestMetadata<global::REDACTED.Tests>
                {
                    TestId = $"global::TUnit.Core.MethodDataSourceAttribute:{testMethodDataIndex}:TL-EMDS0:{testMethodDataIndex}:REDACTED.Tests.REDACTED_MethodName(REDACTED.TestData):0",
                    TestClassArguments = [],
                    TestMethodArguments = [methodData],
                    TestClassProperties = [],
                    CurrentRepeatAttempt = 0,
                    RepeatLimit = 0,
                    MethodInfo = methodInfo,
                    ResettableClassFactory = resettableClassFactory,
                    TestMethodFactory = (classInstance, cancellationToken) => AsyncConvert.Convert(() => classInstance.REDACTED_MethodName(methodData)),
                    TestExecutor = DefaultExecutor.Instance,
                    ParallelLimit = null,
                    TestFilePath = @"REDACTED_PATH",
                    TestLineNumber = 165,
                    AttributeTypes = [ typeof(global::TUnit.Core.TestAttribute), typeof(global::System.Runtime.CompilerServices.NullableContextAttribute), typeof(global::System.Runtime.CompilerServices.NullableAttribute), typeof(global::System.Runtime.Versioning.TargetFrameworkAttribute), typeof(global::System.Reflection.AssemblyCompanyAttribute), typeof(global::System.Reflection.AssemblyConfigurationAttribute), typeof(global::System.Reflection.AssemblyFileVersionAttribute), typeof(global::System.Reflection.AssemblyInformationalVersionAttribute), typeof(global::System.Reflection.AssemblyProductAttribute), typeof(global::System.Reflection.AssemblyTitleAttribute), typeof(global::System.Reflection.AssemblyVersionAttribute), typeof(global::System.Reflection.AssemblyMetadataAttribute) ],
                    DataAttributes = [  ],
                    ObjectBag = objectBag,
                });
                resettableClassFactory = resettableClassFactoryDelegate();
            }
        }
        catch (global::System.Exception exception)
        {
            nodes.Add(new FailedInitializationTest
            {
                TestId = $"global::TUnit.Core.MethodDataSourceAttribute:{testMethodDataIndex}:TL-EMDS0:{testMethodDataIndex}:REDACTED.Tests.REDACTED_MethodName(REDACTED.TestData):0",
                TestClass = typeof(global::REDACTED.Tests),
                ReturnType = typeof(global::REDACTED.Tests).GetMethod("REDACTED_MethodName", 0, [typeof(global::REDACTED.TestData)]).ReturnType,
                ParameterTypeFullNames = [typeof(global::REDACTED.TestData)],
                TestName = "REDACTED_MethodName",
                TestFilePath = @"REDACTED_PATH",
                TestLineNumber = 165,
                Exception = exception,
            });
        }
        return nodes;
    }
}

[thomhurst]

What do you mean by resiliancepipeline?

[Vijay-Braidable]

It's a retry method encouraged by MS: https://learn.microsoft.com/en-us/dotnet/core/resilience/?tabs=dotnet-cli

The short version is that it creates a cyclic stack trace when it throws multiple exceptions (the pipeline retries fail).

[Vijay-Braidable]

I span up a PR to try to resolve the issue, I think I found the source (json serialization), but I'm not certain is I'm very new to this project: #1020

[thomhurst]

The change you've made is only for the rpc tests. If you're experiencing an issue with standard test execution then the rpc test project is completely separate to that so wouldn't resolve it

[Vijay-Braidable]

Ah, you're right, my test isn't an RPC test. I'll cancel the PR, though you might want to consider adding that in later to ensure correct serialization when types are cyclical.

Is there anything else I can grab from my test runs that might help locate the issue?

[thomhurst]

You say it works normally - can you provide source code with this resilience pipeline? I'm unfamiliar with it

[Vijay-Braidable]

The smallest toy example I can make quickly is without a pipeline:

    [Test]
    public void WhenThrowing_CyclicalException_TestShouldFinish2()
    {
        // Arrange
        var exceptionA = new Exception("Exception A");
        var exceptionB = new Exception("Exception B", exceptionA);

        // Use reflection to set exceptionA's InnerException to exceptionB, creating a cycle
        FieldInfo innerExceptionField = typeof(Exception).GetField("_innerException", BindingFlags.Instance | BindingFlags.NonPublic);
        innerExceptionField.SetValue(exceptionA, exceptionB);

        // Act: Throw exceptionA, which now has a cyclic InnerException reference
        throw exceptionA;
    }

When running this via TUnit it never finishes running the test for me.

[Vijay-Braidable]

Just in case it's relevant, here is the csproj file used when I made that test:

<Project Sdk="Microsoft.NET.Sdk">
  
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>

    <PackageId>REDACTED.$(AssemblyName)</PackageId>
    <RootNamespace>REDACTED.$(MSBuildProjectName.Replace(" ", "_"))</RootNamespace>
    <AssemblyName>$(MSBuildProjectName.Replace(" ", "_"))</AssemblyName>

  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="TUnit" />
    <PackageReference Include="FluentAssertions" />
    <PackageReference Include="Moq" />
  </ItemGroup>

</Project>

[thomhurst]

The smallest toy example I can make quickly is without a pipeline:

    [Test]
    public void WhenThrowing_CyclicalException_TestShouldFinish2()
    {
        // Arrange
        var exceptionA = new Exception("Exception A");
        var exceptionB = new Exception("Exception B", exceptionA);

        // Use reflection to set exceptionA's InnerException to exceptionB, creating a cycle
        FieldInfo innerExceptionField = typeof(Exception).GetField("_innerException", BindingFlags.Instance | BindingFlags.NonPublic);
        innerExceptionField.SetValue(exceptionA, exceptionB);

        // Act: Throw exceptionA, which now has a cyclic InnerException reference
        throw exceptionA;
    }

When running this via TUnit it never finishes running the test for me.

Thanks! I'm out right now so it might be a couple of days before I can look but I'll get on it when I have a chance

[Vijay-Braidable]

I embedded the test in TUnit.TestProject, I can break on all the pre test running lines, but as soon as the exception is thrown it vanishes into the ether. I'm starting to wonder is this is a deeper problem than the library, but in the new testing platform. If you think of a way for me to easily verify this while you're away please let me know!

[thomhurst]

It very well could be the new testing platform. Since you think it's related to the json rpc, TUnit doesn't deal with that specifically. And since I might not be available for a couple of days, would you be able to also raise it with Microsoft regarding the new platform?

Their repo is https://github.com/microsoft/testfx

[Vijay-Braidable]

Confirmed to exist in the latest version of MSTest, so it seems as though it's an underlying issue.

[thomhurst]

Glad to know it's not TUnit haha. But also hoping it gets solved soon!

[Vijay-Braidable]

Fingers crossed! It's killing my commit tests atm, I just swapped to TUnit and the new platform. Now do decorate all of my tests with timeouts...

[Vijay-Braidable]

So it turns out that the Timeout(TIME_IN_MS) attribute doesn't have a force stop, it's literally just the CancellationToken. I think this means I have to leave my test swap over in a branch and hope it doesn't get too stale while we wait for the fix, unless there is another "Force stop" option I'm missing?

[thomhurst]

So it turns out that the Timeout(TIME_IN_MS) attribute doesn't have a force stop, it's literally just the CancellationToken. I think this means I have to leave my test swap over in a branch and hope it doesn't get too stale while we wait for the fix, unless there is another "Force stop" option I'm missing?

This is by design. This isn't really isolated to TUnit either. There isn't really a way to abort threads other than requesting cancellation and then honouring it.

TUnit attempts to shut down early but it's entirely possible the original test method could still be executing in the background if it didn't honour any cancellations.

So I don't know what to suggest here other than use the cancellation token given to you. Aborting threads is a no-no (also TUnit doesn't start threads manually, it leverages the standard thread pool)

[Vijay-Braidable]

Sadly, as the thread lives forever once the exception is thrown, using the cancellation token appears to have no impact. Guess I’ll dig into the MSTest code before bed!

[thomhurst]

I'll need a repro to try and investigate myself, as like I said I'm unfamiliar with it. But if you can provide a simple repro repo then I'll be happy to take a look

[Vijay-Braidable]

The toy example I provided should be suitable for reproducing the error and cannot leverage a cancellation token. Even if the nested code could use it, when a cyclical exception is thrown (in this case by the MS pipeline library AFAIK) the code that is preventing the test from completing is not in the user's control.

[thomhurst]

[Test]
public void WhenThrowing_CyclicalException_TestShouldFinish2()
{
// Arrange
var exceptionA = new Exception("Exception A");
var exceptionB = new Exception("Exception B", exceptionA);

    // Use reflection to set exceptionA's InnerException to exceptionB, creating a cycle
    FieldInfo innerExceptionField = typeof(Exception).GetField("_innerException", BindingFlags.Instance | BindingFlags.NonPublic);
    innerExceptionField.SetValue(exceptionA, exceptionB);

    // Act: Throw exceptionA, which now has a cyclic InnerException reference
    throw exceptionA;
}

This seems to hang in NUnit too by the way

[thomhurst]

The toy example I provided should be suitable for reproducing the error and cannot leverage a cancellation token. Even if the nested code could use it, when a cyclical exception is thrown (in this case by the MS pipeline library AFAIK) the code that is preventing the test from completing is not in the user's control.

Got you. I think we need to leave this in the hands of the Microsoft.Testing.Platform guys, as they'll need to tweak how they handle the JSON serialization

[thomhurst]

Dotnet have closed the issue as by design, and it happens on the throw so it's not TUnit related. Closing

[Vijay-Braidable]

They only said it's by design as private reflection was used, I'll make another example today/tomorrow when I get time that doesn't require this.

I'm confident that something is causing the test runner or some related process to hang. I can replicate it 100% (as in at least one test hangs) of the time when I run a batch of tests in my solution on the new runner + TUnit, but I can't replicate it even once with XUnit and the non-testing-platform approach to testing.

[Vijay-Braidable]

Just looped back around to this. Seems the exception theory was incorrect, will have to find time to dig into this more. Ultimately I'm not using reflection and when I run my test suite with TUnit it hangs forever, but it does not with XUnit. I'm assuming it's TUnit or the testing platform causing the issue as they're the only things being swapped out, but I don't have time to keep trying to test it to figure out what exactly is going on.

If you have any tips on debugging the infinite running of certain tests (I can't share the tests) please let me know, there is an issue, I was just wrong about the cause.

[thomhurst]

Not sure what to suggest. If it's only happening on a subset of tests, try and figure out what those tests have in common that others don't?

In order for me or Microsoft to fix, it'll be much easier if we have a repro.

We don't need your exact test but if you can make a new test without sensitive data that reproduces the problem then we can likely solve it.