As this is not a priority for 2.2 for now, moving to the backlog.

@mkArtakMSFT just to clarify the concrete change we should make in 2.2:

We should make it so that it's possible to boot up the WebApplicationFactory (or another derived type) without a test server. It makes functional testing of your application absolutely trivial and the changes required to do this should be small.

/cc @javiercn

I'm not sure I follow. How would these changes make it easier to use something like puppeteer and chrome headless/selenium for automated browser testing. What is a real server?

@steveoh the current set up allows for very convenient Unit Testing by spinning up the App/WebHost and talking to it 'in memory." So no HTTP, no security issue, you're basically talking HTTP without actually putting bytes on the wire (or localhost). Super useful.

But if you want to use Automated Browser Testing and have it driven by a Unit Test Framework, you are hampered by the concrete TestServer class and have to do a little dance (above) to fake it out and start up the WebHost yourself. I'd propose no regular person could/would figure that out.

David is saying that if WebApplicationFactory could be started without the fake TestServer we could easily and cleanly do Unit Testing AND Browser Automation Testing with the piece we have here.

This is why we love Scott! Always the advocate for the mainstream developers out there wanting to use MS tools, but stymied by various obscure limitations. Thank you sir!

The provided workaround has problems.

The problems start with the fact that now we have 2 hosts. One from the TestServer, and another built to work with http. And WebApplicationFactory.Server references the first one, which we are not testing against. And to make things worse, calls to methods that configure the builder, such as WebApplicationFactory.WithWebHostBuilder will not work with the dummy TestServer.

Because of these problems we cannot easily interact with the real host, the one being tested. It is very common that I change some backend service and configure it before a test is run. Suppose I need to access some service that is not callable during development, only production. I replace that service when I configure the services collection with a fake, and then configure it to respond the way I want it to respond. I can't do that through WebApplicationFactory.Server.Host.Services.

The resulting code I have works, but it is ugly as hell, it is an ugly ugly hack.

I hope we can move this forward and do not require TestServer, maybe an IServer. I thought about forking the whole TestServer and WebApplicationFactory infrastructure, but as this is somewhat planning I'll wait. I hope it gets fixed soon. I am just commenting to complement that the provided workaround is not enough and to really work around you have to avoid WebApplicationFactory.Server and create a terrible work around it.

One way I solved this is to stop using WebApplicationFactory<T> all together.

Refactored Program.Main() to:

public static Task<int> Main(string[] args)
{
   return RunServer(args);
}
public static async Task<int> RunServer(string[] args,
                                        CancellationToken cancellationToken = default)
{
    ...
    CreateWebHostBuilder()
          .Build()
          .RunAsync(cancellationToken)
}

So, my unit test fixtures new up a var cts = new CancellationTokenSource(), then pass the cancellation token by calling Program.RunServer(new string[0], cts.Token). The server starts up as normal without having to create a separate process.

Make real HTTP calls like normal. When your done and your unit test completes, call cts.Cancel() to clean up and shutdown the HTTP server.

One down side is you need to copy appsettings.json to an output directory from your test project; potentially managing two appsettings.json files (one in your server project, and one in your test project). Maybe a linked project file could help eliminate the issue.

YMMV.

🤔 ✨ "Do you ponder the manner of things... yeah yeah... like glitter and gold..."

Hello everyone, this issue is still unresolved and seems to keep being postponed. Just so we know the planning, are you considering resolving this for ASP.NET Core 3.0? If not, do you have a workaround that does not incur on the problems I mentioned earlier (#4892 (comment))?

This would be very welcome. The workaround mentioned here is great, except when you have html files and what not that you would also need to copy over.

WebApplicationFactory is designed for a very specific in-memory scenario. Having it start Kestrel instead and wire up HttpClient to match is a fairly different feature set. We actually do have components that do this in Microsoft.AspNetCore.Server.IntegrationTesting but we've never cleaned them up for broader usage. It might make more sense to leave WebApplicationFactory for in-memory and improve the IntegrationTesting components for this other scenario.

I'm fine with that, as long as we have a good end to end testing story.

Thanks guys for this discussion and specially to @bchavez
I succeeded to run Selenium tests in Azure pipelines with the idea to start a local server on the agent, run tests and stop the server.

My test class:

public class SelenuimSampleTests : IDisposable
{
    private const string TestLocalHostUrl = "http://localhost:8080";

    private readonly CancellationTokenSource tokenSource;

    public SelenuimSampleTests()
    {
        this.tokenSource = new CancellationTokenSource();

        string projectName = typeof(Web.Startup).Assembly.GetName().Name;

        string currentDirectory = Directory.GetCurrentDirectory();
        string webProjectDirectory = Path.GetFullPath(Path.Combine(currentDirectory, $@"..\..\..\..\{projectName}"));

        IWebHost webHost = WebHost.CreateDefaultBuilder(new string[0])
            .UseSetting(WebHostDefaults.ApplicationKey, projectName)
            .UseContentRoot(webProjectDirectory) // This will make appsettings.json to work.
            .ConfigureServices(services =>
            {
                services.AddSingleton(typeof(IStartup), serviceProvider =>
                {
                    IHostingEnvironment hostingEnvironment = serviceProvider.GetRequiredService<IHostingEnvironment>();
                    StartupMethods startupMethods = StartupLoader.LoadMethods(
                        serviceProvider, 
                        typeof(TestStartup),
                        hostingEnvironment.EnvironmentName);

                    return new ConventionBasedStartup(startupMethods);
                });
            })
            .UseEnvironment(EnvironmentName.Development)
            .UseUrls(TestLocalHostUrl)
            //// .UseStartup<TestStartUp>() // It's not working
            .Build();

        webHost.RunAsync(this.tokenSource.Token);
    }

    public void Dispose()
    {
        this.tokenSource.Cancel();
    }

    [Fact]
    public void TestWithSelenium()
    {
        string assemblyLocation = System.Reflection.Assembly.GetExecutingAssembly().Location;
        string currentDirectory = Path.GetDirectoryName(assemblyLocation);

        using (ChromeDriver driver = new ChromeDriver(currentDirectory))
        {
            driver.Navigate().GoToUrl(TestLocalHostUrl);
            IWebElement webElement = driver.FindElementByCssSelector("a.navbar-brand");

            string expected = typeof(Web.Startup).Assembly.GetName().Name;

            Assert.Equal(expected, webElement.Text);

            string appSettingValue = driver.FindElementById("myvalue").Text;
            const string ExpectedAppSettingsValue = "44";
            Assert.Equal(ExpectedAppSettingsValue, appSettingValue);
        }
    }
}

Very similar to the @bchavez 's example, I'm starting the CreateDefaultBuilder. That approach gives me a little bit more flexibility to set custom settings. For example using a TestStartup. the .UseStartup<TestStartUp>() wasn't working, so I used logic from WebHostBuilderExtensions.UseStartUp to set WebHostDefaults.ApplicationKey and configure services.

I'm sure there is better approach somewhere, but after а few days of research, nothing helped me. So I share this solution in case someone need it.

The test method is just for an example, there are different approaches to initialize browser driver.

@M-Yankov I had your code work by using IWebHostBuilder.UseStartup<>, and IWebHost.StartAsync as well as IWebHost.StopAsync, without CancellationTokenSource.
=> .NetCore 2.2

The idea of the IWebHostBuilder.UseStartup<> is to simplify the code. But for me it was not enough: after applying the UseStartup<TestStartup>(), ChromeWEbDriver cannot open the home page.
I didn't mention that the TestStartup class inherits the real startup class with overriding two custom methods (if it matters).
That's why I've used the .ConfigureServices(services => ... approach.
About the IWebHost.StartAsync & IWebHost.StopAsync - it seems that they are working. Thanks.

This is a solution that worked for me in the mean time. It ensures that everything on WebApplicationFactory, like the Services property keep working as expected:

    protected override void ConfigureWebHost(IWebHostBuilder builder) {
        if (builder == null) throw new ArgumentNullException(nameof(builder));

        IPEndPoint endPoint;
        // Assign available TCP port
        using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)){
            socket.Bind(new IPEndPoint(IPAddress.Loopback, 0));
            socket.Listen(1);
            endPoint = (IPEndPoint)socket.LocalEndPoint;
        }

        return builder
            .ConfigureKestrel(k => k.Listen(new IPEndPoint(IPAddress.Loopback, 0)));
    }

    protected override TestServer CreateServer(IWebHostBuilder builder)
    {
        // See: https://github.com/aspnet/AspNetCore/issues/4892
        this._webHost = builder.Build();

        var testServer = new TestServer(new PassthroughWebHostBuilder(this._webHost));
        var address = testServer.Host.ServerFeatures.Get<IServerAddressesFeature>();
        testServer.BaseAddress = new Uri(address.Addresses.First());

        return testServer;
    }

    private sealed class PassthroughWebHostBuilder : IWebHostBuilder
    {
        private readonly IWebHost _webHost;

        public PassthroughWebHostBuilder(IWebHost webHost)
        {
            this._webHost = webHost;
        }

        public IWebHost Build() => this._webHost;

        public IWebHostBuilder ConfigureAppConfiguration(Action<WebHostBuilderContext, IConfigurationBuilder> configureDelegate){
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(this.ConfigureAppConfiguration)}");
            return this;
        }

        public IWebHostBuilder ConfigureServices(Action<WebHostBuilderContext, IServiceCollection> configureServices) {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(ConfigureServices)}");
            return this;
        }

        public IWebHostBuilder ConfigureServices(Action<IServiceCollection> configureServices)
        {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(ConfigureServices)}");
            return this;
        }

        public string GetSetting(string key) => throw new NotImplementedException();

        public IWebHostBuilder UseSetting(string key, string value)
        {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(this.UseSetting)}({key}, {value})");
            return this;
        }
    }

Just a note: with .net core 3.0 apps using IHostBuilder, @danroth27's workaround no longer works.

Now I just have a test script that runs the server, then runs the tests, waits for them to finish, then kills the server. Have to start up the server manually when running in the Visual Studio test runner. Not a great experience.

I'm not using IWebHostBuilder so CreateServer is never called.

There's an override you can define create a custom IHostBuilder and another to create a custom IHost with a builder passed in, and that's likely where the solution would be. However, it didn't work when I tried it. I don't know what part of the internals of WebApplicationFactory adds the in-memory restrictions but it might to be outside of those two overloads. I've already spent too much time on it, and running the app directly without WebApplicationFactory seems to work fine for now.

What is the status on this? @davidfowl did this ever make it to 2.2? In that case, how do we use it?

@ffMathy not much progress has been made on this scenario, it's still in the backlog.

Alright. Is there an ETA? Rough estimate?

@ffMathy this is an uncommitted feature, there's no ETA until we decide to move it from the backlog to a milestone.

@Sebazzz Your code above does not compile.

• You are returning ConfigureKestrel, even though the method is void.
• You are assigning a variable _webHost in CreateServer that is not in the listing. Is this just a class variable or is it supposed to be coming from somewhere else?
• You setup an endpoint in ConfigureWebHost and then you don't use it, you just create another one in the call to ConfigureKestrel.

@lukos That's not correct. IServerAddressesFeature is absolutely there. If you have a piece of code that reproduces the issue please paste it here.

The code has churned a bit since then but I think it was simply the following code, taken from Scott's example and modified for IHostBuilder that was the problem:

protected override IHostBuilder CreateHostBuilder()
{
    var builder = base.CreateHostBuilder();
    // Logging added here
    return builder;
}

protected override IHost CreateHost(IHostBuilder builder)
{
    host = builder.Build();
    host.Start();
    var features = host.Services.GetRequiredService<IServer>().Features;
    RootUri = features.Get<IServerAddressesFeature>().Addresses.FirstOrDefault();  // Null reference exception
}

I haven't tried with a more complex scenario (and I'm a complete noob) - but it seems to work fine with a KestrelServer if I simply ignore the type being wrong (both KestrelServer and TestServer implement IServer?): Repro

Am I missing the point?

Does anyone have an update on how to do this now? It seems like WebApplicationFactory doesn't exist anymore... (in Microsoft.AspNetCore.Mvc.Testing version 5.0.4)

I found this: https://www.meziantou.net/automated-ui-tests-an-asp-net-core-application-with-playwright-and-xunit.htm
Based on which I'm able to create a running IHost that serves all content except blazor WASM resources, so my selenium tests fail with _framework/blazor.webassembly.js returning 404. All my static content, controllers, and the generated index page work correctly. Just the requests for files that are part of blazor seem to fail.

This is my host setup for the tests:

return new HostBuilder()
    .ConfigureWebHost(webHostBuilder => webHostBuilder
        .UseEnvironment("Development")
        .UseConfiguration(configuration)
        .UseKestrel()
        .UseContentRoot(Path.GetFullPath(Path.Join(solutionFolder, "Client/wwwroot")))
        .UseWebRoot(Path.GetFullPath(Path.Join(solutionFolder, "Client/wwwroot")))
        .UseStaticWebAssets()
        .UseStartup<TStartup>()
        .UseUrls("http://127.0.0.1:0"))
    .Build();

Full code: https://github.com/Revolutionary-Games/ThriveDevCenter/blob/f0e0ee5a3ce0eac19c6a0cbbb9fcf03bd5c3e546/AutomatedUITests/Fixtures/WebHostServerFixture.cs

So does anyone know how I can get this server setup to serve blazor framework files?

@hhyyrylainen Did you try this extension method UseBlazorFrameworkFiles

@arkiaconsulting If I try to add that to the IWebHostBuilder, I get (as expected because auto complete didn't suggest it to me):

Error CS1061 : 'IWebHostBuilder' does not contain a definition for 'UseBlazorFrameworkFiles' and no accessible extension method 'UseBlazorFrameworkFiles' accepting a first argument of type 'IWebHostBuilder' could be found

I don't think I'm missing any using statement, that would make that work. So I think that IWebHostBuilder does not have that method.

My startup class, does call app.UseBlazorFrameworkFiles();

For now, I decided to go with the approach of separately running the server manually and then running the tests. I ran into issues here as well that the server refused to serve the blazor or static files, when I tried to add a custom Testing environment. I eventually solved that by just using the Development environment (which for some reason, with the exact same code in my startup running, does serve the static files), and adding a special environment variable that makes my project load the special testing settings.

This method is available on the type IApplicationBuilder, not on IWebHostBuilder

Then, do you have a suggestion as to how I should change my code? I shared the code in this comment: #4892 (comment)
As I said, my Startup class calls app.UseBlazorFrameworkFiles(); already, so I'm not sure what I should do differently.

@hhyyrylainen I think the problem in your case is that the UseStaticWebAssets cannot find its configuration file. So, you need to set it explicitly using something like the following:

        return new HostBuilder()
            .ConfigureHostConfiguration(config =>
            {
                // Make UseStaticWebAssets work
                var applicationPath = typeof(TProgram).Assembly.Location;
                var applicationDirectory = Path.GetDirectoryName(applicationPath);
                var name = Path.ChangeExtension(applicationPath, ".StaticWebAssets.xml");

                var inMemoryConfiguration = new Dictionary<string, string>
                {
                    [WebHostDefaults.StaticWebAssetsKey] = name,
                };

                config.AddInMemoryCollection(inMemoryConfiguration);
            })
            .ConfigureWebHost(webHostBuilder => webHostBuilder
                .UseKestrel()
                .UseSolutionRelativeContentRoot(typeof(TProgram).Assembly.GetName().Name)
                .UseStaticWebAssets()
                .UseStartup<Startup>()
                .UseUrls($"http://127.0.0.1:0")) // :0 allows to choose a port automatically
            .Build();

The full code is available on my blog: https://www.meziantou.net/automated-ui-tests-an-asp-net-core-application-with-playwright-and-xunit.htm

Thank you, @meziantou that makes the static assets work. A bit embarrassing that I did find your blog post before, but I thought it only applied to MVC, which I don't use. So I didn't even think of using that when I ran into problems. Let's hope that future updates don't break this really complicated configuration needed to have this working.

@meziantou Is your code supposed to work for 'hosted' Blazor Webassembly too? It works for a static Blazor WASM app, but not sure how to handle a hosted one :/

@ThumbGen
I'm making a ASP.net hosted Blazor WASM app, and the code provided here helped me get this kind of testing working. You can see my code here:
https://github.com/Revolutionary-Games/ThriveDevCenter/blob/f282b6a8d766bcc869c80366fc1320791c637627/AutomatedUITests/Fixtures/WebHostServerFixture.cs

Sorry Im just dumping code but this is what I am using:

internal class TestApplicationFactory<TStartup> : WebApplicationFactory<TStartup>
        where TStartup : class
    {

        private readonly Action<IServiceCollection> _configureServicesAction;
        private IWebHost _externalHost;

        private string _externalUri;
        public string RootUri
        {
            get
            {
                return _externalUri ?? "http://localhost";
            }
        }

        public override IServiceProvider Services => _externalHost?.Services ?? base.Services;

        public TestApplicationFactory(Action<IServiceCollection> serviceConfiguration,
            bool createExternalTestServer)
        {
            _configureServicesAction = serviceConfiguration ?? NullActionHelper.Null<IServiceCollection>();

            if (createExternalTestServer)
                CreateServer(CreateWebHostBuilder());
        }

        #region EndToEndTesting
        // see https://github.com/dotnet/aspnetcore/issues/4892
        //https://github.com/dotnet/aspnetcore/issues/4892#issuecomment-391901173
        protected override TestServer CreateServer(IWebHostBuilder builder)
        {
            ClientOptions.BaseAddress = new Uri("https://localhost");

            _externalHost = builder.Build();
            _externalHost.Start();
            _externalUri = _externalHost.ServerFeatures.Get<IServerAddressesFeature>().Addresses.LastOrDefault();

            // not used but needed in the CreateServer method logic
            var externalServerHostBuilder = WebHost.CreateDefaultBuilder(Array.Empty<string>());
            SetupWebHostBuilder(externalServerHostBuilder);
            var testServer = new TestServer(externalServerHostBuilder);

            return testServer;
        }

        protected override IWebHostBuilder CreateWebHostBuilder()
        {
            var builder = WebHost.CreateDefaultBuilder(Array.Empty<string>());
            SetupWebHostBuilder(builder);
            builder.ConfigureTestServices(services =>
            {
                _configureServicesAction(services);
                services.AddAfterTestsCleanup();
            });

            return builder;
        }
        #endregion

        protected override IHost CreateHost(IHostBuilder builder)
        {
            var dir = Directory.GetCurrentDirectory();
            builder.UseContentRoot(dir);
            return base.CreateHost(builder);
        }

        protected override void ConfigureWebHost(IWebHostBuilder builder)
        {
            builder.ConfigureServices(services =>
            {
                _configureServicesAction(services);
                services.AddAfterTestsCleanup();
            });

            base.ConfigureWebHost(builder);
        }

        private void SetupWebHostBuilder(IWebHostBuilder builder)
        {
            builder.UseStartup<TStartup>();
            builder.UseSolutionRelativeContentRoot(typeof(TStartup).Assembly.GetName().Name);
        }

        public HttpClient CreateConfiguredHttpClient()
        {
            var client = this.CreateClient(new WebApplicationFactoryClientOptions
            {
                AllowAutoRedirect = false
            });

            client.DefaultRequestHeaders.Authorization =
                new AuthenticationHeaderValue("Test");

            return client;
        }

        protected override void Dispose(bool disposing)
        {
            base.Dispose(disposing);
            if (disposing)
            {
                _externalHost?.Dispose();
            }
        }
    }

It will not surprise me if this could be cleaned up or if I have misunderstood , but it does work for me using .Net 5 for both integration and end-to-end via the createExternalTestServer boolean. I am able to replace services, and the app's server and testing code are also using the same service provider by assigning the service provider from the application factory to a custom fixture class and only using that to perform setup in my tests.

Hopefully someone who knows more than me can take this a step further.

The workarounds provided in the comments seems to be not compatible with .net6 and minimal api hosting as creating webhost returns null (due to minimal api model) and host builder/resolver/factory that is used by WebApplicationFactory is not exposed. So as I can see there is no way to start Api and bind to real port when using .net6 and minimal api?

I have a demo app available here that uses WebApplicationFactory<T> to implement UI tests with real browsers pointed at a real HTTP server running on a port, if anyone who's commented on this issue would find it useful: https://github.com/martincostello/dotnet-minimal-api-integration-testing#readme

I was unable to find the "magic" from your example as to how to make things work.
Luckily it turns out that the approach I used before was pretty easy to update to work. I just had to change one line:

-                    var name = Path.ChangeExtension(applicationPath, ".StaticWebAssets.xml");
+                    var name = Path.ChangeExtension(applicationPath, ".staticwebassets.runtime.json");

Here's where's that in my source code if anyone wants to take a look at the entire thing: https://github.com/Revolutionary-Games/ThriveDevCenter/blob/140f920dd28668eaaee1e166dca687777aa75018/AutomatedUITests/Fixtures/WebHostServerFixture.cs#L117

Finding this page https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-6.0 it seems that the magic might be now in the WebApplicationFactory class, though I don't think I can switch to using that without porting my project structure to be like how a freshly created 6.0 blazor app sets itself up.

If I've understood you correctly, I think the "magic" is a mix of adding a reference to the Microsoft.AspNetCore.Mvc.Testing package to your project file (code). That includes the MSBuild targets that will copy the runtime assets to your test output (code).

The WebApplicationFactory class will also check that it's there if you use it, and throw an exception if it's missing:

The class also contains a bunch of code for setting the content root and some other things:

perhaps this doc should be updated to use the current versions and recommended approaches. it is currently using v3 nugets, netcoreapp3.1, the legacy Startup approach to hosting, etc