Even if you close all windows of an application on Windows, it may still continue running in the background or the close button may not function at all. However, Microsoft has introduced a new taskbar feature in Windows 11 that aims to solve this problem.
This new improvement on the Windows 11 taskbar enables you to quickly shut down an application directly from the taskbar with just a right-click on the app. Currently, most apps offer only two options upon right-clicking - you can either close the window or pin the app to the taskbar. Although there are no screenshots available yet, Microsoft has confirmed that you will be able to shut down apps "without needing the Task Manager." This presumably refers to the capability of killing or ending a task or all tasks related to an app.
Using the Task Manager can be confusing for some people or just a plain annoyance for others. Therefore, this option will provide those people with the ability to close an unresponsive or malfunctioning app entirely without having to sift through active tasks to kill something specific.
Windows Insiders should see this addition roll out to them within the next few days or weeks, and that should be followed by a stable release sometime after that.
Source: Microsoft