InstrumentationTests don't fail when failure is expected

[PhilLab]

⚠️ Before posting ⚠️

• This is a bug, not a question or an enhancement.
• I've searched for similar issues and didn't find a duplicate.
• I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
• I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

1. Open Instrumentation test, e.g. DrawerActivityIT.kt
2. Add an assertion which must fail into the activity scenario:

   @Test
   @UiThread
   fun switchAccountViaAccountList() {
       launchActivity<FileDisplayActivity>().use { scenario ->
           scenario.onActivity { sut ->
               onIdleSync {
                   EspressoIdlingResource.increment()
                   sut.setUser(user1)
   
                   Assert.assertEquals(true, false)  //<----------------- impossible assertion
   

3. Start the test, either via UI or via gradlew command

Expected behaviour

The test should fail

Actual behaviour

1. The test doesn't fail
2. and also doesn't really execute the UI steps.

But when adding the impossible assertion outside, it fails as expected:

@Test
@UiThread
fun switchAccountViaAccountList() {
    Assert.assertEquals(true, false)  //<----------------- impossible assertion
    launchActivity<FileDisplayActivity>().use { scenario ->
        scenario.onActivity { sut ->
            onIdleSync {
                EspressoIdlingResource.increment()
                sut.setUser(user1)

Android version

API 35

Device brand and model

Emulator (Google free)

Stock or custom OS?

Stock

Nextcloud android app version

current master (e1a80b6)

Nextcloud server version

Using a reverse proxy?

I don't know

Android logs

No response

Server error logs

Additional information

Screen recording showing successful tests despite impossible assertion. First via UI, then via gradlew command

Nextcloud-Android-Test-success.mp4

Screen recording showing how the same steps yield the expected test failure if the assertion is added outside of the scenario

Nextcloud-Android-Test-fail.mp4

[PhilLab]

@alperozturk96 @tobiasKaminsky can you take a look? If true, this would be pretty consequential for the development.

[alperozturk96]

@PhilLab

Hi

Thanks for pointing this out. It appears that using the @UiThread annotation and onIdleSync causes assertion failures to be ignored.

I will update the tests accordingly.

Thank you. @tobiasKaminsky fyi