Support for Java-based application module aware Flyway migrations

[cleemansen]

Hello,

we use Application-Module Aware Flyway Migrations (sql migration files).

Are Java-based migrations also supported per module?

Cheers

[cleemansen]

After debugging org.springframework.modulith.runtime.flyway.SpringModulithFlywayMigrationStrategy, it should actually work. I'm not sure if that's intentional. But at first glance, I don't see a problem.

Setup:

• Project Root package: com.example.green
• Configuration

spring:
  flyway:
    locations: classpath:db/migration,classpath:com/example/green

With this setup I can create Java-based migrations in each module (like com.example.green.core.migration.V5__Iteration.kt

• Any concerns?
• Should the documentation be expanded to include this?

[odrotbohm]

I guess that depends on what you consider “works” to mean. The Java-based migrations get executed as the configurations we set up will run those migrations due to the default setup of Spring Boot. As the test executions customize the component scanning, you will also see migrations of modules not included in a test run not executed.

What's missing, though, is the ordering of the migrations according to the module dependency structure. Unfortunately, such an analysis requires significantly more infrastructure to be bootstrapped compared to what we do today, but we can certainly consider this.

[cleemansen]

Thanks for your reply!

I can't say I fully understand it. This is what I took to be the main issue:

What's missing, though, is the ordering of the migrations according to the module dependency structure.

Does this refer to the execution of the Spring Boot app or the ApplicationModuleTests?

I’ve created a Minimal Working Example in the hope that it will help clarify things.

• It contains the two modules core and plant.
• plant uses core.
• Both modules have Java-based migrations and SQL file migrations
• A plant migration defines an explicit reference to a table from core

The Spring Boot app starts successfully. The ApplicationModuleTests for both modules pass.

I would appreciate your review. What scenario have I overlooked?

[odrotbohm]

In your example, nothing guarantees the migration in core to be executed before plant. It might be the case, but that's accidental. I guess you might be able to break it if the files do not accidentally lexically order correctly.

The core of Spring Modulith's Flyway support is to reliably execute migrations in a deterministic order following the application module dependency structure.

[cleemansen]

Okay, thanks for the explanation. I see your point now.

Unfortunately, I wasn't able to reproduce the issue after adding the new alpha module (which uses core). But I understand that it might have started correctly by accident.

Schema history table "PUBLIC"."flyway_schema_history_core" does not exist yet
[..]
Schema history table "PUBLIC"."flyway_schema_history_alpha" does not exist yet
[..]
Schema history table "PUBLIC"."flyway_schema_history_plant" does not exist yet

I've updated my example. Maybe it will help us find a stable solution.

[cleemansen]

I certainly don’t have a complete overview of everything! But what I find interesting is that the identifiers in SpringModulithFlywayMigrationStrategy are in the correct module order. This has to be guaranteed, otherwise the SQL file-based migrations wouldn’t work. Correct?

This list is now processed in the correct order in augmentWithApplicationModule (+ the __root module). There, locations are calculated for EVERY module:

0 = “classpath:com/example/green/__root”
1 = “classpath:db/migration/__root”

0 = “classpath:com/example/green/core”
1 = “classpath:db/migration/core”

0 = “classpath:com/example/green/alpha”
1 = “classpath:db/migration/alpha”

0 = “classpath:com/example/green/plant”
1 = “classpath:db/migration/plant”

This was my initial point: that the Java-based migrations may already be handled by application-module-aware Flyway (SQL file) migrations. Sorry if I'm missing something basic and this isn't helpful at all.

[odrotbohm]

You're not missing anything, but I am fully aware of the situation 🙃. The file-based migrations work exactly because we can set up a dedicated Flyway migration per module and execute them. The identifiers being properly ordered guarantees that the migrations are executed in the correct order. That ordered list of identifiers can be extracted from the metadata file we generate at test or AOT build time and check for at runtime.

For Java-based migrations, the scenario is more complicated. When iterating over the module identifiers, from all the Spring beans available, we have to find the JavaMigration instances belonging to the current module. That decision is currently only available via a full ApplicationModules instance, which needs ArchUnit to be available and executed on the runtime classpath. The aforementioned generated metadata file avoids that need for a lot of similar functionality (ApplicationModuleInitializers, for example). I guess we could try to implement the “does this migration belong to module X” decision solely based on the metadata, too, but that'd mean we'd duplicate that logic between the code operating on the metadata and the one in ApplicationModules, which is suboptimal.

Additional thought has to go into how all of this fits into the existing abstractions in the spring-modulith-runtime module. So nothing terribly complicated, but stuff that I guess I'll have to think about for a while.

[cleemansen]

For Java-based migrations, the scenario is more complicated. When iterating over the module identifiers, from all the Spring beans available, we have to find the JavaMigration instances belonging to the current module.

Isn't the search scope for all available Spring Beans already correctly restricted? Isn't the classpath limited to the core module here?

0 = “classpath:com/example/green/core”
1 = “classpath:db/migration/core”