Update STFL pipeline

[Guiliano99]

Description

• Add a new pipelien file to pre-generate XMSS and XMSS^MT keys (cache_xmss_and_xmssmt_keys.yml)
• Remove the XMSS and XMSS^MT generated keys from the repository.

Motivation and Context

• Addresses Stateful sigs segfault on macOS and Linux #121 issue of storing keys and test data inside the repository. Solved with CI-caching.

Note

• The pipeline for generating the cache, must be restarted, because it takes longer than 6 hours.
• Pipeline starts only after the merge.

[Guiliano99]

Should SLH-DSA be included in this PR, or addressed in a follow-up?

[Guiliano99]

I wouldn't want to add a new dependency to liboqs-python in general. Would this be only for testing? And perhaps we should move the discussion over to #130.

@dstebila Yes, it would only be used for testing, unless you also want to support key export for SLH-DSA, ML-KEM, and ML-DSA (to PEM).

Since I install a stable version of liboqs, this could make the tests safer: issues related to key loading would be identified directly when testing against the latest version. But maybe this makes it more complicated.

[dstebila]

@dstebila Yes, it would only be used for testing, unless you also want to support key export for SLH-DSA, ML-KEM, and ML-DSA (to PEM).

I don't have my own opinions of what the scope of this Python wrapper should be, whether key export to PEM is desirable (I guess #78 is somewhat along these lines), and the extent to which other tools in the ecosystem already provide that functionality. I'm somewhat inclined to keep our scope relatively narrow, in part due to limited resources within the project.

Since I install a stable version of liboqs, this could make the tests safer: issues related to key loading would be identified directly when testing against the latest version. But maybe this makes it more complicated.

I'm not sure I understand. Is this basically talking about compatibility between one version of the liboqs-python wrapper and the next?

[Guiliano99]

@dstebila What I meant is that the current solution was introduced because generating the XMSS and XMSS^MT keys takes a very long time — approximately 8 hours. Because of that, the pipeline had to be split across multiple runs in order to generate all required keys.

For LMS/HSS, there was previously also an issue with loading the generated keys. Based on the recent liboqs PRs, I think this may now be fixed, although I have not tested it yet.

My idea was to avoid storing the generated keys directly in the project, as discussed in previous comments. Instead, the pipeline could fetch the required keys from my keyutils-py package, rather than regenerating them each time.

Regarding the versioning aspect: the keys were generated with version 0.15.0. If the key-loading or serialization logic changes in the future, this would become visible immediately, because keys generated with the older version would no longer be loadable. To make the setup easier, I could expose functionality to load the raw private and public key bytes directly.

This was just an idea, but I thought it could simplify the pipeline while still avoiding committed key material in the repository.