Skip to content

Explicit error terms testing and documentation#1064

Open
janzill wants to merge 87 commits intoActivitySim:explicit_error_termsfrom
outerl:explicit_error_terms
Open

Explicit error terms testing and documentation#1064
janzill wants to merge 87 commits intoActivitySim:explicit_error_termsfrom
outerl:explicit_error_terms

Conversation

@janzill
Copy link
Copy Markdown

@janzill janzill commented Apr 3, 2026

This PR

  1. adds testing and documentation for explicit error term (EET) functionality
  2. addresses runtime issues in location choice sampling when running sampling with EET
  3. removes an inconsistency in EET when using shadow pricing
  4. removes an inconsistency in EET that can show up in interaction_sample choices when the sampled set changes (e.g., for changed location samples in location choice)

The implementation for the inconsistency fix for sampled choices is based on #1065, thanks @m-richards!

Regarding the RNG reset across shadow pricing iterations, this currently only applies to EET to make sure there are no changes to any Monte Carlo (MC) functionality. It would be good to discuss using this for MC as well, but I'd like to avoid doing this as part of this PR because it would mean updating tests and I want to make sure that the EET implementation does not change any MC results.

Some further discussion points regarding testing:

  • Regarding integration tests with the two external models, we cannot compare outcomes between MC and EET due to the small household sample sizes. The current tests also take a long time and running things consecutively would increase that even more. It would be good to review this as part of the on-going testing discussions.
  • We included EET functionality in semcog tests and noticed a small difference for runs with 1 and 2 processes. Debugging revealed an edge case for probabilistic scheduling. This is independent of EET, the model just looks up probabilities from provided tables and therefore does not have an EET equivalent. It turns out that how failed trips are grouped together can lead to very small differences between single and multi-processing runs for small test sets. This applies to MC as well. Changing the base seed did not trigger that edge case, but worth discussing during the engineering meeting.
  • We noticed that ARC tests are disabled but wanted to add these back in because it's the only test model that uses trip_scheduling_choice afaik. It turns out the regress trips had small differences for trip schedules when running with MC. Git history shows the test was disabled in BayDAG Contribution #16: Parking Locations in Trip Matrices #840. It looks like the test files were later touched in Trip Scheduling Choice -- Same Results Single Process and Multi-Process #1005 but the test was never re-enabled. I cannot see why the test was disabled in the first place, I might be missing some context here but I decided to update the MC regress file and re-enable the test, as well as add one for EET.

janzill and others added 30 commits March 19, 2026 15:21
Extend logit tests, add model tests, add simulate tests, draft docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants