Skip to content

Infinite retries bugfix, Spot instances, and some code cleanup #284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 59 commits into
base: ec2-new-implementation
Choose a base branch
from
Draft

Infinite retries bugfix, Spot instances, and some code cleanup #284

wants to merge 59 commits into from
+1,216 −479

Conversation

@anthony-yip
Copy link

@anthony-yip anthony-yip commented Oct 27, 2025 •
edited
Loading

Fixes #123, #279, #277, #273.

Sorry for the gigantic PR but all the little fixes built off of each other.

Summary of changes:

  • Preliminary mypy support and corresponding type annotations
    • Implementation of VMMS and Local/Remote as Protocols
  • Code cleanup (removing dead code) in which a job always has a preVM (provided by the jobManager)
  • Functional semantics for local/remote data structures (since Redis is weird)
  • Fixing infinite retries (coherence bug in makeDead)
    • Immutable Attribtues for TangoJob for better maintenance
  • Spot instances added
    • Errors at all stages of the pipeline (e.g. copy in) now trigger a rescheduling (instead of just failing). This is because the spot termination can occur an an arbitrary stage
    • Enforcing the invariant that detachVM is always called upon a a worker's termination

Also associated with autolab/Autolab#2301

KesterTan and others added 30 commits January 26, 2025 00:58
@KesterTan
added in validation for input files and permissions while creating di…
e5de5f5 
…rectory
@KesterTan
removed unnecessary files
2283fbe
@KesterTan
added thread pool for scp commands
5cc5004
@KesterTan
added more in-depth print statements
81422e1
@KesterTan
reverted threading and added logging
9760759
@KesterTan
reverted thread pool
f508a15
@KesterTan
remove dump
e460603
@evanyeyeye
Small fixces
0b3f866
@KesterTan
reverted threading
9a27e47
@KesterTan
small fixes
52326da
@coder6583
stressTest.py working
014ad49
@KesterTan
Incorporated PDL correctness changes
1c506a2
@KesterTan
Merge branch 'ec2-cli-testing' into copy-in
02c3f11
@KesterTan
Added some stabilization time
b753118
@KesterTan
Changed stabilization
e6c19f3
@KesterTan
Added timeout using aws waiter
6facb4e
@KesterTan
Syntax issues
53a2858
@KesterTan
Syntax issues and logging
e23e8e3
@coder6583
readme updated
d78f88b
@KesterTan
Fixed client region
c8263ee
@KesterTan
Revert stop before
3c22e1f
@KesterTan
Revert sleep
1b83595
@KesterTan
Revert aws waiter
47388c9
@evanyeyeye
Update .gitignore
4d24ee5
@evanyeyeye
Clean dump
f609b07
@evanyeyeye
Quality improvements
158f0af
@evanyeyeye
Fix gitignore
20c8994
@evanyeyeye
Update .gitignore
128a2e7
@anthony-yip
Fixed the change by making the retries attribute private (and also th…
ce52b32 
…e assigned attribute), improving logging surrounding retries and giving up
@anthony-yip
Made all of tangoJob's attributes read only
28db9e5
coder6583 and others added 25 commits April 7, 2025 15:05
@coder6583
Merge branch 'ec2-cli-testing' into copy-in
ce23c9d
@anthony-yip
fixed the local/remote desyncing issues, left some print statements i…
9d57597 
…n there
@anthony-yip
Better encapsulation of TangoJob._remoteLocation
3bd80cf
@anthony-yip
Fixing uses of makeDead that was changed 2 commits ago
f0c0d32
@anthony-yip
Merge branch 'copy-in' into anthonyyip/infinite_retries_bugfix
5b5cbc0
@anthony-yip
comments, logging and todos
ba83509
@anthony-yip
Code to enable spot instances
600803c
@KesterTan
copy boto.cfg
8f59e44
@KesterTan
docker installation and install ping
4950256
@KesterTan
copy boto if exist
85b9a60
@anthony-yip
type annotations
d450112
@anthony-yip
finished mypy stuff up to not using --check-untyped-defs
029c1a9
@anthony-yip
doesn't quite work, but wanted a checkpoint before muddling with the …
85bdf7c 
…preallocator
@anthony-yip
more typing changes, streamlining the used of the TangoDictionary in …
7e6751e 
…preallocator (do note that my initial worry about incorrectness was unfounded due to weird Redis sharing behavior)
@anthony-yip
Merge branch 'copy-in' into anthonyyip/infinite_retries_bugfix
df044e3
@anthony-yip
Merge branch 'anthonyyip/infinite_retries_bugfix' into anthonyyip/spo…
4c8a521 
…t_instances
@anthony-yip
refactored afterJobExecution and detachVM
03e1f31
@anthony-yip
error messages
bc8c8a7
@anthony-yip
code cleanup: worker always gets initialized with a preallocated VM (…
20b833b 
…else preVM.name will fail)
@KesterTan
Removed detachVM argument
97a1792
@KesterTan
chnaged request obj
0b134f0
@KesterTan
removed vmms in request obj
e1815c3
@anthony-yip
Merge branch 'copy-in' into anthonyyip/spot_instances
d2e069c
@anthony-yip
fixed logging type safety
138e332
@anthony-yip
always assert that detachVM is called (taken care of with .keep_for_d…
d1a8b27 
…ebuggin)
@anthony-yip anthony-yip requested a review from KesterTan October 27, 2025 02:38
Base automatically changed from copy-in to ec2-new-implementation October 30, 2025 21:57
KesterTan
KesterTan requested changes Oct 30, 2025
View reviewed changes
Copy link

@KesterTan KesterTan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we resolve the merge conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@KesterTan KesterTan KesterTan requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@anthony-yip @KesterTan @evanyeyeye @coder6583