|
STAR
Cal STAR — firmware, DAQ server, and engine design
|
Top-level sequencer service. More...
#include <SequencerService.hpp>
Classes | |
| struct | Command |
| struct | HoldRule |
| One state's timed hold: how long, where it lands, and whether a client may set the length. More... | |
Public Member Functions | |
| SequencerService ()=default | |
| ~SequencerService () | |
| bool | init (const std::string &config_path="config/config.toml") |
| Load config, CSVs, connect to Elodin. | |
| bool | transitionTo (const std::string &state_name) |
| Request a state transition by name (e.g. | |
| bool | transitionTo (const std::string &state_name, uint32_t requested_hold_ms, std::string *refusal_reason=nullptr) |
| bool | transitionTo (State to) |
| Resolve-free overload. | |
| bool | transitionTo (State to, uint32_t requested_hold_ms, std::string *refusal_reason=nullptr) |
| Transition, optionally overriding the target state's hold with a client-supplied window. | |
| bool | setDebugMode (bool enabled) |
| Enable or disable debug mode. | |
| bool | manualActuator (const std::string &name, int pos, std::string *refusal_reason=nullptr) |
| Debug mode only: manual OPEN/CLOSE for one role. | |
| bool | extendFire () |
| Extend the FIRE window (only valid while in FIRE state). | |
| std::string | scriptStatusReport () const |
| What this process made of its dynamic states at startup, as newline-terminated text. | |
| State | currentState () const |
| bool | isDebugMode () const |
Private Member Functions | |
| void | commandLoop () |
| bool | enqueueAndWait (std::function< bool()> fn) |
| Post work and block until the worker has run it. | |
| void | enqueueDetached (std::function< bool()> fn) |
| Post work and return immediately. | |
| bool | doTransitionTo (State to, uint32_t requested_hold_ms=0, std::string *refusal_reason=nullptr) |
| bool | doSetDebugMode (bool enabled) |
| bool | doManualActuator (const std::string &name, int pos, std::string *refusal_reason=nullptr) |
| bool | doExtendFire () |
| ScriptEnv | makeScriptEnv (const DynamicState &ds) |
| Build the callbacks a script is allowed to reach the world through. | |
| std::string | configDir () const |
| Directory containing config_path_, or "." — where scripts/ is resolved against. | |
| void | publishState () |
| void | publishStateTransition (State from, State to) |
| bool | tryConnectElodin () |
| Connect to Elodin and do everything that depends on the connection: register the ACT_CMD VTables, hand the client to the commander, publish initial state. | |
| void | startElodinRetry () |
| void | stopElodinRetry () |
| bool | loadConfig (const std::string &path) |
| bool | applyConfig (const fsw::config::Config &cfg) |
| Everything a config load has to apply, in the one order that works. | |
| void | notifyControllerFire (bool active) |
| Send FIRE_START / FIRE_STOP to controller_service. | |
| bool | notifyControllerFireOnce (bool active) |
| One attempt: connect, send, wait for the controller's "OK". | |
Static Private Member Functions | |
| static bool | isAbortState (State s) |
Private Attributes | |
| std::deque< Command > | cmd_queue_ |
| std::mutex | cmd_mutex_ |
| std::condition_variable | cmd_cv_ |
| std::thread | cmd_thread_ |
| std::atomic< bool > | cmd_stop_ {false} |
| StateMachine | state_machine_ |
| ActuatorCommander | actuator_commander_ |
| AbortBroadcaster | abort_broadcaster_ |
| HoldTimer | hold_timer_ |
| ScriptRunner | script_runner_ |
| PressureFeed | pressure_feed_ |
| Live calibrated pressures for pressure(), on its own Elodin connection. | |
| fsw::elodin::ElodinClient | elodin_ |
| std::atomic< State > | current_state_ {State::IDLE} |
| std::atomic< bool > | debug_mode_ {false} |
| std::map< State, HoldRule > | hold_rules_ |
| Holdable states, by target. | |
| std::map< State, DynamicState > | dynamic_states_ |
| Dynamic states whose scripts loaded and validated. | |
| std::map< State, std::string > | refused_states_ |
| States that declare a script which did NOT hold up, with the reason. | |
| std::mutex | config_mutex_ |
| std::string | config_path_ |
| std::string | config_content_ |
| std::string | daq_bind_address_ {"0.0.0.0"} |
| Local address board-facing UDP leaves from, resolved once in init(). | |
| std::vector< fsw::elodin::BoardChannels > | actuator_boards_ |
| std::vector< fsw::elodin::BoardChannels > | ball_valve_boards_ |
| std::string | elodin_host_ = "127.0.0.1" |
| uint16_t | elodin_port_ = 2240 |
| std::thread | elodin_retry_thread_ |
| std::atomic< bool > | elodin_retry_stop_ {false} |
| State | fire_state_ {State::UNKNOWN} |
| State | fire_expiry_state_ {State::UNKNOWN} |
| uint32_t | fire_duration_ms_ {6000} |
| uint32_t | fire_extended_ms_ {10000} |
| std::string | controller_host_ {"127.0.0.1"} |
| uint16_t | controller_port_ {8000} |
| bool | sequencer_owns_valves_ {false} |
| [controller_service].sequencer_owns_valves. | |
Static Private Attributes | |
| static constexpr int | kCommandTimeoutSeconds = 10 |
| How long a caller waits for its command to run before giving up and reporting failure. | |
| static constexpr uint32_t | kAbortDoneDelayMs = 3000 |
| Delay between ABORT and ABORT_DONE. | |
| static constexpr int | kElodinRetrySeconds = 5 |
Top-level sequencer service.
The physical ABORT broadcast must not queue behind the sequencer's own housekeeping.
Owns and wires together:
All external commands arrive via the TCP server in sequencer_main.cpp and are dispatched to the methods below.
── Threading contract ────────────────────────────────────────────────────── Every public command method is a thin wrapper that puts work on cmd_queue_ and waits for the single worker thread (commandLoop) to run it. Exactly one thread is ever inside the state machine. Callers may come from anywhere: the TCP command path, and the FireManager timer thread when a burn expires.
This replaced a design where nothing was serialized at all. Individual fields were atomic, but a transition is a seven-step sequence built out of them — stop the republish loop, apply actuators, restart the loop, assign the state, arm the burn — and making each step atomic does nothing for the sequence. Two threads interleaving it could move-assign onto a still-joinable std::thread, which is a std::terminate: the sequencer died outright if an operator aborted at the moment the fire timer expired.
A mutex was the obvious fix and does not work here. An operator thread holding it calls fire_manager_.stop(), which joins the timer thread — while the timer thread is inside its expiry callback waiting for that same mutex. A recursive_mutex does not help; the re-entry is cross-thread. A queue sidesteps it entirely: the timer thread never enters the state machine, it posts a request and returns immediately (see kDetached below).
transitionTo() used to fire abort_broadcaster_.triggerAbort() near the END of the function, after stopContinuousLoop() (joins the republish thread, up to ~100 ms), after fire_manager_.stop() — which notifies controller_service over TCP — and after a full actuator batch. None of that is a precondition for telling the boards to abort, and the controller leg was the worst of it: connect() on a blocking socket with no connect timeout, so a controller host that drops SYNs rather than refusing them (powered off but routable, or a partition) held the abort for the kernel's full SYN retry budget. Measured on this machine: 133,348 ms.
The abort broadcast is now the first thing transitionTo() does. This test pins that ordering by making the controller unreachable in exactly the way that used to hurt, and asserting the ABORT datagram still lands promptly.
── Why 192.0.2.1 ────────────────────────────────────────────────────────────────────────────── TEST-NET-1 (RFC 5737) is guaranteed not to be routed anywhere, so SYNs are dropped rather than refused. A closed port on localhost is NOT a substitute: it answers RST immediately, so connect() fails fast and the regression is invisible.
── Destination, not just ordering ───────────────────────────────────────────────────────────── The broadcaster now takes its destination and port from [server_heartbeat] rather than sending to the limited broadcast 255.255.255.255 on a hardcoded 5005. This test writes those keys and binds what it wrote, so it also covers the routing: if configure() were skipped, or the address plumbed through wrongly, the datagram would go to the shipped default 192.168.2.255 and never arrive here.
A high port is used rather than 5005 so the test does not collide with a real board stack (or another service) on the developer's machine.
Definition at line 69 of file SequencerService.hpp.
|
default |
Definition at line 56 of file SequencerService.cpp.
|
private |
Everything a config load has to apply, in the one order that works.
Separate from init() so that order is stated once. It has to be exact: [[states]] is adopted before anything resolves a state name, both CSVs are parsed against that table, and [fire] is resolved last, against the same table rather than the compiled enum.
Definition at line 210 of file SequencerService.cpp.
|
private |
Definition at line 76 of file SequencerService.cpp.
|
private |
Directory containing config_path_, or "." — where scripts/ is resolved against.
Definition at line 179 of file SequencerService.cpp.
|
inline |
Definition at line 150 of file SequencerService.hpp.
|
private |
Definition at line 987 of file SequencerService.cpp.
|
private |
Definition at line 964 of file SequencerService.cpp.
|
private |
Definition at line 947 of file SequencerService.cpp.
|
private |
Definition at line 666 of file SequencerService.cpp.
|
private |
Post work and block until the worker has run it.
Returns false if the service is shutting down or the worker did not get to it within kCommandTimeoutSeconds.
Definition at line 109 of file SequencerService.cpp.
|
private |
Post work and return immediately.
Used by the fire-expiry callback: waiting there would reintroduce the join deadlock described above.
Definition at line 129 of file SequencerService.cpp.
Extend the FIRE window (only valid while in FIRE state).
Definition at line 981 of file SequencerService.cpp.
| bool sequencer::SequencerService::init | ( | const std::string & | config_path = "config/config.toml" | ) |
Load config, CSVs, connect to Elodin.
| config_path | Path to config.toml (falls back to standard locations). |
Definition at line 570 of file SequencerService.cpp.
|
staticprivate |
Definition at line 626 of file SequencerService.cpp.
|
inline |
Definition at line 153 of file SequencerService.hpp.
|
private |
Definition at line 149 of file SequencerService.cpp.
|
private |
Build the callbacks a script is allowed to reach the world through.
Definition at line 1170 of file SequencerService.cpp.
| bool sequencer::SequencerService::manualActuator | ( | const std::string & | name, |
| int | pos, | ||
| std::string * | refusal_reason = nullptr ) |
Debug mode only: manual OPEN/CLOSE for one role.
Overrides persist until a state transition (which clears overrides and applies the new state's CSV).
Allowed in a dynamic state too, where it outranks the running script's position for that valve. The script cannot notice, and the two will fight over it — but the operator with their hand on the panel is the last line, and taking that away from them to protect a script is the wrong trade.
| refusal_reason | optional; on a false return, set to a short phrase for a protocol reply. One validation, one wording — the TCP layer does not get its own copy of these rules to drift out of step with. |
Definition at line 957 of file SequencerService.cpp.
|
private |
Send FIRE_START / FIRE_STOP to controller_service.
Tell controller_service the burn gate changed.
The single place anything tells the controller the burn gate changed. Retries once if the first attempt is not acknowledged.
One TCP connection per message, 1 s send timeout.
This lives here rather than in the hold timer so exactly one component talks to the controller — previously the timer opened its own socket AND the backend independently saw the FIRE edge and sent the same messages, so a safety-critical gate had two writers in two processes.
Definition at line 1005 of file SequencerService.cpp.
|
private |
One attempt: connect, send, wait for the controller's "OK".
Returns whether it was acknowledged — a successful send() is not evidence of delivery, since the kernel buffers for an application that may never read.
Definition at line 1023 of file SequencerService.cpp.
|
private |
Definition at line 1116 of file SequencerService.cpp.
|
private |
Definition at line 1245 of file SequencerService.cpp.
| std::string sequencer::SequencerService::scriptStatusReport | ( | ) | const |
What this process made of its dynamic states at startup, as newline-terminated text.
One line per state that declares a script: SCRIPT:<id>:<name>:OK:<statements>:<timeout_ms> SCRIPT:<id>:<name>:REFUSED:<reason> terminated by "END\n".
Read-only and answerable from any thread — the answer was decided once, at config load, and is fixed for the life of the run. It exists so the panel can show a refused state as dead with a reason on hover rather than as a button that errors when pressed.
Definition at line 1206 of file SequencerService.cpp.
| bool sequencer::SequencerService::setDebugMode | ( | bool | enabled | ) |
Enable or disable debug mode.
Debug mode: all state transitions allowed, manual actuator commands accepted.
Definition at line 941 of file SequencerService.cpp.
|
private |
Definition at line 1274 of file SequencerService.cpp.
|
private |
Definition at line 1289 of file SequencerService.cpp.
| bool sequencer::SequencerService::transitionTo | ( | const std::string & | state_name | ) |
Request a state transition by name (e.g.
"Fuel Fill"). Validates via StateMachine unless debug mode is on. Applies actuators, fires abort/FIRE lifecycle handlers, publishes to Elodin.
Definition at line 634 of file SequencerService.cpp.
| bool sequencer::SequencerService::transitionTo | ( | const std::string & | state_name, |
| uint32_t | requested_hold_ms, | ||
| std::string * | refusal_reason = nullptr ) |
Definition at line 638 of file SequencerService.cpp.
| bool sequencer::SequencerService::transitionTo | ( | State | to | ) |
Resolve-free overload.
Internal callers use this so a state never round-trips through its own name, which is what let a rename refuse the fire timer's expiry transition.
Definition at line 655 of file SequencerService.cpp.
| bool sequencer::SequencerService::transitionTo | ( | State | to, |
| uint32_t | requested_hold_ms, | ||
| std::string * | refusal_reason = nullptr ) |
Transition, optionally overriding the target state's hold with a client-supplied window.
requested_hold_ms of 0 means "none supplied" — the state's configured default is used, and this is the form every internal caller uses. A non-zero value is only honoured for a state whose hold rule is marked gui_settable, which the burn state never is; anything else is REFUSED WITHOUT TRANSITIONING. That last part is the point: a client that queues a command, disconnects and has it replayed on reconnect must end up where it started, not in a hold it cannot be talked out of.
| refusal_reason | optional; on a false return, set to a short phrase suitable for a protocol reply. One validation, one wording — the TCP layer does not get its own copy of these rules to drift out of step with. |
Definition at line 659 of file SequencerService.cpp.
|
private |
Connect to Elodin and do everything that depends on the connection: register the ACT_CMD VTables, hand the client to the commander, publish initial state.
Safe to call repeatedly — a reconnect must re-register, since the VTables live in the db process.
Definition at line 1253 of file SequencerService.cpp.
|
private |
Definition at line 225 of file SequencerService.hpp.
|
private |
Definition at line 273 of file SequencerService.hpp.
Definition at line 224 of file SequencerService.hpp.
|
private |
Definition at line 274 of file SequencerService.hpp.
|
private |
Definition at line 168 of file SequencerService.hpp.
|
private |
Definition at line 167 of file SequencerService.hpp.
|
private |
Definition at line 166 of file SequencerService.hpp.
|
private |
Definition at line 170 of file SequencerService.hpp.
|
private |
Definition at line 169 of file SequencerService.hpp.
|
private |
Definition at line 258 of file SequencerService.hpp.
|
mutableprivate |
Definition at line 255 of file SequencerService.hpp.
|
private |
Definition at line 257 of file SequencerService.hpp.
|
private |
Definition at line 335 of file SequencerService.hpp.
|
private |
Definition at line 336 of file SequencerService.hpp.
|
private |
Definition at line 235 of file SequencerService.hpp.
|
private |
Local address board-facing UDP leaves from, resolved once in init().
"0.0.0.0" means the host has no NIC on the board subnet (a dev box) and traffic stays unpinned.
Definition at line 265 of file SequencerService.hpp.
|
private |
Definition at line 236 of file SequencerService.hpp.
|
private |
Dynamic states whose scripts loaded and validated.
Guarded by config_mutex_.
Definition at line 244 of file SequencerService.hpp.
Definition at line 230 of file SequencerService.hpp.
|
private |
Definition at line 280 of file SequencerService.hpp.
|
private |
Definition at line 281 of file SequencerService.hpp.
|
private |
Definition at line 283 of file SequencerService.hpp.
|
private |
Definition at line 282 of file SequencerService.hpp.
|
private |
Definition at line 333 of file SequencerService.hpp.
|
private |
Definition at line 329 of file SequencerService.hpp.
|
private |
Definition at line 334 of file SequencerService.hpp.
|
private |
Definition at line 328 of file SequencerService.hpp.
|
private |
Holdable states, by target.
Rebuilt on every config load; read on the transition path. RELOAD_CONFIG runs on a TCP handler thread while a transition may be running on another, hence the mutex.
Definition at line 241 of file SequencerService.hpp.
Definition at line 226 of file SequencerService.hpp.
|
staticconstexprprivate |
Delay between ABORT and ABORT_DONE.
Was an AbortBroadcaster ctor default; kept here beside the other fire/abort timings until a config key is wanted for it.
Definition at line 269 of file SequencerService.hpp.
|
staticconstexprprivate |
How long a caller waits for its command to run before giving up and reporting failure.
Generous: the point is that a wedged worker cannot pin every TCP thread forever, not to impose a deadline on normal work (a transition is milliseconds).
Definition at line 175 of file SequencerService.hpp.
|
staticconstexprprivate |
Definition at line 279 of file SequencerService.hpp.
Live calibrated pressures for pressure(), on its own Elodin connection.
Definition at line 229 of file SequencerService.hpp.
|
private |
States that declare a script which did NOT hold up, with the reason.
These are not enterable. doTransitionTo refuses them in its validation block, before the abort broadcast and before any valve moves, and allowedBitmask() omits them so nothing ever advertises one as reachable in the first place. Guarded by config_mutex_.
Definition at line 253 of file SequencerService.hpp.
|
private |
Definition at line 227 of file SequencerService.hpp.
|
private |
[controller_service].sequencer_owns_valves.
When true the fire hold is given no notifier at all, so there is no FIRE_START to suppress and no flag to forget to check.
Definition at line 339 of file SequencerService.hpp.
Definition at line 223 of file SequencerService.hpp.