STAR
Cal STAR — firmware, DAQ server, and engine design
Loading...
Searching...
No Matches
SequencerService Class Reference

Top-level sequencer service. More...

#include <SequencerService.hpp>

Collaboration diagram for SequencerService:
[legend]

Classes

struct  Command
struct  HoldRule
 One state's timed hold: how long, where it lands, and whether a client may set the length. More...

Public Member Functions

 SequencerService ()=default
 ~SequencerService ()
bool init (const std::string &config_path="config/config.toml")
 Load config, CSVs, connect to Elodin.
bool transitionTo (const std::string &state_name)
 Request a state transition by name (e.g.
bool transitionTo (const std::string &state_name, uint32_t requested_hold_ms, std::string *refusal_reason=nullptr)
bool transitionTo (State to)
 Resolve-free overload.
bool transitionTo (State to, uint32_t requested_hold_ms, std::string *refusal_reason=nullptr)
 Transition, optionally overriding the target state's hold with a client-supplied window.
bool setDebugMode (bool enabled)
 Enable or disable debug mode.
bool manualActuator (const std::string &name, int pos, std::string *refusal_reason=nullptr)
 Debug mode only: manual OPEN/CLOSE for one role.
bool extendFire ()
 Extend the FIRE window (only valid while in FIRE state).
std::string scriptStatusReport () const
 What this process made of its dynamic states at startup, as newline-terminated text.
State currentState () const
bool isDebugMode () const

Private Member Functions

void commandLoop ()
bool enqueueAndWait (std::function< bool()> fn)
 Post work and block until the worker has run it.
void enqueueDetached (std::function< bool()> fn)
 Post work and return immediately.
bool doTransitionTo (State to, uint32_t requested_hold_ms=0, std::string *refusal_reason=nullptr)
bool doSetDebugMode (bool enabled)
bool doManualActuator (const std::string &name, int pos, std::string *refusal_reason=nullptr)
bool doExtendFire ()
ScriptEnv makeScriptEnv (const DynamicState &ds)
 Build the callbacks a script is allowed to reach the world through.
std::string configDir () const
 Directory containing config_path_, or "." — where scripts/ is resolved against.
void publishState ()
void publishStateTransition (State from, State to)
bool tryConnectElodin ()
 Connect to Elodin and do everything that depends on the connection: register the ACT_CMD VTables, hand the client to the commander, publish initial state.
void startElodinRetry ()
void stopElodinRetry ()
bool loadConfig (const std::string &path)
bool applyConfig (const fsw::config::Config &cfg)
 Everything a config load has to apply, in the one order that works.
void notifyControllerFire (bool active)
 Send FIRE_START / FIRE_STOP to controller_service.
bool notifyControllerFireOnce (bool active)
 One attempt: connect, send, wait for the controller's "OK".

Static Private Member Functions

static bool isAbortState (State s)

Private Attributes

std::deque< Command > cmd_queue_
std::mutex cmd_mutex_
std::condition_variable cmd_cv_
std::thread cmd_thread_
std::atomic< bool > cmd_stop_ {false}
StateMachine state_machine_
ActuatorCommander actuator_commander_
AbortBroadcaster abort_broadcaster_
HoldTimer hold_timer_
ScriptRunner script_runner_
PressureFeed pressure_feed_
 Live calibrated pressures for pressure(), on its own Elodin connection.
fsw::elodin::ElodinClient elodin_
std::atomic< State > current_state_ {State::IDLE}
std::atomic< bool > debug_mode_ {false}
std::map< State, HoldRule > hold_rules_
 Holdable states, by target.
std::map< State, DynamicState > dynamic_states_
 Dynamic states whose scripts loaded and validated.
std::map< State, std::string > refused_states_
 States that declare a script which did NOT hold up, with the reason.
std::mutex config_mutex_
std::string config_path_
std::string config_content_
std::string daq_bind_address_ {"0.0.0.0"}
 Local address board-facing UDP leaves from, resolved once in init().
std::vector< fsw::elodin::BoardChannels > actuator_boards_
std::vector< fsw::elodin::BoardChannels > ball_valve_boards_
std::string elodin_host_ = "127.0.0.1"
uint16_t elodin_port_ = 2240
std::thread elodin_retry_thread_
std::atomic< bool > elodin_retry_stop_ {false}
State fire_state_ {State::UNKNOWN}
State fire_expiry_state_ {State::UNKNOWN}
uint32_t fire_duration_ms_ {6000}
uint32_t fire_extended_ms_ {10000}
std::string controller_host_ {"127.0.0.1"}
uint16_t controller_port_ {8000}
bool sequencer_owns_valves_ {false}
 [controller_service].sequencer_owns_valves.

Static Private Attributes

static constexpr int kCommandTimeoutSeconds = 10
 How long a caller waits for its command to run before giving up and reporting failure.
static constexpr uint32_t kAbortDoneDelayMs = 3000
 Delay between ABORT and ABORT_DONE.
static constexpr int kElodinRetrySeconds = 5

Detailed Description

Top-level sequencer service.

The physical ABORT broadcast must not queue behind the sequencer's own housekeeping.

Owns and wires together:

All external commands arrive via the TCP server in sequencer_main.cpp and are dispatched to the methods below.

── Threading contract ────────────────────────────────────────────────────── Every public command method is a thin wrapper that puts work on cmd_queue_ and waits for the single worker thread (commandLoop) to run it. Exactly one thread is ever inside the state machine. Callers may come from anywhere: the TCP command path, and the FireManager timer thread when a burn expires.

This replaced a design where nothing was serialized at all. Individual fields were atomic, but a transition is a seven-step sequence built out of them — stop the republish loop, apply actuators, restart the loop, assign the state, arm the burn — and making each step atomic does nothing for the sequence. Two threads interleaving it could move-assign onto a still-joinable std::thread, which is a std::terminate: the sequencer died outright if an operator aborted at the moment the fire timer expired.

A mutex was the obvious fix and does not work here. An operator thread holding it calls fire_manager_.stop(), which joins the timer thread — while the timer thread is inside its expiry callback waiting for that same mutex. A recursive_mutex does not help; the re-entry is cross-thread. A queue sidesteps it entirely: the timer thread never enters the state machine, it posts a request and returns immediately (see kDetached below).

transitionTo() used to fire abort_broadcaster_.triggerAbort() near the END of the function, after stopContinuousLoop() (joins the republish thread, up to ~100 ms), after fire_manager_.stop() — which notifies controller_service over TCP — and after a full actuator batch. None of that is a precondition for telling the boards to abort, and the controller leg was the worst of it: connect() on a blocking socket with no connect timeout, so a controller host that drops SYNs rather than refusing them (powered off but routable, or a partition) held the abort for the kernel's full SYN retry budget. Measured on this machine: 133,348 ms.

The abort broadcast is now the first thing transitionTo() does. This test pins that ordering by making the controller unreachable in exactly the way that used to hurt, and asserting the ABORT datagram still lands promptly.

── Why 192.0.2.1 ────────────────────────────────────────────────────────────────────────────── TEST-NET-1 (RFC 5737) is guaranteed not to be routed anywhere, so SYNs are dropped rather than refused. A closed port on localhost is NOT a substitute: it answers RST immediately, so connect() fails fast and the regression is invisible.

── Destination, not just ordering ───────────────────────────────────────────────────────────── The broadcaster now takes its destination and port from [server_heartbeat] rather than sending to the limited broadcast 255.255.255.255 on a hardcoded 5005. This test writes those keys and binds what it wrote, so it also covers the routing: if configure() were skipped, or the address plumbed through wrongly, the datagram would go to the shipped default 192.168.2.255 and never arrive here.

A high port is used rather than 5005 so the test does not collide with a real board stack (or another service) on the developer's machine.

Definition at line 69 of file SequencerService.hpp.

Constructor & Destructor Documentation

◆ SequencerService()

◆ ~SequencerService()

Member Function Documentation

◆ applyConfig()

Everything a config load has to apply, in the one order that works.

Separate from init() so that order is stated once. It has to be exact: [[states]] is adopted before anything resolves a state name, both CSVs are parsed against that table, and [fire] is resolved last, against the same table rather than the compiled enum.

Definition at line 210 of file SequencerService.cpp.

◆ commandLoop()

Definition at line 76 of file SequencerService.cpp.

◆ configDir()

std::string sequencer::SequencerService::configDir ( ) const
private

Directory containing config_path_, or "." — where scripts/ is resolved against.

Definition at line 179 of file SequencerService.cpp.

◆ currentState()

Definition at line 150 of file SequencerService.hpp.

◆ doExtendFire()

Definition at line 987 of file SequencerService.cpp.

◆ doManualActuator()

bool sequencer::SequencerService::doManualActuator ( const std::string & name,
int pos,
std::string * refusal_reason = nullptr )
private

Definition at line 964 of file SequencerService.cpp.

◆ doSetDebugMode()

bool sequencer::SequencerService::doSetDebugMode ( bool enabled)
private

Definition at line 947 of file SequencerService.cpp.

◆ doTransitionTo()

bool sequencer::SequencerService::doTransitionTo ( State to,
uint32_t requested_hold_ms = 0,
std::string * refusal_reason = nullptr )
private

Definition at line 666 of file SequencerService.cpp.

◆ enqueueAndWait()

bool sequencer::SequencerService::enqueueAndWait ( std::function< bool()> fn)
private

Post work and block until the worker has run it.

Returns false if the service is shutting down or the worker did not get to it within kCommandTimeoutSeconds.

Definition at line 109 of file SequencerService.cpp.

◆ enqueueDetached()

void sequencer::SequencerService::enqueueDetached ( std::function< bool()> fn)
private

Post work and return immediately.

Used by the fire-expiry callback: waiting there would reintroduce the join deadlock described above.

Definition at line 129 of file SequencerService.cpp.

◆ extendFire()

Extend the FIRE window (only valid while in FIRE state).

Definition at line 981 of file SequencerService.cpp.

◆ init()

bool sequencer::SequencerService::init ( const std::string & config_path = "config/config.toml")

Load config, CSVs, connect to Elodin.

Parameters
config_pathPath to config.toml (falls back to standard locations).
Returns
true on success.

Definition at line 570 of file SequencerService.cpp.

◆ isAbortState()

Definition at line 626 of file SequencerService.cpp.

◆ isDebugMode()

Definition at line 153 of file SequencerService.hpp.

◆ loadConfig()

bool sequencer::SequencerService::loadConfig ( const std::string & path)
private

Definition at line 149 of file SequencerService.cpp.

◆ makeScriptEnv()

ScriptEnv sequencer::SequencerService::makeScriptEnv ( const DynamicState & ds)
private

Build the callbacks a script is allowed to reach the world through.

Definition at line 1170 of file SequencerService.cpp.

◆ manualActuator()

bool sequencer::SequencerService::manualActuator ( const std::string & name,
int pos,
std::string * refusal_reason = nullptr )

Debug mode only: manual OPEN/CLOSE for one role.

Overrides persist until a state transition (which clears overrides and applies the new state's CSV).

Allowed in a dynamic state too, where it outranks the running script's position for that valve. The script cannot notice, and the two will fight over it — but the operator with their hand on the panel is the last line, and taking that away from them to protect a script is the wrong trade.

Parameters
refusal_reasonoptional; on a false return, set to a short phrase for a protocol reply. One validation, one wording — the TCP layer does not get its own copy of these rules to drift out of step with.

Definition at line 957 of file SequencerService.cpp.

◆ notifyControllerFire()

Send FIRE_START / FIRE_STOP to controller_service.

Tell controller_service the burn gate changed.

The single place anything tells the controller the burn gate changed. Retries once if the first attempt is not acknowledged.

One TCP connection per message, 1 s send timeout.

This lives here rather than in the hold timer so exactly one component talks to the controller — previously the timer opened its own socket AND the backend independently saw the FIRE edge and sent the same messages, so a safety-critical gate had two writers in two processes.

Definition at line 1005 of file SequencerService.cpp.

◆ notifyControllerFireOnce()

One attempt: connect, send, wait for the controller's "OK".

Returns whether it was acknowledged — a successful send() is not evidence of delivery, since the kernel buffers for an application that may never read.

Definition at line 1023 of file SequencerService.cpp.

◆ publishState()

Definition at line 1116 of file SequencerService.cpp.

◆ publishStateTransition()

Definition at line 1245 of file SequencerService.cpp.

◆ scriptStatusReport()

What this process made of its dynamic states at startup, as newline-terminated text.

One line per state that declares a script: SCRIPT:<id>:<name>:OK:<statements>:<timeout_ms> SCRIPT:<id>:<name>:REFUSED:<reason> terminated by "END\n".

Read-only and answerable from any thread — the answer was decided once, at config load, and is fixed for the life of the run. It exists so the panel can show a refused state as dead with a reason on hover rather than as a button that errors when pressed.

Definition at line 1206 of file SequencerService.cpp.

◆ setDebugMode()

Enable or disable debug mode.

Debug mode: all state transitions allowed, manual actuator commands accepted.

Definition at line 941 of file SequencerService.cpp.

◆ startElodinRetry()

Definition at line 1274 of file SequencerService.cpp.

◆ stopElodinRetry()

Definition at line 1289 of file SequencerService.cpp.

◆ transitionTo() [1/4]

bool sequencer::SequencerService::transitionTo ( const std::string & state_name)

Request a state transition by name (e.g.

"Fuel Fill"). Validates via StateMachine unless debug mode is on. Applies actuators, fires abort/FIRE lifecycle handlers, publishes to Elodin.

Returns
true if transition was accepted and executed.

Definition at line 634 of file SequencerService.cpp.

◆ transitionTo() [2/4]

bool sequencer::SequencerService::transitionTo ( const std::string & state_name,
uint32_t requested_hold_ms,
std::string * refusal_reason = nullptr )

Definition at line 638 of file SequencerService.cpp.

◆ transitionTo() [3/4]

Resolve-free overload.

Internal callers use this so a state never round-trips through its own name, which is what let a rename refuse the fire timer's expiry transition.

Definition at line 655 of file SequencerService.cpp.

◆ transitionTo() [4/4]

bool sequencer::SequencerService::transitionTo ( State to,
uint32_t requested_hold_ms,
std::string * refusal_reason = nullptr )

Transition, optionally overriding the target state's hold with a client-supplied window.

requested_hold_ms of 0 means "none supplied" — the state's configured default is used, and this is the form every internal caller uses. A non-zero value is only honoured for a state whose hold rule is marked gui_settable, which the burn state never is; anything else is REFUSED WITHOUT TRANSITIONING. That last part is the point: a client that queues a command, disconnects and has it replayed on reconnect must end up where it started, not in a hold it cannot be talked out of.

Parameters
refusal_reasonoptional; on a false return, set to a short phrase suitable for a protocol reply. One validation, one wording — the TCP layer does not get its own copy of these rules to drift out of step with.

Definition at line 659 of file SequencerService.cpp.

◆ tryConnectElodin()

Connect to Elodin and do everything that depends on the connection: register the ACT_CMD VTables, hand the client to the commander, publish initial state.

Safe to call repeatedly — a reconnect must re-register, since the VTables live in the db process.

Definition at line 1253 of file SequencerService.cpp.

Member Data Documentation

◆ abort_broadcaster_

Definition at line 225 of file SequencerService.hpp.

◆ actuator_boards_

◆ actuator_commander_

◆ ball_valve_boards_

◆ cmd_cv_

std::condition_variable sequencer::SequencerService::cmd_cv_
private

Definition at line 168 of file SequencerService.hpp.

◆ cmd_mutex_

Definition at line 167 of file SequencerService.hpp.

◆ cmd_queue_

Definition at line 166 of file SequencerService.hpp.

◆ cmd_stop_

std::atomic<bool> sequencer::SequencerService::cmd_stop_ {false}
private

Definition at line 170 of file SequencerService.hpp.

◆ cmd_thread_

Definition at line 169 of file SequencerService.hpp.

◆ config_content_

Definition at line 258 of file SequencerService.hpp.

◆ config_mutex_

std::mutex sequencer::SequencerService::config_mutex_
mutableprivate

Definition at line 255 of file SequencerService.hpp.

◆ config_path_

Definition at line 257 of file SequencerService.hpp.

◆ controller_host_

std::string sequencer::SequencerService::controller_host_ {"127.0.0.1"}
private

Definition at line 335 of file SequencerService.hpp.

◆ controller_port_

Definition at line 336 of file SequencerService.hpp.

◆ current_state_

std::atomic<State> sequencer::SequencerService::current_state_ {State::IDLE}
private

Definition at line 235 of file SequencerService.hpp.

◆ daq_bind_address_

std::string sequencer::SequencerService::daq_bind_address_ {"0.0.0.0"}
private

Local address board-facing UDP leaves from, resolved once in init().

"0.0.0.0" means the host has no NIC on the board subnet (a dev box) and traffic stays unpinned.

Definition at line 265 of file SequencerService.hpp.

◆ debug_mode_

std::atomic<bool> sequencer::SequencerService::debug_mode_ {false}
private

Definition at line 236 of file SequencerService.hpp.

◆ dynamic_states_

std::map<State, DynamicState> sequencer::SequencerService::dynamic_states_
private

Dynamic states whose scripts loaded and validated.

Guarded by config_mutex_.

Definition at line 244 of file SequencerService.hpp.

◆ elodin_

◆ elodin_host_

std::string sequencer::SequencerService::elodin_host_ = "127.0.0.1"
private

Definition at line 280 of file SequencerService.hpp.

◆ elodin_port_

Definition at line 281 of file SequencerService.hpp.

◆ elodin_retry_stop_

std::atomic<bool> sequencer::SequencerService::elodin_retry_stop_ {false}
private

Definition at line 283 of file SequencerService.hpp.

◆ elodin_retry_thread_

Definition at line 282 of file SequencerService.hpp.

◆ fire_duration_ms_

Definition at line 333 of file SequencerService.hpp.

◆ fire_expiry_state_

Definition at line 329 of file SequencerService.hpp.

◆ fire_extended_ms_

Definition at line 334 of file SequencerService.hpp.

◆ fire_state_

Definition at line 328 of file SequencerService.hpp.

◆ hold_rules_

Holdable states, by target.

Rebuilt on every config load; read on the transition path. RELOAD_CONFIG runs on a TCP handler thread while a transition may be running on another, hence the mutex.

Definition at line 241 of file SequencerService.hpp.

◆ hold_timer_

◆ kAbortDoneDelayMs

uint32_t sequencer::SequencerService::kAbortDoneDelayMs = 3000
staticconstexprprivate

Delay between ABORT and ABORT_DONE.

Was an AbortBroadcaster ctor default; kept here beside the other fire/abort timings until a config key is wanted for it.

Definition at line 269 of file SequencerService.hpp.

◆ kCommandTimeoutSeconds

int sequencer::SequencerService::kCommandTimeoutSeconds = 10
staticconstexprprivate

How long a caller waits for its command to run before giving up and reporting failure.

Generous: the point is that a wedged worker cannot pin every TCP thread forever, not to impose a deadline on normal work (a transition is milliseconds).

Definition at line 175 of file SequencerService.hpp.

◆ kElodinRetrySeconds

int sequencer::SequencerService::kElodinRetrySeconds = 5
staticconstexprprivate

Definition at line 279 of file SequencerService.hpp.

◆ pressure_feed_

Live calibrated pressures for pressure(), on its own Elodin connection.

Definition at line 229 of file SequencerService.hpp.

◆ refused_states_

std::map<State, std::string> sequencer::SequencerService::refused_states_
private

States that declare a script which did NOT hold up, with the reason.

These are not enterable. doTransitionTo refuses them in its validation block, before the abort broadcast and before any valve moves, and allowedBitmask() omits them so nothing ever advertises one as reachable in the first place. Guarded by config_mutex_.

Definition at line 253 of file SequencerService.hpp.

◆ script_runner_

Definition at line 227 of file SequencerService.hpp.

◆ sequencer_owns_valves_

[controller_service].sequencer_owns_valves.

When true the fire hold is given no notifier at all, so there is no FIRE_START to suppress and no flag to forget to check.

Definition at line 339 of file SequencerService.hpp.

◆ state_machine_


The documentation for this class was generated from the following files: