Maximise robustness with fast startup and graceful shutdown.

  • Processes are dispoable.
    • Enables elastic scaling, rapid deployment of code or Configuration and robust production deploys.
  • Minimise startup time.
    • Ideally processes take seconds to be ready to serve transactions from cold.
    • Process manager can more easily relocate processes with shorter startup time, aiding robustness.
  • Processes shut down gracefully on receipt of p.r.comp.os.linux.signal.sigterm (Private).
    • Stop listening on the service port.
    • Let current transactions complete.
      • Requires shorter response time.
      • Long polling allows another server to pick up the request.
    • Exit.
  • Worker processes shutdown gracefully by returning in-flight transactions to the work queue:
    • For instance via a RabbitMQ NACK.
    • Releasing lock on the job record.
    • All jobs must be Reentrant.
  • Processes should be robust against sudden death in case of hardware failure:
    • Use a queueing backend to tolerate processing failures.
    • Return jobs to the queue on failure.