Using a Scheduler

The backbone of this pipeline is spotify’s luigi batch process management framework. Luigi uses the concept of a scheduler in order to manage task execution. Two types of scheduler are available, a local scheduler and a central scheduler. The local scheduler is useful for development and debugging while the central scheduler provides the following two benefits:

• Make sure two instances of the same task are not running simultaneously
• Provide visualization of everything that’s going on

While in the recon-pipeline shell, running tools install luigi-service will copy the luigid.service file provided in the repo to its appropriate systemd location and start/enable the service. The result is that the central scheduler is up and running easily.

The other option is to add --local-scheduler to your scan command from within the recon-pipeline shell.