After the introduction of the previous post, let’s now see an example that I code to better teach myself the use of Luigi’s pipeline.
A Task in Luigi
Here follows a simple Luigi Task:
This result in the creation of the file ‘simple_plot.png’.
To create a Luigi Task, we create a Class that import
luigi.Task, this turn the class into a Luigi Task.
Witnin is present a parameter
name at this stage name is used just to indicate to luigi the name of the plot and, a method
Run() is one of the three methods that Luigi search within a Task, the other two being:
The task of
run() is to execute the code contained and create the plot.
output() returns the location/name of the plot. This method is used to pass LocalTarget from a Task to another, for now nothing is done.
Two Luigi Tasks
In the next example we add a other Task that calls
CreatePlotTask and pass the parameter
name to it.
In this code we add the new Task
PassPlotNameTask the only aim of this task is to execute
CreatePlotTask by the method
requires(). Luigi sees this method and before doing the other present, Luigi execute what is inside requires.
That would be
CreatePlotTask that create and save a plot, like we did before.
There is no limit of how many Tasks can follow one, but it is good norm that each task will requires a single other Task, return in output a single value.
Three Luigi Tasks
As example we can expande
CreatePlotTask and making it require an other task:
This code will create a plot called ‘simple_plot.png’ and a folder folder: