Tutorial on Luigi pipeline, part 2: Examples

After the introduction of the previous post, let’s now see an example that I code to better teach myself the use of Luigi’s pipeline.

A Task in Luigi

Here follows a simple Luigi Task:

This result in the creation of the file ‘simple_plot.png’.

To create a Luigi Task, we create a Class that import luigi.Task, this turn the class into a Luigi Task.

Witnin is present a parameter name at this stage name is used just to indicate to luigi the name of the plot and, a method run().

Run() is one of the three methods that Luigi search within a Task, the other two being: output() and requires().

The task of run() is to execute the code contained and create the plot.

The method output() returns the location/name of the plot. This method is used to pass LocalTarget from a Task to another, for now nothing is done.

Two Luigi Tasks

In the next example we add a other Task that calls CreatePlotTask and pass the parameter name to it.

In this code we add the new Task PassPlotNameTask the only aim of this task is to execute CreatePlotTask by the method requires(). Luigi sees this method and before doing the other present, Luigi execute what is inside requires.

That would be CreatePlotTask that create and save a plot, like we did before.

There is no limit of how many Tasks can follow one, but it is good norm that each task will requires a single other Task, return in output a single value.

Three Luigi Tasks

As example we can expande CreatePlotTask and making it require an other task: MakeDirectory

This code will create a plot called ‘simple_plot.png’ and a folder folder:

0 0 vote
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments