Pipeline#

class lumen.pipeline.Pipeline(*, source, table, **params)#

Pipeline encapsulates filters and transformations applied to a lumen.sources.base.Source table.

A Pipeline ingests data from a lumen.sources.base.Source table or another Pipeline applying the declared lumen.filters.base.Filter, lumen.transforms.base.Transform and lumen.transforms.sql.SQLTransform definitions. It can be used to drive one or more visual outputs or leveraged as a standalone component to encapsulate multiple data processing steps.


Parameters#

auto_update

type: bool
default: True
Whether changes in filters, transforms and references automaticallytrigger updates in the data or whether an update has to be triggeredmanually using the update event or the update button in the UI.

data

type: pandas.DataFrame
default: None
The current data on this source.

filters

type: list[lumen.Filter]
default: []
A list of Filters to apply to the source data.

name

type: str
default: 'Pipeline'
String identifier for this object.

pipeline

type: lumen.Pipeline
default: None
Optionally a pipeline may be chained to another pipeline.

schema

type: dict
default: None
The schema of the input data.

source

type: lumen.Source
default: None
The Source this pipeline is fed by.

sql_transforms

type: list[lumen.transforms.SQLTransform]
default: []
A list of SQLTransforms to apply to the source data.

table

type: str
default: ''
The name of the table driving this pipeline.

transforms

type: list[lumen.Transform]
default: []
A list of Transforms to apply to the source data.

update

type: bool
default: False
Update event trigger (if manual update is set).


Methods#

Pipeline.add_filter(filt: Filter | Type[Filter] | Widget, field: str | None = None, **kwargs)#

Add a filter to the pipeline.

Parameters:
  • filt (Filter | Type[Filter]) – The filter instance or filter type to add.

  • field (str | None) – The field to filter on (required to instantiate Filter type).

Pipeline.add_transform(transform: Transform, **kwargs)#

Add a (SQL)Transform to the pipeline.

Parameters:

filt (Transform) – The Transform instance to add.

Pipeline.chain(filters: List[Filter] | None = None, transforms: List[Transform] | None = None, sql_transforms: List[Transform] | None = None, **kwargs)#

Chains additional filtering, transform and sql_transform operations on an existing pipeline. Note that if one or more sql_transforms are provided the pipeline is cloned rather than applying the operations on top of the existing pipeline.

Parameters:
  • filters (List[Filter] | None) – Additional filters to apply on top of existing pipeline.

  • transforms (List[Transform] | None) – Additional transforms to apply on top of existing pipeline.

  • sql_transforms (List[SQLTransform] | None) – Additional filters to apply on top of existing pipeline.

Return type:

Pipeline

Pipeline.clone(**params) Pipeline#

Create a new instance of the pipeline with optionally overridden parameter values.

Pipeline.precache(queries: Dict[str, Dict[str, List[Any]]] | List[Dict[str, Dict[str, Any]]]) None#

Populates the cache of the lumen.sources.base.Source with the provided queries.

Queries can be provided in two formats:

  • A dictionary containing ‘filters’ and ‘variables’ dictionaries each containing lists of values to compute a cross-product for, e.g.

    {
    ‘filters’: {

    <filter>’: [‘a’, ‘b’, ‘c’, …], …

    }, ‘variables’: {

    <variable>: [0, 2, 4, …], …

    }

    }

  • A list containing dictionaries of explicit values for each filter and variables.

    [{

    ‘filters’: {<filter>: ‘a’}, ‘variables’: {<variable>: 0}

    }, {

    ‘filters’: {<filter>: ‘a’}, ‘variables’: {<variable>: 1}

    ]

Pipeline.servable(title: Optional[str] = None, location: bool | 'Location' = True, area: str = 'main', target: Optional[str] = None) Viewable#

Serves the object or adds it to the configured pn.state.template if in a panel serve context, writes to the DOM if in a pyodide context and returns the Panel object to allow it to display itself in a notebook context.

Parameters:
  • title (str) – A string title to give the Document (if served as an app)

  • location (boolean or panel.io.location.Location) – Whether to create a Location component to observe and set the URL location.

  • area (str (deprecated)) – The area of a template to add the component too. Only has an effect if pn.config.template has been set.

  • target (str) – Target area to write to. If a template has been configured on pn.config.template this refers to the target area in the template while in pyodide this refers to the ID of the DOM node to write to.

Return type:

The Panel object itself

Pipeline.show(title: Optional[str] = None, port: int = 0, address: Optional[str] = None, websocket_origin: Optional[str] = None, threaded: bool = False, verbose: bool = True, open: bool = True, location: bool | 'Location' = True, **kwargs) threading.Thread | 'Server'#

Starts a Bokeh server and displays the Viewable in a new tab.

Parameters:
  • title (str | None) – A string title to give the Document (if served as an app)

  • port (int (optional, default=0)) – Allows specifying a specific port

  • address (str) – The address the server should listen on for HTTP requests.

  • websocket_origin (str or list(str) (optional)) – A list of hosts that can connect to the websocket. This is typically required when embedding a server app in an external web site. If None, “localhost” is used.

  • threaded (boolean (optional, default=False)) – Whether to launch the Server on a separate thread, allowing interactive use.

  • verbose (boolean (optional, default=True)) – Whether to print the address and port

  • open (boolean (optional, default=True)) – Whether to open the server in a new browser tab

  • location (boolean or panel.io.location.Location) – Whether to create a Location component to observe and set the URL location.

Returns:

server – Returns the Bokeh server instance or the thread the server was launched on (if threaded=True)

Return type:

bokeh.server.Server or threading.Thread

Pipeline.to_spec(context: Dict[str, Any] | None = None) Dict[str, Any]#

Exports the full specification to reconstruct this component.

Parameters:

context (Dict[str, Any]) – Context contains the specification of all previously serialized components, e.g. to allow resolving of references.

Return type:

Declarative specification of this component.

Pipeline.traverse(type) List[Transform] | List[Filter]#

Returns all Filter or Transform objects in a potentially chained pipeline.