Source#

class lumen.sources.base.Source(*, cache_data, cache_dir, cache_metadata, cache_per_query, cache_schema, cache_with_dask, metadata_func, root, shared, name)#

Source components provide allow querying all kinds of data.

A Source can return one or more tables queried using the .get_tables method, a description of the data returned by each table in the form of a JSON schema accessible via the .get_schema method and lastly a .get method that allows filtering the data.

The Source base class also implements both in-memory and disk caching which can be enabled if a cache_dir is provided. Data cached to disk is stored as parquet files.

Parameters#

cache_data

type: bool
default: True
Whether to cache actual data.

cache_dir

type: str
default: None
Whether to enable local cache and write file to disk.

cache_metadata

type: bool
default: True
Whether to cache metadata.

cache_per_query

type: bool
default: True
Whether to query the whole dataset or individual queries.

cache_schema

type: bool
default: True
Whether to cache table schemas.

cache_with_dask

type: bool
default: True
Whether to read and write cache files with dask if available.

metadata_func

type: Any
default: None
Function to implement custom metadata lookup for tables.Given a list of tables it should return a dictionary of the form:{

: {“description”: …, “columns”: {“column_name”: “…”}}}May be used to override the default _get_table_metadataimplementation of the Source.

name

type: str
default: 'Source'
String identifier for this object.

root

type: pathlib.Path
default: None
Root folder of the cache_dir, default is config.root

shared

type: bool
default: False
Whether the Source can be shared across all instances of thedashboard. If set to True the Source will be loaded oninitial server load.

Methods#

Source.clear_cache(*events: Event)#: Clears any cached data.

Source.get(table: str, **query) → DataFrame#

Return a table; optionally filtered by the given query.

Parameters:

table (str) – The name of the table to query
query (dict) – A dictionary containing all the query parameters

Returns:

A DataFrame containing the queried table.

Return type:

DataFrame

Source.get_metadata(table: str | list[str] | None) → dict#

Returns metadata for one, multiple or all tables provided by the source.

The metadata for a table is structured as:

{

“description”: …, “columns”: {

<COLUMN>: {
“description”: …, “data_type”: …,

}

}, **other_metadata

}

If a list of tables or no table is provided the metadata is nested one additional level:

{

“table_name”: {

{

“description”: …, “columns”: {

<COLUMN>: { “description”: …, “data_type”: …, }

}, **other_metadata

}

Parameters:: table (str | list[str] | None) – The name of the table to return the schema for. If None returns schema for all available tables.
Returns:: metadata – Dictionary of metadata indexed by table (if no table was was provided or individual table metdata.
Return type:: dict

Source.get_schema(table: str | None = None, limit: int | None = None, shuffle: bool = False) → dict[str, dict[str, Any]] | dict[str, Any]#

Returns JSON schema describing the tables returned by the Source.

Parameters:

table (str | None) – The name of the table to return the schema for. If None returns schema for all available tables.
limit (int | None) – Limits the number of rows considered for the schema calculation

Returns: