IntakeSource type: intake#
- class lumen.sources.intake.IntakeSource(*, catalog, dask, uri, load_schema, cache_data, cache_dir, cache_metadata, cache_per_query, cache_schema, cache_with_dask, metadata_func, root, shared, name)#
An IntakeSource loads data from an Intake catalog.
Intake is a lightweight set of tools for loading and sharing data in data science projects using convenient catalog specifications.
The IntakeSource can be given a dictionary catalog specification OR a URI pointing to a catalog.yaml file on disk.
Parameters#
type: dict
default: None
An inlined Catalog specification.
type: bool
default: False
Whether to return a dask DataFrame.
type: bool
default: True
Whether to load the schema
type: str
default: ''
URI of the catalog file.
Methods#
- IntakeSource.clear_cache(*events: Event)#
Clears any cached data.
- IntakeSource.get(table, **query)#
Return a table; optionally filtered by the given query.
- Parameters:
table (str) – The name of the table to query
query (dict) – A dictionary containing all the query parameters
- Returns:
A DataFrame containing the queried table.
- Return type:
DataFrame
- IntakeSource.get_metadata(table: str | list[str] | None) dict#
Returns metadata for one, multiple or all tables provided by the source.
The metadata for a table is structured as:
- {
“description”: …, “columns”: {
- <COLUMN>: {
“description”: …, “data_type”: …,
}
}, **other_metadata
}
If a list of tables or no table is provided the metadata is nested one additional level:
- {
- “table_name”: {
- {
“description”: …, “columns”: {
<COLUMN>: { “description”: …, “data_type”: …, }
}, **other_metadata
}
}
}
- Parameters:
table (str | list[str] | None) – The name of the table to return the schema for. If None returns schema for all available tables.
- Returns:
metadata – Dictionary of metadata indexed by table (if no table was was provided or individual table metdata.
- Return type:
dict
- IntakeSource.get_schema(table: str | None = None, limit: int | None = None, shuffle: bool = False) dict[str, dict[str, Any]] | dict[str, Any]#
Returns JSON schema describing the tables returned by the Source.
- Parameters:
table (str | None) – The name of the table to return the schema for. If None returns schema for all available tables.
limit (int | None) – Limits the number of rows considered for the schema calculation
- Returns:
JSON schema(s) for one or all the tables.
- Return type:
dict
- IntakeSource.get_tables()#
Returns the list of tables available on this source.
- Returns:
The list of available tables on this source.
- Return type:
list
- IntakeSource.to_spec(context: dict[str, Any] | None = None) dict[str, Any]#
Exports the full specification to reconstruct this component.
- Return type:
Resolved and instantiated Component object