Crawler CLI
Contents
Crawler CLI#
List of Crawler CLI commands:
crawler#
Crawler CLI.
crawler [OPTIONS] COMMAND [ARGS]...
Options
- --version#
Show the version and exit.
collections#
Show source collections available in the data store.
Returned source collections can optionally be filtered by identifier, service type, target, and whether or it has been extracted, transformed or ingested. For example, to return Mars collections not yet ingested but extracted, run:
crawler collections --target=mars --not-ingested --extracted
crawler collections [OPTIONS]
Options
- --id <id>#
Collection ID filter.
- --service-type <service_type>#
Service type filter.
- --target <target>#
Target filter.
- --extracted#
Filter to return only extracted collections.
- --not-extracted#
Filter to return collections not yet extracted.
- --transformed#
Filter to return only transformed collections.
- --not-transformed#
Filter to return collections not yet transformed.
- --ingested#
Filter to return only ingested collections.
- --not-ingested#
Filter to return collections not yet ingested.
config#
Show Crawler configuration.
crawler config [OPTIONS]
extract#
Extract source collection metadata files from source data catalog service.
crawler extract [OPTIONS]
Options
- --id <id>#
Collection ID.
- --overwrite, --no-overwrite#
Overwrite existing source collection files.
ingest#
Ingest transformed STAC catalog files to destination STAC API Catalog.
crawler ingest [OPTIONS]
Options
- --id <id>#
Collection ID.
- --update, --no-update#
Update destination STAC collection if exists
initds#
Initialise data store.
crawler initds [OPTIONS]
process#
Process all or a filtered selection of source collections.
Use options to filter source collections from the data store. If you’re unsure about the filtering result, first use the collections command to preview the list of source collections given your input filters.
Source collections will currently be processed sequentially, going through extraction, transformation and ingestion. Note that all STAC collections are grouped into a single STAC catalog, with one catalog per target body. Ingestion of all collections could be done at once when they all have been transformed.
crawler process [OPTIONS]
Options
- --id <id>#
Collection ID filter.
- --service-type <service_type>#
Service type filter.
- --target <target>#
Target filter.
- --extracted#
Filter to return only extracted collections.
- --not-extracted#
Filter to return collections not yet extracted.
- --transformed#
Filter to return only transformed collections.
- --not-transformed#
Filter to return collections not yet transformed.
- --ingested#
Filter to return only ingested collections.
- --not-ingested#
Filter to return collections not yet ingested.
- --overwrite, --no-overwrite#
Overwrite existing source collection files.
registry#
Show internal and external registered services.
Optionally display service information and collections using the service option. For example:
crawler registry --service-title='PDS ODE API'
crawler registry -s 'PDS ODE API'
crawler registry [OPTIONS]
Options
- -s, --service-title <service_title>#
Show service information/collections for a given service title.
schemas#
Get schemas information.
crawler schemas [OPTIONS]
Options
- --get, --no-get#
Get schema JSON representation.
- --name <name>#
Name of the schema.
- --type <type>#
Type of the schema object: collection or item.
transform#
Transform extracted source collection files to STAC catalog files.
crawler transform [OPTIONS]
Options
- --id <id>#
Collection ID.
- -o, --overwrite, --no-overwrite#
Overwrite existing STAC catalog files.