Crawler Python API
Contents
Crawler Python API#
crawler module#
Crawler module.
- class crawler.crawler.Crawler[source]#
Bases:
objectCrawler class acting as controller and high-level interface.
Used by the Crawler CLI and Airflow DAGs.
- extract_collection(collection_id: str, overwrite=False) None[source]#
Extract source collection file(s) from the “data catalog” service associated to a given collection identifier.
- get_registered_collections(retrieve=False) List[crawler.datastore.SourceCollectionModel][source]#
- get_registered_services(retrieve=False) List[crawler.registry.Service][source]#
- get_source_collection(collection_id) crawler.datastore.SourceCollectionModel[source]#
Returns the source collection of a given identifier.
- get_source_collections(collection_id='', service_type=None, target=None, extracted=None, transformed=None, ingested=None) [<class 'crawler.datastore.SourceCollectionModel'>][source]#
Returns the definition of the source collection matching a set of input filters.
- ingest_collection(collection_id, update=False)[source]#
Ingest a STAC collection into the destination STAC catalog service.
- list_source_collections(collection_id='', service_type=None, target=None, extracted=None, transformed=None, ingested=None)[source]#
Display all, or a filtered list of source collections indexed in the data store.
- process_collections(collection_id='', service_type=None, target=None, extracted=None, transformed=None, ingested=None, overwrite=False) None[source]#
Process all or a filtered selection of collections.
registry module#
PDSSP Crawler RegistryInterface module.
- pydantic model crawler.registry.ExternalService[source]#
Bases:
crawler.registry.ServiceShow JSON schema
{ "title": "ExternalService", "type": "object", "properties": { "title": { "title": "Title", "type": "string" }, "description": { "title": "Description", "type": "string" }, "providers": { "title": "Providers", "type": "array", "items": { "$ref": "#/definitions/ServiceProvider" } }, "type": { "$ref": "#/definitions/ExternalServiceType" }, "url": { "title": "Url", "type": "string" }, "ping_url": { "title": "Ping Url", "type": "string" }, "ssys:targets": { "title": "Ssys:Targets", "type": "array", "items": { "type": "string" } }, "extra_params": { "title": "Extra Params", "type": "object" } }, "required": [ "title", "description", "providers", "type", "url" ], "definitions": { "ProviderRole": { "title": "ProviderRole", "description": "An enumeration.", "enum": [ "producer", "licensor", "processor", "host" ] }, "ServiceProvider": { "title": "ServiceProvider", "type": "object", "properties": { "name": { "title": "Organization name", "type": "string" }, "description": { "title": "Organization description", "type": "string" }, "roles": { "title": "Organization roles", "type": "array", "items": { "$ref": "#/definitions/ProviderRole" } }, "url": { "title": "Organization homepage", "type": "string" } }, "required": [ "name" ] }, "ExternalServiceType": { "title": "ExternalServiceType", "description": "An enumeration.", "enum": [ "WFS", "PDSODE", "EPNTAP" ] } } }
- field extra_params: Optional[dict] = None#
- field type: crawler.registry.ExternalServiceType [Required]#
- class crawler.registry.ExternalServiceType(value)[source]#
Bases:
enum.EnumAn enumeration.
- EPNTAP = 'EPNTAP'#
- PDSODE = 'PDSODE'#
- WFS = 'WFS'#
- class crawler.registry.HealthcheckrRegistry(url='')[source]#
Bases:
crawler.registry.RegistryInterfaceClass that represents the PDSSP Services Registry.
End-point: https://pdssp.ias.universite-paris-saclay.fr/registry/services
- class crawler.registry.LocalRegistry(path='')[source]#
Bases:
crawler.registry.RegistryInterfaceClass that represents a local registry defining external services, not compliant to the PDSSP data model.
- class crawler.registry.ProviderRole(value)[source]#
Bases:
enum.EnumAn enumeration.
- host = 'host'#
- licensor = 'licensor'#
- processor = 'processor'#
- producer = 'producer'#
- class crawler.registry.RegistryInterface(url='', path='')[source]#
Bases:
objectAbstract registry class that defines a common interface for the children HealthcheckrRegistry and Local RegistryInterface classes.
- get_services() List[crawler.registry.Service][source]#
- pydantic model crawler.registry.Service[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "Service", "type": "object", "properties": { "title": { "title": "Title", "type": "string" }, "description": { "title": "Description", "type": "string" }, "providers": { "title": "Providers", "type": "array", "items": { "$ref": "#/definitions/ServiceProvider" } }, "type": { "$ref": "#/definitions/ServiceType" }, "url": { "title": "Url", "type": "string" }, "ping_url": { "title": "Ping Url", "type": "string" }, "ssys:targets": { "title": "Ssys:Targets", "type": "array", "items": { "type": "string" } } }, "required": [ "title", "description", "providers", "type", "url" ], "definitions": { "ProviderRole": { "title": "ProviderRole", "description": "An enumeration.", "enum": [ "producer", "licensor", "processor", "host" ] }, "ServiceProvider": { "title": "ServiceProvider", "type": "object", "properties": { "name": { "title": "Organization name", "type": "string" }, "description": { "title": "Organization description", "type": "string" }, "roles": { "title": "Organization roles", "type": "array", "items": { "$ref": "#/definitions/ProviderRole" } }, "url": { "title": "Organization homepage", "type": "string" } }, "required": [ "name" ] }, "ServiceType": { "title": "ServiceType", "description": "An enumeration.", "enum": [ "STAC", "WFS", "WMS", "WMTS", "XYZ" ] } } }
- Fields
- field description: str [Required]#
- field ping_url: Optional[str] = None#
- field providers: List[crawler.registry.ServiceProvider] [Required]#
- field ssys_targets: Optional[List[str]] = None (alias 'ssys:targets')#
- field title: str [Required]#
- field type: crawler.registry.ServiceType [Required]#
- field url: str [Required]#
- pydantic model crawler.registry.ServiceProvider[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "ServiceProvider", "type": "object", "properties": { "name": { "title": "Organization name", "type": "string" }, "description": { "title": "Organization description", "type": "string" }, "roles": { "title": "Organization roles", "type": "array", "items": { "$ref": "#/definitions/ProviderRole" } }, "url": { "title": "Organization homepage", "type": "string" } }, "required": [ "name" ], "definitions": { "ProviderRole": { "title": "ProviderRole", "description": "An enumeration.", "enum": [ "producer", "licensor", "processor", "host" ] } } }
- Fields
- field description: Optional[str] = None#
- field name: str [Required]#
- field roles: Optional[List[crawler.registry.ProviderRole]] = None#
- field url: Optional[str] = None#
schemas module#
Schemas module for the definition and retrieval of "collection" and "item"-type metadata dictionary
compliant to the following schemas/models:
Schema |
Collection metadata |
Item metadata |
|---|---|---|
|
||
|
||
|
||
|
||
|
References/examples:
https://github.com/radiantearth/stac-spec/blob/master/item-spec
https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md
https://github.com/ivoa-std/EPNTAP/blob/master/example-record.xml
- pydantic model crawler.schemas.EPNTAP_Collection[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "EPNTAP_Collection", "type": "object", "properties": {} }
- pydantic model crawler.schemas.EPNTAP_Granule[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "EPNTAP_Granule", "type": "object", "properties": { "granule_uid": { "title": "Granule Uid", "type": "string" }, "granule_gid": { "title": "Granule Gid", "type": "string" }, "obs_id": { "title": "Obs Id", "type": "string" }, "dataproduct_type": { "title": "Dataproduct Type", "type": "string" }, "target_name": { "title": "Target Name", "type": "string" }, "target_class": { "title": "Target Class", "type": "string" }, "time_min": { "title": "Time Min", "type": "string" }, "time_max": { "title": "Time Max", "type": "string" }, "time_sampling_step_min": { "title": "Time Sampling Step Min", "type": "string" }, "time_sampling_step_max": { "title": "Time Sampling Step Max", "type": "string" }, "time_exp_min": { "title": "Time Exp Min", "type": "string" }, "time_exp_max": { "title": "Time Exp Max", "type": "string" }, "spectral_range_min": { "title": "Spectral Range Min", "type": "string" }, "spectral_range_max": { "title": "Spectral Range Max", "type": "string" }, "spectral_sampling_step_min": { "title": "Spectral Sampling Step Min", "type": "string" }, "spectral_sampling_step_max": { "title": "Spectral Sampling Step Max", "type": "string" }, "spectral_resolution_min": { "title": "Spectral Resolution Min", "type": "string" }, "spectral_resolution_max": { "title": "Spectral Resolution Max", "type": "string" }, "c1min": { "title": "C1Min", "type": "string" }, "c1max": { "title": "C1Max", "type": "string" }, "c2min": { "title": "C2Min", "type": "string" }, "c2max": { "title": "C2Max", "type": "string" }, "c3min": { "title": "C3Min", "type": "string" }, "c3max": { "title": "C3Max", "type": "string" }, "s_region": { "title": "S Region", "type": "string" }, "c1_resol_min": { "title": "C1 Resol Min", "type": "string" }, "c1_resol_max": { "title": "C1 Resol Max", "type": "string" }, "c2_resol_min": { "title": "C2 Resol Min", "type": "string" }, "c2_resol_max": { "title": "C2 Resol Max", "type": "string" }, "c3_resol_min": { "title": "C3 Resol Min", "type": "string" }, "c3_resol_max": { "title": "C3 Resol Max", "type": "string" }, "spatial_frame_type": { "title": "Spatial Frame Type", "type": "string" }, "incidence_min": { "title": "Incidence Min", "type": "string" }, "incidence_max": { "title": "Incidence Max", "type": "string" }, "emergence_min": { "title": "Emergence Min", "type": "string" }, "emergence_max": { "title": "Emergence Max", "type": "string" }, "phase_min": { "title": "Phase Min", "type": "string" }, "phase_max": { "title": "Phase Max", "type": "string" }, "instrument_host_name": { "title": "Instrument Host Name", "type": "string" }, "instrument_name": { "title": "Instrument Name", "type": "string" }, "measurement_type": { "title": "Measurement Type", "type": "string" }, "processing_level": { "title": "Processing Level", "type": "string" }, "creation_date": { "title": "Creation Date", "type": "string" }, "modification_date": { "title": "Modification Date", "type": "string" }, "release_date": { "title": "Release Date", "type": "string" }, "service_title": { "title": "Service Title", "type": "string" }, "access_url": { "title": "Access Url", "type": "string" }, "access_format": { "title": "Access Format", "type": "string" }, "access_estsize": { "title": "Access Estsize", "type": "string" }, "file_name": { "title": "File Name", "type": "string" }, "publisher": { "title": "Publisher", "type": "string" } }, "required": [ "granule_uid", "granule_gid", "obs_id", "dataproduct_type", "target_name", "target_class", "time_min", "time_max", "time_sampling_step_min", "time_sampling_step_max", "time_exp_min", "time_exp_max", "spectral_range_min", "spectral_range_max", "spectral_sampling_step_min", "spectral_sampling_step_max", "spectral_resolution_min", "spectral_resolution_max", "c1min", "c1max", "c2min", "c2max", "c3min", "c3max", "s_region", "c1_resol_min", "c1_resol_max", "c2_resol_min", "c2_resol_max", "c3_resol_min", "c3_resol_max", "spatial_frame_type", "incidence_min", "incidence_max", "emergence_min", "emergence_max", "phase_min", "phase_max", "instrument_host_name", "instrument_name", "measurement_type", "processing_level", "creation_date", "modification_date", "release_date", "service_title", "access_url", "access_format", "access_estsize", "file_name", "publisher" ] }
- Fields
- field access_estsize: str [Required]#
- field access_format: str [Required]#
- field access_url: str [Required]#
- field c1_resol_max: str [Required]#
- field c1_resol_min: str [Required]#
- field c1max: str [Required]#
- field c1min: str [Required]#
- field c2_resol_max: str [Required]#
- field c2_resol_min: str [Required]#
- field c2max: str [Required]#
- field c2min: str [Required]#
- field c3_resol_max: str [Required]#
- field c3_resol_min: str [Required]#
- field c3max: str [Required]#
- field c3min: str [Required]#
- field creation_date: str [Required]#
- field dataproduct_type: str [Required]#
- field emergence_max: str [Required]#
- field emergence_min: str [Required]#
- field file_name: str [Required]#
- field granule_gid: str [Required]#
- field granule_uid: str [Required]#
- field incidence_max: str [Required]#
- field incidence_min: str [Required]#
- field instrument_host_name: str [Required]#
- field instrument_name: str [Required]#
- field measurement_type: str [Required]#
- field modification_date: str [Required]#
- field obs_id: str [Required]#
- field phase_max: str [Required]#
- field phase_min: str [Required]#
- field processing_level: str [Required]#
- field publisher: str [Required]#
- field release_date: str [Required]#
- field s_region: str [Required]#
- field service_title: str [Required]#
- field spatial_frame_type: str [Required]#
- field spectral_range_max: str [Required]#
- field spectral_range_min: str [Required]#
- field spectral_resolution_max: str [Required]#
- field spectral_resolution_min: str [Required]#
- field spectral_sampling_step_max: str [Required]#
- field spectral_sampling_step_min: str [Required]#
- field target_class: str [Required]#
- field target_name: str [Required]#
- field time_exp_max: str [Required]#
- field time_exp_min: str [Required]#
- field time_max: str [Required]#
- field time_min: str [Required]#
- field time_sampling_step_max: str [Required]#
- field time_sampling_step_min: str [Required]#
- pydantic model crawler.schemas.MARSSI_WFS_Feature[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "MARSSI_WFS_Feature", "type": "object", "properties": {} }
- pydantic model crawler.schemas.MARSSI_WFS_Layer[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "MARSSI_WFS_Layer", "type": "object", "properties": {} }
- crawler.schemas.METADATA_SCHEMAS = {'EPNTAP': {'collection': <class 'crawler.schemas.EPNTAP_Collection'>, 'item': <class 'crawler.schemas.EPNTAP_Granule'>}, 'MARSSI_WFS': {'collection': <class 'crawler.schemas.MARSSI_WFS_Layer'>, 'item': <class 'crawler.schemas.MARSSI_WFS_Feature'>}, 'PDSODE': {'collection': <class 'crawler.schemas.PDSODE_IIPTSet'>, 'item': <class 'crawler.schemas.PDSODE_Product'>}, 'PDSSP_STAC': {'collection': <class 'crawler.schemas.PDSSP_STAC_Collection'>, 'item': <class 'crawler.schemas.PDSSP_STAC_Item'>}, 'PDSSP_WFS': {'collection': <class 'crawler.schemas.PDSSP_WFS_Layer'>, 'item': <class 'crawler.schemas.PDSSP_WFS_Feature'>}}#
Mapping of netadata schemas and their corresponding
"collection"and"item"classes.
- pydantic model crawler.schemas.PDSODE_Collection[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSODE_Collection", "type": "object", "properties": { "iiptset": { "$ref": "#/definitions/PDSODE_IIPTSet" }, "stac_extensions": { "title": "Stac Extensions", "type": "array", "items": { "type": "string" } } }, "required": [ "iiptset" ], "definitions": { "PDSODE_ValidTargets": { "title": "PDSODE_ValidTargets", "type": "object", "properties": { "ValidTarget": { "title": "Validtarget", "anyOf": [ { "type": "string" }, { "type": "array", "items": { "type": "string" } } ] } }, "required": [ "ValidTarget" ] }, "PDSODE_IIPTSet": { "title": "PDSODE_IIPTSet", "type": "object", "properties": { "ODEMetaDB": { "title": "Odemetadb", "type": "string" }, "IHID": { "title": "Ihid", "type": "string" }, "IHName": { "title": "Ihname", "type": "string" }, "IID": { "title": "Iid", "type": "string" }, "IName": { "title": "Iname", "type": "string" }, "PT": { "title": "Pt", "type": "string" }, "PTName": { "title": "Ptname", "type": "string" }, "DataSetId": { "title": "Datasetid", "type": "string" }, "ValidTargets": { "$ref": "#/definitions/PDSODE_ValidTargets" }, "NumberProducts": { "title": "Numberproducts", "type": "integer" } }, "required": [ "ODEMetaDB", "IHID", "IHName", "IID", "IName", "PT", "PTName", "DataSetId", "ValidTargets", "NumberProducts" ] } } }
- field iiptset: crawler.schemas.PDSODE_IIPTSet [Required]#
- field stac_extensions: Optional[list[str]] = None#
- pydantic model crawler.schemas.PDSODE_IIPTSet[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSODE_IIPTSet", "type": "object", "properties": { "ODEMetaDB": { "title": "Odemetadb", "type": "string" }, "IHID": { "title": "Ihid", "type": "string" }, "IHName": { "title": "Ihname", "type": "string" }, "IID": { "title": "Iid", "type": "string" }, "IName": { "title": "Iname", "type": "string" }, "PT": { "title": "Pt", "type": "string" }, "PTName": { "title": "Ptname", "type": "string" }, "DataSetId": { "title": "Datasetid", "type": "string" }, "ValidTargets": { "$ref": "#/definitions/PDSODE_ValidTargets" }, "NumberProducts": { "title": "Numberproducts", "type": "integer" } }, "required": [ "ODEMetaDB", "IHID", "IHName", "IID", "IName", "PT", "PTName", "DataSetId", "ValidTargets", "NumberProducts" ], "definitions": { "PDSODE_ValidTargets": { "title": "PDSODE_ValidTargets", "type": "object", "properties": { "ValidTarget": { "title": "Validtarget", "anyOf": [ { "type": "string" }, { "type": "array", "items": { "type": "string" } } ] } }, "required": [ "ValidTarget" ] } } }
- Fields
- field DataSetId: str [Required]#
- field IHID: str [Required]#
- field IHName: str [Required]#
- field IID: str [Required]#
- field IName: str [Required]#
- field NumberProducts: int [Required]#
- field ODEMetaDB: str [Required]#
- field PT: str [Required]#
- field PTName: str [Required]#
- field ValidTargets: crawler.schemas.PDSODE_ValidTargets [Required]#
- pydantic model crawler.schemas.PDSODE_Product[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSODE_Product", "type": "object", "properties": { "ode_id": { "title": "Ode Id", "type": "string" }, "pdsid": { "title": "Pdsid", "type": "string" }, "ihid": { "title": "Ihid", "type": "string" }, "iid": { "title": "Iid", "type": "string" }, "pt": { "title": "Pt", "type": "string" }, "Label_Product_Type": { "title": "Label Product Type", "type": "string" }, "Data_Set_Id": { "title": "Data Set Id", "type": "string" }, "PDSVolume_Id": { "title": "Pdsvolume Id", "type": "string" }, "RelativePathtoVol": { "title": "Relativepathtovol", "type": "string" }, "LabelFileName": { "title": "Labelfilename", "type": "string" }, "Product_creation_time": { "title": "Product Creation Time", "type": "string" }, "Target_name": { "title": "Target Name", "type": "string" }, "Product_version_id": { "title": "Product Version Id", "type": "string" }, "Observation_id": { "title": "Observation Id", "type": "string" }, "Observation_number": { "title": "Observation Number", "type": "string" }, "Observation_type": { "title": "Observation Type", "type": "string" }, "Producer_id": { "title": "Producer Id", "type": "string" }, "Producer_name": { "title": "Producer Name", "type": "string" }, "Product_release_date": { "title": "Product Release Date", "type": "string" }, "Activity_id": { "title": "Activity Id", "type": "string" }, "Predicted_dust_opacity": { "title": "Predicted Dust Opacity", "type": "string" }, "Predicted_dust_opacity_text": { "title": "Predicted Dust Opacity Text", "type": "string" }, "Observation_time": { "title": "Observation Time", "type": "string" }, "SpaceCraft_clock_start_count": { "title": "Spacecraft Clock Start Count", "type": "string" }, "SpaceCraft_clock_stop_count": { "title": "Spacecraft Clock Stop Count", "type": "string" }, "Stop_orbit_number": { "title": "Stop Orbit Number", "type": "string" }, "UTC_start_time": { "title": "Utc Start Time", "type": "string" }, "UTC_stop_time": { "title": "Utc Stop Time", "type": "string" }, "Emission_angle": { "title": "Emission Angle", "type": "number" }, "Emission_angle_text": { "title": "Emission Angle Text", "type": "string" }, "Phase_angle": { "title": "Phase Angle", "type": "number" }, "Phase_angle_text": { "title": "Phase Angle Text", "type": "number" }, "Incidence_angle": { "title": "Incidence Angle", "type": "number" }, "Incidence_angle_text": { "title": "Incidence Angle Text", "type": "number" }, "Map_resolution": { "title": "Map Resolution", "type": "number" }, "Map_resolution_text": { "title": "Map Resolution Text", "type": "number" }, "Map_scale": { "title": "Map Scale", "type": "number" }, "Map_scale_text": { "title": "Map Scale Text", "type": "number" }, "Solar_distance": { "title": "Solar Distance", "type": "number" }, "Solar_distance_text": { "title": "Solar Distance Text", "type": "number" }, "Solar_longitude": { "title": "Solar Longitude", "type": "number" }, "Center_georeferenced": { "title": "Center Georeferenced", "type": "string" }, "Center_latitude": { "title": "Center Latitude", "type": "string" }, "Center_longitude": { "title": "Center Longitude", "type": "string" }, "Center_latitude_text": { "title": "Center Latitude Text", "type": "string" }, "Center_longitude_text": { "title": "Center Longitude Text", "type": "string" }, "BB_georeferenced": { "title": "Bb Georeferenced", "type": "string" }, "Easternmost_longitude": { "title": "Easternmost Longitude", "type": "string" }, "Maximum_latitude": { "title": "Maximum Latitude", "type": "string" }, "Minimum_latitude": { "title": "Minimum Latitude", "type": "string" }, "Westernmost_longitude": { "title": "Westernmost Longitude", "type": "string" }, "Easternmost_longitude_text": { "title": "Easternmost Longitude Text", "type": "string" }, "Maximum_latitude_text": { "title": "Maximum Latitude Text", "type": "string" }, "Minimum_latitude_text": { "title": "Minimum Latitude Text", "type": "string" }, "Westernmost_longitude_text": { "title": "Westernmost Longitude Text", "type": "string" }, "Footprint_geometry": { "title": "Footprint Geometry", "type": "string" }, "Footprint_C0_geometry": { "title": "Footprint C0 Geometry", "type": "string" }, "Footprint_GL_geometry": { "title": "Footprint Gl Geometry", "type": "string" }, "Footprint_NP_geometry": { "title": "Footprint Np Geometry", "type": "string" }, "Footprint_SP_geometry": { "title": "Footprint Sp Geometry", "type": "string" }, "Footprints_cross_meridian": { "title": "Footprints Cross Meridian", "type": "string" }, "Pole_state": { "title": "Pole State", "type": "string" }, "Footprint_souce": { "title": "Footprint Souce", "type": "string" }, "Comment": { "title": "Comment", "type": "string" }, "Description": { "title": "Description", "type": "string" }, "External_url": { "title": "External Url", "type": "string" }, "External_url2": { "title": "External Url2", "type": "string" }, "External_url3": { "title": "External Url3", "type": "string" }, "FilesURL": { "title": "Filesurl", "type": "string" }, "ProductURL": { "title": "Producturl", "type": "string" }, "LabelURL": { "title": "Labelurl", "type": "string" }, "Product_files": { "$ref": "#/definitions/PDSODE_Product_file_key" } }, "required": [ "ode_id", "pdsid", "ihid", "iid", "pt", "Data_Set_Id", "PDSVolume_Id", "RelativePathtoVol", "LabelFileName", "Product_creation_time", "Target_name", "UTC_start_time", "UTC_stop_time", "Product_files" ], "definitions": { "PDSODE_Product_file": { "title": "PDSODE_Product_file", "type": "object", "properties": { "Description": { "title": "Description", "type": "string" }, "FileName": { "title": "Filename", "type": "string" }, "KBytes": { "title": "Kbytes", "type": "string" }, "Type": { "title": "Type", "type": "string" }, "URL": { "title": "Url", "type": "string" } }, "required": [ "Description", "FileName", "KBytes", "Type", "URL" ] }, "PDSODE_Product_file_key": { "title": "PDSODE_Product_file_key", "type": "object", "properties": { "Product_file": { "title": "Product File", "type": "array", "items": { "$ref": "#/definitions/PDSODE_Product_file" } } }, "required": [ "Product_file" ] } } }
- Fields
- field Activity_id: Optional[str] = None#
Label Activity id
- field BB_georeferenced: Optional[str] = None#
T if the product has a footprint bounding box.
- field Center_georeferenced: Optional[str] = None#
T if the product has a footprint center.
- field Center_latitude: Optional[str] = None#
Planetocentric footprint center latitude.
- field Center_latitude_text: Optional[str] = None#
Text found in the center latitude label keyword if the center latitude is not a valid number.
- field Center_longitude: Optional[str] = None#
Longitude 0-360 center longitude.
- field Center_longitude_text: Optional[str] = None#
Text found in the center longitude label keyword if the center longitude is not a valid number.
- field Comment: Optional[str] = None#
Any associated comment.
- field Data_Set_Id: str [Required]#
PDS Data Set Id.
- field Description: Optional[str] = None#
Label description
- field Easternmost_longitude: Optional[str] = None#
Longitude 0-360 Easternmost longitude of the footprint.
- field Easternmost_longitude_text: Optional[str] = None#
Text found in the easternmost longitude label keyword if the easternmost longitude is not a valid number.
- field Emission_angle: Optional[float] = None#
Emission angle.
- field Emission_angle_text: Optional[str] = None#
Emission angle text from the product label.
- field External_url: Optional[str] = None#
URL to an external reference to the product. Product type specific but usually something like the HiRISE site.
- field External_url2: Optional[str] = None#
URL to an external reference to the product. Product type specific but usually something like the HiRISE site.
- field External_url3: Optional[str] = None#
URL to an external reference to the product. Product type specific but usually something like the HiRISE site.
- field FilesURL: Optional[str] = None#
- field Footprint_C0_geometry: Optional[str] = None#
Planetocentric, longitude -180-180 product footprint in WKT format. Only if there is a valid footprint. Note - this is a cylindrical projected footprint. The footprint has been split into multiple polygons when crossing the -180/180 longitude line and any footprints that cross the poles have been adjusted to add points to and around the pole. It is meant for use in cylindrical projects and is not appropriate for spherical displays.
- field Footprint_GL_geometry: Optional[str] = None#
Planetocentric, longitude 0-360 product footprint in WKT format. Only if there is a valid footprint. This is not a projected footprint.
- field Footprint_NP_geometry: Optional[str] = None#
Stereographic south polar projected footprint in WKT format. Only if there is a valid footprint. This footprint has been projected into meters in stereographic north polar projection.
- field Footprint_SP_geometry: Optional[str] = None#
Stereographic south polar projected footprint in WKT format. Only if there is a valid footprint. This footprint has been projected into meters in stereographic south polar projection.
- field Footprint_geometry: Optional[str] = None#
Cylindrical projected planetocentric, longitude 0-360 product footprint in WKT format. Only if there is a valid footprint. Note - this is a cylindrical projected footprint. The footprint has been split into multiple polygons when crossing the 0/360 longitude line and any footprints that cross the poles have been adjusted to add points to and around the pole. It is meant for use in cylindrical projects and is not appropriate for spherical displays.
- field Footprint_souce: Optional[str] = None#
A brief description of where the footprint came from.
- field Footprints_cross_meridian: Optional[str] = None#
T if the footprint crosses the 0/360 longitude line (anti-meridian).
- field Incidence_angle: Optional[float] = None#
Incidence angle.
- field Incidence_angle_text: Optional[float] = None#
Incidence angle text from the product label.
- field LabelFileName: str [Required]#
The file name of the product label.
- field LabelURL: Optional[str] = None#
- field Label_Product_Type: Optional[str] = None#
Label product type (if it exists in the label and is different from the ODE_Product_Type).
- field Map_resolution: Optional[float] = None#
Map resolution.
- field Map_resolution_text: Optional[float] = None#
Map resolution text from the product label.
- field Map_scale: Optional[float] = None#
Map scale.
- field Map_scale_text: Optional[float] = None#
Map scale text from the product label.
- field Maximum_latitude: Optional[str] = None#
Planetocentric maximum latitude of the footprint.
- field Maximum_latitude_text: Optional[str] = None#
Text found in the maximum latitude label keyword if the maximum latitude is not a valid number.
- field Minimum_latitude: Optional[str] = None#
Planetocentric minimum latitude of the footprint.
- field Minimum_latitude_text: Optional[str] = None#
Text found in the minimum latitude label keyword if the minimum latitude is not a valid number.
- field Observation_id: Optional[str] = None#
Identifies a scientific observation within a data set.
- field Observation_number: Optional[str] = None#
Monotonically increasing ordinal counter of the EDRs generated for a particular OBSERVATION_ID.
- field Observation_time: Optional[str] = None#
Observation time (mid-point between the start and end of the observation).
- field Observation_type: Optional[str] = None#
Identifies the general type of an observation.
- field PDSVolume_Id: str [Required]#
Volume Id.
- field Phase_angle: Optional[float] = None#
Phase angle.
- field Phase_angle_text: Optional[float] = None#
Phase angle text from the product label.
- field Pole_state: Optional[str] = None#
String of “none”, “north”, or “south”.
- field Predicted_dust_opacity: Optional[str] = None#
Predicted dust opacity.
- field Predicted_dust_opacity_text: Optional[str] = None#
Predicted dust opacity text.
- field Producer_id: Optional[str] = None#
Producer id.
- field Producer_name: Optional[str] = None#
Producer name.
- field ProductURL: Optional[str] = None#
- field Product_creation_time: str [Required]#
Product creation time (UTC).
- field Product_files: crawler.schemas.PDSODE_Product_file_key [Required]#
Associated product files.
- field Product_release_date: Optional[str] = None#
Product release date.
- field Product_version_id: Optional[str] = None#
Product version.
- field RelativePathtoVol: str [Required]#
The relative path from the volume root to the product label file.
- field Solar_distance: Optional[float] = None#
Solar distance.
- field Solar_distance_text: Optional[float] = None#
Solar distance text from the product label.
- field Solar_longitude: Optional[float] = None#
Solar longitude.
- field SpaceCraft_clock_start_count: Optional[str] = None#
Spacecraft clock start.
- field SpaceCraft_clock_stop_count: Optional[str] = None#
Spacecraft clock stop.
- field Stop_orbit_number: Optional[str] = None#
Stop orbit number.
- field Target_name: str [Required]#
Product target (example: Mars).
- field UTC_start_time: str [Required]#
Observation start time in UTC.
- field UTC_stop_time: str [Required]#
Observation stop time in UTC.
- field Westernmost_longitude: Optional[str] = None#
Longitude 0-360 Westernmost longitude of the footprint.
- field Westernmost_longitude_text: Optional[str] = None#
Text found in the westernmost longitude label keyword if the westernmost longitude is not a valid number.
- field ihid: str [Required]#
Instrument host id.
- field iid: str [Required]#
Instrument id.
- field ode_id: str [Required]#
An internal ODE product identifier.
- field pdsid: str [Required]#
PDS Product Id.
- field pt: str [Required]#
ODE Product type.
- pydantic model crawler.schemas.PDSODE_Product_file[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSODE_Product_file", "type": "object", "properties": { "Description": { "title": "Description", "type": "string" }, "FileName": { "title": "Filename", "type": "string" }, "KBytes": { "title": "Kbytes", "type": "string" }, "Type": { "title": "Type", "type": "string" }, "URL": { "title": "Url", "type": "string" } }, "required": [ "Description", "FileName", "KBytes", "Type", "URL" ] }
- field Description: str [Required]#
- field FileName: str [Required]#
- field KBytes: str [Required]#
- field Type: str [Required]#
- field URL: str [Required]#
- pydantic model crawler.schemas.PDSODE_Product_file_key[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSODE_Product_file_key", "type": "object", "properties": { "Product_file": { "title": "Product File", "type": "array", "items": { "$ref": "#/definitions/PDSODE_Product_file" } } }, "required": [ "Product_file" ], "definitions": { "PDSODE_Product_file": { "title": "PDSODE_Product_file", "type": "object", "properties": { "Description": { "title": "Description", "type": "string" }, "FileName": { "title": "Filename", "type": "string" }, "KBytes": { "title": "Kbytes", "type": "string" }, "Type": { "title": "Type", "type": "string" }, "URL": { "title": "Url", "type": "string" } }, "required": [ "Description", "FileName", "KBytes", "Type", "URL" ] } } }
- field Product_file: list[crawler.schemas.PDSODE_Product_file] [Required]#
- pydantic model crawler.schemas.PDSODE_ValidTargets[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSODE_ValidTargets", "type": "object", "properties": { "ValidTarget": { "title": "Validtarget", "anyOf": [ { "type": "string" }, { "type": "array", "items": { "type": "string" } } ] } }, "required": [ "ValidTarget" ] }
- field ValidTarget: Union[str, list[str]] [Required]#
- pydantic model crawler.schemas.PDSSP_STAC_Asset[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Asset", "type": "object", "properties": { "href": { "title": "Href", "type": "string" }, "title": { "title": "Title", "type": "string" }, "description": { "title": "Description", "type": "string" }, "type": { "title": "Type", "type": "string" }, "roles": { "title": "Roles", "type": "array", "items": { "type": "string" } } }, "required": [ "href" ] }
- Fields
- field description: Optional[str] = None#
- field href: str [Required]#
- field roles: Optional[list[str]] = None#
- field title: Optional[str] = None#
- field type: Optional[str] = None#
- pydantic model crawler.schemas.PDSSP_STAC_Collection[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Collection", "type": "object", "properties": { "type": { "title": "Type", "type": "string" }, "stac_version": { "title": "Stac Version", "default": "1.0.0", "type": "string" }, "stac_extensions": { "title": "Stac Extensions", "type": "array", "items": { "type": "string" } }, "id": { "title": "Id", "type": "string" }, "title": { "title": "Title", "type": "string" }, "description": { "title": "Description", "type": "string" }, "keywords": { "title": "Keywords", "type": "array", "items": { "type": "string" } }, "licence": { "title": "Licence", "type": "string" }, "providers": { "title": "Providers", "type": "array", "items": { "$ref": "#/definitions/PDSSP_STAC_Provider" } }, "extent": { "$ref": "#/definitions/PDSSP_STAC_Extent" }, "summaries": { "title": "Summaries", "type": "object" }, "links": { "title": "Links", "type": "array", "items": { "$ref": "#/definitions/PDSSP_STAC_Link" } }, "assets": { "title": "Assets", "type": "object" }, "extra_fields": { "title": "Extra Fields", "type": "object" } }, "required": [ "type", "id", "description", "licence", "extent" ], "definitions": { "PDSSP_STAC_Provider": { "title": "PDSSP_STAC_Provider", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "description": { "title": "Description", "type": "string" }, "roles": { "title": "Roles", "type": "array", "items": { "type": "string" } } }, "required": [ "name" ] }, "PDSSP_STAC_SpatialExtent": { "title": "PDSSP_STAC_SpatialExtent", "type": "object", "properties": { "bbox": { "title": "Bbox", "type": "array", "items": { "type": "array", "items": { "type": "number" } } } }, "required": [ "bbox" ] }, "PDSSP_STAC_TemporalExtent": { "title": "PDSSP_STAC_TemporalExtent", "type": "object", "properties": { "interval": { "title": "Interval", "type": "array", "items": { "type": "array", "items": { "type": "string" } } } }, "required": [ "interval" ] }, "PDSSP_STAC_Extent": { "title": "PDSSP_STAC_Extent", "type": "object", "properties": { "spatial": { "$ref": "#/definitions/PDSSP_STAC_SpatialExtent" }, "temporal": { "$ref": "#/definitions/PDSSP_STAC_TemporalExtent" } }, "required": [ "spatial", "temporal" ] }, "PDSSP_STAC_Link": { "title": "PDSSP_STAC_Link", "type": "object", "properties": { "href": { "title": "Href", "type": "string" }, "rel": { "title": "Rel", "type": "string" }, "type": { "title": "Type", "type": "string" }, "title": { "title": "Title", "type": "string" } }, "required": [ "href", "rel" ] } } }
- field assets: Optional[dict] = None#
- field description: str [Required]#
- field extent: crawler.schemas.PDSSP_STAC_Extent [Required]#
- field extra_fields: Optional[dict] = None#
- field id: str [Required]#
- field keywords: Optional[list[str]] = None#
- field licence: str [Required]#
- field links: Optional[list[crawler.schemas.PDSSP_STAC_Link]] = None#
- field providers: Optional[list[crawler.schemas.PDSSP_STAC_Provider]] = None#
- field stac_extensions: Optional[list[str]] = None#
- field stac_version: str = '1.0.0'#
- field summaries: Optional[dict] = None#
- field title: Optional[str] = None#
- field type: str [Required]#
- pydantic model crawler.schemas.PDSSP_STAC_Extent[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Extent", "type": "object", "properties": { "spatial": { "$ref": "#/definitions/PDSSP_STAC_SpatialExtent" }, "temporal": { "$ref": "#/definitions/PDSSP_STAC_TemporalExtent" } }, "required": [ "spatial", "temporal" ], "definitions": { "PDSSP_STAC_SpatialExtent": { "title": "PDSSP_STAC_SpatialExtent", "type": "object", "properties": { "bbox": { "title": "Bbox", "type": "array", "items": { "type": "array", "items": { "type": "number" } } } }, "required": [ "bbox" ] }, "PDSSP_STAC_TemporalExtent": { "title": "PDSSP_STAC_TemporalExtent", "type": "object", "properties": { "interval": { "title": "Interval", "type": "array", "items": { "type": "array", "items": { "type": "string" } } } }, "required": [ "interval" ] } } }
- Fields
- field spatial: crawler.schemas.PDSSP_STAC_SpatialExtent [Required]#
Potential spatial extents covered by the Collection.
- field temporal: crawler.schemas.PDSSP_STAC_TemporalExtent [Required]#
Potential temporal extents covered by the Collection.
- pydantic model crawler.schemas.PDSSP_STAC_Item[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Item", "type": "object", "properties": { "type": { "title": "Type", "type": "string" }, "stac_version": { "title": "Stac Version", "default": "1.0.0", "type": "string" }, "stac_extensions": { "title": "Stac Extensions", "type": "array", "items": { "type": "string" } }, "id": { "title": "Id", "type": "string" }, "geometry": { "title": "Geometry" }, "bbox": { "title": "Bbox", "type": "array", "items": { "type": "number" } }, "properties": { "title": "Properties", "type": "object" }, "links": { "title": "Links", "type": "array", "items": { "$ref": "#/definitions/PDSSP_STAC_Link" } }, "assets": { "title": "Assets", "type": "object" }, "collection": { "title": "Collection", "type": "string" }, "extra_fields": { "title": "Extra Fields", "type": "object" } }, "required": [ "type", "id", "properties", "assets" ], "definitions": { "PDSSP_STAC_Link": { "title": "PDSSP_STAC_Link", "type": "object", "properties": { "href": { "title": "Href", "type": "string" }, "rel": { "title": "Rel", "type": "string" }, "type": { "title": "Type", "type": "string" }, "title": { "title": "Title", "type": "string" } }, "required": [ "href", "rel" ] } } }
- Fields
- field assets: dict [Required]#
- field bbox: Optional[list[float]] = None#
- field collection: Optional[str] = None#
- field extra_fields: Optional[dict] = None#
- field geometry: object = None#
- field id: str [Required]#
- field links: Optional[list[crawler.schemas.PDSSP_STAC_Link]] = None#
- field properties: dict [Required]#
- field stac_extensions: Optional[list[str]] = None#
- field stac_version: str = '1.0.0'#
- field type: str [Required]#
- pydantic model crawler.schemas.PDSSP_STAC_Link[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Link", "type": "object", "properties": { "href": { "title": "Href", "type": "string" }, "rel": { "title": "Rel", "type": "string" }, "type": { "title": "Type", "type": "string" }, "title": { "title": "Title", "type": "string" } }, "required": [ "href", "rel" ] }
- field href: str [Required]#
The actual link in the format of an URL. Relative and absolute links are both allowed.
- field rel: str [Required]#
Relationship between the current document and the linked document.
- field title: Optional[str] = None#
A human readable title to be used in rendered displays of the link.
- field type: Optional[str] = None#
Media type of the referenced entity.
- pydantic model crawler.schemas.PDSSP_STAC_Processing_Properties[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Processing_Properties", "type": "object", "properties": { "processing:expression": { "title": "Processing:Expression" }, "processing:lineage": { "title": "Processing:Lineage", "type": "string" }, "processing:level": { "title": "Processing:Level", "type": "string" }, "processing:facility": { "title": "Processing:Facility", "type": "string" }, "processing:software": { "title": "Processing:Software", "type": "object" } } }
- Fields
- field processing_expression: Optional[object] = None (alias 'processing:expression')#
- field processing_facility: Optional[str] = None (alias 'processing:facility')#
- field processing_level: Optional[str] = None (alias 'processing:level')#
- field processing_lineage: Optional[str] = None (alias 'processing:lineage')#
- field processing_software: Optional[dict] = None (alias 'processing:software')#
- pydantic model crawler.schemas.PDSSP_STAC_Properties[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Properties", "type": "object", "properties": { "title": { "title": "Title", "type": "string" }, "description": { "title": "Description", "type": "string" }, "datetime": { "title": "Datetime", "type": "string" }, "created": { "title": "Created", "type": "string" }, "updated": { "title": "Updated", "type": "string" }, "start_datetime": { "title": "Start Datetime", "type": "string" }, "end_datetime": { "title": "End Datetime", "type": "string" }, "license": { "title": "License", "type": "string" }, "platform": { "title": "Platform", "type": "string" }, "instruments": { "title": "Instruments", "type": "array", "items": { "type": "string" } }, "constellation": { "title": "Constellation", "type": "string" }, "mission": { "title": "Mission", "type": "string" }, "gsd": { "title": "Gsd", "type": "number" } }, "required": [ "datetime" ] }
- Fields
- field constellation: Optional[str] = None#
- field created: Optional[str] = None#
- field datetime: str [Required]#
- field description: Optional[str] = None#
- field end_datetime: Optional[str] = None#
- field gsd: Optional[float] = None#
- field instruments: Optional[list[str]] = None#
- field license: Optional[str] = None#
- field mission: Optional[str] = None#
- field platform: Optional[str] = None#
- field start_datetime: Optional[str] = None#
- field title: Optional[str] = None#
- field updated: Optional[str] = None#
- pydantic model crawler.schemas.PDSSP_STAC_Provider[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_Provider", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "description": { "title": "Description", "type": "string" }, "roles": { "title": "Roles", "type": "array", "items": { "type": "string" } } }, "required": [ "name" ] }
- field description: Optional[str] = None#
Multi-line description to add further provider information such as processing details for processors and producers, hosting details for hosts or basic contact information. CommonMark 0.29 syntax MAY be used for rich text representation.
- field name: str [Required]#
The name of the organization or the individual.
- field roles: Optional[list[str]] = None#
Roles of the provider. Any of licensor, producer, processor or host.
- pydantic model crawler.schemas.PDSSP_STAC_SSYS_Properties[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_SSYS_Properties", "type": "object", "properties": { "ssys:targets": { "title": "Ssys:Targets", "type": "array", "items": { "type": "string" } }, "ssys:solar_longitude": { "title": "Ssys:Solar Longitude", "type": "number" }, "ssys:solar_distance": { "title": "Ssys:Solar Distance", "type": "number" }, "ssys:incidence_angle": { "title": "Ssys:Incidence Angle", "type": "number" }, "ssys:emission_angle": { "title": "Ssys:Emission Angle", "type": "number" }, "ssys:phase_angle": { "title": "Ssys:Phase Angle", "type": "number" } } }
- Fields
- field ssys_emission_angle: Optional[float] = None (alias 'ssys:emission_angle')#
- field ssys_incidence_angle: Optional[float] = None (alias 'ssys:incidence_angle')#
- field ssys_phase_angle: Optional[float] = None (alias 'ssys:phase_angle')#
- field ssys_solar_distance: Optional[float] = None (alias 'ssys:solar_distance')#
- field ssys_solar_longitude: Optional[float] = None (alias 'ssys:solar_longitude')#
- field ssys_targets: Optional[list[str]] = None (alias 'ssys:targets')#
- pydantic model crawler.schemas.PDSSP_STAC_SpatialExtent[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_SpatialExtent", "type": "object", "properties": { "bbox": { "title": "Bbox", "type": "array", "items": { "type": "array", "items": { "type": "number" } } } }, "required": [ "bbox" ] }
- Fields
- field bbox: list[list[float]] [Required]#
Potential spatial extents covered by the Collection.
- pydantic model crawler.schemas.PDSSP_STAC_TemporalExtent[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_STAC_TemporalExtent", "type": "object", "properties": { "interval": { "title": "Interval", "type": "array", "items": { "type": "array", "items": { "type": "string" } } } }, "required": [ "interval" ] }
- Fields
- field interval: list[list[str]] [Required]#
Potential temporal extents covered by the Collection.
- pydantic model crawler.schemas.PDSSP_WFS_Feature[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_WFS_Feature", "type": "object", "properties": {} }
- pydantic model crawler.schemas.PDSSP_WFS_Layer[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "PDSSP_WFS_Layer", "type": "object", "properties": {} }
- crawler.schemas.create_schema_object(metadata: dict, name: str, object_type: str) Optional[pydantic.main.BaseModel][source]#
Create a collection or item metadata object from an input metadata dictionary and schema name.
datastore module#
PDSSP Crawler data store module.
- crawler.datastore.COLLECTIONS_JSON_TYPE = 'SourceCollections'#
JSON Source Collections file type
- class crawler.datastore.DataStore(source_data_dir='', stac_data_dir='', collections=None)[source]#
Bases:
objectDataStore class.
- add_source_collections(collections: [<class 'crawler.datastore.SourceCollectionModel'>])[source]#
Add collections to the source collections index table
- delete_source_collections(collections: [<class 'crawler.datastore.SourceCollectionModel'>])[source]#
Delete collections to the source collections index table
- get_source_collection(collection_id: str) crawler.datastore.SourceCollectionModel[source]#
Returns the source collection corresponding to input identifier.
- get_source_collections(collection_id='', service_type=None, target=None, extracted=None, transformed=None, ingested=None) [<class 'crawler.datastore.SourceCollectionModel'>][source]#
Returns source collections matching input filters.
- list_source_collections(collection_id='', service_type=None, target=None, extracted=None, transformed=None, ingested=None)[source]#
Display all, or a filtered list of source collections indexed in the data store.
- reset_source_collections(collections=None)[source]#
Reset source collections index to an empty collections or an input collections list.
- pydantic model crawler.datastore.SourceCollectionModel[source]#
Bases:
pydantic.main.BaseModelShow JSON schema
{ "title": "SourceCollectionModel", "type": "object", "properties": { "collection_id": { "title": "Collection Id", "type": "string" }, "service": { "title": "Service", "anyOf": [ { "$ref": "#/definitions/Service" }, { "$ref": "#/definitions/ExternalService" } ] }, "source_schema": { "title": "Source Schema", "type": "string" }, "target": { "title": "Target", "type": "string" }, "stac_extensions": { "title": "Stac Extensions", "type": "array", "items": { "type": "string" } }, "n_products": { "title": "N Products", "type": "integer" }, "extracted": { "title": "Extracted", "default": false, "type": "boolean" }, "extracted_files": { "title": "Extracted Files", "default": [], "type": "array", "items": {} }, "transformed": { "title": "Transformed", "default": false, "type": "boolean" }, "stac_dir": { "title": "Stac Dir", "default": "", "type": "string" }, "ingested": { "title": "Ingested", "default": false, "type": "boolean" }, "stac_url": { "title": "Stac Url", "default": "", "type": "string" } }, "required": [ "collection_id" ], "definitions": { "ProviderRole": { "title": "ProviderRole", "description": "An enumeration.", "enum": [ "producer", "licensor", "processor", "host" ] }, "ServiceProvider": { "title": "ServiceProvider", "type": "object", "properties": { "name": { "title": "Organization name", "type": "string" }, "description": { "title": "Organization description", "type": "string" }, "roles": { "title": "Organization roles", "type": "array", "items": { "$ref": "#/definitions/ProviderRole" } }, "url": { "title": "Organization homepage", "type": "string" } }, "required": [ "name" ] }, "ServiceType": { "title": "ServiceType", "description": "An enumeration.", "enum": [ "STAC", "WFS", "WMS", "WMTS", "XYZ" ] }, "Service": { "title": "Service", "type": "object", "properties": { "title": { "title": "Title", "type": "string" }, "description": { "title": "Description", "type": "string" }, "providers": { "title": "Providers", "type": "array", "items": { "$ref": "#/definitions/ServiceProvider" } }, "type": { "$ref": "#/definitions/ServiceType" }, "url": { "title": "Url", "type": "string" }, "ping_url": { "title": "Ping Url", "type": "string" }, "ssys:targets": { "title": "Ssys:Targets", "type": "array", "items": { "type": "string" } } }, "required": [ "title", "description", "providers", "type", "url" ] }, "ExternalServiceType": { "title": "ExternalServiceType", "description": "An enumeration.", "enum": [ "WFS", "PDSODE", "EPNTAP" ] }, "ExternalService": { "title": "ExternalService", "type": "object", "properties": { "title": { "title": "Title", "type": "string" }, "description": { "title": "Description", "type": "string" }, "providers": { "title": "Providers", "type": "array", "items": { "$ref": "#/definitions/ServiceProvider" } }, "type": { "$ref": "#/definitions/ExternalServiceType" }, "url": { "title": "Url", "type": "string" }, "ping_url": { "title": "Ping Url", "type": "string" }, "ssys:targets": { "title": "Ssys:Targets", "type": "array", "items": { "type": "string" } }, "extra_params": { "title": "Extra Params", "type": "object" } }, "required": [ "title", "description", "providers", "type", "url" ] } } }
- field collection_id: str [Required]#
- field extracted: Optional[bool] = False#
- field extracted_files: Optional[list] = []#
- field ingested: Optional[bool] = False#
- field n_products: Optional[int] = None#
- field service: Optional[Union[crawler.registry.Service, crawler.registry.ExternalService]] = None#
- field source_schema: Optional[str] = None#
- field stac_dir: Optional[str] = ''#
- field stac_extensions: Optional[list[str]] = None#
- field stac_url: Optional[str] = ''#
- field target: Optional[str] = None#
- field transformed: Optional[bool] = False#
extractor module#
PDSSP Crawler extractor module (extract and read).
To add an Extractor handling a new service type:
class NEW_Extractor(AbstractExtractor):
def __init__(self, service=None):
super().__init__(service=service)
pass
EXTRACTORS = {
ExternalServiceType.EPNTAP: EPNTAP_Extractor,
ExternalServiceType.PDSODE: PDSODE_Extractor,
ExternalServiceType.WFS: WFS_Extractor,
ExternalServiceType.NEW: NEW_Extractor
}
- class crawler.extractor.AbstractExtractor(collection=None, service=None)[source]#
Bases:
objectAbstract Extractor class.
- set_collection(collection: crawler.datastore.SourceCollectionModel)[source]#
Set extracted source collection from input SourceCollectionModel object and derive Extractor properties from it.
- class crawler.extractor.EPNTAP_Extractor(collection=None, service=None)[source]#
Bases:
crawler.extractor.AbstractExtractorEPNTAP_Extractor class.
- crawler.extractor.EXTRACTORS = {<ExternalServiceType.EPNTAP: 'EPNTAP'>: <class 'crawler.extractor.EPNTAP_Extractor'>, <ExternalServiceType.PDSODE: 'PDSODE'>: <class 'crawler.extractor.PDSODE_Extractor'>, <ExternalServiceType.WFS: 'WFS'>: <class 'crawler.extractor.WFS_Extractor'>}#
List of available extractors and their corresponding ServiceType.
- crawler.extractor.Extractor(collection=None, service_type='', service=None)[source]#
Extractor function serving as Extractor objects factory.
- class crawler.extractor.PDSODE_Extractor(collection=None, service=None)[source]#
Bases:
crawler.extractor.AbstractExtractorPDSODE_Extractor class.
- extract(collection_id, output_dir_path='', service=None, overwrite=False)[source]#
Extract source collection files required to retrieve collection and product metadata.
- get_collection_metadata(collection_id, service=None) crawler.datastore.SourceCollectionModel[source]#
- class crawler.extractor.WFS_Extractor(collection=None, service=None)[source]#
Bases:
crawler.extractor.AbstractExtractorWFS_Extractor class.
transformer module#
Transformer module.
- class crawler.transformer.AbstractTransformer(collection=None, source_schema=None, destination_schema='PDSSP_STAC')[source]#
Bases:
object- get_assets(source_metadata: pydantic.main.BaseModel, object_type='item') Dict[str, crawler.schemas.PDSSP_STAC_Asset][source]#
- get_extension_fields(source_metadata: pydantic.main.BaseModel, stac_extension, object_type='item') pydantic.main.BaseModel[source]#
- get_extension_properties(source_metadata: pydantic.main.BaseModel, stac_extension, object_type='item') pydantic.main.BaseModel[source]#
- get_links(source_metadata: pydantic.main.BaseModel, object_type='item') list[pydantic.main.BaseModel][source]#
- get_processing_properties(source_metadata: pydantic.main.BaseModel, object_type='item') pydantic.main.BaseModel[source]#
- get_proj_properties(source_metadata: pydantic.main.BaseModel, object_type='item') pydantic.main.BaseModel[source]#
- get_resto_keywords(source_metadata: pydantic.main.BaseModel) dict[source]#
Returns RESTO-specific STAC Items keywords.
- get_ssys_properties(source_metadata: pydantic.main.BaseModel, object_type='item') pydantic.main.BaseModel[source]#
- set_source_collection(collection: crawler.datastore.SourceCollectionModel)[source]#
Set transformed source collection from input SourceCollectionModel object and derive Transfornmer properties from it.
- transform(source_collection_file_path='', output_dir_path='', stac_extensions=[], overwrite=False) None[source]#
Transform (extracted) source collection files into PDSSP STAC catalog.
Destination STAC catalog may contain one or several collections, related to only one reference target.
- transform_source_metadata(source_metadata: pydantic.main.BaseModel, object_type='item', stac_extensions=[]) Union[crawler.schemas.PDSSP_STAC_Item, crawler.schemas.PDSSP_STAC_Collection][source]#
Transform input source metadata into output PDSSP STAC metadata schema object.
- class crawler.transformer.EPNTAP_STAC(collection=None, source_schema=None, destination_schema='PDSSP_STAC')[source]#
Bases:
crawler.transformer.AbstractTransformer- get_assets(source_metadata: pydantic.main.BaseModel, object_type='item') list[pydantic.main.BaseModel][source]#
- exception crawler.transformer.InvalidModelObjectTypeError[source]#
Bases:
ExceptionCustom error that is raised when invalid STAC object type is passed.
- class crawler.transformer.MARSSI_STAC(collection=None, source_schema=None, destination_schema='PDSSP_STAC')[source]#
Bases:
crawler.transformer.AbstractTransformer- get_assets(source_metadata: pydantic.main.BaseModel, object_type='item') list[pydantic.main.BaseModel][source]#
- get_links(source_metadata: pydantic.main.BaseModel, object_type='item') list[pydantic.main.BaseModel][source]#
- class crawler.transformer.PDSODE_STAC(collection=None, source_schema=None, destination_schema='PDSSP_STAC')[source]#
Bases:
crawler.transformer.AbstractTransformer- get_assets(source_metadata: pydantic.main.BaseModel, object_type='item') Dict[str, crawler.schemas.PDSSP_STAC_Asset][source]#
- get_extent(source_metadata: pydantic.main.BaseModel) crawler.schemas.PDSSP_STAC_Extent[source]#
- get_processing_properties(source_metadata: pydantic.main.BaseModel, object_type='item') pydantic.main.BaseModel[source]#
- crawler.transformer.Transformer(collection: Optional[crawler.datastore.SourceCollectionModel] = None, source_schema=None, destination_schema='PDSSP_STAC')[source]#
Transformer function serving as Transformer objects factory.
ingestor module#
- crawler.ingestor.INGEST_STRATEGIES = {'both': {'ingest_catalog': True, 'ingest_feature': True}, 'catalog': {'ingest_catalog': True, 'ingest_feature': False}, 'feature': {'ingest_catalog': True, 'ingest_feature': False}, 'none': {'ingest_catalog': False, 'ingest_feature': False}}#
Ingest strategies define what is ingested i.e. “collection”, “feature”, “both” or “none”.
- class crawler.ingestor.Ingestor(stac_api_parent_url='', auth_token='', source_collection=None)[source]#
Bases:
objectIngestion of STAC catalog or collection into destination STAC API service (eg: PDSSP RESTO).
- ingest(stac_file='', stac_api_url='', update_if_exists=False, ingest_strategy='catalog')[source]#
Ingest input STAC file into destination STAC API catalog service.
Input STAC file can be a catalog, a collection or an item JSON file. Different ingestion strategies define what is ingested i.e. ‘catalog’, ‘feature’, ‘both’ or ‘none’.
~ stac2resto (“process_stuff(url, lookup_table”))