API and Code¶
Below is the API documention for the the Stetl Python code.
Main Entry Points¶
There are several entry points through which Stetl can be called. The most common is to use the commandline script bin/stetl. This command should be available after doing an install.
In some contexts like integrations you may want to call Stetl via Python. The entries are then.
-
stetl.main.
main
()[source]¶ The main function, to be called from commandline, like python src/main.py -c etl.cfg.
- Args:
-c --config <config_file> the Stetl config file. -s –section <section_name> the section in the Stetl config (ini) file to execute (default is [etl]). -a –args <arglist> sero or more substitutable args for symbolic, {arg}, values in Stetl config file, in format -a arg1=foo -a arg2=bar etc. -d –doc <class> Get component documentation like its configuration parameters, e.g. stetl –doc stetl.inputs.fileinput.FileInput -h –help get help info
-
stetl.main.
print_doc
(class_name)[source]¶ Print documentation for class in particular config options
Core Framework¶
The core framework is directly under the directory src/stetl. Below are the main seven classes. Their interrelation is as follows:
One or more stetl.chain.Chain
objects are built from
a Stetl ETL configuration via the stetl.factory.Factory
class.
A stetl.chain.Chain
consists of a set of connected stetl.component.Component
objects.
A stetl.component.Component
is either an stetl.input.Input
, an stetl.output.Output
or a stetl.filter.Filter
. Data and status flows as stetl.packet.Packet
objects
from an stetl.input.Input
via zero or more stetl.filter.Filter
objects to a final stetl.output.Output
.
As a trivial example: an stetl.input.Input
could be an XML file, a stetl.filter.Filter
could represent
an XSLT file and an stetl.output.Output
a PostGIS database. This is effected by specialized classes in
the subpackages inputs, filters, and outputs. New in 1.1.0: stetl.Splitter
to split data to multiple Outputs
and stetl.Merger
to combine multiple Inputs.
-
class
stetl.factory.
Factory
[source]¶ Object and class Factory (Pattern). Based on: http://stackoverflow.com/questions/2226330/instantiate-a-python-class-from-a-name
-
class
stetl.component.
Component
(configdict, section, consumes='none', produces='none')[source]¶ Abstract Base class for all Input, Filter and Output Components.
-
input_format
()[source]¶ CONFIG
- The specific input format if the consumes parameter is a list or the format to be converted to the output_format.
- type: str
- required: False
- default: None
-
invoke
(packet)[source]¶ Components override for Component-specific behaviour, typically read, filter or write actions.
-
-
class
stetl.component.
Config
(ptype=<type 'str'>, default=None, required=False)[source]¶ Decorator class to tie config values from the .ini file to object instance property values. Somewhat like the Python standard @property but with the possibility to define default values, typing and making properties required.
Each property is defined by @Config(type, default, required). Basic idea comes from: https://wiki.python.org/moin/PythonDecoratorLibrary#Cached_Properties
-
class
stetl.chain.
Chain
(chain_str, config_dict)[source]¶ Holder for single invokable pipeline of components A Chain is basically a singly linked list of Components Each Component executes a part of the total ETL. Data along the Chain is passed within a Packet object. The compatibility of input and output for linked Components is checked when adding a Component to the Chain.
-
get_by_class
(clazz)[source]¶ Get Component instance from Chain by class, mainly for testing. :param clazz: :return Component:
-
get_by_id
(id)[source]¶ Get Component instance from Chain, mainly for testing. :param name: :return Component:
-
-
class
stetl.packet.
FORMAT
[source]¶ Format of Packet (enumeration).
Current possible values:
- ‘none’
- ‘xml_line_stream’
- ‘line_stream’
- ‘etree_doc’
- ‘etree_element’
- ‘etree_feature_array’
- ‘xml_doc_as_string’
- ‘string’
- ‘record’
- ‘record_array’
- ‘struct’
- ‘geojson_feature’
- ‘geojson_collection’
- ‘ogr_feature’
- ‘ogr_feature_array’
- ‘any’
-
class
stetl.packet.
Packet
(data=None)[source]¶ Represents units of (any) data and status passed along Chain of Components.
-
class
stetl.input.
Input
(configdict, section, produces)[source]¶ Bases:
stetl.component.Component
Abstract Base class for all Input Components.
-
class
stetl.output.
Output
(configdict, section, consumes)[source]¶ Bases:
stetl.component.Component
Abstract Base class for all Output Components.
-
class
stetl.filter.
Filter
(configdict, section, consumes, produces)[source]¶ Bases:
stetl.component.Component
Maps input to output. Abstract base class for specific Filters.
-
class
stetl.splitter.
Splitter
(config_dict, child_list)[source]¶ Bases:
stetl.component.Component
Component that splits a single input to multiple output Components. Use this for example to produce multiple output file formats (GML, GeoJSON etc) or to publish to multiple remote services (SOS, SensorThings API) or for simple debugging: target Output and StandardOutput.
-
class
stetl.merger.
Merger
(config_dict, child_list)[source]¶ Bases:
stetl.component.Component
Component that merges multiple Input Components into a single Component. Use this for example to combine multiple input streams like API endpoints. The Merger will embed Child Components to which actions are delegated. A Child Component may be a sub-Chain e.g. (Input|Filter|Filter..) sequence. Hence the “next” should be coupled to the last Component in that sub-Chain with the degenerate case where the sub-Chain is a single (Input) Component. NB this Component can only be used for Inputs.
Components: Inputs¶
-
class
stetl.inputs.dbinput.
DbInput
(configdict, section, produces)[source]¶ Bases:
stetl.input.Input
Input from any database (abstract base class).
-
class
stetl.inputs.dbinput.
PostgresDbInput
(configdict, section)[source]¶ Bases:
stetl.inputs.dbinput.SqlDbInput
Input by querying records from a Postgres database. Input is a query, like SELECT * from mytable. Output is zero or more records as record array (array of dict) or single record (dict).
produces=FORMAT.record_array (default) or FORMAT.record
-
host
()[source]¶ CONFIG
- host name or host IP-address, defaults to ‘localhost’
- type: str
- required: False
- default: localhost
-
password
()[source]¶ CONFIG
- User password, defaults to ‘postgres’
- type: str
- required: False
- default: postgres
-
-
class
stetl.inputs.dbinput.
SqlDbInput
(configdict, section)[source]¶ Bases:
stetl.inputs.dbinput.DbInput
Input using a query from any SQL-based RDBMS (abstract base class).
-
column_names
()[source]¶ CONFIG
- Column names to populate records with. If empty taken from table metadata.
- type: str
- required: False
- default: None
-
read_once
()[source]¶ CONFIG
- Read once? i.e. only do query once and stop
- type: bool
- required: False
- default: False
-
-
class
stetl.inputs.dbinput.
SqliteDbInput
(configdict, section)[source]¶ Bases:
stetl.inputs.dbinput.SqlDbInput
Input by querying records from a SQLite database. Input is a query, like SELECT * from mytable. Output is zero or more records as record array (array of dict) or single record (dict).
produces=FORMAT.record_array (default) or FORMAT.record
-
class
stetl.inputs.fileinput.
ApacheLogFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Parses Apache log files. Lines are converted into records based on the log format. Log format should follow Apache Log Format. See ApacheLogParser for details.
produces=FORMAT.record
-
class
stetl.inputs.fileinput.
CsvFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Parse CSV file into stream of records (dict structures) or a one-time record array. NB raw version: CSV needs to have first line with fieldnames.
produces=FORMAT.record or FORMAT.record_array
-
class
stetl.inputs.fileinput.
FileInput
(configdict, section, produces)[source]¶ Bases:
stetl.input.Input
Abstract base class for specific FileInputs, use derived classes.
-
depth_search
()[source]¶ CONFIG
- Should we recurse into sub-directories to find files?
- type: bool
- required: False
- default: False
-
file_path
()[source]¶ CONFIG
- Path to file or files or URLs: can be a dir or files or URLs or even multiple, comma separated. For URLs only JSON is supported now.
- type: str
- required: False
- default: None
-
-
class
stetl.inputs.fileinput.
GlobFileInput
(configdict, section, produces=['string', 'line_stream'])[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Returns file names based on the glob.glob pattern given as filename_filter.
produces=FORMAT.string or FORMAT.line_stream
-
class
stetl.inputs.fileinput.
JsonFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Parse JSON file from file system or URL into hierarchical data struct. The struct format may also be a GeoJSON structure. In that case the output_format needs to be explicitly set to geojson_collection in the component config.
produces=FORMAT.struct or FORMAT.geojson_collection
-
class
stetl.inputs.fileinput.
LineStreamerFileInput
(configdict, section, produces='line_stream')[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Reads text-files, producing a stream of lines, one line per Packet. NB assumed is that lines in the file have newlines !!
-
class
stetl.inputs.fileinput.
StringFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Reads and produces file as String.
produces=FORMAT.string
-
class
stetl.inputs.fileinput.
XmlElementStreamerFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Extracts XML elements from a file, outputs each feature element in Packet. Parsing is streaming (no internal DOM buildup) so any file size can be handled. Use this class for your big GML files!
produces=FORMAT.etree_element
CONFIG
- Comma-separated string of XML (feature) element tag names of the elements that should be extracted and added to the output element stream.
- type: list
- required: True
- default: None
-
class
stetl.inputs.fileinput.
XmlFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Parses XML files into etree docs (do not use for large files!).
produces=FORMAT.etree_doc
-
class
stetl.inputs.fileinput.
XmlLineStreamerFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.LineStreamerFileInput
DEPRECATED Streams lines from an XML file(s) NB assumed is that lines in the file have newlines !! DEPRECATED better is to use XmlElementStreamerFileInput for GML features.
produces=FORMAT.xml_line_stream
-
class
stetl.inputs.fileinput.
ZipFileInput
(configdict, section)[source]¶ Bases:
stetl.inputs.fileinput.FileInput
Parse ZIP file from file system or URL into a stream of records containing zipfile-path and file names.
produces=FORMAT.record
-
class
stetl.inputs.httpinput.
ApacheDirInput
(configdict, section, produces='record')[source]¶ Bases:
stetl.inputs.httpinput.HttpInput
Read file data from an Apache directory “index” HTML page. Uses http://stackoverflow.com/questions/686147/url-tree-walker-in-python produces=FORMAT.record. Each record contains file_name and file_data (other meta data like date time is too fragile over different Apache servers).
-
file_ext
()[source]¶ CONFIG
- The file extension for target files in Apache dir.
- type: str
- required: False
- default: xml
-
-
class
stetl.inputs.httpinput.
HttpInput
(configdict, section, produces='any')[source]¶ Bases:
stetl.input.Input
Fetch data from remote services like WFS via HTTP protocol. Base class: subclasses will do datatype-specific formatting of the returned data.
produces=FORMAT.any
Add authorization from config data. Authorization scheme-specific. May be extended or overloaded for additional schemes.
Parameters: request – the HTTP Request Returns:
-
auth
()[source]¶ CONFIG
Authentication data: Flat JSON-like struct dependent on auth type/schema. Only the type field is required, other fields depend on auth schema. Supported values :
type: basic|token
If the type is
basic
(HTTP Basic Authentication) two additional fieldsuser
andpassword
are required. If the type istoken
(HTTP Token) additional two additional fieldskeyword
andtoken
are required.Any required Base64 encoding is provided by
HttpInput
.Examples:
# Basic Auth url = https://some.rest.api.com auth = { type: basic, user: myname password: mypassword } # Token Auth url = https://some.rest.api.com auth = { type: token, keyword: Bearer token: mytoken }
- type: dict
- required: False
- default: None
-
format_data
(data)[source]¶ Format response data, override in subclasses, defaults to returning original data. :param packet: :return:
-
parameters
()[source]¶ CONFIG
Flat JSON-like struct of the parameters to be appended to the url.
Example: (parameters require quotes):
url = http://geodata.nationaalgeoregister.nl/natura2000/wfs parameters = { service : WFS, version : 1.1.0, request : GetFeature, srsName : EPSG:28992, outputFormat : text/xml; subtype=gml/2.1.2, typename : natura2000 }
- type: dict
- required: False
- default: None
-
class
stetl.inputs.ogrinput.
OgrInput
(configdict, section)[source]¶ Bases:
stetl.input.Input
Direct GDAL OGR input via Python OGR wrapper. Via the Python API http://gdal.org/python an OGR data source is accessed and from each layer the Features are read. Each Layer corresponds to a “doc”, so for multi-layer sources the ‘end-of-doc’ flag is set after a Layer has been read.
This input can read almost any geospatial dataformat. One can use the features directly in a Stetl Filter or use a converter to e.g. convert to GeoJSON structures.
produces=FORMAT.ogr_feature or FORMAT.ogr_feature_array (all features)
-
data_source
()[source]¶ CONFIG
- String denoting the OGR datasource. Usually a path to a file like “path/rivers.shp” or connection string to PostgreSQL like “PG: host=localhost dbname=’rivers’ user=’postgres’”.
- type: str
- required: True
- default: None
-
source_format
()[source]¶ CONFIG
Instructs GDAL to use driver by that name to open datasource. Not required for many standard formats that are self-describing like ESRI Shapefile.
Examples: ‘PostgreSQL’, ‘GeoJSON’ etc
- type: str
- required: False
- default: None
-
-
class
stetl.inputs.ogrinput.
OgrPostgisInput
(configdict, section)[source]¶ Bases:
stetl.input.Input
Input from PostGIS via ogr2ogr command. For now hardcoded to produce an ogr GML line stream. OgrInput may be a better alternative.
Alternatives: either stetl.input.PostgresqlInput or stetl.input.OgrInput.
produces=FORMAT.xml_line_stream
-
in_pg_sql
()[source]¶ CONFIG
- The input query (string) to fire.
- type: str
- required: False
- default: None
-
in_srs
()[source]¶ CONFIG
- SRS (projection) (ogr2ogr -s_srs) input DB e.g. ‘EPSG:28992’.
- type: str
- required: False
- default: None
-
out_dimension
()[source]¶ CONFIG
- Dimension (OGR: DIM=N) of features in output stream.
- type: str
- required: False
- default: 2
-
out_geotype
()[source]¶ CONFIG
- OGR Geometry type new layer in output stream, e.g. POINT.
- type: str
- required: False
- default: None
-
out_gml_format
()[source]¶ CONFIG
- GML format OGR name in output stream, e.g. ‘GML3’.
- type: str
- required: False
- default: None
-
-
class
stetl.inputs.deegreeinput.
DeegreeBlobstoreInput
(configdict, section)[source]¶ Bases:
stetl.input.Input
Read features from deegree Blobstore DB into an etree doc.
produces=FORMAT.etree_doc
-
end_feature_tag
()[source]¶ CONFIG
- XML tag that ends Feature.
- type: str
- required: False
- default: None
-
Components: Filters¶
-
class
stetl.filters.xsltfilter.
XsltFilter
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Invokes XSLT processor (via lxml) for given XSLT script on an etree doc.
consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc
-
class
stetl.filters.xmlassembler.
XmlAssembler
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Split a stream of etree DOM XML elements (usually Features) into etree DOM docs. Consumes and buffers elements until max_elements reached, will then produce an etree doc.
consumes=FORMAT.etree_element, produces=FORMAT.etree_doc
-
class
stetl.filters.xmlelementreader.
XmlElementReader
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Extracts XML elements from a file, outputs each feature element in Packet. Parsing is streaming (no internal DOM buildup) so any file size can be handled. Use this class for your big GML files!
consumes=FORMAT.string, produces=FORMAT.etree_element
CONFIG
- Comma-separated string of XML (feature) element tag names of the elements that should be extracted and added to the output element stream.
- type: list
- required: True
- default: None
-
class
stetl.filters.xmlvalidator.
XmlSchemaValidator
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Validates an etree doc and prints result to log.
consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc
-
class
stetl.filters.sieve.
AttrValueRecordSieve
(configdict, section)[source]¶ Bases:
stetl.filters.sieve.Sieve
Sieves by attr/value(s) in Record Packets.
-
attr_name
()[source]¶ CONFIG
- Name of attribute whose value(s) are to be sieved.
- type: str
- required: True
- default: None
-
-
class
stetl.filters.sieve.
Sieve
(configdict, section, consumes, produces)[source]¶ Bases:
stetl.filter.Filter
ABC for specific Sieves that pass-through, “sieve”, Packets based on criteria in their data.
-
class
stetl.filters.stringfilter.
StringFilter
(configdict, section, consumes, produces)[source]¶ Bases:
stetl.filter.Filter
Base class for any string filtering
-
class
stetl.filters.stringfilter.
StringSubstitutionFilter
(configdict, section)[source]¶ Bases:
stetl.filters.stringfilter.StringFilter
String filtering using Python advanced String formatting. String should have substitutable values like {schema} {foo} format_args should be of the form format_args = schema:test foo:bar …
consumes=FORMAT.string, produces=FORMAT.string
-
format_args
()[source]¶ CONFIG
Provides a list of format arguments used by the string substitution filter. Formatting of content according to Python String.format(). String should have substitutable values like {schema} {foo}.
Example: format_args = schema:test foo:bar
- type: str
- required: True
- default: None
-
-
class
stetl.filters.templatingfilter.
Jinja2TemplatingFilter
(configdict, section)[source]¶ Bases:
stetl.filters.templatingfilter.TemplatingFilter
Implements Templating using Jinja2. Jinja2 http://jinja.pocoo.org, is a modern and designer-friendly templating language for Python modelled after Django’s templates. A ‘struct’ format as input provides a tree-like structure that could originate from a JSON file or REST service. This input struct provides all the variables to be inserted into the template. The template itself can be configured in this component as a Jinja2 string or -file. An optional ‘template_search_paths’ provides a list of directories from which templates can be fethced. Default is the current working directory. Via the optional ‘globals_path’ a JSON structure can be inserted into the Template environment. The variables in this globals struture are typically “boilerplate” constants like: id-prefixes, point of contacts etc.
consumes=FORMAT.struct, produces=FORMAT.string
-
add_env_filters
(jinja2_env)[source]¶ Register additional Filters on the template environment by updating the filters dict: Somehow min and max of list are not present so add them as well.
-
static
geojson2gml_filter
(value, source_crs=4326, target_crs=None, gml_id=None, gml_format='GML2', gml_longsrs='NO')[source]¶ Jinja2 custom Filter: generates any GML geometry from a GeoJSON geometry. By specifying a target_crs we can even reproject from the source CRS. The gml_format=GML2|GML3 determines the general GML form: e.g. pos/posList or coordinates. gml_longsrs=YES|NO determines the srsName format like EPSG:4326 or urn:ogc:def:crs:EPSG::4326 (long).
-
-
class
stetl.filters.templatingfilter.
StringTemplatingFilter
(configdict, section)[source]¶ Bases:
stetl.filters.templatingfilter.TemplatingFilter
Implements Templating using Python’s internal string.Template. A template string or file should be configured. The input record contains the actual values to be substituted in the template string as a record (key/value pairs). Output is a regular string.
consumes=FORMAT.record or FORMAT.record_array, produces=FORMAT.string
-
safe_substitution
()[source]¶ CONFIG
- Apply safe substitution? With this method, string.Template.safe_substitute will be invoked, instead of string.Template.substitute. If placeholders are missing from mapping and keywords, instead of raising an exception, the original placeholder will appear in the resulting string intact.
- type: bool
- required: False
- default: False
-
-
class
stetl.filters.templatingfilter.
TemplatingFilter
(configdict, section, consumes='any', produces='string')[source]¶ Bases:
stetl.filter.Filter
Abstract base class for specific template-based filters. See https://wiki.python.org/moin/Templating Subclasses implement a specific template language like Python string.Template, Mako, Genshi, Jinja2,
consumes=FORMAT.any, produces=FORMAT.string
-
invoke
(packet)[source]¶ Components override for Component-specific behaviour, typically read, filter or write actions.
-
-
class
stetl.filters.gmlfeatureextractor.
GmlFeatureExtractor
(configdict, section='gml_feature_extractor')[source]¶ Bases:
stetl.filter.Filter
Extract arrays of GML features etree elements from etree docs.
consumes=FORMAT.etree_doc, produces=FORMAT.etree_feature_array
-
class
stetl.filters.gmlsplitter.
GmlSplitter
(configdict, section='gml_splitter')[source]¶ Bases:
stetl.filter.Filter
Split a stream of text XML lines into documents DEPRECATED: use the more robust XmlElementStreamerFileInput+XmlAssembler instead!!! TODO phase out
consumes=FORMAT.xml_line_stream, produces=FORMAT.etree_doc
-
class
stetl.filters.formatconverter.
FormatConverter
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Converts (almost) any packet format (if converter available).
consumes=FORMAT.any, produces=FORMAT.any but actual formats are changed at initialization based on the input to output format to be converted via the input_format and output_format config parameters.
-
converter_args
()[source]¶ CONFIG
- Custom converter-specific arguments.
- type: dict
- required: False
- default: None
-
static
etree_doc2geojson_collection
(packet, converter_args=None)[source]¶ Use converter_args to determine XML tag names for features and GeoJSON feature id. For example
- converter_args = {
- ‘root_tag’: ‘FeatureCollection’, ‘feature_tag’: ‘featureMember’, ‘feature_id_attr’: ‘fid’ }
Parameters: - packet –
- converter_args –
Returns:
-
static
etree_doc2struct
(packet, strip_space=True, strip_ns=True, sub=False, attr_prefix='', gml2ogr=True, ogr2json=True)[source]¶ Parameters: - packet –
- strip_space –
- strip_ns –
- sub –
- attr_prefix –
- gml2ogr –
- ogr2json –
Returns:
-
-
class
stetl.filters.execfilter.
CommandExecFilter
(configdict, section)[source]¶ Bases:
stetl.filters.execfilter.ExecFilter
Executes an arbitrary command and captures the output
consumes=FORMAT.string, produces=FORMAT.string
-
class
stetl.filters.execfilter.
ExecFilter
(configdict, section, consumes, produces)[source]¶ Bases:
stetl.filter.Filter
Executes any command (abstract base class).
-
env_args
()[source]¶ CONFIG
Provides of list of environment variables which will be used when executing the given command.
Example: env_args = pgpassword=postgres othersetting=value~with~spaces
- type: str
- required: False
- default:
-
-
class
stetl.filters.nullfilter.
NullFilter
(configdict, section, consumes='any', produces='any')[source]¶ Bases:
stetl.filter.Filter
Pass-through Filter, does nothing. Mainly used in Test Cases.
-
class
stetl.filters.packetbuffer.
PacketBuffer
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Buffers all incoming Packets, main use is unit-testing to inspect Packets after ETL is done.
-
class
stetl.filters.packetwriter.
PacketWriter
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Writes the payload of a packet as a string to a file.
consumes=FORMAT.any, produces=FORMAT.string
-
class
stetl.filters.regexfilter.
RegexFilter
(configdict, section, consumes='string', produces='record')[source]¶ Bases:
stetl.filter.Filter
Extracts data from a string using a regular expression and returns the named groups as a record. consumes=FORMAT.string, produces=FORMAT.record
-
class
stetl.filters.zipfileextractor.
ZipFileExtractor
(configdict, section)[source]¶ Bases:
stetl.filter.Filter
Extracts a file from a ZIP file, and saves it as the given file name.
consumes=FORMAT.record, produces=FORMAT.string
-
delete_file
()[source]¶ CONFIG
- Delete the file when the chain has been completed?
- type: bool
- required: False
- default: True
-
Components: Outputs¶
-
class
stetl.outputs.fileoutput.
FileOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Pretty print input to file. Input may be an etree doc or any other stringify-able input.
consumes=FORMAT.any
-
class
stetl.outputs.fileoutput.
MultiFileOutput
(configdict, section)[source]¶ Bases:
stetl.outputs.fileoutput.FileOutput
Print to multiple files from subsequent packets like strings or etree docs, file_path must be of a form like: gmlcities-%03d.gml.
consumes=FORMAT.any
-
class
stetl.outputs.standardoutput.
StandardOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Print any input to standard output.
consumes=FORMAT.any
-
class
stetl.outputs.standardoutput.
StandardXmlOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Pretty print XML from etree doc to standard output. OBSOLETE, can be done with StandardOutput
consumes=FORMAT.etree_doc
-
class
stetl.outputs.httpoutput.
HttpOutput
(configdict, section, consumes='any')[source]¶ Bases:
stetl.output.Output
Output via HTTP protocol, usually via POST.
consumes=FORMAT.any
-
content_type
()[source]¶ CONFIG
- The HTTP ContentType request header for target request.
- type: str
- required: False
- default: text/xml
-
create_payload
(packet)[source]¶ Create a HTTP body payload like for POST of an XML or JSON message. Subclasses like WFS and SOS override. :param packet: :return payload as string:
-
host
()[source]¶ CONFIG
- The hostname/IP addr for target request.
- type: str
- required: True
- default: None
-
list_fanout
()[source]¶ CONFIG
- If we consume a list(), should we create a HTTP req for each member?
- type: bool
- required: False
- default: True
-
method
()[source]¶ CONFIG
- The HTTP method for target request.
- type: str
- required: False
- default: POST
-
-
class
stetl.outputs.ogroutput.
Ogr2OgrOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Output from GML etree doc to any OGR2OGR output using the GDAL/OGR ogr2ogr command
consumes=FORMAT.etree_doc
-
class
stetl.outputs.ogroutput.
OgrOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Direct GDAL OGR output via Python OGR wrapper. Via the Python API http://gdal.org/python OGR Features are written.
This output can write almost any geospatial, OGR-defined, dataformat.
consumes=FORMAT.ogr_feature or FORMAT.ogr_feature_array
-
append
()[source]¶ CONFIG
- Add to destination destination if it extists (ogr2ogr -append option).
- type: bool
- required: False
- default: False
-
dest_create_options
()[source]¶ CONFIG
Creation options.
Examples: ..
- type: list
- required: False
- default: []
-
dest_data_source
()[source]¶ CONFIG
- String denoting the OGR data destination. Usually a path to a file like “path/rivers.shp” or connection string to PostgreSQL like “PG: host=localhost dbname=’rivers’ user=’postgres’”.
- type: str
- required: True
- default: None
-
dest_format
()[source]¶ CONFIG
Instructs GDAL to use driver by that name to open data destination. Not required for many standard formats that are self-describing like ESRI Shapefile.
Examples: ‘PostgreSQL’, ‘GeoJSON’ etc
- type: str
- required: False
- default: None
-
dest_options
()[source]¶ CONFIG
- Custom data destination-specific options. Used in gdal.SetConfigOption().
- type: dict
- required: False
- default: None
-
layer_create_options
()[source]¶ CONFIG
- Options for newly created layer (-lco).
- type: list
- required: False
- default: []
-
new_layer_name
()[source]¶ CONFIG
- Layer name for layer created in the destination source.
- type: str
- required: True
- default: None
-
overwrite
()[source]¶ CONFIG
- Overwrite destination if it extists (ogr2ogr -overwrite option).
- type: bool
- required: False
- default: False
-
-
class
stetl.outputs.execoutput.
CommandExecOutput
(configdict, section)[source]¶ Bases:
stetl.outputs.execoutput.ExecOutput
Executes an arbitrary command.
consumes=FORMAT.string
-
class
stetl.outputs.execoutput.
ExecOutput
(configdict, section, consumes)[source]¶ Bases:
stetl.output.Output
Executes any command (abstract base class).
-
class
stetl.outputs.execoutput.
Ogr2OgrExecOutput
(configdict, section)[source]¶ Bases:
stetl.outputs.execoutput.ExecOutput
Executes an Ogr2Ogr command. Input is a file name to be processed. Output by calling Ogr2Ogr command.
consumes=FORMAT.string
-
cleanup_input
()[source]¶ CONFIG
- Flag to indicate whether the input file to ogr2ogr should be cleaned up.
- type: bool
- required: False
- default: False
-
dest_data_source
()[source]¶ CONFIG
- String denoting the OGR data destination. Usually a path to a file like “path/rivers.shp” or connection string to PostgreSQL like “PG: host=localhost dbname=’rivers’ user=’postgres’”.
- type: str
- required: True
- default: None
-
dest_format
()[source]¶ CONFIG
Instructs GDAL to use driver by that name to open data destination. Not required for many standard formats that are self-describing like ESRI Shapefile.
Examples: ‘PostgreSQL’, ‘GeoJSON’ etc
- type: str
- required: False
- default: None
-
gfs_template
()[source]¶ CONFIG
- Name of GFS template file to use during loading. Passed to ogr2ogr as –config GML_GFS_TEMPLATE <name>
- type: str
- required: False
- default: None
-
lco
()[source]¶ CONFIG
- Options for newly created layer (-lco).
- type: str
- required: False
- default: None
-
-
class
stetl.outputs.dboutput.
DbOutput
(configdict, section, consumes)[source]¶ Bases:
stetl.output.Output
Output to any database (abstract base class).
-
class
stetl.outputs.dboutput.
PostgresDbOutput
(configdict, section)[source]¶ Bases:
stetl.outputs.dboutput.DbOutput
Output to PostgreSQL database. Input is an SQL string. Output by executing input SQL string.
consumes=FORMAT.string
-
class
stetl.outputs.dboutput.
PostgresInsertOutput
(configdict, section, consumes='record')[source]¶ Bases:
stetl.outputs.dboutput.PostgresDbOutput
Output by inserting a single record in a Postgres database table. Input is a Stetl record (Python dict structure) or a list of records. Creates an INSERT for Postgres to insert each single record. When the “replace” parameter is True, any existing record keyed by “key” is attempted to be UPDATEd first.
NB a constraint is that the first and each subsequent each record needs to contain all values as an INSERT and UPDATE query template is built once for the columns in the first record.
consumes=[FORMAT.record_array, FORMAT.record]
-
class
stetl.outputs.wfsoutput.
WFSTOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Insert features via WFS-T (WFS Transaction) OGC protocol from an etree doc.
consumes=FORMAT.etree_doc
-
idgen
()[source]¶ CONFIG
- Mode that WFS server generates new Id’s for incoming Features.
- type: str
- required: False
- default: GenerateNew
-
-
class
stetl.outputs.deegreeoutput.
DeegreeBlobstoreOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Insert features into deegree Blobstore from an etree doc.
consumes=FORMAT.etree_doc
-
class
stetl.outputs.deegreeoutput.
DeegreeFSLoaderOutput
(configdict, section)[source]¶ Bases:
stetl.output.Output
Insert features via deegree using deegree’s FSLoader tool from an etree doc.
consumes=FORMAT.etree_doc