wuttasync.importing.handlers

Data Import / Export Handlers

class wuttasync.importing.handlers.FromFileHandler(config, **kwargs)

Handler for import/export which uses input file(s) as data source.

This handler assumes its importer/exporter classes inherit from FromFile for source parent logic.

class wuttasync.importing.handlers.ImportHandler(config, **kwargs)

Base class for all import/export handlers.

Despite the name ImportHandler this can be used for export as well. The logic is no different on a technical level and the “export” concept is mostly only helpful to the user. The latter is important of course and to help with that we track the orientation to distinguish.

The role of the “import/export handler” (instance of this class) is to orchestrate the overall DB connections, transactions and then invoke the importer/exporter instance(s) to do the actual data assessment/transfer. Each of the latter will be an instance of (a subclass of) Importer.

property actioning

Convenience property which effectively returns the orientation in progressive verb tense - i.e. one of:

• 'importing'

• 'exporting'

begin_source_transaction()

Begin a transaction on the source side, if applicable.

This is normally called from begin_transaction().

begin_target_transaction()

Begin a transaction on the target side, if applicable.

This is normally called from begin_transaction().

begin_transaction()

Begin an import/export transaction, on source and/or target side as needed.

This is normally called from process_data().

Default logic will call both:

• begin_source_transaction()

• begin_target_transaction()

commit_source_transaction()

Commit the transaction on the source side, if applicable.

This is normally called from commit_transaction().

commit_target_transaction()

Commit the transaction on the target side, if applicable.

This is normally called from commit_transaction().

commit_transaction()

Commit the current import/export transaction, on source and/or target side as needed.

This is normally called from process_data().

Default logic will call both:

• commit_target_transaction()

• commit_source_transaction()

Note

By default the target transaction is committed first; this is to avoid edge case errors when the source connection times out. In such cases we want to properly cleanup the target and then if an error happens when trying to cleanup the source, it is less disruptive.

consume_kwargs(kwargs)

This method is called by process_data().

Its purpose is to give handlers a hook by which they can update internal handler state from the given kwargs, prior to running the import/export task(s).

Any kwargs which pertain only to the handler, should be removed before they are returned. But any kwargs which (also) may pertain to the importer/exporter instance, should not be removed, so they are passed along via get_importer().

Parameters:

kwargs – Dict of kwargs, “pre-consumption.” This is the same kwargs dict originally received by process_data().

Returns:

Dict of kwargs, “post-consumption.”

define_importers()

This method must “define” all importer/exporter classes available to the handler. It is called from the constructor.

This should return a dict keyed by “model name” and each value is an importer/exporter class. The end result is then assigned as importers (in the constoructor).

For instance:

return {
    'Widget': WidgetImporter,
}

Note that the model name will be displayed in various places and the caller may invoke a specific importer/exporter by this name etc. See also get_importer().

dry_run = False

Flag indicating whether data import/export should truly happen vs. dry-run only.

If true, the data transaction will be rolled back at the end; if false then it will be committed.

See also rollback_transaction() and commit_transaction().

get_importer(key, **kwargs)

Returns an importer/exporter instance corresponding to the given key.

The key will be the “model name” mapped to a particular importer/exporter class and thus must be present in importers.

This method is called from process_data() but may also be used by ad-hoc callers elsewhere.

It will call get_importer_kwargs() and then construct the importer/exporter instance using those kwargs.

Parameters:

key – Model key for desired importer/exporter.

Returns:

Instance of (subclass of) Importer.

get_importer_kwargs(key, **kwargs)

Returns a dict of kwargs to be used when construcing an importer/exporter with the given key. This is normally called from get_importer().

Parameters:

• key – Model key for the desired importer/exporter, e.g. 'Widget'

• **kwargs – Any kwargs we have so collected far.

Returns:

Final kwargs dict for new importer/exporter.

classmethod get_key()

Returns the “full key” for the handler. This is a combination of source_key and target_key and orientation.

For instance in the case of CSV → Wutta, the full handler key is to_wutta.from_csv.import.

Note that more than one handler may return the same full key here; but only one will be configured as the “default” handler for that key. See also get_spec().

get_source_title()

Returns the display title for the data source.

See also get_title() and get_target_title().

classmethod get_spec()

Returns the “class spec” for the handler. This value is the same as what might be used to configure the default handler for a given key.

For instance in the case of CSV → Wutta, the default handler spec is wuttasync.importing.csv:FromCsvToWutta.

See also get_key().

get_target_title()

Returns the display title for the data target.

See also get_title() and get_source_title().

get_title()

Returns the full display title for the handler, e.g. "CSV → Wutta".

Note that the orientation is not included in this title.

It calls get_source_title() and get_target_title() to construct the full title.

importers = None

This should be a dict of all importer/exporter classes available to the handler. Keys are “model names” and each value is an importer/exporter class. For instance:

{
    'Widget': WidgetImporter,
}

This dict is defined during the handler constructor; see also define_importers().

Note that in practice, this is usually an OrderedDict so that the “sorting” of importer/exporters can be curated.

If you want an importer/exporter instance you should not use this directly but instead call get_importer().

orientation = 'import'

Orientation for the data flow. Must be a value from Orientation:

• Orientation.IMPORT (aka. 'import')

• Orientation.EXPORT (aka. 'export')

Note that the value may be displayed to the user where helpful:

print(handler.orientation.value)

See also actioning.

It’s important to understand the difference between import/export and source/target; they are independent concepts. Source and target indicate where data comes from and where it’s going, whereas import vs. export is mostly cosmetic.

How a given data flow’s orientation is determined, is basically up to the developer. Most of the time it is straightforward, e.g. CSV → Wutta would be import, and Wutta → CSV would be export. But confusing edge cases certainly exist, you’ll know them when you see them. In those cases the developer should try to choose whichever the end user is likely to find less confusing.

process_data(*keys, **kwargs)

Run import/export operations for the specified models.

Parameters:

*keys – One or more importer/exporter (model) keys, as defined by the handler.

Each key specified must be present in importers and thus will correspond to an importer/exporter class.

A transaction is begun on the source and/or target side as needed, then for each model key requested, the corresponding importer/exporter is created and invoked. And finally the transaction is committed (assuming normal operation).

See also these methods which may be called from this one:

• consume_kwargs()

• begin_transaction()

• get_importer()

• process_data() (on the importer/exporter)

• rollback_transaction()

• commit_transaction()

rollback_source_transaction()

Rollback the transaction on the source side, if applicable.

This is normally called from rollback_transaction().

rollback_target_transaction()

Rollback the transaction on the target side, if applicable.

This is normally called from rollback_transaction().

rollback_transaction()

Rollback the current import/export transaction, on source and/or target side as needed.

This is normally called from process_data(). It is “always” called when dry_run is true, but also may be called if errors are encountered.

Default logic will call both:

• rollback_target_transaction()

• rollback_source_transaction()

Note

By default the target transaction is rolled back first; this is to avoid edge case errors when the source connection times out. In such cases we want to properly cleanup the target and then if an error happens when trying to cleanup the source, it is less disruptive.

source_key = None

Key identifier for the data source.

This should “uniquely” identify the data source, within the context of the data target. For instance in the case of CSV → Wutta, csv is the source key.

Among other things, this value is used in get_key().

target_key = None

Key identifier for the data target.

This should “uniquely” identify the data target. For instance in the case of CSV → Wutta, wutta is the target key.

Among other things, this value is used in get_key().

enum wuttasync.importing.handlers.Orientation(value)

Enum values for ImportHandler.orientation.

Valid values are as follows:

IMPORT = <Orientation.IMPORT: 'import'>

EXPORT = <Orientation.EXPORT: 'export'>

class wuttasync.importing.handlers.ToSqlalchemyHandler(config, **kwargs)

Handler for import/export which targets a SQLAlchemy ORM (DB).

begin_target_transaction()

Establish a new db session via make_target_session() and assign the result to target_session.

commit_target_transaction()

Commit the target_session.

make_target_session()

Make and return a new db session for the import/export.

Subclass must override this; default logic is not implemented.

rollback_target_transaction()

Rollback the target_session.

target_session = None

Reference to the SQLAlchemy db session for the target side.

This may often be a session for the app database (i.e. for importing to Wutta DB) but it could also be any other.

This will be None unless an import/export transaction is underway. See also begin_target_transaction().