wuttasync.importing.csv

Importing from CSV

class wuttasync.importing.csv.FromCsv(config, **kwargs)

Base class for importer/exporter using CSV file as data source.

Note that this assumes a particular “format” for the CSV files. If your needs deviate you should override more methods, e.g. open_input_file().

The default logic assumes CSV file is mostly “standard” - e.g. comma-delimited, UTF-8-encoded etc. But it also assumes the first line/row in the file contains column headers, and all subsequent lines are data rows.

input_reader

While the input file is open, this will reference a csv.DictReader instance.

csv_encoding = 'utf_8'

Encoding used by the CSV input file.

You can specify an override if needed when calling process_data().

get_input_file_name()

By default this returns the importer/exporter model name plus CSV file extension, e.g. Widget.csv

It calls get_model_title() to obtain the model name.

get_source_objects()

This returns a list of data records “as-is” from the CSV source file (via input_reader).

Since this uses csv.DictReader by default, each record will be a dict with key/value for each column in the file.

open_input_file()

Open the input file for reading, using a CSV parser.

This tracks the file handle via input_file and the CSV reader via input_reader.

It also updates the effective fields list per the following logic:

First get the current effective field list, e.g. as defined by the class and/or from caller params. Then read the column header list from CSV file, and discard any which are not found in the first list. The result becomes the new effective field list.

class wuttasync.importing.csv.FromCsvToSqlalchemyHandlerMixin

Mixin class for CSV → SQLAlchemy ORM import handlers.

This knows how to dynamically generate importer classes to target the particular ORM involved. Such classes will inherit from FromCsvToSqlalchemyMixin, in addition to whatever FromImporterBase and ToImporterBase reference.

This all happens within define_importers().

FromImporterBase

This must be set to a valid base class for the CSV source side. Default is FromCsv which should typically be fine; you can change if needed.

alias of FromCsv

ToImporterBase = None

For a handler to use this mixin, this must be set to a valid base class for the ORM target side. The define_importers() logic will use this as base class when dynamically generating new importer/exporter classes.

define_importers()

This mixin overrides typical (manual) importer definition, and instead dynamically generates a set of importers, e.g. one per table in the target DB.

It does this based on the target model, as returned by get_target_model(). It calls make_importer_factory() for each model class found.

get_target_model()

This should return the app model or a similar module containing data model classes for the target side.

The target model is used to dynamically generate a set of importers (e.g. one per table in the target DB) which can use CSV file as data source. See also define_importers().

Subclass must override this if needed; default behavior is not implemented.

make_importer_factory(model_class, name)

Generate and return a new importer class, targeting the given data model class.

The newly-created class will inherit from:

• FromCsvToSqlalchemyMixin

• FromImporterBase

• ToImporterBase

Parameters:

• model_class – A data model class.

• name – The “model name” for the importer/exporter. New class name will be based on this, so e.g. Widget model name becomes WidgetImporter class name.

Returns:

The new class, meant to process import/export targeting the given data model.

class wuttasync.importing.csv.FromCsvToSqlalchemyMixin(config, **kwargs)

Mixin class for CSV → SQLAlchemy ORM importers.

Meant to be used by FromCsvToSqlalchemyHandlerMixin.

This mixin adds some logic to better handle uuid key fields which are of UUID data type (i.e. on the target side). Namely, when reading uuid values as string from CSV, convert them to proper UUID instances, so the key matching between source and target will behave as expected.

class wuttasync.importing.csv.FromCsvToWutta(config, **kwargs)

Handler for CSV → Wutta app database import.

This uses FromCsvToSqlalchemyHandlerMixin for most of the heavy lifting.

ToImporterBase

alias of ToWutta