The Big Tour

Here we try to give a "full tour" of the Rattail Project in one page. It's intentionally brief in many areas but plenty wordy (and technical) still; it aims to at least touch on "all" the birds-eye-view details.

Really this page tries to answer the question for the reader - who is assumed to be a developer or sysadmin! - "Should I even be interested in Rattail?" By the time you've read this page you should have the answer to that.

In a nutshell, Rattail tries to offer a "basic and unobtrusive, yet opinionated and convenient" Swiss Army Knife / toolkit for building custom back-end / automation software for integrating with other systems and interacting with that data. Retailers were the assumed market for this software but in practice it can be used for many other purposes / organizations as well.

To some extent each of the primary aspects of the Rattail Project (below) can be used independently, so feel free to start wherever you like in practice.

Remember, the word "Poser" here really means "Custom Project" - i.e. your custom project - so replace that name mentally with whatever you like, i.e. whatever you want to name your custom project. ;)

Configuration

Here are a few things worth knowing about how a Poser/Rattail app is configured.

Files

The primary method of config is via so-called INI files, which contain "sections" and "option / value" pairs.

Such config files may "inherit" from each other, so that your app may require one config file but then several others are indirectly loaded due to this inheritance. The inherited "values" will be overridden by any values present in a "descendent" file, e.g. the one your app uses.

It also is possible for the app to specify multiple config files instead of just the one; in this case each of these will still inherit (as defined in each file).

DB Settings

If there is a Poser database present for the app, then it usually is helpful to allow config to be read from the Settings table in this DB, as well as the normal files. This is not required though and is off by default.

The big win here is, you can then edit settings in the database directly from the Poser web app, and they take effect across the app "immediately" - whereas if you edit a config file, you must restart the app for it to take effect. This allows for (more convenient) so-called feature flags, meaning you launch/upgrade the app with support for a certain new feature hidden behind a setting which you then "turn on" whenever you're ready.

Note that some config values will never be read from DB and only can come from the files (e.g. database connection parameters). Generally speaking any "sensitive" values should probably only be stored in the files (with appropriate security).

Database

It often is useful to maintain one or more Poser databases, to facilitate automation or other organization-defined workflow. Note that the web app requires a database.

Schema Definition

Rattail provides a "core" ("common") database schema, with tables for users, products, customers, etc. Each custom Poser app is free to define "extensions" to this schema, to track things not represented by the core schema. (It's generally expected that most Poser apps would do so.) These schema definitions are made using the SQLAlchemy ORM library/syntax.

Note that whereas you can pick and choose which parts of Rattail software you want to run, the schema is "all or nothing" - meaning each DB built using the Rattail framework will effectively contain this full "core + extensions" schema. In practice that's no big deal - just ignore any tables you don't need within app logic, as the core schema is sure to define some you don't need!

Location / Purpose

The assumption is that generally each Poser app will require one such database, which will live in PostgreSQL. At the very least, this DB would be needed to store user accounts and roles/permissions for the Poser web app, if that is deployed. Beyond that, it's often useful to treat this DB as a "pseudo-authority" - meaning, with the right setup you can expect this DB to always contain the "latest" authoritative data (assuming datasync etc.) in which case you can then develop further reports and app features based on this "custom but very accurate" DB. Additionally it can be useful to treat this DB as a "workspace" for moving data between other systems but with extra processing / validation steps etc.

In some cases (e.g. minimal integration) no such database is needed; in other cases (e.g. multi-store) there may be multiple Poser apps with each requiring its own database.

Schema Upgrades

Regardless of how many Poser databases must exist, they should generally be kept up-to-date such that the schema of each is the same. Among other reasons, this is because the datasync daemon will usually try to sync all fields for a record, in which case those field definitions must match between app nodes. Alembic is the tool used to track migrations and keep schema for each database upgraded.

Data

As for how data makes its way into this database, that will vary of course but by and large will be via "integration" (with other systems) and/or "web app" (aka. integrating with users) - both of which are discussed further below.

Note that if there are multiple Poser databases, then often a new data change in one node, will be propagated to all other nodes, typically by way of a "host" (master) node. When applicable, this would be handled via datasync.

Data Versions

Poser's database includes support for "data versioning" via SQLAlchemy-Continuum. The versioning is a slightly tricky feature, in that it must be turned on or off at app startup, and the versioning can't be turned back off/on without restarting the app. In practice that's not normally too big a deal, and there are workarounds available when needed. (The main issue is that some importers and batches can be slow to process when versioning is enabled in the web app.)

But enough of that, now for the good news. You are in fact encouraged to turn on versioning wherever you like, it's a pretty solid feature. The idea is that for a given "versioned" table, a second table is created to match it basically, and this table will include a new "version" record for each time the "main" record is changed in any way. With versioning turned on, this will happen automatically; with it off, it won't.

The version history of a given "primary" record then can be made available within the Poser web app. Note that due to technical issues with how this feature works (again, tricky), this version history can only be displayed in the web UI if the versioning feature is currently turned ON.

Integrations

By "integration" we mean the providing and/or receiving of data and/or commands between the Poser/Rattail system and an arbitrary number of other systems.

Next we'll expand on that definition a bit, then go over the technical "styles" of integration provided by Rattail.

Concepts

Let's say you're a retailer with a POS (point of sale) system, which serves as a defacto CRM (customer relations management) system also. In other words "all your customer records" live in the POS. Let's say you also have an online email campaign (mailing list) to which you would like to push the email addresses of new customers as they are added to your POS. This is one of many "integration" scenarios which may be handled via Poser/Rattail.

Continuing with this example, you have some options. It's possible that your POS already has a plugin which could provide the solution. Or, you could use Poser to dynamically read customer data from POS (presumably via ODBC/SQL), and from the email campaign also (presumably via web API), then push any "missing" addresses to the campaign. It may be helpful to keep a "local cache" of data from one or both systems, within a Poser database. This can make for a more efficient app, albeit with more moving parts. But then you could also run a Poser web app to give a read-only view of "all" the data in one place.

Pretty much all integrations are like this in some sense. Poser would need to have a way to read and/or write data from/to the given system(s). When and how it does so would ultimately be up to Poser business logic etc. The Poser web app (and even the Poser database) is optional until you want a feature which would require it.

Note that in this example, we mentioned your POS may already have a plugin to do the job. To expand on that point, the Poser/Rattail approach is not "novel" per se - most of what it does can be accomplished in some other way, easily enough. Sometimes this "other way" would resemble a Poser approach in many ways, but using a different tech stack etc. Especially when only "one" integration feature is considered at a time, there may be no compelling reason to use the Poser/Rattail approach. But what the latter does provide, is a common framework which is capable of tying all integration efforts into one cohesive whole.

Data Import / Export

The "meat and potatoes" of integrations are really the import/export of data between systems. Rattail provides an "importer framework" which is meant to do most of the heavy lifting, and in many cases a new importer need do little more than define which fields it supports, and how to obtain a value for each, for a given object/record.

Note that from this framework's perspective, the "import vs. export" distinction is largely meaningless; it always assumes (essentially) a "Source -> Target" scenario which may be thought of as "Target imports from Source". You may declare a given importer to really be an "exporter" in which case that would be thought of as "Source exports to Target" instead - but everything about how it actually works, would be the same. Declaring an importer to be of "export" persuasion is only for the user interface, so it makes more sense to the humans.

The general idea of the importer framework, is that for a given pair of systems, and data flow direction (e.g. POS -> Poser), there will be one "import handler" defined. This is responsible for, among other things, establishing a database transaction for either/both of the systems, and committing it (or rolling back, if dry run) at the end. The handler also will be configured with one or more "importers" - each of which is responsible for a "model" and how to map data for a record from one system, to the other. So the "handler" is for the DB and each "importer" corresponds to a table, conceptually. This is a simplistic overview and many importers in the wild bend these definitions somewhat.

The framework provides a fairly robust command line interface for each import handler defined. The commands are optimized for efficiency and allow for toggling the create/update/delete support, as well as restricting which fields are updated in a given run, etc. Such commands may be scheduled via cron and voilĂ , you have (poor man's / pseudo) data sync between systems. It also has a "warnings" mode which basically means "no changes are expected, so send email report if any are found" - as well as a "dry run" mode which can be useful when testing ad-hoc commands.

Generally importer "jobs" (runs) are expected to be either automated (via cron) or else ran manually via command line. In some cases the web app may support running certain types of importer jobs, as needed. It is even possible to convert a new/incoming importer job to a "batch" so its contents may be fully inspected within the web app prior to "execution".

Real-Time Data Sync

If the poor man's data sync (i.e. running importers via cron) isn't good enough, you might want to go with the Rattail "DataSync" proper. This is a daemon with multiple threads running, some of which will "watch" for new data changes which appear in a given system, while others will then "consume" the changes by pushing updates to some other system.

This "datasync" daemon is meant to leverage the "importer" framework as much as possible. So for instance, one watcher thread will notice say, a new Customer record has appeared in the POS system. Then a consumer thread will create the appropriate importer, and tell it to "import" the new customer record (which might say, push a new email address to the mailing list). In this way you're able to keep the logic as "DRY" as possible. (Adding datasync support rarely requires much new code although does assume importers contain most of the logic.)

However this datasync daemon does require a Poser database - because it uses a table within it as a "queue" for incoming changes which await processing.

While your Poser app may have multiple import handlers defined, you will need only one datasync daemon per app node (if indeed, you need one at all). Multiple importers can be "handled" within a single datasync daemon; it merely creates additional threads as configured.

Data Batches

The "batch" concept is a bit like an importer, but the real idea for a batch is that certain data will be brought (from one or more systems) into a "workspace / temp" table in the Poser DB, and then some user(s) will review and perhaps modify this temp data, and ultimately "execute" the batch - which would update Poser and/or other systems as appropriate.

So, while you can usually create a batch "from" an importer job (run), and thereby delay "execution" for the importer job (pending user input), that is not the typical use case. Most batches are "purpose-built" in some way, and serve a different function than normal importers would. Some common examples are:

After saying most batches are "not" importers, I still threw in that last example! To highlight the difference, a "true" importer in Rattail-speak is meant to be part of the automation, whereas a batch by definition is meant to require user input (and therefore not "automated" - at least not fully). But it's often the case that someone in charge of adding products to the system, is very Excel-savvy and wants a tool to do so directly from some spreadsheet of their own design (for instance). It can then be helpful to bring this data in via batch instead of proper importer, so the user can be more involved and confident in the details. And, while "any" importer can be used to create a "dynamic" batch, a purpose-built batch type will provide a better experience. (And they're not that hard!)

Really you can use batches to do just about anything, where you want the user to get to preview the entire job before hitting the execute button. Some batches are "immediately" populated with data when first made, e.g. a Receiving batch might be populated with contents of a PO or invoice. Other batches are slowly populated over time, e.g. Purchasing where a user scans shelf tags while building the order. And yet, because it's a free country, you can automate a batch if you want - e.g. automatically create/populate/execute it with no user intervention.

Since batches generally require user input, they also will generally require a Poser web app.

Note that while we refer to a "temp table" here, really each batch type has its own set of persistent tables, so each "batch" is really just a set of records within those tables. The default behavior is for these records to "stick around" after the batch has been executed, which may be useful as an audit trail of sorts. Each batch may be executed only once, after which it is frozen in time. (Who created and executed the batch is also recorded.) You may want to purge old batch records occasionally, if any tables get heavy use.

File Monitor

Rattail's "FileMon" is another daemon, somewhat like the "DataSync" daemon, but instead of watching databases and syncing those changes, the filemon...monitors files. And due to the nature of files, this isn't always about the "data" per se, meaning sometimes this gets more into the "command" side of things.

The general idea though again is not unlike datasync, in that the filemon daemon has a bunch of threads running, some of which will "monitor" ("watch") a given folder for new files to appear, and other threads will then "process" these files in some way. In most cases the files are either deleted outright, or perhaps archived to another folder, after processing.

The filemon can be useful for keeping tabs on a POS system, if it's the type to generate new files in a configured location, for e.g. new transactions which occur at the register. Another use would be to actually send "commands" between Poser nodes, in a "resilient" manner. For instance, to print a set of labels for all stores, from a multi-store host node, the host web app can generate a file with all the details, and then effectively hand it off to the filemon for distributing around to the other nodes.

But in the end, anytime you need to watch a folder for new files, and process them, you would probably want to use filemon.

Web App

TODO

App Emails

TODO

Config Management

TODO