Data Models #

Overview #

Data Models are the core abstraction layer in FunnelStory for ingesting, standardizing, and processing customer data. They act as the bridge between raw data from various external sources (like databases, warehouses, or SaaS applications) and our internal system of entities, activities, and metrics.

The entire process is managed by a robust refresh mechanism that periodically fetches data, detects changes, derives activities, and updates our internal entity records.

The Model Refresh Process #

The lifecycle of a data model refresh is managed by the core.RefreshModels and core.RefreshSingle functions.

Triggering: A refresh can be triggered automatically based on a schedule (model.RefreshIntervalDuration()) or manually via an API call (force=true).
Locking: To prevent concurrent refreshes of the same model, a distributed lock is acquired using the model_refreshes_locks table. If a lock is already held, the new refresh job will skip.
Data Fetching & Joining:
- The system uses a temporary in-memory SQLite database for each refresh.
- Using the datajoin package, it fetches data from the primary data source and any configured joins. This allows for the combination of data from multiple tables or sources into a single, unified view before processing.
Mapping to Standardized Properties:
- The raw data from the source query is transformed into a standardized model.Record (a map[string]any).
- This is done by applying the mappings configured for the model, which connect source columns (e.g., organizations.id) to FunnelStory’s internal properties (e.g., account_id).
- During this step, data types are validated and normalized (e.g., converting various date formats into Unix timestamps). AI-powered analysis for sentiment or text features is also triggered here if configured (e.g., <fs_analyze>).

Aggregation, Diffing, and Activity Derivation #

The core of our change detection system relies on hashing and comparing model records against their last known state.

Hashing and Aggregation:
- Each row fetched from the source is processed into a standardized model.Record.
- A stable hash is computed for the JSON representation of this record.
- The refresh.Aggregator collects all records, grouping them by their unique key (e.g., a combination of account_id and user_id for a user model).
State Comparison (Diffing):
- Before processing the new records, the system queries the model_history table to retrieve the hash of each record from the last successful refresh.
- For each new record, its hash is compared with the previously stored hash.
  - If the hashes match, the record has not changed. The system simply updates the last_refresh_id in the model_history table to mark it as seen in the current refresh.
  - If the hashes do not match (or the record is new), the record is considered “dirty.” A new entry is inserted into model_history with the new hash and data, and this change triggers activity derivation.
Activity Derivation:
- When a record is identified as new or changed, the system consults the configured Model Rules.
- If a rule’s conditions are met, a new record is inserted into the activity_history table. This record captures the activity type, the associated account/user, the timestamp, and a reference to the model state that triggered it.
- For special activity types like account_property_changed, the system performs a deep comparison between the old and new record data to see if any of the specified properties have changed before creating the activity.

Model Types and Entity Creation #

The final and most crucial step of the refresh process is to update FunnelStory’s core entity records. Based on the model.Type, the standardized data is used to create or update records in our primary entity tables. This ensures our internal representation of the customer is always synchronized with the latest data from the source.

The following table outlines the primary model types and the corresponding entities that are created or updated:

Model Type (`model.Type`)	Corresponding Entity	Description
`account`	`entity.Account`	Creates or updates the central account record. Properties are populated, and account assignments to CSMs are evaluated and updated.
`user`	`entity.User`	Creates or updates user records associated with accounts. Handles traits, role tags, and source attribution.
`subscription`	`entity.Subscription`	Creates or updates subscription records, linking them to accounts and plans.
`plan`	`entity.Plan`	Creates or updates service plan records, defining available features and dimensions.
`usage`	`entity.DimensionUsage`	Records time-series data for specific usage dimensions, linked to an account.
`meeting`	`entity.Meeting`	Creates records for meetings (e.g., from Zoom, Gong), storing metadata, participants, and analysis results.
`conversation`	`entity.Conversation`	Creates records for communications like support tickets or Slack messages. Handles sentiment, topics, and contact resolution.
`note`	`notes.Note`	Creates external note records, linking them to accounts and associating them with users via email.
`strategy`	`entity.Strategy`	Creates or updates account-specific strategy records, which can contain structured text or markdown.

How User Entities Are Created From Activities #

A key feature of the system is its ability to automatically create user entities on-the-fly when an activity is detected, even if a user model isn’t explicitly defined. This ensures that user interactions are never lost.

The process, found within the activity detection loop in doRefreshSingle, is as follows:

An activity is derived from a model record (e.g., a support ticket or a product event) that contains a user_id and an email, but no FunnelStory User ID (fs_user_id).
The system first attempts to find an existing entity.User within the associated account_id using the provided user_id.
If no user is found, the system uses the entityDAO.UpsertUserWithAccount function. This function creates a new entity.User record, populating it with the account_id, user_id, and email from the model record.
A new, unique FunnelStory ID (fs_id) is generated for this user.
This fs_id is then immediately used to correctly associate the activity_history record with the newly created user entity.

This “just-in-time” user creation ensures data integrity and allows us to build a complete picture of user engagement from the very first interaction.