Destinations Event Model
The foundation for Fullstory's data model is an Event. This data serves as a record of user interactions with your website or app as well as any server-side events you send to Fullstory. Each event comes with a set of properties that describe the event and the user's context at the time of the event.
The data model is designed to be stable at the outer level, meaning that the fields and their types will not change. However, the data model is extensible, so new fields can be added to the data model without breaking existing integrations or requiring expensive schema migrations.
Important Timestamps
There are three timestamp fields that are relevant for destinations: event_time, processed_time, and updated_time.
event_timeis an immutable field that records the timestamp of each event according to the user's device.processed_timeis a timestamp field indicating when the event was processed by Fullstory's servers. On average, 95% of events are captured and processed within 20 minutes of the original event time. Several events, including server side events, may reach Fullstory's servers much later than the original event time. Depending on the contents of these late events, Fullstory may need to reprocess them to report on the most accurate metrics, including session length, page active time, etc. In these scenarios, theprocessed_timefor all events in the session or on the page will be updated and the events will be re-synced to the warehouse.updated_timeindicates when a record was last modified (inserted or updated) and is populated by a built-in function in the warehouse during the sync. This field tracks the last time a record in your warehouse was changed and can be used as a filter to determine which new records to pull into your query.
Sync Latency
Sync latency is a concept that tracks the cadence with which Fullstory syncs new events to the destination.
Captured events flow into Fullstory constantly, then are processed on a defined cadence to be sent to the destination.
The interval is dependent on processed_time, which indicates when our servers processed the event for the particular destination.
For example, when syncing to Snowflake, the sync latency is
Approximately
processed_timerounded to the next hour + 1 hour.
This is because Fullstory writes data to a file based on the processed_time, with each file containing events that
were processed in a given hour. The file is then merged into the database table within the next hour.
This timing is approximate because syncing to the destination depends on how large the file is and how much compute is available to run the merge.
Update Window
Due to the nature of web traffic, distributed systems, and worldwide users, not all events will hit Fullstory's servers chronologically. There are some scenarios where events come out of order, like swan song events that appear hours after they occur. Fullstory's reporting pipeline treats pages and sessions as top-level entities. All events that flow into the system will be associated with a page, a session, or both. This relationship is then used to derive metrics like "session length" and "active time on page." When a late event is captured, Fullstory will reprocess the page or associated session to "put the events back in order." When this reprocessing happens, Fullstory will also push the updated events out to the warehouse, sometimes overwriting the data previously associated with that event. This is called the Update Window, or the rolling look-back period where Fullstory may update events in the warehouse. For a small set of edge cases, events may be reprocessed after the update window, causing minor discrepancies between the Fullstory UI and the data warehouse data. A very long update window reduces discrepancies but can make it harder to "close the books" from an accounting perspective.
For destinations that support updates, the update window for each destination is stated in terms of the expected percentage of events that will no longer be modified after a certain time period.
Destinations
| Destination | Sync Interval | Update Window* | Relevant Timestamps | Setup Guide |
|---|---|---|---|---|
| BigQuery | Approximately processed_time rounded to the next hour + 1 hour | 99% stable within 3 hours | event_time updated_time processed_time | BigQuery Setup |
| Redshift | Approximately processed_time rounded to the next hour + 1 hour | 99% stable within 3 hours | event_time updated_time processed_time | Redshift Setup |
| Snowflake | Approximately processed_time rounded to the next hour + 1 hour | 99% stable within 3 hours | event_time updated_time processed_time | Snowflake Setup |
* Update Window: These are observed aggregates and may vary based on the specifics of the data capture for your site or app.
Differences between V1 and V2
The current version of Destinations takes a drastically different approach to getting data out of Fullstory compared to V1.
The primary approach in V1 was utilizing webhooks to push data out of Fullstory. However, these are limited in what data is available, so they are now considered legacy. They are still available for use and are documented here, but will not be receiving any new features or updates.