Query Sync with Legend State and IndexedDB #

This document outlines the process for implementing synchronization between Legend State observables and IndexedDB for persistent data storage in the webapp.

Overview #

Our webapp uses Legend State for reactive state management combined with IndexedDB for client-side persistence. This setup allows data to persist across browser sessions and provides optimistic updates with backend synchronization.

Architecture #

The sync system consists of three main components:

Table Configuration (persist.ts) - Defines table names and database versioning
Observable Creation - Creates synced observables using syncedDataStateQueryObservable
Initialization (use-init-sync.ts) - Ensures all observables are properly initialized

Adding a New Table #

When creating a new data table with Legend State and IndexedDB persistence, follow these steps:

Step 1: Define Table Configuration #

In apps/webapp/src/shared/utils/persist.ts:

Increment the database version:

// Increment the DB_VERSION when adding a new table
const DB_VERSION = 12; // Previously was 11

Add the table name constant:

export const TABLE_YOUR_FEATURE = "yourFeature";

Add the table name to the tableNames array:

export const getPersistPlugin = () => {
  if (!persistPluginInstance) {
    persistPluginInstance = observablePersistIndexedDB({
      databaseName: DEFAULT_PERSIST_DB_NAME,
      version: DB_VERSION,
      tableNames: [
        TABLE_ENTITIES,
        TABLE_DOCUMENTS,
        TABLE_INVOICES,
        // ... existing tables
        TABLE_YOUR_FEATURE, // Add your new table here
      ],
    });
  }
  return persistPluginInstance;
};

Step 2: Create the Observable #

Create a new file for your feature (e.g., apps/webapp/src/shared/sync/your-feature.ts):

import type { Observable } from "@legendapp/state";
import { syncedDataStateQueryObservable } from "@/shared/utils/query";
import type { DbState } from "@/shared/sync/types";
import { TABLE_YOUR_FEATURE } from "@/shared/utils/persist";
import { auth$ } from "@/shared/sync/auth";
import { doesCurrentEntityExist$, ui$ } from "@/shared/sync/ui";

// Define your data interface
interface YourFeatureData {
  id: string;
  name: string;
  // ... other properties
}

// Define the state interface extending DbState
interface YourFeatureState extends DbState<YourFeatureData[]> {
  // Add any custom methods here
  getById: (id: string) => YourFeatureData | undefined;
}

// Create the observable
const yourFeature$: Observable<YourFeatureState> =
  syncedDataStateQueryObservable(
    TABLE_YOUR_FEATURE,
    () => ({
      list: getYourFeatureQueryOptions({
        entity_id: ui$.currentEntityId.get() || "",
      }),
      transform: {
        list: (data) => data?.data ?? [],
      },
      persist: true,
      enabledQueryObserver: true,
      staleTime: 5000,
      refetchInterval: 10000,
      mode: "set",
    }),
    {
      dependencies: [ui$.currentEntityId],
      waitFor: [auth$.isAuthenticated, doesCurrentEntityExist$],
      extensions: {
        getById: (data$) => (id: string) => {
          return data$.get().find((item) => item.id === id);
        },
      },
    }
  );

export default yourFeature$;

Step 3: Register in the State Tree #

In apps/webapp/src/shared/sync/index.ts:

import yourFeature$ from "@/shared/sync/your-feature";

export const state$ = observable({
  auth: auth$,
  db: {
    entities: entities$,
    documents: documents$,
    invoices: invoices$,
    yourFeature: yourFeature$, // Add your observable here
    // ... other tables
  },
  ui: ui$,
});

Step 4: Initialize the Observable #

In apps/webapp/src/shared/sync/hooks/use-init-sync.ts, add your observable to the initialization:

const initSync = async (): Promise<void> => {
  // ... existing initialization code

  try {
    // Initialize by directly accessing the data$ properties
    state$.db.entities.data$.get();
    state$.db.documents.data$.get();
    state$.db.invoices.data$.get();
    state$.db.yourFeature.data$.get(); // Add this line

    // ... rest of initialization
  } catch (err) {
    // ... error handling
  }
};

Updating Existing Tables #

When modifying an existing table structure (adding/removing fields, changing indexes, etc.):

Step 1: Update Database Version #

In apps/webapp/src/shared/utils/persist.ts:

// Increment the DB_VERSION
const DB_VERSION = 12; // Previously was 11

Step 2: Update Initialization #

In apps/webapp/src/shared/sync/hooks/use-init-sync.ts, ensure your observable’s data$.get() is called in the initialization:

const initSync = async (): Promise<void> => {
  try {
    // Make sure to include your updated observable
    state$.db.yourUpdatedFeature.data$.get();

    // ... other initializations
  } catch (err) {
    // ... error handling
  }
};

Real-World Examples #

Example 1: Entities Table #

const entities$: Observable<EntitiesState> = syncedDataStateQueryObservable(
  TABLE_ENTITIES,
  {
    list: getGetEntitiesQueryOptions(),
    transform: {
      list: (data) => data?.data || [],
    },
    persist: true,
    enabledQueryObserver: true,
    staleTime: 5000,
    refetchInterval: 10000,
    mode: 'set',
  },
  {
    extensions: {
      getById: (data$) => (id: string) => {
        return data$.get().find((entity) => entity.id === id);
      },
    },
  }
);

Example 2: Accounting Data with Dependencies #

const accountingAccounts$: Observable<AccountingAccountsState> =
  syncedDataStateQueryObservable(
    TABLE_ACCOUNTING_ACCOUNTS,
    () => ({
      list: getAccountingAccountsQueryOptions(
        accountingIntegrations$.data$?.get()?.[0]?.id || '',
        {
          entity_id: ui$.currentEntityId.get() || '',
        }
      ),
      transform: {
        list: (data) => data?.data ?? [],
      },
      persist: true,
      enabledQueryObserver: true,
      staleTime: 3000,
      refetchInterval: 3000,
      mode: 'set',
    }),
    {
      dependencies: [ui$.integrationId, ui$.currentEntityId],
      waitFor: [auth$.isAuthenticated, doesCurrentEntityExist$],
    }
  );

Important Notes #

Database Versioning #

Always increment DB_VERSION when adding new tables or updating existing table structures
This applies to both creating new tables and modifying existing ones
This ensures proper IndexedDB schema migration

Development Environment #

When developing locally, close the browser tab before making table changes
This prevents version conflicts during development

Naming Conventions #

Table name constants should be in SCREAMING_SNAKE_CASE
Observable names should end with $ (e.g., entities$)
The table name should match the first parameter of syncedDataStateQueryObservable

Dependencies and Wait Conditions #

Use dependencies array for observables that should trigger re-queries when changed
Use waitFor array to ensure required conditions are met before starting queries
Common wait conditions include authentication and entity existence

Error Handling #

Always include proper error handling in your observables
Use the logger utility for consistent error reporting
Consider showing user-friendly error messages with toast notifications

Troubleshooting #

Common Issues #

Table not found errors: Ensure the table name is added to both the constant and the tableNames array
Data not persisting: Check that persist: true is set in the observable configuration
Initialization errors: Verify that the observable’s data$.get() is called in use-init-sync.ts
Version conflicts: Make sure to increment DB_VERSION when making schema changes

Performance Considerations #

Set appropriate staleTime and refetchInterval values based on data update frequency
Use waitFor conditions to prevent unnecessary queries
Consider using extensions for computed values to avoid recalculation

This documentation should be updated as the sync system evolves and new patterns emerge.