Agentic Workflows #

Capability Summary #

The agentic workflows enable the automated generation and delivery of reports using Large Language Models (LLMs). These workflows allow users to configure reports, schedule them, and receive the generated content via email.

UI Page #

The UI page for managing agentic reports is available at https://app.funnelstory.ai/internal/agent-reports.

How It Works #

Report Configuration: Users create and configure reports with specific settings, including email recipients and schedules. This is done through the PostReports API endpoint.
Report Execution: Reports are executed based on schedules or manual triggers. The execution involves querying datasets, generating content using LLMs, and sending emails. This is managed by the Run function in the report_runs package.
State Machine: A state machine manages the execution flow of report generation. It transitions between states based on LLM responses and tool executions. The state machine is implemented in the runStateMachine function.
Tools: Various tools are available within the state machine to perform specific actions. These include:
- Query Datasets: Using the QueryDatasetsToolFunc function.
- Get Account Details: Using the GetAccountDetailsToolFunc function.
- Query Accounts Table: Using the QueryAccountsTableToolFunc function.
- Send Notification Email: Using the SendReportEmailToolFunc function.
Scheduling and Ticking: Reports are scheduled using cron expressions. The ticking mechanism, triggered at regular intervals (e.g., every 10 minutes), checks for reports that need to be executed based on their schedules and triggers their execution if necessary. This is managed by the Tick function and the tick mechanism in server.go.

Key Components #

Reports Table: Stores report configurations.

CREATE TABLE reports (
    workspace_id uuid NOT NULL,
    id uuid NOT NULL,
    name text NOT NULL,
    config jsonb NOT NULL DEFAULT '{}'::jsonb,
    created_at timestamp with time zone NOT NULL,
    updated_at timestamp with time zone NOT NULL,
    created_by uuid,
    PRIMARY KEY (workspace_id, id),
    FOREIGN KEY (created_by) REFERENCES users(id) ON DELETE SET NULL,
    FOREIGN KEY (workspace_id) REFERENCES workspaces(id) ON DELETE CASCADE
);

Report Runs Table: Records the execution details of each report run.

CREATE TABLE report_runs (
    workspace_id uuid NOT NULL,
    id uuid NOT NULL,
    report_id uuid NOT NULL,
    schedule_id integer,
    started_at timestamp with time zone NOT NULL,
    completed_at timestamp with time zone,
    llm_trace jsonb,
    success boolean,
    error text,
    PRIMARY KEY (workspace_id, id),
    FOREIGN KEY (workspace_id, report_id) REFERENCES reports(workspace_id, id) ON DELETE CASCADE,
    FOREIGN KEY (workspace_id) REFERENCES workspaces(id) ON DELETE CASCADE
);

Report Runs Locks Table: Ensures that only one instance of a report run is processed at a time.

CREATE TABLE report_runs_locks (
    workspace_id uuid NOT NULL,
    PRIMARY KEY (workspace_id),
    FOREIGN KEY (workspace_id) REFERENCES workspaces(id) ON DELETE CASCADE
);

API Endpoints #

POST /reports: Create a new report using the PostReports function.
GET /reports: Retrieve all reports using the GetReports function.
GET /reports/{report_id}: Retrieve a specific report using the GetReport function.
PUT /reports/{report_id}: Update a specific report using the PutReport function.
DELETE /reports/{report_id}: Delete a specific report using the DeleteReport function.
POST /reports/{report_id}/run: Trigger a report run using the PostReportRun function.
POST /reports/tick: Trigger the ticking mechanism to check and execute scheduled reports using the PostReportRunTick function.

State Machine and Tools #

The state machine, implemented in the runStateMachine function, manages the execution flow of report generation. It uses various tools to perform actions such as querying datasets, getting account details, and sending emails. These tools are defined within the state machine and are called based on the current state and LLM responses.

Scheduling and Ticking #

Reports can be scheduled using cron expressions. The ticking mechanism, triggered at regular intervals (e.g., every 10 minutes), checks for reports that need to be executed based on their schedules and triggers their execution if necessary. This is managed by the Tick function and the tick mechanism in server.go.

Testing the Workflow #

To test the agentic workflows, follow these steps:

Create a Report: Use the POST /reports endpoint to create a new report with the desired configuration. Here’s an example config:

{
  "schedule": {
    "type": "cron",
    "expr": "30 14 * * *"
  },
  "email": {
    "system_prompt": "",
    "prompt": "Give me a email report with a list all accounts have had CRM sync failures or connection sync failures or model refresh failures in last 24 hours. The email should have a table with the following columns:\n            1. Account name along with a hyperlink to the account. Links would be in the format https://app.funnelstory.ai/accounts/{account_id}?workspace_id=fbb744b0-589c-4278-a15f-3bb74fe09e26\n\t\t\t2. Total CRM sync failures.\n\t\t\t3. Total Connection sync failures.\n\t\t\t4. Total Model referesh failures.\n            Also mention the time period for which the report is generated. Do not repeat the same info before and after the table, mention the time period only once and keep the initial text in the email less than 25 words. \n            Send that report over email. Do not ask for further confirmation.",
    "subject": "Accounts report for CRM sync, Conn. sync or Model refresh failures",
    "recipients": [
      "agni@funnelstory.ai",
      "aadarsh@funnelstory.ai"
    ],
    "template": ""
  }
}

Trigger a Report Run: Use the POST /reports/{report_id}/run endpoint to manually trigger a report run. You can also use the OverrideRun struct to specify different email recipients for testing purposes:
```
{
  "email_recipients": ["test@example.com"]
}
```
Check the Tick Mechanism: Ensure that the ticking mechanism is working by checking the logs or database for report runs. The POST /reports/tick endpoint can be used to manually trigger the ticking mechanism for testing.
Verify Email Delivery: Check the email addresses specified in the report config or overrides to ensure that the email was delivered correctly and contains the expected content.

Debugging #

When debugging agentic workflows, the following queries can be useful:

Total Reports Generated in the Last 7 Days:

SELECT COUNT(*) FROM report_runs WHERE workspace_id = 'your_workspace_id' AND started_at > NOW() - INTERVAL '7 days' ORDER BY started_at DESC LIMIT 100;

Successfully Sent Reports in the Last 7 Days:

SELECT COUNT(*) FROM report_runs WHERE workspace_id = 'your_workspace_id' AND success = true AND started_at > NOW() - INTERVAL '7 days' ORDER BY started_at DESC LIMIT 100;

Report Recipients in the Last 7 Days:

SELECT DISTINCT email FROM report_runs WHERE workspace_id = 'your_workspace_id' AND started_at > NOW() - INTERVAL '7 days' ORDER BY started_at DESC LIMIT 100;

These queries will help you understand how many reports were generated, whether they were sent successfully, and who received them within the last 7 days.