Home AI agent Enable AI agent without blowing up your support inbox

Enable AI agent without blowing up your support inbox

Last updated on Feb 23, 2026

The safest way to deploy an AI agent is to start supervised and gradually give it more autonomy. Enable the AI agent, review its work, then expand as you build confidence.

New to AI agents? Read our Introduction to the AI agent guide first to understand the basics, including Knowledge sources, Guidance, and Shopify Actions.

1. Start with Require Approval Mode

Turn on Require approval in AI agent → Settings before enabling the agent on any inbox. In this mode, the AI drafts messages and plans actions, but nothing reaches the customer until a human agent reviews it.

For each pending item, your team can:

  • Approve the draft as-is

  • Adjust it (loads the draft into the reply editor so you can edit before sending. The AI stays on the conversation and picks back up when the customer replies.)

  • Hand over to take full control of the conversation

Conversations waiting for review show up in AI agent → Needs approval in the sidebar.

This lets your team see exactly how the AI handles real tickets without any risk of bad responses reaching customers.

2. Test in the Playground

Alongside live approvals, use the AI agent → Playground to test specific scenarios safely. The Playground simulates conversations using your actual Shopify customer data:

  • Chat directly with your AI agent using different customer scenarios

  • Replay existing conversations from your ticket history

  • Select specific customers to test order-related scenarios with real order data

  • Test Shopify Actions safely (simulated only, no real changes made)

Testing checklist:

  • Test common scenarios with 5-10 real customer messages

  • Use the Replay feature on historical tickets

  • Verify response accuracy, tone, and any automated actions

  • Check edge cases and unexpected customer phrasing

  • Use Revise answer to correct poor responses

Learn more: Read our detailed Playground guide for step-by-step testing instructions.

Need help writing effective guidance? Check our Knowledge & Guidance guide for detailed examples and best practices.

3. Write Guidance for Patterns You See

After a few days of approving AI responses, you'll notice patterns: the same types of questions keep coming up. Write guidance rules for these scenarios so the AI handles them consistently.

Start with your highest-volume, lowest-complexity requests:

  • "Customer asking about order status"

  • "Customer wants to cancel their order"

  • "Customer received wrong item"

  • "Customer asking about return policy"

  • "Customer reporting damaged product"

Test each guidance rule in the Playground, then continue monitoring through approval mode until you're confident.

4. Turn Off Approval and Restrict to Enabled Guidance

Once you have at least 5 well-tested guidance scenarios and you're confident in the AI's responses, you can turn off Require approval and let the AI respond autonomously.

To stay in control, enable "Restrict to enabled intents" in AI agent settings:

  • When enabled: The AI only responds to tickets that match your guidance scenarios. Everything else gets handed over.

  • When disabled: The AI may attempt to answer any message using general knowledge, which can lead to unexpected replies.

⚠️ Minimum 5 guidance scenarios required. With fewer than 5, the AI may incorrectly match customer messages to unrelated scenarios. The AI needs enough variety to distinguish between different types of requests.

Alternative: If you want the AI to handle a broader range of tickets using general knowledge, leave "Restrict to enabled intents" off. Just monitor responses closely.

5. Use Clear AI Agent Identification

Configure your agent's name and signature to make it obvious that responses come from AI, not a human agent.

Recommended format:

  • Name: "AI Assistant" or "[Your Company] Bot"

  • Signature: "This response was generated by AI. Need human help? Just ask!"

This sets proper customer expectations and reduces confusion if the AI responds incorrectly.

6. Disable Conflicting Automations

Before enabling your AI agent, turn off these features to prevent message conflicts:

  • Business hours settings that send automated responses

  • "Send message" automations such as auto-replies

These can trigger unnecessary messages and create confusing customer experiences when combined with AI agents.

7. Set Up Knowledge Sources

Before going live, review the knowledge sources your AI agent has access to:

Essential knowledge sources:

  • Help articles - Your help center content (can be kept in draft status)

  • Canned responses - Templates your human agents use

  • Text snippets - Short knowledge pieces covering policies, FAQs, and specific topics

  • Shopify products - Your product catalog (automatically synced)

  • Shopify website pages - Store content and policies

Knowledge management tips:

  • Review AI agent → Knowledge sources to see what's available

  • Exclude irrelevant or outdated content that might confuse responses

  • Add text snippets for common scenarios not covered elsewhere

  • Keep help articles in draft if you only want the AI to access them

Detailed setup: Our Knowledge & Guidance guide explains how to configure each knowledge source.

8. Configure Shopify Actions (If Needed)

If your guidance scenarios involve order changes, enable the appropriate Shopify Actions:

Popular starter actions:

  • Get latest order information

  • Add order tags

  • Change shipping addresses

  • Cancel unfulfilled orders

Setup requirements:

  • Navigate to AI agent → Actions to enable specific actions

  • If you installed Commslayer before August 2025, reauthorize your Shopify connection

  • Each action requires individual activation and condition setup

Important: Actions only execute when the AI agent is confident. Uncertain situations automatically escalate to human agents.

Learn more: Read our Shopify Actions guide for complete setup instructions and available actions.

Rollback Plan

Have a clear process for quickly disabling the AI agent if issues arise:

  1. Immediately disable problematic guidance scenarios in AI agent → Guidance

  2. Notify your support team of any customer confusion

  3. Review and adjust guidance instructions based on issues found

  4. Re-test thoroughly in the Playground before re-enabling

Good to Know

  • AI agents include natural response delays on emails and social media comments

  • The AI agent is programmed to hand over tickets when uncertain. This prevents hallucinations.

  • The AI agent can ask your team for input instead of handing over entirely. When stuck on a question it can't answer, it pauses and asks a human agent internally. Enable this in AI agent → Settings.

  • Regular Playground testing helps identify knowledge gaps and improve responses

Related Guides

Quick Start Checklist

  • Read the Introduction to the AI agent overview

  • Enable "Require approval" in AI agent settings

  • Enable the AI agent on your inbox

  • Review and approve AI responses for a few days

  • Write guidance for the most common ticket types you see

  • Test each guidance rule in the Playground

  • Once you have 5+ tested guidances, turn off approval and enable "Restrict to enabled intents"

  • Configure clear AI agent name and signature

  • Set up relevant Knowledge sources

  • Enable Shopify Actions if needed

  • Disable conflicting business hours and auto-reply settings

  • Monitor performance and expand gradually