Interested in Multi-Agent Infrastructure for Test Automation?

Check my short summary and insights from yesterday's webinar led by Founder of TestZeus

Yesterday, I attended a webinar led by Robin Gupta, Founder at TestZeus, where he presented their open-source platform leveraging AI Agents for test automation.

The multi-agent infrastructure of Hercules (that’s how the tool is called) for test automation operates by deploying a network of specialized AI agents, each designed to handle specific tasks within the testing process. This architecture is built upon the AutoGen framework, which facilitates the creation and management of these agents.

Key Components:

• Agents: Each agent focuses on a particular domain, such as UI interactions, API testing, or data validation. These agents work collaboratively, communicating and coordinating their actions to achieve comprehensive test coverage.

• Tools: Agents utilize a set of predefined tools to perform atomic actions, like clicking a button or sending an API request. Each tool execution returns a natural language description of its outcome, enabling agents to adapt their strategies based on real-time feedback.

Workflow:

1. Input: Users provide test scenarios in natural language or Gherkin format, detailing the desired end-to-end workflows to be tested.

2. Task Allocation: The system parses these scenarios and assigns specific tasks to the appropriate agents based on their specializations.

3. Execution: Agents independently execute their assigned tasks using the relevant tools. For instance, a UI agent might simulate user interactions in a web browser, while an API agent sends requests to verify backend responses.

4. Collaboration: Agents share their findings and statuses with each other to ensure synchronized progress and to handle dependencies between tasks effectively.

5. Reporting: Upon completion, the system aggregates the results from all agents, providing a comprehensive report of the test outcomes.

As a test automation enthusiast, I see great potential in this approach, which shifts the role of a quality engineer from writing detailed test scripts to simply defining the task—such as "create a lead in CRM"—while the Agents figure out how to achieve it.

The Determinism Debate: Rethinking Test Automation

One of the most frequently asked questions during the webinar was how we can ensure that the tests remain deterministic—that they follow the same execution path every time.

But do they really need to?

In the automated testing, our goal is to ensure that a user can complete a task successfully. The exact path doesn’t always have to be the same, as long as the outcome is correct. In fact, I see this flexibility as a strength rather than a weakness, especially when considering maintenance costs— since tests are written at a high level of abstraction, not every minor change in the code requires updates to the test suite. This minimizes the time and effort spent on maintaining brittle UI tests, which is often a bottleneck in traditional test automation strategies.

Key Questions That Need Answers

While the concept is promising, there are important challenges to address:

• Reliability: How do we ensure that the AI Agents are correctly flagging test assertions as passed or failed? If we rely on them to self-validate, we need robust mechanisms to verify their correctness.

• Cost: Of course we all know how much LLMs work, at this point I can't imaging running extensive test suites every change or even just every day.

• Performance: Web UI tests are already slow, and introducing a network of AI Agents that generate test content dynamically could slow them down even further. In a CI/CD context, where rapid feedback loops are crucial, execution time optimization is a must.

  Btw. in terms of CI/CD you can check my article:

The Bigger Picture: Automating the Entire SDLC with Agentic Infrastructure

If we extend this idea beyond test automation, we can envision an entire software development lifecycle (SDLC) driven by AI Agents:

• User Data Analysis: Agents analyze user behavior and suggest improvements.

• Task & Backlog Management: Automating prioritization and backlog refinement.

• Requirements Analysis & Test Case Generation: Agents generate Gherkin use cases, feeding the test automation process.

• System Architecture & Design: AI-driven architecture diagrams and technical specifications.

• Coding: AI Agents assist in writing and optimizing code.

• Quality Assurance: The AI-driven test automation system described above.

Next Steps

I plan to experiment with this tool and share my hands-on insights. There’s no doubt that AI-driven, multi-agent infrastructure is the future of test automation—and perhaps, eventually, the future of software development itself.

Oh…

here is the repo in case you are interested to check:
https://github.com/test-zeus-ai/testzeus-hercules