Analysis of 5 User Interview Transcripts - Day Zero & Day 30 Experience
This report synthesizes insights from 5 in-depth user interviews conducted with IBM professionals who work directly with watsonx Orchestrate customers. Participants included Client Engineers (Cameron, Joe, Ankitha), Account Technical Leader (Richard), Customer Success Engineer (Dana), Technology Sales Leader (Sai), and Chief Architect (Ahmed). The research focused on understanding pain points in the onboarding experience and gathering feedback on proposed Day Zero and Day 30 landing page designs.
Problem: The current model where all users see the same interface and have access to all features doesn't align with enterprise security and workflow expectations.
Joseph (Client Engineer):
"The problem is orchestrate everybody's access to everything, at least that's how it is. If you think of a traditional from a traditional RBAC enterprise approach, typically connections is something done by admins. Typically it's something that usually has somebody who actually has the credentials to access the external systems."
Joseph:
"An AI engineer typically does not have the credentials to connect to the back end systems. So if for example I wanna build an ask IT agent... the AI engineer designing the Ask IT solution wouldn't be able to connect to a ServiceNow instead because they usually don't have admin level access... and at that point basically they're stuck."
Impact:
Recommendation: Implement role-based landing pages with different views for Admin, AI Engineer/Builder, and Business User personas.
Admin Persona:
AI Engineer/Builder Persona:
Business User Persona:
Joseph:
"If somebody is their main job is to just put in credentials and manage credentials and make connections to systems, then they really don't want to see a lot of stuff that's here... Some people are just interested in looking at analytics and status and make sure everybody is green, everything is green... But they don't. They're not gonna build agents."
Problem: Connecting to external systems (tools and knowledge sources) is where users struggle most and spend the most time.
Joseph:
"A lot of questions will come in around like integration. Integrations and domain agents. Those are usually a lot of the questions that come up in terms of how can I do this, how can I connect to this external function, how to integrate with it."
Cameron (Client Engineer):
"I think integration points is definitely the hardest point to kind of figure out from, you know, looking around or the documentation. I think that's something that's kinda hard."
Common Integration Needs:
Recommendation: Create integration-focused quick start guides and make connection status visible to builders.
Cameron:
"Maybe like a top three cookbooks of, you know, here's what you can do with orchestrate... One using data ingestion with watsonx.data. How you connect to Milvus? Maybe you have another one about document processing things that like clients kind of come in with a specific use case and it's not immediately clear how to do that in orchestrate."
Suggested Cookbook Topics:
Problem: The proposed "Chat with documents" example is seen as generic RAG, not true agentic AI, and doesn't differentiate Orchestrate.
Joseph:
"Chat with documents have... I guess to some level people think it's old news and they don't think necessarily it's agents. Chat with documents is Gen. AI is really more generative AI than agentic AI... I think Orchestra can do a lot more than that. And so I think it gives us an idea about some things that Orchestra can do, yeah, but I think it may be not exposing the right capabilities."
Joseph:
"People will not be very excited about chat with documents because it's just been there for a long time. There are many, many solutions that exist and do that, but it's definitely something Orchestrator can do. But I don't think it's the only thing it's like it's main power."
Problem: Invoice extraction is too narrow and doesn't resonate unless users have that exact use case.
Richard (Account Technical Leader):
"We're only presenting invoice extractor as an option. And maybe, what's the purpose of that? Just help me understand that. Why would we present this?"
Sai (Sales Leader):
"If my job is to do pre-authorize, when I log in, if you show me this, these are interesting, but this is not my job. My job is to build agents around pre-auth. So invoice extractor and chat with documents is going to be meaningless."
Better Approach: Use "Document Extraction" as the category with invoice as an example, making it clear it applies to contracts, receipts, forms, etc.
Recommendation: Show Client Zero use cases that demonstrate multi-agent orchestration and tool integration.
Cameron:
"I think a lot of clients resonate with the client 0 strategy and they'll see like a demo and ask procurement, ask HR and they kind of get it. They get orchestrate and they're like, OK, so how do I do that? You know, so maybe if they saw a client 0 use case."
Suggested Examples:
Problem: Users don't understand the difference between Instructions, Guardrails, and Behavior fields.
Joseph:
"There are a lot of options to configure an agent. I don't think it's always clear to them what the different options mean... A lot of time we spend extra effort explaining what each step means. So what's the right way to create a description?"
Joseph:
"One question I don't even know very clearly is what's the difference between instructions and guardrails, right? And behaviour. It's still not clear to me. I mean I I don't use guardrails, but I know we added them for a reason, but I don't personally know what they're for and documentation says something but it's not very clear how to use those."
Impact: Users struggle to tune agents to work correctly, especially as complexity increases with more tools and knowledge.
Joseph:
"The other big effort that we deal a lot with is is is when things don't work, when basically you do something with the agent, you build something. And maybe your description you put is not very good, or maybe the tools you provided are not very very clear in terms of natural language, or the behavior you provided is not adequate."
Recommendation: Provide in-context help, examples, and AI assistance for writing effective descriptions, instructions, and guardrails.
Problem: "Create my first agent" and "Guide me to create an agent" feel redundant.
Cameron:
"Wonder if those 3 can all be consolidated into one, right? Like you're all creating an agent... It's not immediately clear to me what the difference between the the first create my own agent and then guide me to create my own agent. It feels like almost the same action."
Joseph:
"Create my agent and got my agent would be 1 tile and explore agent will be another tile. So there'll be two big tiles instead of four."
Joseph:
"As documentation, I think it's always good thing to provide, but in my mind you can provide that as a as as an agent in as a pop-up agent, right? It doesn't have to be right. I mean tile in in your first interface... You could have it up there a button next to the user profile or you could have it at the bottom agent at the bottom right."
Recommendation: Move documentation to a persistent help agent (bottom right) rather than a main tile.
Joseph:
"Again, who's your persona? If you're talking to the AI engineer, the answer is no. If you're talking to an admin, the answer is yes. So if you don't have separation personas, I would say no because your main user would be AI engineers."
Richard:
"It's kind of taken up real estate that doesn't matter. If we'll never use this screen after the first agent."
Cameron:
"A lot of people are kind of coming in with external agents. They're not building with Orchestrate and it feels like at least for the things I've heard from the product team or on the governance and things like that, that's almost like the new positioning because there's so many different apps that create agents nowadays."
Joseph:
"Maybe three ones, right? One is create my first agent, second is explored catalog, third one is bring my own agent or or yeah or or or yeah, integrated with my own and bring my own agent."
Key Success Factors:
Dana (Customer Success):
"I like this. This is very similar to what I had in mind as a second, you know, as a frequent user. I would naturally go to the needs attention, especially if I see higher risks and things like that. So, you know, view traces and so on. That would be what would attract me right away."
Dana:
"I think this is more of an end to end platform where you're doing all of the governance in-house... In the current orchestrate, because we didn't have agent monitoring. The analytics where clients were always saying the analytics were not enough."
Ahmed (Chief Architect):
"If you ask me the same questions, I would actually give the highest score actually for everything here. It's really amazing."
Problem: Day 30 dashboard focuses on monitoring but doesn't provide obvious way to create new agents.
Sai:
"This looks like an operational Asian dashboard to me... From here, I can't get to build an agent mode at all, right? Like, I mean, because if you look at everything you're saying is degradation of performance, root cause analysis, you want to fix the latest agent. This is inadvertently giving me that this is an operations team."
Ahmed:
"Where is where is the link here to build to create a new agent? Front and center... I guess this is nice for managing the existing agents... I guess it's just missing the creation, the development port."
Recommendation: Add prominent "Create Agent" button and access to agent catalog in Day 30 view.
Cameron:
"I think it would be helpful to have like a tutorial almost like maybe like a prebuilt agent that you can go into and preview off to the side. This is like my first agent and this has like you know an example knowledge base and things like that."
Richard:
"Having even a sample invoice extractor or a sample chat with documents where you could... I mean, you could even have this be not even like A scripted demo that's not really using the backend tools."
Ahmed:
"Maybe we could have more guided experience more than create my first agent... I like that you mentioned actually 3 to 5 minutes, because I guess the user would need something very fast actually in the beginning."
Positive Development: Team is building an AI agent trained on lab guides, documentation, and best practices.
Joseph:
"We have an agent that can help customers answer questions around those. So that's something that we're doing... It has three supporting agents. One of them is Ask Docs. It's integrated with all IBM docs and answer questions about that. The second agent is Ask Lab Guide... And then there's a third one for kind of like general best practices around building with the product."
Recommendation: Integrate this agent prominently in the onboarding experience.
Cameron:
"Maybe we could even do like when you first get on the page you give like a rating one to five how familiar you are with agentic tools or AI and then you know if someone gives A1. Then you can maybe have a chat with documents, 'cause that might be good for someone who doesn't really know what they're doing. But if you're coming in as an experienced AI engineer, you give it like a little four, like maybe don't show that option or show a more advanced option."
Recommendation: Consider skill-level based onboarding paths or at minimum show estimated time and complexity for each option.
Joseph:
"The main thing was the ease of building, right? It is easy to build an agent in a few clicks and you have something that seems to work, right? And I think that's a positive experience. That's the one that that's the main thing I would say, especially for somebody who's not a a programmer."
Goal: Maintain this strength while addressing integration and configuration challenges.
Joseph:
"The create my first agent doesn't give them any an expectation in terms of effort for how long it's gonna take them to build an agent... If we can do something maybe with effects or something in terms of like get started quickly or quick start versus or build an agent in whatever 20 minutes."
Cameron:
"Maybe even having like if you don't wanna do a survey, you can have like a little at the bottom of each tile estimated 20 minutes experience level medium or something like that, just so that they know what they're getting in into."
Joseph:
"Another key big item is evaluate my agent, right? Create my agent, explore the catalog, but create, evaluate my agent. And I think that's something could be a very interesting starting point because somebody may have built an agent and they just want to figure out how orchestra can help them evaluate and govern it... because that's a big blocker for a lot of people because it's easy to build, it's hard to govern."
Use Case: Users who built agents elsewhere (LangChain, custom code) want to use Orchestrate for governance and monitoring.
Recommendation: Add "Evaluate/Govern My Agent" as a primary Day Zero option alongside "Create" and "Explore Catalog".
Participants: 5 IBM professionals with direct customer-facing experience
Method: Semi-structured interviews with prototype walkthrough
Duration: 45-60 minutes per session
Focus Areas: