Azure Integration Services and AI: Key Learnings from Integrate 2024

This year I once again had the pleasure of attending Integrate 2024 London, a conference which has always been important to me for both its technical content and unparalleled access to representatives from Azure Integration Services’ various Product Teams.

There were was a plethora of sessions from both the Microsoft delegates and mainstays of the Integration community alike but one session stood out in particular to me, one of Kent Weare’s, (Principal PM of Azure Logic Apps and all round stand up fellow).

His session was, quite frankly in my opinion, one of the most important sessions of the conference.

Kent’s session covered a lot of new features, but it was his AI Demo which was important, the broad strokes of what was demonstrated are as follows:

• Firstly, he delved into how Integration Services can be combined with Generative AI in Azure OpenAI to provide more powerful integrations with low code solutions.
• Secondly, he then showed how to implement the Retrieval and Augmentation Pattern (RAG Pattern) with Logic Apps. What would normally require a few hundred lines of Python was reduced to using about a dozen workflow designer shapes.
• Thirdly, he then enhanced this demo by integrating these workflows into an Azure OpenAI language model using the new (and still in preview) Assistants Feature.
• Lastly, using a simple web request client to call his language model and utilize those functions, demonstrating the back-end APIs for a specialized Co-Pilot experience tailored to his demo company.

Crucially I want to explain why his demo was so important.

From low-code to natural language processing

Ease of use

I have written some reasonably complex applications in the course of my career and I’m comfortable in creating both high-code and low-code solutions.

That said, having the option of building something with low-code which would ordinarily take me hours upon hours to do with High Code tools is well worthy of consideration when deciding what you’re going to use to build a given solution.

I’ll not harp on about this much longer, but ease of use democratizes the solutions to complex problems.

Using natural language to make requests

Kent just used natural language to make his requests.

This is critical!

To understand why, we need to take a step back from Co-Pilots for a moment and rewind time by about a decade.

“Since she’s powered by Bing, she understands the entire internet…”

Look familiar? Cortana was demo’d at Build 2014, when Chat Bots and Assistants were all the rage and were going to change the world. Let us not forget Cortana had contemporaries, Alexa, Siri et.al. All came with SDKs to build integrations to make them ostensibly “smarter”. What these SDKs actually accomplished was that the agents just became more capable, but definitely not smarter.

There are two faults which fundamentally scuppered many of these chat bots:

1. The agents would do what we said, not what we meant.
2. Our interactions with them were still very robotic and procedural.

We’ll take a moment to explore these faults a bit further and show where modern LLM’s bring improvements over their ancestors.

The limitations of existing voice assistants and chatbots in everyday tasks

Let’s take the Alexa unit in my kitchen – What I Said vs What I Meant

I like to listen to music sometimes when I cook. I have an Alexa unit in my kitchen, and she’s configured to integrate with Spotify. However, I can’t just ask Alexa to play something from my Spotify list using casual language.

My query must be specifically constructed to achieve the desired end goal.

The meaning (semantics) behind my request is irrelevant because it is the content and structure (syntax) of my request that matters. Put it another way the content must be specifically formatted for Alexa to derive intent. Consider the following sample command.

“Alexa, from Spotify play my Best of Brian Fallon and The Gaslight Anthem playlist.”

This command is onerous to articulate and not natural, but it works. I would much rather ask.

“Alexa, play my Brian Fallon playlist”.

This is because Alexa must be specifically told to launch Spotify, and she’s not smart enough to know my Brian Fallon playlist unless a get its name exactly right. Content and Syntax, vs Semantics.

To be clear I know I know, the underlying semantics of the API calls between Alexa and Spotify are far more nuanced as to why my request is hard to perform, but non-the less, as a consumer it’s a not great experience.

Ordering food from chatbots – Procedural Conversations

Once upon a time, I was testing a chatbot for ordering food from a fast-food giant. To say it was a cumbersome experience would be an understatement.

I couldn’t say “Order a medium cheeseburger, fries, and a strawberry milkshake, hold the gherkins.”

I had to move through a workflow where my responses to prompts would be in a structured and sequential manner.

The prompts:

1. What Burger would you like? “Cheeseburger.”
2. What size would you like. “Medium.”
3. Would you like any extras? “No.”
4. What sides would you like? “Fries.”
5. What size? “Medium” and so on and so on.

These experiences were not great, they’re slow, limited and they haven’t really in my opinion gotten much better as the years have passed. You’ll see a lot of these chat bot assistants online and they’re nearly always a poor replacement for a human being.

Transforming general-purpose LLMs into domain-specific Co-Pilots

Domain Knowledge Extensibility

Large Language Models (LLMs) can’t provide meaningful answers to questions on information they don’t know about. Or can they?

That’s what RAG is all about, pulling data into our models in a way in which they can then answer questions on that data, which they do elegantly.

To be clear, this is not my LLM regurgitating my data at me blindly, this is how my LLM can understand the semantics of that data and answer questions relating to that data with a high degree of accuracy. There’s a whole lot of vector mathematics and higher dimensional trigonometry involved which I won’t even pretend to understand in detail but in short RAG turns your data into vectors (a multidimensional array of floating-point numbers which plot a line). LLM’s turn your prompts into vectors and looks for comparable vectors to determine a response. The distance between 2 vectors determines how related the vectors are, with the shorter the better.

Action Extensibility Logic Apps and Azure Open AI Assistants

Azure Open AI Assistants allowed Kent to quickly refine his general purpose LLM into a domain specific Co-Pilot with what seemed little effort. By defining several Logic Apps and importing them as functions his LLM could retrieve data (using the RAG pattern) and even take actions in the real world based on natural language prompts.

Rather than his Co-Pilot just talking a good game it could go out and do it! The subtle but important part of this is that the model understood how and when to use the Functions it had been provided and didn’t just use them blindly.

I left that session deep in thought.

Kent had shown how easy it is to build domain specific agents/co-pilots/chat bots using PAAS and SaaS components in a way which could be tailored to any specific problem domain. This resulted in a Co-Pilot that was actually useful!

Not a clunky chat-bot experience we would have had to assemble with blood sweat and tears in years past for a passable user experience.

I was resolved to build one of my own.

Woke up and Chose to Hack

So, fresh from the conference (well relatively fresh, I had to take some days off on account of accidentally breaking my foot on the return trip!) I was eager to strike while the iron was hot and build a Co-Pilot.

The timing could not have been better as there was a Microsoft AI Hackathon coming up, a summer sequel to the Spring Hackathon I recently posted on LinkedIn about (a shout out to Robin Lester for organising these Hacks).

After making a case to our CTO, Javid Khan, who to be fair didn’t need much of a case to be made, CloudGuard formed a team for the Hack:

Myself, Jonathan Hartdegen and, Yakub Desai.

With Jav’s blessing we were off to the races!

Well, that Hack was last week, you might be wondering. How did it go?

In short, the Summer Hack was a blast. We each earned another badge to add to our nascent collections. Proud to show mine here.

We achieved nearly all our stretch goals in creating our Co-Pilot and I’ll be continuing to work on and improve that prototype in the near future.

Slightly longer story, there was some head bending concepts in there We went off script a bit (we ended up using both Semantic Kernel AND Azure OpenAI Assistants), but we achieved our goals and learned a lot about extending LLM’s because of it.

I am not going to go into depth on the Hack in this post (I will in the near future).

Wrap Up!

I wanted to conclude on a few points to which I hope I’ve given some evidence to support:

1. If you’re a Chief Technology Officer or Senior Manager with a training budget.
   Send your team to conferences! Why? To learn new things! To ideate! To Network! I am now 2 for 2 for Integrate attendances in the last 2 years where I have come back and built something because of the ideas and tools I have been shown at a conference.
2. If you’re a software professional.
   Make a case to go to conferences to your manager and make a case for doing some Hack events! Build, experiment, fail, succeed, learn! Hacks are like pressure cookers which make diamonds and as I’m rediscovering in my “renaissance” of Hackathon attendance they’re great for pushing yourself to new heights.

That about sums up this post. Thanks for reading all. As promised, I’ll post again soon with some more technical content on my working in Azure AI in the integration space but until then, take care.