Quoting from https://opentelemetry.io/ – “OpenTelemetry is an open source observability framework for cloud native software. It provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application.”

Here we focus on Open Telemetry – or OTel for short – Traces.

The Anemic OTel Trace Antipattern

Due to an error in my demo prep, I ended up showing sparse OTel Traces – definitely not producing meaningful telemetry so observability will be subpar (or terrible). I am calling this the Anemic OTel Trace Antipattern. This antipattern comes through in the four screenshots that follow. The first screenshot shows the overall traces view listing one trace-per row. This is actually fine and normal as these are reasonable top-level traces. But drilling into any of these individual traces revealed no nesting and no context.

The Flat Trace OTel Trace Antipattern

Consider the traces below. If GetJob is triggered by an HTTP GET to the jobs endpoint, then my suggestion is they should be nested – GetJob under the corresponding HTTP GET /jobs/guid. As shown below they are flat, appearing as siblings rather than hierarchical. This is another OTel Trace Antipattern – let’s call it the Flat Trace OTel Trace Antipattern. We have this great “Trace”/”Span” nesting support, but still our signals look like old-school flat log entries. Definitely not optimal!

Properly Nested Traces and Spans

Let’s tighten up terminology. An OTel Trace represents the complete journey of a request through a system, and it is made up of one or more Spans that form a (logically nested) tree where each span is a unit of work. Within a trace it can make sense that some spans are siblings and others nested – it should mimic the actual flow through the system. The tree is reconstructed by following parent_span_id references. A trace can span multiple services (distributed tracing for the win!). Each service creates its own spans, propagating the Trace ID and parent Span ID via context propagation headers (e.g., traceparent in W3C Trace Context). Each Span in a Trace will share the same trace_id but have its own unique span_id.

So, using our vocabulary from above, the remedy for Anemic is to add more spans, and the remedy for Flat is to reuse span parents – passing them down to child processes rather than creating new spans.

With proper span structure, here is the SAME application again, except with OTel Traces and Spans more thoughtfully configured.

Now I can click on any of these and there will be spans nested within. You can tell the number and types of spans from the Spans column. The following span is from when the job was submitted: starts with an HTTP POST, stores some stuff in an Azure Storage Blob, creates a message in an Azure Storage Queue, then returns an HTTP 202 STATUS (“Accepted”) with a JobId.

Note above that movie we are requesting to assess is “best picture winner from 1988” – which is not a movie name you’ll find on IMDB. But a human will at least know what you mean. As will an LLM.

Now let’s double-click on the “RunJob” trace for the same movie – this is also around 20 seconds after the job was created since processing is asynchronous – and starts when our movie makes it to the front of the Azure Storage Queue queue:

Since we have an AI Agent, the movie request we made earlier (via HTTP POST) was asking to assess “best picture winner from 1988” and the name of the movie actually assessed was “Rain Man” as you can see. AI is working for us. For visibility in our monitoring and debugging, we added those details as properties in the OTel span. The helps us know exactly which business operation we are looking at when we review the telemetry.

Here’s one more span from the RunJob trace, this one showing some OTel Semantic Conventions for GenAI – the gen_ai.request_model and other span properties – but see also the next section for more on this:

Finally, here’s the trace for that same movie request being retrieved after processing has completed:

OTel GenAI Semantic Conventions

OTel drives consistency across solutions and vendors by specifying semantic conventions. Specifically for GenAI they specify many identifiers (see example above – two screenshots ago). In this screenshot you can see a bunch of them in action. For more information, check out these resources:

• https://opentelemetry.io/docs/specs/semconv/gen-ai/
• https://opentelemetry.io/docs/specs/semconv/gen-ai/azure-ai-inference/
• https://opentelemetry.io/docs/specs/semconv/gen-ai/mcp/

Source Code

• Repo is here: https://github.com/CrankingAI/goat-agent

Presentation

• PowerPoint Deck is here:

Connect

Connect with Bill

• LinkedIn
• WordPress
• Bluesky
• GitHub
• X

Connect with Boston Azure AI

• LinkedIn
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

Connect with Agent Framework Dev Project

• GitHub
• Link
• Twitter
• X

Thank You to the Speakers, Sponsors, and Organizers

Thank you to all the speakers:

Anirban Tarafder · Bala Subra · Bill Wilder · Bob German · Bryan Hogan · Chris Seferlis · Cole Flenniken · Dave Davis · Dave Finn · Dekel Cohen Sharon · Fnu Tarana · Gleb Bahmutov · Harry Kimpel · Jason Haley · Jeff Blanchard · Jesse Liberty · Jim Wilcox · John Miner · Joseph Parzel · Josh Goldberg · Juan Pablo Garcia Gonzalez · Keith Fitts · Matt Ferguson · Matthew Norberg · Michael Mintz · Pavan Kumar Kasani · Richard Crane · Sunil Kadimdiwan · Taiob Ali · Ty Augustine · Udaiappa Ramachandran · Varsham Papikian · Vijaya Vishwanath · Viswa Mohanty

And thank you sponsors:

Hosting: Microsoft · Gold: MILL5 · Silver: Pulsar Security · Progress Telerik · Triverus · Brightstar · In-kind: Sessionize

Making Agents Work

My session was Making Agents Work which highlighted some of “the boring side” of building an AI Agent – but these boring details can be super-valuable. The talk was inspired by work I did in my day job as CTO at Open Admissions. I am using an AI Agent to scale a 30 year-old methodology that can be used to help people understand themselves better and use those insights to choose a more aligned college, major, job, or other consequential life decision. Doing this with an AI Agent is a huge responsibility and, as I shared, putting together the initial agent was the easy part – being confident it is consistent, accurate, well behaved, robust if attacked or misused – but still easy to use – that was the hard and boring part!

The talk uses a different AI Agent – a simple one that accepts a movie and returns a rating summary – to illuminate some of the points. For example, it uses Agent Framework and has a fan-out/fan-in workflow in the internal agent architecture, uses Microsoft Foundry, a modern tech stack, and Azure Monitor for OTel-aligned Observability.

The full description, link to github repo, and slides follow.

But first, please find some elaboration on OTel Traces, inspired by my OTel demo snafu at the live event. That blog post is here: https://blog.codingoutloud.com/2026/03/30/otel-traces-for-the-win/

OTel Traces for the Win

Speaking of OTel… Due 100% to user error (that would be me!), the demo I had prepared to show the incredible power of OTel had a technical glitch. So I have attempted to remedy that with a blog post I’m calling OTel Traces for the Win. So please hop over there if you are interested.

Making Agents Work – the official talk description

Building more powerful AI Agents seems to be getting easier by the day. They are powered by incredible models, have access to tools, and can work in teams. But how can we have confidence in non-deterministic systems that make consequential decisions?

This talk explores four approaches for building that confidence.

1. Observability platforms – You can’t improve what you can’t see. We’ll explore tools that make the hard-to-see stuff visible.

2. Evals (evaluations) – Moving beyond LGTM (looks good to me), evals wrap agents in formal testing structures to measure accuracy, consistency, and edge case handling – both before and after your Agent goes live.

3. Safety guardrails – Content filtering, PII detection, and hallucination detection from both platform vendors and standalone models. Let’s see how they fit into your agent stack.

4. Selective determinism – Sometimes we make better AI solutions by knowing when NOT to use AI. We will discuss mixing in deterministic logic with our non-deterministic behaviors.

Concepts are platform-agnostic, but demos will use Microsoft Foundry and the Agent Framework (currently in preview). (In case you haven’t been following along, Microsoft Foundry was previously know as known as Azure AI Foundry, and before that was Azure AI Studio. And Agent Framework is the next generation of both Semantic Kernel and AutoGen.)

Target audience: Those new to building production agent systems seeking approaches beyond the “hello world” tutorials – which described me not too long ago.

Source Code

• The code used to implement the Movie Rating Agent is in the GitHub repo here: https://github.com/CrankingAI/movie-trivia-agent

Presentation

• The slides I presented are here:

Connect

Connect with Bill

• LinkedIn
• WordPress
• Bluesky
• GitHub
• X

Connect with Boston Azure AI

• LinkedIn
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

Connect with Agent Framework Dev Project

• GitHub
• Link
• Twitter
• X

The description, link to github repo, and slides follow.

Making Agents Work

Building more powerful AI Agents seems to be getting easier by the day. They are powered by incredible models, have access to tools, and can work in teams. But how can we have confidence in non-deterministic systems that make consequential decisions?

This talk explores four approaches for building that confidence.

1. Observability platforms – You can’t improve what you can’t see. We’ll explore tools that make the hard-to-see stuff visible.

2. Evals (evaluations) – Moving beyond LGTM (looks good to me), evals wrap agents in formal testing structures to measure accuracy, consistency, and edge case handling – both before and after your Agent goes live.

3. Safety guardrails – Content filtering, PII detection, and hallucination detection from both platform vendors and standalone models. Let’s see how they fit into your agent stack.

4. Selective determinism – Sometimes we make better AI solutions by knowing when NOT to use AI. We will discuss mixing in deterministic logic with our non-deterministic behaviors.

Concepts are platform-agnostic, but demos will use Microsoft Foundry and the Agent Framework (currently in preview). (In case you haven’t been following along, Microsoft Foundry was previously know as known as Azure AI Foundry, and before that was Azure AI Studio. And Agent Framework is the next generation of both Semantic Kernel and AutoGen.)

Target audience: Those new to building production agent systems seeking approaches beyond the “hello world” tutorials – which described me not too long ago.

• Repo is here: https://github.com/CrankingAI/goat-agent
• Deck is here:

Connect with Bill and Boston Azure AI

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

First, thank you to the organizers, sponsors, and speakers who have been making this possible since 2003!

MCP – Model Context Protocol – is coming up on its first birthday and adoption is currently on fire accelerating the creation and adoption of new MCP servers.

Photo above from Robert Hurlbut’s LinkedIn post.

Anthropic’s original MCP specification:

• https://www.anthropic.com/news/model-context-protocol

Tools and Libraries for building, testing, and consuming MCP servers:

• https://github.com/modelcontextprotocol/csharp-sdk
• https://learn.microsoft.com/en-us/dotnet/ai/get-started-mcp
• https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/telemetry
• https://modelcontextprotocol.io/docs/tools/inspector

Registries of MCP Servers (these are a couple of examples of reputable ones, but be cautious about any registries, especially rando registries out there!):

• https://registry.modelcontextprotocol.io
• https://github.com/mcp

Photo above courtesy of Udaiappa Ramachandran (who runs https://www.meetup.com/nashuaug/).

Talk description:

Agency is the capacity to act autonomously, make choices, and shape outcomes. The Model Context Protocol (MCP) brings this agency to AI systems at scale.

In this session, we’ll explain the gap MCP fills, highlight key use cases, and explore the rapidly growing ecosystem of tools and marketplaces. We’ll demonstrate MCP in action and walk through how an MCP tool is built and deployed.

You’ll leave knowing what MCP is, why it matters, and how it connects systems and data to make AI more effective – and more agentic. And as Spider-Man reminds us, with great power comes great responsibility: we’ll close by looking at the risks and governance challenges.

Above photo from Veronika Kolesnikova’s post.

I had the opportunity (22-Nov-2025) to present at the 39th running of Boston Code Camp since started in 2003.

And the deck is here:

Connect with Bill and Boston Azure AI

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

The tool used in demos to illustrate an embedding model in action can be found at:

funwithvectors.com.

And the deck is here:

Connect with Bill and Boston Azure AI

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

What I didn’t realize at first was that these would be the same demo.

I started with a solid sample C#/.NET MCP server ready to be deployed as an Azure Function (one of several languages offered). The sample implemented a couple of tools and my goal was to implement an additional tool that would accept an IP address and return the country where that IP address is registered. The IP to country code mapping functionality if available as part of Azure Maps.

I started to hand-implement it, then… I decided to see how far GitHub Copilot Agent mode would get me. I’ve used it many times before and it can be helpful, but this ask was tricky. One challenge being that there was IaC in the mix: Bicep files to support the azd up deployment, AVM modules, and many code files implementing the feature set. And MCP is still new. And the MCP support within Azure Functions was newer still.

Give GitHub Copilot Agent a Goal

The first step was to give the GitHub Copilot Agent a goal that matches my needs. In my case, I gave Agent mode this prompt:

The .NET project implements a couple of Model Context Protocol (MCP) tools – a couple for snippets and one that says hello. Add a new MCP tool that accepts an IPv4 IP address and returns the country where that IP address is registered. For example, passing in 8.8.8.8, which is Google’s well-known DNS server address, would return “us” because it is based in the USA. To look up the country of registration, use the Azure Maps API.

And here’s what happened – as told through some screenshots from what scrolled by in the Agent chat pane – in a sequence that took around 12 minutes:

I can see some coding progress along the way:

A couple of times the Agent paused to see if I wanted to continue:

It noticed an error and didn’t stop – it just got busy overcoming it:

It routinely asked for permissions before certain actions:

Again, error identification – then overcoming errors, sometimes by getting more up-to-date information:

Second check to make sure I was comfortable with it continuing – this one around 10 minutes after starting work on the goal:

In total 9 files were changed and 11 edit locations were identified:

Deploy to Azure

Using azd up, get it deployed into Azure.

Add MCP Reference to VS Code

Once up and running, then I installed it in VS Code as a new Tool – first click on the wrench/screwdriver:

Then from the pop-up, scroll the the bottom, then choose + Add More Tools…

Then follow the prompts (and see also instructions in the GitHub repo):

Exercise in VS Code

Now that you’ve added the MCP server (running from an Azure Function) into the MCP host (which is VS Code), you can invoke the MCT tool that accepts an IP and returns a country code:

domain-availability-checker% dig A en.kremlin.ru +short
95.173.136.70
95.173.136.72
95.173.136.71
domain-availability-checker%

Using the first of the three returned IP addresses, I ask within the Agent chat area “where is 95.173.136.70 located?” – assuming that the LLM used by the chat parser will recognize the IP address – and the need for a location – and figure out the right MCT tool to invoke:

I give it one-time permission and it does its thing:

Victory!

Check Code Changes into GitHub

Of course, using GitHub Copilot to generate a commit message:

Done!

Connect with Bill and Boston Azure AI

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

There is a lot in that statement, so let me elaborate.

First, what do you mean by “the best way to allow my LLM to call my code” — why is the LLM calling my code at all? Don’t we invoke the LLM via its API, not the other way around? Good question, but LLMs can actually invoke your code. Because this is how LLMs are empowered to do more as AI Agents. Think about an AI Agent as an LLM + a Goal (prompts) + Tools (code, such as provided by MCP servers). The LLM uses the totality of the prompt (system prompt + user prompt + RAG data + any other context channeled in via prompt) to understand the goal you’ve given it then it figures out which tools to call to get that done.

In the simple Azure AI Agent I presented, its goal is to deliver an HTML snippet that follows HTML Accessibility best practices in linking to a logo it tracks down for us. One of the tools is web search to find the link to the logo. Another tool validates that the proposed link to the logo actually resolves to a legit image. And another tool could have been to create a text description of the image, but I made the design choice to leave that up to the Agent’s LLM since it was multimodel. (My older version had a separate tool for this that used a different LLM than the one driving the agent. This was an LLM with vision capabilities – which is still a reasonable idea here for multiple reasons, but kept it simple here.)

Second, what do you mean by “in a standard way” – aren’t all LLMs different? It is actually the differences between LLMs that drives the benefits of a standard way. It has been possible for a while to allow your LLM to call out to tools, but there were many ways to do this. Now doing so according to a cross-vendor agreed-upon standard, which MCP represents, lowers the bar for creating reusable and independently testable tools. And marketplaces!

Remember many challenges remain ahead. There are a few others in the deck, but here are two:

First screenshot reminds that there are limits to how many MCP tools an LLM (or host) can juggle; here, GitHub Copilot currently is capping at 128 tools, but you can get there quickly!

Second screenshot reminds that these are complex operational systems. This “major outage” (using Anthropic’s terminology) was shortly before this talk so complicated my planned preparation timel. But it recovered before the talk timeslot. Phew.

Connect with Bill and Boston Azure AI

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

Links from the talk

1. Assorted Cranking AI resources ➞ https://github.com/crankingai
2. Code for the Agent ➞ https://github.com/crankingai/logo-agent
3. Code for the Logo Validator MCP tool ➞ https://github.com/crankingai/logo-validator-mcp
4. Code for the Brave Web Search MCP tool ➞ https://github.com/crankingai/brave-search-mcp
5. Images I used in the example ➞ https://github.com/crankingai/bad-images (https://raw.githubusercontent.com/crankingai/bad-images/refs/heads/main/JPEG_example_flower-jpg.png)

Anthropic status page ➞ https://status.anthropic.com/ (see screenshot above).

Model Context Protocol (MCP) Resources

Standards & Cross-vendor Cooperation

• Model Context Protocol Introduction – Official introduction to MCP, described as “a USB-C port for AI applications” that standardizes how applications provide context to LLMs.
• Anthropic’s Model Context Protocol Announcement – The original announcement from Anthropic open-sourcing MCP as a universal standard for connecting AI systems with data sources.
• Official C# SDK for Model Context Protocol – The official C# SDK maintained in collaboration with Microsoft, available as a NuGet package: ModelContextProtocol.
• Microsoft’s Partnership with Anthropic for C# SDK – Microsoft’s announcement of their collaboration with Anthropic to create the official C# SDK for MCP.
• https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/ – A2A – Agent to Agent protocol
• https://techcrunch.com/2025/05/07/microsoft-adopts-googles-standard-for-linking-up-ai-agents/ – A2A from Google, being adopted by Microsoft
• Nanda Lab at MIT Media Lab – Research lab working on projects related to AI systems and tools at scale – a highly distributed AI future

SDKs & Samples

• Integrating MCP Tools with Semantic Kernel – A step-by-step guide for integrating MCP tools with Microsoft’s Semantic Kernel framework.
• Azure AI Agent with Semantic Kernel (C#) – Microsoft documentation on building AI agents with Azure and Semantic Kernel using C#.
• MCP Examples – Official examples of using the Model Context Protocol in various scenarios.

MCP Servers & Implementations

Popular MCP Servers

• GitHub MCP Server – GitHub’s official MCP server that provides seamless integration with GitHub APIs for automating workflows, extracting data, and building AI-powered tools. In case you’d like to create a Personal Access Token to allow your GitHub MCP tools to access github.com on your behalf ➞ https://github.com/settings/personal-access-tokens
• Playwright MCP Server – Microsoft’s MCP server that provides browser automation capabilities using Playwright, enabling LLMs to interact with web pages through structured accessibility snapshots.
• MCP Servers Repository – Collection of official reference implementations of MCP servers.
• Popular MCP Servers Directory – Curated list of popular MCP server implementations.

MCP Inspector Tool ➞ Check this out for sure

• MCP Inspector – Interactive debugging tool for testing and inspecting MCP servers.
• MCP Inspector on GitHub – Source code repository for the MCP Inspector tool.

Download the deck from the talk ➞

I delivered a new iteration of my talk on how it is possible to have Human Language as the new UI, thanks to LLMs and Embedding models. There was an engaged and inquisitive group! The resources I used during the presentation, including my deck, are all included below.

Connect with Bill or other related resources:

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

Links from the talk:

1. Assorted Cranking AI resources ➞ https://github.com/crankingai
2. The funwithvectors.com app used in the talk ➞ https://funwithvectors.com and OSS repo
3. The repo with code for the “next-token” project that I used to show how tokens have probabilities and how they are selected (and can be influenced by Temperature and Top-P which is also known as nucleus sampling) ➞ https://github.com/crankingai/next-token
4. The OpenAI Tokenizer shown in the talk ➞ https://platform.openai.com/tokenizer/

The deck from the talk:

1. The deck from the talk ➞

It was similar to, but not identical to, the recent talk I gave. And next time it will be different again.

This is from the https://funwithvectors.com app I used to show vectors in action:

┃┃┃┃┃┃┃┃┃┃┃┃┃······· ⟪0.64⟫ → ‘doctor’ vs ‘physician’
┃┃┃┃┃┃┃┃┃┃┃┃┃······· ⟪0.67⟫ → ‘doctor’ vs ‘dr.’
┃┃┃┃┃┃┃┃┃┃·········· ⟪0.48⟫ → ‘physician’ vs ‘dr.’

The above is intended to illustrate the non-transitive nature of the “nearness” of two vectors. Just because “doctor” & “physician” are close and “doctor” & “dr.” are close does NOT mean “dr.” & “physician” are as close.

Connect with Bill or other related resources:

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

Links from the talk:

1. Cranking AI resources (including source to funwithvectors.com app) ➞ https://github.com/crankingai
2. The funwithvectors.com app used in the talk ➞ https://funwithvectors.com
3. The OpenAI Tokenizer shown in the talk ➞ https://platform.openai.com/tokenizer/

The deck from the talk:

1. The deck from the talk ➞ https://blog.codingoutloud.com/wp-content/uploads/2025/04/memphisglobalai-humanlanguageisnewui-25-apr-2025_pub.pptx

This is from the https://funwithvectors.com app I used to show vectors in action:

Connect with Bill or other related resources:

• LinkedIn
• WordPress
• Bluesky
• GitHub
• Meetup
• Twitter
• X
• YouTube

Links from the talk:

1. Cranking AI resources (including source to funwithvectors.com app) ➞ https://github.com/crankingai
2. The funwithvectors.com app used in the talk ➞ https://funwithvectors.com
3. The OpenAI Tokenizer shown in the talk ➞ https://platform.openai.com/tokenizer/

The deck from the talk:

1. The deck from the talk ➞

Today I attended and contributed a talk to the Boston Code Camp 38 (yes, impressively, the 38th edition of this event). I made the trip with Maura (she gave a talk combining Cryptocurrency And Agentic AI) and Kevin (he gave a talk on Top 10 AI Security Risks). and we got to hang out with and chat with so many cool people from the Boston technology community.

The description of my Let’s Build a Goal-Oriented AI Agent Using Semantic Kernel talk follows, followed by a couple of relevant links, then the slide deck.

Imagine an AI not limited to answering individual questions or chatting, but actively working towards a goal you’ve assigned to it.

In this session, we’ll explore the building of an AI Agent – an autonomous actor that can execute tasks and achieve objectives on your behalf.

Along the way we will demystify:

1. LLMs – What is a Large Language Model (LLM)
2. Tokens – What is a token and what are its roles
3. Embeddings – What are embedding models and vectors and what can they do for us
4. Prompts – Beyond the basics
5. Tools – How can these be created and accessed using Semantic Kernel
6. Agents – Let’s put all these concepts to work!

The end result will be the core (or perhaps ‘kernel’ ) of an AI Agent – your virtual coworker willing to handle tasks on your behalf without. It will be built in C# using the open source, cross-platform Semantic Kernel library.

This talk assumes user-level familiarity with LLMs like ChatGPT or Microsoft Copilot and basic prompting. Anything else will be explained.

(stole photo from Robert Hurlbut)

A couple of prominent links from the talk are:

You can find the Fun with Vectors tool here: https://funwithvectors.com/

You can find the OpenAI Tokenizer tool here: https://platform.openai.com/tokenizer

Download the slides here:

In between the initial announcement in 2008 and the GA date in 2010, Boston Azure was born. On Oct 22, 2009, Boston Azure debuted as the first community group of its kind – the first one dedicated to learning about the Azure platform. As of this writing, it has around 3.500 members according to Meetup.com.

(For a long time after we started hosting events we had people attending other events see our signage and curiously pop their head in to ask “What’s Azure?” When I’d answer “that’s Microsoft’s public cloud platform” they would very often react with a puzzled look and a follow-up question: “Cloud? What’s a cloud?” So yes, those were early days.)

That first event was held at the Microsoft NERD building in Cambridge MA. Mike Werner said a few words, I gave a short talk about cloud benefits and the coming opportunity (and somehow managed to reference “the Internet is … a series of tubes” comment by Alaska Senator Ted Stevens) and Brian Lambert (@softwarenerd) was the featured speaker who talked about queuing patterns in Azure Storage which was part of my journey of getting interested in cloud-relevant patterns (which culminated in me writing a book – Cloud Architecture Patterns – a few years later). Michael Stiefel was Boston Azure’s second-ever speaker.

Heady days from 2008 to 2010!

The times they are a-changin’

“You better start swimmin’ or you’ll sink like a stone, for the times they are a-changin’.” –Bob Dylan (https://www.youtube.com/watch?v=DL_kPNFL3dY and do yourself a favor and check out the movie A Complete Unknown).

A few things have changed since then. The PDC conference is no longer – though content has been subsumed into the Build conference. Windows Azure is now just Azure. There are hundreds of Azure services, not two. And Ray Ozzie is no longer at Microsoft (but has the Blues in the best sense of the word).

And Boston Azure is still at it. We’ve delivered more than 150 free events and still going strong. Now also delivering events virtually since the you-know-what made in-person events so difficult. Back in the early days George Babey and Nazik Huq signed on to help me run things. These days – and for some time now – our leadership team is Jason Haley (@haleyjason), Veronika Kolesnikova (@veronika_dev1), and me.

But technology continues to evolve, and we need to evolve too. Today Artificial Intelligence is playing a role similar to the role played by public cloud platforms back 15 years ago: everything is different so what does that mean? what will come out next? what does this make possible? how can I make use of this? how do I learn this stuff? This is exciting, right??

Changing Call Sign to Boston Azure AI

With no end in sight for AI to slow down, the three of us running Boston Azure – Veronika Kolesnikova (@veronika_dev1), Jason Haley (@haleyjason), and myself (@codingoutloud) – have decided to update our community group for 2025 by changing call signs – Boston Azure is now Boston Azure AI.

We’ve been emphasizing AI topics for a while already. Veronika is a long-time Microsoft MVP for AI, Jason is a long-time Microsoft MVP for Azure who last year was reclassified to the AI category, and myself as a long-time Azure MVP was re-classified last year as Dev Tools (presumably due to giving so many talks on GitHub Copilot, the AI coding assistant, in the prior year), so this emphasis also aligns with where the group’s leadership is spending time. At any rate, this rename should at least help us more clearly communicate to the community what we intend to offer.

Where to Find Boston Azure AI

With the rename, we are retooling some of our properties. Some are new, some are renamed from bostonazure version. You can find us at the following destinations:

• X/Twitter:  https://x.com/bostonazureai – renamed, so if you followed before you are still following
• Bluesky: https://bsky.app/profile/bostonazureai.org – yeah, we did the fancy domain version
• YouTube:  https://www.youtube.com/@bostonazureai (renamed) – we have more than 50 videos posted
• GitHub:  https://github.com/bostonazureai – created a new GitHub Organization for this and will be migrating over the old content, including the C# + Semantic Kernel + Azure OpenAI hands-on workshop materials shortly (see bottom of this post for more – we are running an event on Jan 31)
• Meetup:  https://meetup.com/bostonazureai – renamed, so if you were a member before you are still a member
• Website: https://bostonazureai.org – coming soon
• Email: hello@bostonazureai.org – we used a gmail address for the first 15 years, but now we are getting fancy with the bostonazureai.org domain. Hit us up if you want to offer a talk (in person or virtual) or have a suggested topic for us!

And, fittingly, we also have a shiny new logo.

Hands-on AI Coding Workshop: C#, .NET 9, and Semantic Kernel on Azure OpenAI

In another evolution, Jason Haley and I are experimenting with offering Boston Azure AI in-person hands-on AI coding workshops during the workday. The community events we’ve historically offered have been only nights and weekends – non-working hours. We’ll see how this works out. We have our second such in-person during-the-workday hands-on coding workshop focused on using C#, .NET 9, and Semantic Kernel on Azure OpenAI coming up on Fri Jan 31, 2025 held in Cambridge MA. You can sign up here. Free.

And we have a weekend event on the schedule to participate in Boston Azure edition Boston Azure AI edition of the Global AI Bootcamp in March. You can sign up here. Free.

Buckle in. Looking forward to an exciting next few years!

Last night I was guest speaker at the Boston .NET Architecture community group. I learned they are now 21-year-olds. That’s a long track record! The audience had some insightful questions, which I always appreciate.

My talk focused on the perspective of the application architect – and not the data scientist, for example – in how the process works and what are some areas I would need to dig into.

Here’s the alternative talk description I offered a few days ago:

Interested in understanding how LLMs are created and how they work internally, including all the in-depth data science and machine learning techniques? If so, then this is not that talk. Rather, this talk steps back to treat the LLM as a black box. And then steps further back to treat the LLM as a part of a cohesive system offered over the internet through an API. It is from that perspective that we begin our exploration.

How exactly does an application make use of LLM services? Is this thing secure? Is it private? Am I operating according to Responsible AI principles? (Oh, and what are Responsible AI principles?) Is it accurate? Is it portable? And of course, when does it stop being a Chatbot and start being an Agent?

These are some of the key types of application architecture considerations we will discuss as we start with “the humble chatbot demo” then turn it into an Agent and then see what it would take to put that into production.

The deck is here:

The recording is here: https://youtu.be/UJutO4eFLZg

There was an incredibly energetic group of inquisitive people at my talk which was on how you can give your AI LLM a goal and some tools and let it figure out how to move ahead! Lots of questions came from this highly engaged group.

The details of my talk follow.

Building an AI Agent with Semantic Kernel

The classic approach to managing complexity is through abstraction. While also useful in the physical world (you can know how to use a “car” without needing to know about all the parts under the hood), it is an essential tool in software.

To program against the current generative AI models you can use the model’s native abstraction (their SDK). But there are other options too, one of which is to use Semantic Kernel, an open-source library from Microsoft.

In this talk we will understand the first-class abstractions representable using Semantic Kernel, from the granular Function and building up to an Agent, and a couple of steps in between.

This talk will be a mix of explaining AI-relevant and Semantic Kernel-relevant topics + some explanatory sample code. We may also sneak in a little Prompty.

By the end of this talk you will appreciate why you might (or might not) want to build your AI solution with Semantic Kernel (SK) and how you would approach it.

This talk will assume you have used LLMs (like ChatGPT or others) and know the very basics of iterating on prompts and experiencing that GenAI systems have an ability to make decisions from human language. Anything beyond this will be explained in the talk.

The sample application used in the talk can be found here:

https://github.com/semantickerneldev/icon-agent

The deck used in the talk can be found here:

Much of my talk was focused on how to use Semantic Kernel (in C# and .NET 8) to give your AI a goal and have it solve it. The deck I presented and a recording of the talk follow. <I will likely update this post to link to code used in demo and as other artifacts become available>

Jason Haley’s code and materials are here: https://jasonhaley.com/2024/06/25/boston-azure-june-2024/

For my part, I pulled content from Generative AI for .NET Developers and Getting Started with Azure AI Studio and blended in some of my own. The combined mega-deck is attached to this post, though the deck spans much more than I had time to go through.

If you attended and have not had opportunity to give some feedback to Microsoft, there are only a few quick questions.

Take the survey here: aka.ms/AttendeeSurveySeasonOfAI

Additional Resources

Also complements of the Season of AI team, check out these resources.

Join the Azure AI Community on Discord

Connect with fellow enthusiasts, engage with Microsoft experts and MVPs, discuss your favorite sessions, and delve into AI discussions. Your space to ask, share, and explore!

aka.ms/AzureAI/Discord

Get started skilling with AI on Microsoft Learn

Build AI skills, connect with the community, earn Microsoft Credentials, learn from experts, and take the Cloud Skills Challenge.

aka.ms/LearnAtAITour

Download Deck Bill Presented

A diagram we spent time on (slightly updated, source here):

And finally, the deck I used follows:

And here are a couple of the links I showed during the talk that got a lot of discussion or attention:

1. https://platform.openai.com/tokenizer
2. https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4

Recall the third one shown – Telugu – was wildly more expensive (in terms of token count) than English (50 tokens) and Chinese (75 tokens) – where Telugu weighed in at 353 tokens.

• Veronika and Jason’s highlights can be found at https://jhaleysftp.blob.core.windows.net/files/MicrosoftBuild2024Recap.pdf.
• My highlights are covered in this deck attached to this post.

Here’s a direct link to the video on the Boston Azure YouTube channel: https://www.youtube.com/watch?v=odwHlnk_tzI and the same video is embedded immediately below.

This weekend I create a web front-end for it and deployed as an Azure Static Web App with an Azure Function supporting the refactored C# logic to execute the Azure OpenAI service calls.

The new app is running here: https://hello-ai.doingazure.com

You can find the source code here: https://github.com/codingoutloud/hello-ai

Note that while the additional grounding fails to stop all of the hallucinations, it does help with the most obvious one (so we are making progress) but there’s more to be done.

After Pamela’s talk around 15 people stuck around to participate in our first ever “AI mini-workshop” hands-on experience. The remainder of this post is about that mini-workshop.

The AI mini-workshop was a facilitated hands-on coding experience with the following goals:

1. Demystify Azure OpenAI

As background, OpenAI’s ChatGPT burst onto the scene in November 2022. That led to an explosion of people learning about AI and associated technologies such as “LLMs” which is the common shorthand for Large Language Models.

The vast majority of people interact with LLMs via chat interfaces such as available from OpenAI’s public version of ChatGPT or via Copilot on Microsoft Bing search. There’s also a more integrated programming experience surfaced through GitHub Copilot for use with VS Code and several other popular IDEs.

But what about programming your own solution that uses an LLM? Microsoft has done a great job of providing an enterprise-grade version of the OpenAI LLM as a set of services known as Azure OpenAI.

The first goal of this AI mini-workshop was to demystify this programming experience.

This was accomplished by giving the mini-workshop participants a working C# or Python program that fit on a page. And there are only around 10 lines of meaningful code needed to interact with the AI service. This is NOT that complex.

Creating a production-grade application has additional requirements, but at its core, it is straight-forward to interact with Azure OpenAI service programmatically.

The hoped for “Aha!” moment was this:

Aha #1! I can do this! I can programmatically interact with the Azure OpenAI LLM. It isn’t that mysterious after all.

Aha #2! This is possible without much code! In the Python and C# solutions shared there were only around 10 lines of core code.

2. Understand Some AI Concepts

Part of the mini-workshop exercise was to recognize a hallucination and fix it through some additional grounding using a very simple form of RAG.

The hope here is for some “Aha!” moments:

Aha #3! Here’s a concrete, understandable example of a hallucination!

Aha #4! And here’s a concrete, simple example use of RAG pattern to better ground the AI so that it no longer hallucinates about today’s date! But do note that other hallucinations remain…

3. Wield Great Power

The ability to program a LLM to generate unique content is something that essentially NO DEVELOPER COULD DO, EVER, before the super-powerful LLMs that were developed at costs of hundreds of millions of dollars and democratized by the Microsoft Azure OpenAI services (as well as by OpenAI themselves).

The hands-on AI mini-workshop required either (a) a functional Python 3 environment, or (b) a functional C#/.NET environment – everything else was provided, including sufficient access to the Azure OpenAI LLM service to complete the mini-workshop.

But in the end with very little coding you can get to the 5th Aha! moment which is:

Aha #5! I have at my command capabilities that have not been possible in all of the history of computers. The magic of LLMs available via Azure OpenAI gives me superpowers that we are only in the very beginning of understanding the ways this can be put to use.

The source code for the AI mini-working is available here. Note that the API key has subsequently been rolled (invalidated), but the code works pretty well otherwise.

My original thinking was to distribute the keys separately (like this). If this was an in-person workshop I would have kept the configuration values separated from the source, but given the added challenge of doing this with an online distributed audience I decided to simplify the mini workshop by included the configuration values directly in the source code. Looking back, I believe it was a good concession for minimizing obstacles to learning. So I’d do it again next time.

Always great to hang out with the greater Boston tech community. Today I attended and contributed a talk to Boston Code Camp 36 (the 36th edition of this event).

I made the trip with Maura (she gave a talk on blockchain). and we met a lot of cool people and had a great time.

I spoke on GitHub Copilot. Much of my talk was demo and discussion – you have to see this in action (or use it) to appreciate what’s happening. I consider this a glimpse into the future – it will surely become then norm to have an AI assistant when programming.

It is fun have one AI (GitHub Copilot) help us program another AI (Azure OpenAI). After Copilot Chat was able to explain that Azure OpenAI did not have any notion of “today” we used Copilot to implement a trivial version of RAG to anchor the prompt to the current day.

We saw how the agents like @workspace can explain a body of code and even help us figure out where to implement a new feature (such as the --joke command line param).

Another demo was to get Copilot to write unit tests for me. The demo gods were not helpful and I ran into an error. I moved on without fixing it since time was short. I diagnosed it later and it turns out I had double-pasted (classic demo failure!) which caused the problem. We did use /tests to create unit tests, which were initially NUnit test, but then we asked Copilot to recast them as xUnit tests, then to more efficiently create test cases using the InlineData attribute to consolidate similar test cases.We didn’t get to run the tests at the end, but hopefully the power of GitHub Copilot in helping to create unit tests came through.

I also had the opportunity to hang out with some smart soon-to-be graduates from my alma mater – University of Massachusetts at Boston (some of them were Rohini Deshmukh, Master’s in Information Technology, Kunal Sahjwani, Master’s in Information Technology, and Shounak Kulkarni, Master’s in Business Analytics). Great to see our profession is in such capable hands from chatting with these very smart and capable technologists, analysts, and future leaders.

Here is the published talk abstract for the talk I delivered – and though much of the session was demos, the PowerPoint deck is attached after the abstract.

Meet GitHub Copilot, Your AI Coding Assistant

Imagine an assistant who anticipates your needs as you code, handling mundane and time-consuming steps, allowing you to focus on more complex challenges (the fun stuff). Allow me to introduce you to GitHub Copilot.

GitHub Copilot is an AI-powered coding assistant that integrates with your developer IDE adding many powerful productivity features. Backed by the same OpenAI Large Language Model (LLM) behind ChatGPT, it has an uncanny ability to suggest code snippets that you were about to type in. But suggesting code snippets is just the beginning.

In this demo-heavy talk, we’ll show usage basics, distinguish scenarios where it excels vs. some it finds challenging, and point out a few common anti-patterns so you can avoid them.

Since it is still early days, big features are still showing up at a fast clip, so we’ll highlight some recent features and some others just emerging. At the end we’ll squeeze in just a bit of prognosticating about what it might mean for the future of software development.

As you’ll learn in more depth during this session, the promise of GitHub Copilot is to help you be more productive – go faster, stay in flow, build even more robust software. We are not fully there but we are on the way. This imperfect tool is still a game changer.

I believe the rise of AI coding assistants is unstoppable at this point – and that’s a good thing. By the end of this session, you might agree. And maybe you’ll join the revolution early.

Had a great time hanging out with the Orlando tech community at their annual Code Camp. I made the trip with Maura (she gave a talk on blockchain) and we met a lot of cool people and had a great time.

I spoke on GitHub Copilot. Much of my talk was demo and discussion – you have to see this in action (or use it) to appreciate what’s happening. I consider this a glimpse into the future – it will surely become then norm to have an AI assistant when programming.

One part of the demo showed using one AI (GitHub Copilot) to help program another AI (Azure OpenAI).

It is invigorating to engage with a vibrant community of technologists. Thank you Orlando Code Camp organizers, sponsors, and all those in the tech community!

The deck I used in the talk is both attached for download and available on slideshare.

According to legend, programmers back in the stone age would write code without IntelliSense and refactoring tools. The next generation of developers will wonder how our generation got anything done without AI-powered assistants. If you don’t know what GitHub Copilot is all about then come on by to get a glimpse of the future.

In this fast-paced demo-heavy talk we will see how you can go faster, stay in flow, and maybe even do more (unit tests anyone?) with GitHub Copilot, which became commercially available during 2023. Along the way we’ll learn to talk like an AI nerd by explaining and examining terms like “prompt engineering” (how to get Copilot to do what we really want), prompts vs. suggestions, what is a “conversational AI”, what do you mean by “non-deterministic”, and how does this relate to ChatGPT (and its underlying LLM). And hallucinations. All will be explained.

The deck I used:

Always great to engage with the OG Code Camp crew at Boston Code Camp. My talk description:

Haven’t heard about DORA yet? You will.

The annual DevOps Research and Assessment Report — affectionately known as the “DORA Report” — is a data-driven, research-backed set of practices and metrics that will make engineers happier and more productive while improving not just dev, ops, and security outcomes, but also business outcomes. DORA tends to also shine a light on practices that are common within teams that are measurably more effective than industry averages – examples will be drawn from cloud technologies, automation, and security outcomes.

In this talk we’ll explore the DORA report as a data-driven toolbox for helping you “get better at getting better” in the software delivery realm. We will give some historical context, then zoom in on findings from the recently published 2023 report. The goal is for you to leave this talk with an overall appreciation of the breadth of coverage and the impact of DORA as well as some specific metrics and capabilities you’ll want to focus on to level up your own teams.

The deck I used:

Always great to engage with the OG Code Camp crew at Boston Code Camp. My talk description:

According to legend, programmers back in the stone age would write code without IntelliSense and refactoring tools. The next generation of developers will wonder how our generation got anything done without AI-powered assistants. If you don’t know what GitHub Copilot is all about then come on by to get a glimpse of the future.

In this fast-paced demo-heavy talk we will see how you can go faster, stay in flow, and maybe even do more (unit tests anyone?) with GitHub Copilot, which became commercially available during 2023. Along the way we’ll learn to talk like an AI nerd by explaining and examining terms like “prompt engineering” (how to get Copilot to do what we really want), prompts vs. suggestions, what is a “conversational AI”, what do you mean by “non-deterministic”, and how does this relate to ChatGPT (and its underlying LLM). And hallucinations. All will be explained.

The deck I used:

I gave an extended demo of GitHub Copilot at the Boston Azure AI event tonight. Most of the session was a demo, but I did also walk through some slides. Those slides are attached.

I also showed these links or services:

• Microsoft Responsible AI Principles – https://www.microsoft.com/en-us/ai/responsible-ai
• Bing Chat (AI-powered chat) – https://chat.bing.com
• Open AI ChatGPT – https://chat.openai.com

Keeping with the AI theme, I opened with a talk about GitHub Copilot where I gave a brief demo and explanation, then attempted to put it into context among other AI tools (including other Microsoft Copilot initiatives).

The deck I used is attached to this post:

Veronika Kolesnikova, Jason Haley, and myself jointly hosted the Boston Azure edition of Global Azure Bootcamp. It was a sold out show – excellent turnout (around 50 participants).

I presented a couple of segments, one of which was to inspire use of AI tools. For that I gave a demo of GitHub Copilot, explained how to use the free trial, and was there to support the occasional questions by new users of GitHub Copilot during the bootcamp.

The opening talk was also AI:

My talks were later – GitHub Copilot the rolled into hacking on Azure for the rest of the day. And of course, those who opted to do so used GitHub Copilot to help them do so more efficiently.

DORA is short for the DevOps Report Assessment and provides a research-backed assessment of how your technical, process, and cultural practices impact business outcomes. DORA can be used as a guide for choosing how to improve.

This topic is important because I predict research-backed practices (e.g., DORA) will become pervasive/routine in 2024.

This important work helps inform many aspects of software development (and DevX), including many ways it is applicable to cloud computing, including Microsoft Azure, and futures like where is AI on the adoption curve and how impactful is it appearing to be on productivity (inclding for SPACE metrics). So much to talk about.

The deck is here, followed by the published description of the talk:

Haven’t heard about DORA yet? You will. 

The annual DevOps Research and Assessment Report — affectionately known as the “DORA Report” — is a data-driven, research-backed set of practices and metrics that will make engineers really happy while improving not just dev, ops, and security outcomes, but also business outcomes. 

In this talk we’ll explore the DORA report – the background, the talent behind it, the current state, the famous 4 (or is it 5?) DORA metrics, where it is going, and how you and your team can leverage it today to make effective investment decisions for improving your software development and delivery processes.

The last couple of years I was more focused on Security and Compliance-enablement in Microsoft Azure. This year I focused on Robustness. When running in Azure – like other distributed system environments – failures are real and retries are a common remedy to support robust applications. In the session I discussed the need and the mechanisms, with many examples in Azure.

Here is the talk description:

Fail and Retry

Does your application reach out to services or databases over a network. Do you assume that these calls always succeed? In this talk we will cover different reasons for failures and organize them into those that make sense to retry (e.g., “transient” failures) and those that don’t. Where retrying makes sense, we’ll cover some tools and techniques to handle retries automatically, sort out how to select appropriate retry parameters depending on the scenario (e.g., batch job vs. interactive UI with a user waiting), and consider how to test such scenarios (hint: there are some tools to help force transient errors). Planned examples will include C# in Azure, but open to demonstrating other languages and platforms. You can hit me up on Twitter to request a specific scenario (https://twitter.com/codingoutloud/status/1574138969134088192), though the concepts and patterns are generic.

If you want to experience MORE AZURE please check out https://meetup.com/bostonazure (currently operating as part of “Virtual Boston Azure”).

If you are someone who would like to SPEAK at Virtual Boston Azure, please get in touch. (Twitter is a good way to reach me – I am @codingoutloud – or you can address it to @bostonazure.)

Today I had the opportunity to speak at the Granite State Code Camp (#GSCC2021) in Manchester, NH. This was the first time I’ve given an in-person talk since the start of COVID and it was great to see so many smiling facing (even when partially obscured by a mask!).

Last year my focus was a more in-the-weeds talk called Running Azure Securely – which of these Azure security features are for me?. This year I stepped back a level and focused on Compliance. In the session I discussed security vs. compliance, the shared responsibility model, and touched on a few other features, but spent a good bit of time focused on what I am thinking about as the “Policy stack” where one can gather lots of insight about your workload’s compliance with technology controls indicated by various compliance standards – based on the Azure Policy capabilities, a pillar of governance, and rolled up and available from Azure Security Center Microsoft Defender for Cloud.

Azure Security Center as a brand is no more – it is part of a rebranding to Microsoft Defender for Cloud. I assume this renaming, announced at Ignite, is because it is a feature set that can span beyond Azure – for example, keeping an on on-premises resources and resources in non-Azure clouds like AWS.

The session was interactive (as preferred!) and many thanks to Kevin and Vishwas and the nice lady whose name I didn’t catch who I think worked for the college for help in overcoming technical limitations in the room I was speaking from.

If you want to experience MORE AZURE please check out https://meetup.com/bostonazure (currently operating as part of “Virtual Boston Azure”).

If you are someone who would like to SPEAK at Virtual Boston Azure, please get in touch. (Twitter is a good way to reach me – I am @codingoutloud – or you can address it to @bostonazure.)

Slide deck is attached.

• GraniteStateCodeCamp-BillWilder-2021-Nov-6-RunningAzureSecurelyAndCompliantly-final

Slides:

YouTube:

https://www.youtube.com/watch?v=OWXTtCUNmes&feature=youtu.be

Slide deck attached.

• GraniteStateCodeCamp-BillWilder-2019-Nov-02.AllTheseSecurityFeatures.PUBLISHED

Also perhaps of interest – a similar talk from the other Burlington – at the recent VT Code Camp – which has a few add’l resources listed: https://blog.codingoutloud.com/2019/09/28/talk-running-azure-securely-are-all-these-security-features-for-me/

The command

dig CNAME bostonazuredemo.azuresecurely.com +short

will check public DNS records for a CNAME, returning whatever it is mapped to, if anything.

In the above screenshot:

1. nothing returned from dig – this is before any DNS entry was created for the demo subdomain
2. a cascade of CNAMES are returned from dig – this is after a DNS entry was created for the demo subdomain – and it pointed at an Azure Web App — the cascade here includes my subdomain => an azurewebsites.net subdomain (bostonazuredemo.azurewebsites.net) => a second azurewebsites.net subdomain (waws-prod-dm1-139.sip….) => a cloudapp.net domain => and finally an IP address
3. a single CNAME is returned from dig – this is after the Azure Web App was deleted (), but the DNS subdomain entry (bostonazuredemo.azuresecurely.com) was left intact – creating a dangling subdomain at risk of being hijacked — anyone who registered bostonazuredemo.azurewebsites.net (and it was open for anyone) would automatically have bostonazuredemo.azuresecurely.com already wired up to it.
4. a cascade of CNAMES are returned from dig – but different than the first – this is after bostonazuredemo.azurewebsites.net was registered again, by a hacker, and bostonazuredemo.azuresecurely.com was hijacked

Some other notes from the session:

• Here’s the heat map from Brian Krebs from the Mirai botnet attack.
• Let’s Encrypt config instructions –
  • https://gooroo.io/GoorooTHINK/Article/16420/Lets-Encrypt-Azure-Web-Apps-the-Free-and-Easy-Way/21872#.Xa5iY5NKgcg
  • https://www.hanselman.com/blog/SecuringAnAzureAppServiceWebsiteUnderSSLInMinutesWithLetsEncrypt.aspx
  • https://royaljay.com/development/free-ssl-cert-for-azure-web-apps/
• About DNS: https://www.linode.com/docs/networking/dns/dns-records-an-introduction/
• Managing multiple DNS sources: https://github.com/StackExchange/dnscontrol
• Sample code (nothing interesting): https://github.com/codingoutloud/claimdump/blob/master/ClaimDump/Views/Home/Index.cshtml

Subdomain takeover examples:

• https://hackerone.com/reports/576857
• https://hackerone.com/hacktivity?querystring=subdomain

Some Resources Mentioned

• Azure is in 54 regions around the world
• Which services support Managed Identities? (works with Azure Active Directory – AAD)
• Azure Sentinel (SIEM) https://docs.microsoft.com/en-us/azure/sentinel/overview
• Azure Playbooks (automated actions from Azure Security Center): https://docs.microsoft.com/en-us/azure/security-center/security-center-playbooks
• Not mentioned, but might be of interest (more structured than the interactive format used today): https://blog.codingoutloud.com/2019/09/14/talk-running-sql-azure-securely-sql-saturday-877-14-sep-2019/

The deck

VermontCodeCamp-BillWilder-2019-Sep-28.AllTheseSecurityFeatures

Talk description

Action Photo

(if I can find one)

Some Resources Mentioned

• Azure is in 54 regions around the world
• Azure has met so many compliance standards, they had to get the UX team involved! #goodproblemtohave
• Who logged into my Azure SQL Database?
• Setting up Azure SQL Database auditing (prerequisite for the above technique to figure out who logged in)
• Enable baseline protection policies to better protect accounts
• Business Continuity
• Backing up my Azure SQL Databases (full backups)
• Automatic short-term (up to 35 days) backup window via Azure SQL Database Point in Time Restore
• Which services support Managed Identities? (works with Azure Active Directory – AAD)
• Azure SQL Database Managed Instances are not same as SQL Database

The deck

Running Azure SQL DBs Securely – Bill Wilder – SQL Saturday #877 – 14-Sep-2019

Talk description

If you know your way around SQL Server, then you will find Azure SQL Database to be familiar territory. But some aspects are more familiar than others, which is especially true for security-related differences.

In this session we review the key differences around identity management and authentication (including multi-factor authentication), managing server credentials (or, even better, not needing to in some cases), how to audit logins (probably not what you expect), an overview of encryption and data masking options, and the supporting role of Azure Key Vault. We will also touch on compliance and disaster recovery to give the complete picture of powerful features you’ll definitely want to know about to protect your data.

This talk will cover relevant capabilities for both traditional Azure SQL Databases and the newer Azure SQL Managed Instances.

This talk assumes you are already familiar with SQL Server or another enterprise database.

Action Photo

(Credit Taiob Ali @SqlWorldWide)

This was an experimental approach for me and the feedback from the audience tells me it worked pretty well. The group at North Boston Azure was already knowledgeable and engaged, so hopefully made for a interesting experience for all involved (was certainly fun for me).

Azure-DefenseInDepth-BillWilder-2019-July-30

You can follow me on Twitter (@codingoutloud).

You can also follow Boston Azure on Twitter (@bostonazure).

ORGANIZERS

Big thank you to local organizers Olimpia (@olimpiaestela), Veronika (@breakpointv16), Gladis, and Maura (@squdgy). We all worked closely with Jason (@haleyjason) who ran the Burlington MA event. And don’t forget those folks at the Global Azure Bootcamp level providing a platform making this possible for a coordinated day of #Global Azure cloudiness (https://global.azurebootcamp.net/).

SPONSORS

The thanks continue with sponsors: MIT Women in Technology, Insight (formerly Blue Metal – https://www.insight.com/en_US/solve/digital-innovation.html), Finomial, and the Global Sponsors (https://global.azurebootcamp.net/sponsors/).

SPEAKERS

And a big thank you to the speakers – all who gave up a chunk of weekend to join us on a Saturday to share their knowledge (in order of appearance):

• Enterprise Integration using Azure Serverless by
  Juan Pablo Garcia Gonzalez (@liarjo) – slides/sample code: https://github.com/liarjo/LogicApps-MIT-demo/tree/master/Logic-App-Demo
• A SIEM-ple Guide to Azure Sentinel by Jason Nickola (@chm0dx)
• Securing Azure workloads to DoD standards by Michael O’Leary (not finding a personal twitter handle, so providing his employer’s: @F5Networks)
• Cognitive Services by Veronika Kolesnikova (@breakpointv16) – slides/sample code: https://www.slideshare.net/secret/swBgIpZvpPUM7g and https://github.com/Veroni4ka/ArtworkLab
• Build your first Java app by Elizabeth Rosso (@_elizabethrosso)
• Learn to make your own Recommendation System with AzureML by Beth Zeranski (@BethZ)
• I wrapped up the day (Bill Wilder, @codingoutloud) trying to tie together varied points based on audience feedback, with my (wide ranging) slides, samples, and relevant links included below.

Attached are my slides:

• BostonAzureBootcamp-WhoMovedMyCheese-BillWilder-2019-Apr-27
• AzureLogicAndFunctionApps-AzureBootcamp-27-Apr-2019

The above graphic is from here: https://docs.microsoft.com/en-us/azure/event-grid/overview#event-sources

Here are some more links of interest:

1. Some collected links (some repeated below): https://github.com/codingoutloud/bostonazurebootcamp2019/blob/master/README.md
2. C# Script is real – not a hoax! – https://msdn.microsoft.com/en-us/magazine/mt614271.aspx
3. Azure Functions support C# Script (.csx files) – but also regular compiled C# (.cs on .NET Core)
4. Example Azure Function written in regular compiled C#: https://github.com/codingoutloud/opstoolbox (especially https://github.com/codingoutloud/opstoolbox/blob/master/SslCertificateExpirationChecker.cs)
5. Here are some example uses of the above:
   • http://secops.azurewebsites.net/days/blog.codingoutloud.com
   • http://secops.azurewebsites.net/days/twitter.com
   • http://secops.azurewebsites.net/days/domaindoesnotexist98725.com
6. Event Grid:
   1. https://docs.microsoft.com/en-us/azure/event-grid/event-sources
   2. https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-grid
   3. https://madeofstrings.com/2018/06/29/azure-event-grid-filters-with-logic-apps/
   4. “Slide” I showed is below – it is from here: https://docs.microsoft.com/en-us/azure/event-grid/media/overview/functional-model.png
7. Combine Azure Logic Apps with Azure Functions – https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-azure-functions#add-function-logic-app
8. Similar to “follow-along lab” that tied together Subscription changes to an Azure Function using EventGrid
9. Azure Function in JavaScript that fails 75% of the time. Useful for testing retries and seeing how errors are handled: https://gist.github.com/codingoutloud/151976063b1e9367369f1505f6cca66e
10. Azure Blockchain Workbench:
    1. https://azure.microsoft.com/en-us/features/blockchain-workbench/
    2. https://docs.microsoft.com/en-us/azure/blockchain/workbench/
    3. https://docs.microsoft.com/en-us/azure/blockchain/workbench/architecture
    4. https://docs.microsoft.com/en-us/azure/blockchain/workbench/use

To track who is logging into your Azure SQL database, enable auditing (here’s how to do that) with audit entries directed to an Azure storage blob. There are two ways to do this: at the database server level and at the individual database level. Either is fine, but for the example that follows, auditing is assumed to be at the db server level. The example query can be adjusted to work with auditing at the database level, but one of the two auditing options is definitely required to be on!

Run this query to find out all the principals (users) who have logged in so far today into your Azure SQL database.

The output is something like the following, assuming if I’ve logged in 12 times so far today with my AAD account (bill@example.com) and 1 time with a database-specific credential (myadmin):

09-Nov-2019 (Saturday) codingoutloud@example.com 12

09-Nov-2019 (Saturday) myadmin 1

The query might take a while time to run, depending on how much data you are traversing. In one of my test environments, it takes nearly 20 minutes. I am sure it is sensitive the amount of data you are logging, database activity, and maybe settings on your blob (not sure if premium storage is supported, but I’m not using it and didn’t test with it).

Note: There are other ways to accomplish this, but every way I know of requires use of Azure SQL auditing. In this post we pushed them to blobs, but other destinations are available. For example, you could send to Event Hubs for a more on-the-fly tracker.

BostonAzure-RunningAzureSecurely-BillWilder-2018-Oct-18

Please watch for a Part II to be scheduled.

Slides are here:

BostonCodeCamp-RunningAzureSecurely-BillWilder-2018-Apr-07

For those interested in the deck I used, please find it below.

NashvilleAzure-RunningAzureSecurely-BillWilder-2018-Mar-27-Published

My slides are included below.

Azure SQL DB – Not Just A Cloud Version of SQL Server – SQL Saturday RI – 09-Dec-2017

Slides:

• SPS – A Gentle Introduction to Azure – 2017-Oct-28

• Azure SQL DB – Not Just A Cloud Version of SQL Server – SQL Saturday BI – 23-Sep-2017

If you are local, you may be interested in learning more about Azure by checking out @bostonazure and bostonazure.org. You can find me on twitter here: @codingoutloud.

Before I posted the slides (see below), I needed to clean up the 1000+ emails I spammed myself with from my Logic App demo. (Doh!)

The slides are available for review here:

• Serverless Azure – North Boston Azure – 26-Sep-2017

• Serverless Azure – Logic Apps and Functions – Granite State SharePoint Users Group – 07-Sep-2017

Speaking at the “stimulation rich” Microsoft Store — photo credit: @jfj1997 Julie Turner

By the time I turned off the Twitter => Slack Logic App, a lot of messages were posted (in the #demo channel within Boston Azure slack account – which is open – join here):

I will be giving a longer and more general version of this talk at the SharePoint Saturday New England 2017 event on Sat Oct 28 in Burlington MA.

Also planning other variants of this talk in the coming weeks:

1. Azure Functions at Boston Azure – Thu Sep 14 in Cambridge MA (NERD)
2. Serverless Azure at VT Code Camp – Sat Sep 16 in Burlington Vermont
3. Serverless Azure at North Boston Azure – Tue Sep 26 in Burlington MA
4. Logic Apps Thu Oct 5 at NE Microsoft Dev Group
5. Serverless Azure at SharePoint Saturday NE event mentioned above Sat Oct 28 in Burlington MA

• Serverless Azure – Logic Apps – Boston Azure – 06-Sep-2017

I did not have time to show it, but the Slack => Email process did succeed. I saw this later when I checked my email. Subject: bill-tux-full.png, Body: 1504739903F6YKE996C bill-tux-full.png bill-tux-full.png. And you can see the photo for yourself in the Boston Azure slack #demo channel.

If you’d like to learn more about Serverless Azure, check out these upcoming talks: Azure Functions and Logic Apps Thu Sep 7 in NH, Azure Functions Thu Sep 14 in Cambridge MA (NERD), and various playing of Serverless Azure (Azure Functions and Logic Apps) on Sat Sep 16 in Burlington Vermont, Tue Sep 26 in Burlington MA, and (if my talk is accepted for the Sharepoint event) Sat Oct 28 in Burlington MA.

As always, please let me know if you are interested in more talks at Boston Azure.

EDIT: Here is the deck presented by Marija Strazdas from @AlertLogic who spoke about the Shared Security Model for security in the cloud:

Alert Logic Azure Security Presentation

You can find @bostonazure on twitter, and feel free to join us on slack.

The point of the talks wasn’t really that using PowerShell is bad/wrong, more that it might not be the best tool for the job in certain scenarios. In particular, an ARM template is a powerful modeling tool in support of a “no pets” policy, which is interesting to consider as your cloud environments grow more complex while also wanting to make environments easier to manage. Another benefit stems from keeping the ARM template itself as an “infrastructure as code” artifact that can be used to document – and, more to the point, as executable documentation – for stamping out environments predictably. And still another feature: the ARM runtime handles a lot of the complex parts that could come by trying to script one resource at a time via imperative PowerShell scripts – for example, error recovery and retries.

The deck is on the event shared github repo.  There are lots of otherPowerShelly resources on that repo that you may find worth checking out.

(Added 03-June) For those of you who attended my Advanced session, when I attempted to clean up at the end using Remove-AzureRmResourceGroupDeployment, my PowerShell command had an error in it. Here is the correct version. In the first screen shot I show how to ascertain the correct value for  the first the parameter using Get-AzureRmResourceGroupDeployment.

Remove-AzureRmResourceGroupDeployment `
   -Name Microsoft.Template -ResourceGroupName k1

Once that PowerShell command executed, all 8 resources associated with that deployment were removed (deleted, and billing stopped).

Ta da!

Hope to see all you locals at Boston Azure (@bostonazure) in the future for more Azurey action.

Here’s the deck I used:

comparing-azure-sql-database-and-sql-server-North-Boston-Azure-28-Mar-2017-bill-wilder

And here is some of the CLI 2.0 action, including the last line to clean up a resource group:

az login
az account set --subscription MySubscriptionName
./show-resourcegroup-contents.sh nbazure
z group delete -n nbazure --no-wait