Run your own AppSource Validation

Quite honestly, I’m fully into the process of getting our apps (about 30 of them) to AppSource. We chose to have OnPrem implementations first, basically to get the experience, and also because it just still sells better in Belgium (immediate market ;-)). Anyway ..
Recently, there was a call with Microsoft with the topic “what to do in order to pass technical validation”. Given my current state – quite interesting to me, obviously ;-). It was a very good presentation from Freddy, with a clear overview of how Microsoft handled it in the past, how it’s handling it now .. and what to expect in the future.
What I was a bit surprised about was that still quite some partners just uploads a version of their app(s), and let Microsoft figure out what the problems are in terms of technical validation .. . And also that some partners had to wait for quite some time before getting feedback. Well, there was quite a clear explanation that one had to do with the other:

We are much faster in passing an app than in failing it.

And that’s normal. The validation is done by running a script. If the script passes, the validation passes. If the script fails .. Microsoft needs to find out why it fails and report that back – which is a manual process. Basically meaning: the more you check yourself, the faster your (and anyone else’s) validation experience will be!

What can you check yourself?

Well – basically everything that needs to be checked – the entire stack. Later, I will tell you how, but first, let’s see what’s so important to check – and apparently forgotten by a lot of people. Let’s start by this easy link: . It ends up on Microsoft Docs explaining the details. During the call on Tuesday, Freddy highlighted a few common things:


You need to comply with every single rule in the AppSourceCop . Well – that’s what he told on Tuesday. Today, during a session on BCTechTalk – he corrected this. There is actually a ruleset that they apply when checking the apps, which you can find here (not sure how long the ruleset will be found in that link .. ). So, in short – enable it in your settings!

And if convenient for you, just apply the ruleset I mentioned (I don’t – we simply comply with every single AppSourceCop rule)

Breaking changes

When we think of breaking changes, we think of schema-related breaking changes. But that’s not all. In fact. AppSource-related breaking changes can also be:

  • A renamed procedure
  • A renamed or removed control on a page

So .. there are A LOT more breaking changes when we think in terms of AppSource. In fact, it’s important you make yourself familiar with the settings for the AppSourceCop (aka AppSourceCop.json). At minimum, you should refer to a previous version of your extension. And in order for the compiler to take that into account as well, also provide the app-file of that version.
Here is an example of having the AppSourceCop.json (in the root of the app), the setting to my previous release, and the action released app in a folder on the project.

Note – for the VSCode compiler to work, you might have to copy that app in the symbols-folder. I just like to have it separate, so its intention is very clear (and as you can see, I .gitignore my entire symbols folder).


In the screenshot above, you see I also set up an affix. It is important that you reserve your affix at Microsoft (it’s done by email – but I’m reluctant to share emails on a public platform like this .. but look at my resources below, in the documentation you’ll find what mailaddress to contact ;-)), and set it up into the AppSourceCop to make sure you don’t forget ANYTHING to prefix (about anything in extension objects, and about every new object). Small tip – my “CRS AL Language Extension” can help you with the prefixing ;-).
So – set it up in the AppSourceCop.json, and the compiler will make sure you won’t forget!

Code Signing

This is a somewhat trickier thing to check. What do you need to know: for AppSource, you need to codesign your extension with a code signing certificate. The compiler will not sign your app, so this is usually done by a powershell script. But .. that’s not all .. what you should do is also test the resulting (signed) app by publishing and installing it without the “skipverification” check in the publish-commandlet. Don’t forget to check that, because that is the only way to be really sure the codesigning actually was successful!

Publish to all supported countries

Many partners “just” develop against one country localization (like “w1”). But when doing that, it doesn’t mean that your app will be able to deploy against all other countries. So you should also check if your app is able to be published (and even upgraded) against all these localizations that you want to support. PowerShell is your friend: set up a container for the localization, and start publishing your apps!

Name and publisher changes

When you change the name or publisher of your app – if I understood it correctly – at this point, it’s also a breaking change. That might change in the future though .. (only AppId should be leading). The way to check this, is to upgrade your app from your previous version to a new version (so basically: upgrading your app). From VSCode, this once again is difficult to do, other than running a script, which sets up an environment with your previous app, to be able to install to your new app.

How can I do my own validation?

So – a few are easy and configurable in VSCode. For others, you need scripts!
Well, this is a part that is quite interesting for DevOps, if you ask me. Just imagine: a nightly pipeline that tests all the things (and more) above .. which reports back what the current status of your apps are.
And no – you don’t need to create it yourself ;-).


As always – it’s BcContainerHelper to the rescue. Freddy wouldn’t be Freddy to make your life as easy as at all possible. Since the very moment I wrote this post, Freddy released his a function “Run-AlValidation” – specifically meant for validating your app for AppSource. It’s quite a script – I tried to read through it .. wow!
In general (and I’m being VERY general here – you can read the full explanation on Freddy’s blog ;-)), it will:

  • Loop through the countries you specified, and for each country
    • each version you specified
      • Set up a Docker container
      • Install the apps you depend on
      • Run AppSourceCop (basically: compile with AppSourceCop enabled)
      • Install previous version of your app to the environment
      • Upgrade to the new version of your app

As you see, it takes care of all the issues above. If all of this passes .. I think you’re quite good to go and upload your app to Partner Center ;-).
As Freddy explains in his post, it’s still being changed and optimized – so frequently refresh your BcContainerHelper to get the latest and greatest!

DevOps fanciness with ALOps

You might not be using ALOps, still this section might be interesting, as it also explains some neat yaml-stuff 😉

Quite some partners are a bit more demanding in terms of DevOps and development flow optimization. ALOps is one of the tools that make it easy.

So, I started thinking whether this was possible with the building blocks that are already present in ALOps – and taking advantage of the mechanics of DevOps. And of course it is 😉! Quite interesting, even!

Because ALOps is a set of DevOps-specific building blocks – it is actually quite possible to configure a set of yaml-files that does quite a good job in validating your code for AppSource.
How it works is that one yaml-file will be there for the pipeline definition, which calls a yaml-template, which has the stages and steps.

Why two yaml files? Well, doing it like that, makes it possible to create a multistage-pipeline, depending on the countries and versions you set up in the (first) yaml: each combination of country and version would be a stage. And .. stages can run in parallel, so as such, this could tremendously speed up your validation process, when you have a pool with multiple agents ;-). When you have to validate against all countries, and 2 versions .. any performance gain is always welcome ;-).

The files, you can find among our examples here in the folder “AppSourceValidation”. You’ll find 2 files: a template, and a pipeline. Just put them both together in your repo (just use a subfolder “AppSourceValidation” or something like that), and in DevOps, refer to the pipeline-yaml when you create a pipeline.

You see it’s quite readable – and the “magic” (if I can call it that) are the two arays “countries” and “selectVersions”. Those will actually cause that multiple stages will be created when running this pipeline. In the template file, you’ll see the loop here:

Now, the individual steps are a different approach than what Freddy does in his pipeline: Freddy works with app-files, I work with your code. But you can see that we can actually create different templates – like one that would for example download artifacts from a release pipeline – or something like that. It’s quite simple to reconfigure, since all is just a matter of compiling code and publishing apps ;-). I might add other template files in here in the future! Just let me know if you’re interested in that.

The outcome is pretty neat. Here is a successful validation of 4 combinations, in a multistage pipeline:

More interesting, are errors. Just the default pipeline feedback gives you all you need:

And remember, these 4 stages could run in parallel, if you have 4 agents in one pool (which is not so uncommon …), all 4 would just run at the same time!

If we all now start validating our own apps, we would tremendously speed up the validation process at Microsoft. So, let’s just do that, will me? ;-).


Regarding AppSource, there are a few resources that I got on Tuesday that is worth sharing!

5.00 avg. rating (97% score) - 1 vote

Permanent link to this article:


1 ping

Skip to comment form

  1. I plan to keep and update the ruleset on that link. The ContainerHelper grabs it from there. I didn’t want to include the ruleset in ContainerHelper, as I want to be able to update it seperately

      • waldo on December 4, 2020 at 10:21 am

      Ok, perfect! Thanks for clarifying!

    • Hedi on December 4, 2020 at 10:20 am
    • Reply

    Hi Waldo,

    Where do you subscribe to those calls/presentations please? I would love to stay up to date with all those info’s you have and share on your blog 🙂


      • waldo on December 4, 2020 at 10:24 am

      Not sure How I got those .. but if you’re a registered company on partner center and are part of ReadyToGo, you should get these invitations.. .

      • waldo on December 4, 2020 at 10:30 am

      Oh, and don’t forget – today it’s still “BCTechTalk”:

    • Hedi on December 4, 2020 at 10:38 am
    • Reply

    We started recently and are only doing per tenant apps. So i guess we are not invited 🙁 😀

      • waldo on December 4, 2020 at 10:40 am

      Well – read through this and register yourself 😉

  2. Hi Waldo.

    Thank’s for a good writeup! I couldn’t attend the meeting on tuesday so it’s appriciated!

    But if I use the ruleset you say are used on validation I get a whole bunch of warnings on an app that was validated earlier this week. It’s mostly warnings about tooltips and about with statements (that I need to deal with soon…). So it can’t be that an app must have zero (0) warnings using that ruleset to pass validation.

      • waldo on December 4, 2020 at 4:29 pm

      I guess that’s a question for Freddy (I think you’re part of the AppSource group on Yammer?).

      I know there were problems recently with the validation process – and may be yours slipped through, but I’m pretty sure that today, that app wouldn’t get validated anymore.. .

  1. […] Run your own AppSource Validation […]

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.