Speed up your automated tests without really doing anything..

Here’s a Christmas present from me for you: a short, but may be fore some people a very liberating post on how to speed up your tests in your pipeline, without doing anything “real” ;-).

The problem

We have made “tests” mandatory in our company.  What that means: every single pullrequest is being code reviewed, and in the code review, we check for either a changed or new tests.  That has lead to quite a number of tests: 8500 for the product, and we even have a customer with (in the meantime) over 1500 tests.

Over the years, we noticed a growing performance problem when running tests.  For our product, running all tests for all apps, took about 3 hours.  But we were ok with that: we run all those tests at night.  If they take 2, 3 or 5 hours .. who cares?  Nobody is waiting for it.  And for pullrequests, we only run the tests from the app we changed, and that didn’t take that long (the product is split in 50 apps, so we have an average of 170 tests per app, which was quite ok).

But for this customer (and a few projects), it’s a different story, because every single pullrequest needs to execute all the tests.  And it took more than an hour to run the tests – so a pull request took quite a while.  Just imagine we need to do a hotfix: it makes all the sense we reduce that time of a build, and the most significant amount of time to reduce, is the test run!

Just look a this:

At that time, we had about 8 developers constantly developing for this customer, which means: multiple times 1h 20 minutes wait time for a pullrequest validation. That’s simply too much .. something had to happen!

The analysis

After careful analysis of the log file, my colleague noticed a trend that lead us to quite an interesting conclusion:

The more codeunits we had in a test suite, the  longer it took to run the suite.

I’ll say that again in other words:

Every run of a codeunit in one suite, took a fraction longer to run to more the test run advanced in that suite

Or yet again in other words:

Codeunit 1 was fast, codeunit 2 was a fraction slower .. and all following codeunits were (in general) slower than the previous .. 

And this phenomenon was true “per suite“.  So a second suite, first codeunit, would be fast again .. .

Another colleague had an explanation for it which I’m not able to verify (that I know):  the way the test-suite is run from the pipeline, is that it “mimics” a user using the webclient (remember that we need the webclient to be able to execute page testability).  On top of that, you need to be able to track results per codeunit (or even per function): so one way to do that, is loop your way through:

  • Open suite
  • Navigate down to the next codeunit/function
  • Run codeunit/function
  • Save results

Now just imagine you have a lot of codeunits.  “Navigating down” can take a while.. 🤷‍♂️.

Again: not sure that this is the way it works, but it would sure explain this behavior.

(Part of) the solution

We tested what the sweet spots was for the number of codeunits in one suite and the time-gain we had – which (for us) is 25 codeunits.  Now we simply consciously divide codeunits per suite.

We always had an install codeunit that created a test-suite for an app.  Now we simply create multiple suites per app.

Yep – hardcoded and manual.  It’s ok – it’s just a test app ;-).

In the pipeline, we simply run the test-step multiple times:

Where we simply loop this array (by using some simple yaml magic ;-)):


BOOM! 50 minutes time difference for 100% the same tests!

Writing testable code

You might have joined Vjeko’s and my session at BCTechDays – or you might have followed one of Vjeko’s workshops or sessions at Directions – or you might have read his blog posts about the topic, like:

Obviously, what I just shared is by no means a way to avoid “decent coding”, or a replacement for you for not having to dive into “writing testable code” better ;-).  By no means at all!

But it does make sense to never combine all your tests in one suite and run them that way in a pipeline 🤷‍♂️.

Happy Holidays!

4.80 avg. rating (95% score) - 5 votes

Permanent link to this article: https://www.waldo.be/2023/12/28/speed-up-your-automated-tests-without-really-doing-anything/


2 pings

Skip to comment form

    • Manish Kitar on December 28, 2023 at 12:14 pm
    • Reply

    Amazing waldo (as usual). 👌🏼
    Perfect for our team to adopt in upcoming year.

  1. Thanx for sharing this, waldo. Meaningful notion to keep an eye on.

    • Freddy on January 7, 2024 at 9:05 am
    • Reply

    The recommended way of running tests is to NOT have the install codeunit and run the test app based on the appid.
    This will not use the repeater movements etc., but instead use a different mechanism to execute all tests in the test app.
    Did you try to see whether this changes the performance picture?

      • waldo on January 7, 2024 at 10:01 am

      Seriously? 🤔
      I did not test that, simply because (1) I want install codeunits for the convenience of the developers who are continuously adding and using the tests, and (2) it doesn’t sound anywhere near obvious to me that the mechanism would act differently.. . With AppId, i’d always run all tests, so given the behavior, this was a no-go for me.

      But will try.. . Thanks for the feedback.

    • Džoka M. on January 16, 2024 at 2:53 pm
    • Reply

    I looked at our Powershell script to see how we invoke the tests and we do it differently:

    In short, we use Get-TestsFromBcContainer to get the tests we want and then we just loop over every test codeunit and call Run-TestsInBcContainer with the Codeunit id.

    In our main app we have 1500+ tests and it takes roughly 30 mins. (I guess that’s ok)

    So the testsuite Default is used in the process but it’s not populated with all the codeunits.

      • waldo on January 16, 2024 at 2:56 pm

      So what do you do when developers are developing. Are they always “loading test”? Because, with an upgrade-codeunit, they are always just “there”, or updated – which is very convenient.

      • Džoka M. on January 16, 2024 at 4:18 pm

      We added a bunch of helper functions with a page extension to the AL test tool, to make it easier to work with our tests in general. – For devs and consultants.
      For example “SwissSalary / Get test codeunits” gives the user the option to choose the set of tests he wants to add to the current testsuite.

  1. […] Speed up your automated tests without really doing anything.. […]

  2. […] Source : Waldo’s Blog Read more… […]

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.