Hi @vitlyd ,

that isn’t an easy question to answer. It depends on your context and the product/ service you want to test and what your goals are.

Can you share some more information with us?

There is no “right” way to select a test automation tool. One last question, do you want a tool that can automate AI or do you want a test automation tool powered by AI?

Hi Daniel,

I need to create some automation framework based on Phyton, GitHub and Cloud (Azure or AWS) for testing of AI software on chip. For example one of the AI feature: AI SW recognize some dog image and write text: “This is a dog”.  

Hi @vitlyd ,
nice challenge that you have in front of you. So far I never tested an AI software. However, I think before you get into the coding, I would suggest learning more about the algorithm of the AI that should recognize the dog image. If you know the algorithm and the data model behind, I think you need to get into machine learning and how to train the system to understand it.

There is a great article about machine learning here: https://blog.testproject.io/2022/01/17/machine-learning-testing-for-beginners-the-all-in-one-guide/ I recommend you to read.

Once you know more about the underlying model and algorithm, you can start thinking about the automation part.

Hello @vitlyd,

Your question is really good, and I think it would be of great interest to other users as well. I discussed it with @danielknott  yesterday, and we've decided to organize a webinar focused on AI where we can delve deep into your question. While the specifics are yet to be finalized, we are aiming to host the webinar in March. Stay tuned for more details!

Hey @vitlyd,

As mentioned by @danielknott, this is not an easy topic. However, testing AI has been an ongoing discussion at Tricentis since October 2022 when my team and I presented the idea of testing an AI in our GIG community. We have also prototyped parts of it. Test automation is usually referred to as a general term for testing services. However, due to the need for testing AI services, we use the term "Testing an AI" to refer to the test automation service where an AI-based service is tested.

So, if I understood your question correctly and you want to test an AI-based service, you have a few options. First, let's look at how this is being done these days; well, not much is automated. Data scientists, like myself, manually test the different stages of the AI pipeline to check if everything adheres to the standards. In the attempt we made in our prototype, we tried to automate the stages that a data scientist takes to test an AI service. In general, we have defined three different stages. I won't go through all of them right now. The part that I noticed you need, based on the conversation, is testing the AI algorithm to ensure the designed algorithm is working fine. If you don't have access to an automated method, you need to create one.

You need to make sure of whether you have access to the source code of the algorithm or not. If not, you need to change your strategy (moving to stage 3 of testing an AI). If yes, you need to analyse the statistical measure such as precision and recall and if they don’t match with the expected results, the process of confirming the model involves two steps: first, checking the algorithm technology (the logic that the algorithm is using). Secondly, testing the coding of the algorithm. If the algorithm is already a research-based approved algorithm and the model used in your service is confirmed regarding the coding, then you don't really have much to test about the algorithm itself (I am guessing this is the case). You need to focus on the other stages of testing an AI, which I will briefly mention here:

• Data check
• Model validation
• model Explanation 

I might be wrong, but I believe you need to look into this and make sure of what exactly needs to be tested. Sometimes, the "no good" results do not stem from the algorithm but rather from the data (the first stage in the Testing an AI process). Good luck :)