How to choose a basic framework and tools for automated testing of AI?
Best answer by Rhona
View original
Userlevel 4
+1
Hi
that isn’t an easy question to answer. It depends on your context and the product/ service you want to test and what your goals are.
Can you share some more information with us?
There is no “right” way to select a test automation tool. One last question, do you want a tool that can automate AI or do you want a test automation tool powered by AI?
Hi Daniel,
I need to create some automation framework based on Phyton, GitHub and Cloud (Azure or AWS) for testing of AI software on chip. For example one of the AI feature: AI SW recognize some dog image and write text: “This is a dog”.
Userlevel 4
+1
Hi
nice challenge that you have in front of you. So far I never tested an AI software. However, I think before you get into the coding, I would suggest learning more about the algorithm of the AI that should recognize the dog image. If you know the algorithm and the data model behind, I think you need to get into machine learning and how to train the system to understand it.
There is a great article about machine learning here: https://blog.testproject.io/2022/01/17/machine-learning-testing-for-beginners-the-all-in-one-guide/ I recommend you to read.
Once you know more about the underlying model and algorithm, you can start thinking about the automation part.
Userlevel 7
+2
- Community Manager
- 104 replies
-
3 January 2024
Hello
Your question is really good, and I think it would be of great interest to other users as well. I discussed it with
- Employee
- 1 reply
-
11 January 2024
- Answer
Hey
As mentioned by
So, if I understood your question correctly and you want to test an AI-based service, you have a few options. First, let's look at how this is being done these days; well, not much is automated. Data scientists, like myself, manually test the different stages of the AI pipeline to check if everything adheres to the standards. In the attempt we made in our prototype, we tried to automate the stages that a data scientist takes to test an AI service. In general, we have defined three different stages. I won't go through all of them right now. The part that I noticed you need, based on the conversation, is testing the AI algorithm to ensure the designed algorithm is working fine. If you don't have access to an automated method, you need to create one.
You need to make sure of whether you have access to the source code of the algorithm or not. If not, you need to change your strategy (moving to stage 3 of testing an AI). If yes, you need to analyse the statistical measure such as precision and recall and if they don’t match with the expected results, the process of confirming the model involves two steps: first, checking the algorithm technology (the logic that the algorithm is using). Secondly, testing the coding of the algorithm. If the algorithm is already a research-based approved algorithm and the model used in your service is confirmed regarding the coding, then you don't really have much to test about the algorithm itself (I am guessing this is the case). You need to focus on the other stages of testing an AI, which I will briefly mention here:
- Data check
- Model validation
- model Explanation
I might be wrong, but I believe you need to look into this and make sure of what exactly needs to be tested. Sometimes, the "no good" results do not stem from the algorithm but rather from the data (the first stage in the Testing an AI process). Good luck :)
Reply
Login to the community
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.