Emtech Group Podcast

Spencer
Hey, Jeff.
Jeff
Hey, Spencer.
I wanted to reach out to this afternoon to discuss QMT’s test case generation. I noticed there’s minimum versus maximum test generation. And being in QA for the past 25 years, I have a great interest in this, and I want a deep dive into how QMT’s test generation options can help Dr. efficiencies while keeping proper test coverage. Do you have a few? Minutes to discuss.
Jeff
Sure is. Do you want to discuss this because you do not want to discuss the lease?
Spencer
That is, it. You know, I, uh, I am going to keep my keep my hopes up, but uh, not too. Uh, not too positive yet but. You know what the blue and white still have a chance, so let us hold on.
Jeff
OK, we will discuss happier things instead. So, for me, test Max and test min the minimum coverage addresses the fact that it is difficult to.
Speaker
Thank you.
Jeff
Fully test all the different parameters that you will that are involved in each model. When you when you use test, our maximum test coverage. The basics are that there is a model and there are several parameters, things like. Is the is the applicant male or female? What is their age? What? Where do they live? Because there’s jurisdictional impact on policies, other things that affect risk and various things like salary. There is a whole number of parameters, and when you combine all those parameters together to. Fully test. Any model you want to test every single combination of every single parameter, and that can result in many tests. The example that we often use is if you had thirty questions and each of them only had two answers, the number of combinations that you would want to test everything. Would be a billion different combinations and test cases. I am using combinations loosely here. I am not distinguishing between permutations and combinations, but there would be a billion of them, and if you. To test and try to test each one of those and you had to test against a third-party system because you’re doing end to end testing. If it took a second for every single test to execute, that could take 30 years. Now that is a little bit of extreme, but that is how big the problem can be. If you are doing maximum test coverage. Now it is known that not all the various combinations of parameters are important, but it is often not known. Which ones are important? But given that you can’t test every single combination you want to do something else and we have a couple of different options and tech and one of the ones that we have, sorry, in QMT and one of the options we have is what we call minimum and our minimum test coverage is basically trying to it recognizes. We recognize the fact that you cannot test all the combinations, but what we can do is at least make sure that every single possible value is tested once, and that’s what minimum test coverage does. It takes a look at your model and the model may have flows, conditions where it goes to different branches based on the different parameters and in analyzing the model it basically takes a look at all the different inputs that you have and all these different conditions and from that it comes up with a list of all the different parameter values that needed to be. Need to be tested to test all the branches of the model so that everything is test. That at least once, and that is what it is. It does not test all the combinations there. There may be cases where you want to do more than minimum, and that is where we have the option to specify specific combinations of parameters, but the minimum one makes sure that every single path and every single parameter. In the model is tested at least once and. The difference between that one billion test cases and thirty questions with just two answers each is in the latter, and I am doing this off the top of my head, so I’m I may be off, but it would probably be 60 test cases for the minimum test coverage versus a billion. So, it is a drastic decrease in the resources required to test and also in the time that it would take to actually run the test. That was a long.
Spencer
OK, all right.
Jeff
Response did you ask?
Spencer
So let me just understand this. If I understand correctly, then it is. So, let us say we are doing a testing of our system and if I want to do let us say Sup like a smoke test, right, I can use minimum and I will make sure I have coverage of all. Uh, at least all possible uh test routes and decisions and uh my knowledge graph, correct? Yes. OK.
Jeff
Yes, smoke test is a safe way of saying it too. It is like. It will it, you know it will look for the obvious things sending up smoke to show it is a problem. The other thing is that it is a better than happy path test because happy path tests are just, you know, the very minimum this. This covers everything. So, it is something that’s better than happy path. And in the smoke path type. Thing, yes.
Spencer
OK. And then if we actually let’s say we want to drill down on a certain area, then we are just actually we actually put our own inputs in into that area of the decision tree and then that way we can focus in on one part of the decision tree and make sure that that’s working as well.
Jeff
Yes. Yes. And there’s a couple of separate ways you can do that, but that’s. Yes, that is exactly it. Where for where you want tests that you do not think are in the minimum coverage, you can, you can customize it. In that way.
Spencer
OK. I see some, uh, good areas where we could do that specifically if we are introducing new projects which actually are addressing specifics in a. Entry and that way, we would actually use those uh, part of the knowledge graph to actually test that. That would be good.
Jeff
Mm-hmm.
Spencer
What about? What about the how hard is it to update the uh, the knowledge graph? Is it easy to update the knowledge graph so we can expand our testing?
Jeff
Yes, it is trivial when you are using our editor. If, for example, you are testing against the web page and another question was added, you can basically the workflow would represent all those steps as nodes connected to each other. So, you would where the new question was added, you would disconnect two of the nodes. Add the new nodes that represent that new question and. Just reconnect those nodes back in. So, if you say at step five between step five and six, there was some additional steps you disconnect five from six, connect five to the new nodes, connect the new nodes to six and you are done. It is drag and drop; it is. It is intuitive and almost trivial.
Spencer
And then I will generate the brand-new test cases.
Jeff
Yes, you generate the new test cases our test generation step. We will analyze the model. If you introduce new parameters in those additional nodes. Then it will pick those up and appropriately generate the new tests for that as well, and again at that point you can do minimum or maximum on that so that that part is automatic, yes.
Spencer
Excellent. That is perfect. OH OK, that makes things easy because when we want to create new test cases, all we must do is update the model and then regenerate.
Jeff
Yep, and test generation is quick, especially when you consider you know the fact that you could have a billion test cases and how long it would take to execute. Test generation is extremely fast and a lot faster than say a human can do it. Somebody trying to figure out the minimum test case coverage. Uh, it would. It would be a lot quicker or sorry a lot slower than our test generation algorithm. We could do it.
Spencer
Wow, that is a good. I can see many different ways of using this because, uh, when we do our testing, there is a lot of times where we want to a we want to just quickly go through the system, make sure everything is working. And, then when we want to get into deeper coverage, all we must do is select larger amounts of the model, correct? Perfect.
Jeff
Yes, perfect. And then we were like we were discussing this before, you were talking about the possibilities of using this for performance tests and UM, yes, in other cases and I think.
Spencer
Yes, yes. Can you give me an idea of how we would use this for performance testing?
Jeff
Well, I when you mentioned that my first thought was, I was thinking regression test but in one way in in a in a in a way those things are common. Once you generated the test you have that test suite that can be run again and again and again, and you run it. You if you are running it in for regression tests, you just run it, you know, and see whether it the website and everything still passes all the tests. If you are thinking about performance tests well along the same lines, it is like you will for each time you run it, you have a history. You can see how long each of the steps took, and you can see which step is taking a long time. But and so from that way you can drill down and check out the performance of the various aspects of the of the model and the systems you are testing. But then you can also like you could with regression tests, keep rerunning the performance tests and confirm, for example, that if you were to run the performance tests in parallel, run those multiple times, whether the systems you are testing get slow under the strain. Or you can just monitor that these systems. Are regularly performing at the same level. If you run the test one after the other, so it is these models and the generated tests that can be executed are, are, are. They are suitable for both regression and performance testing.
Spencer
Yes. So, I am seeing just many different ways of use. Doing this and I like the fact that you can it is automated, right? Yes. You are just it calls the, uh, the model that that you are using under test it runs the uh test generation you have your test cases ready to be executed so and then you can very easily manipulate that depending on what you are testing.
Speaker
Yes.
Spencer
And then regenerate again so. So, you know, this is a game changer. What we you know traditionally have done in the past. So, I like it.
Jeff
Yes, and. And one of the things in when we when you mentioned automated, it is interesting because one of the things we are doing is we automate the testing of a website. There are tools that do that, and they allow you to record the test site and you can and you can generate automated tests for that. But we take that to another level and that. We automate the generation of suitable tests that can fully test the site. Or whatever it is that we are under test, or we can produce minimal test coverage. And when we do that, the algorithm again, it is automated, it is quicker, it is consistent, there is less chance of the errors that are made when you have somebody manually trying to figure out all the different test cases. So, we are doing. Sort of second level automation here, that is beyond what a lot of the other what people I think are used to which I think really helps as well.
Spencer
I really do. This is good. Because uh. It is significantly better than using manual testing. It is faster. It is. I would expect that it is going to be more accurate, and the coverage is you can select the coverage all the way up to every single scenario imaginable. So, you really have, you know, you really have a sweet spot here. You have you want to be efficient.
Jeff
Yep.
Spencer
You could be automated. And you got your test coverage right. So then you know those are kind of the three things that you look for you know when you’re actually doing testing and I you know I’m really looking forward to actually working with this product because I think this is going to be like I said it’s going to be a game changer.
Jeff
Yeah, I like it.
Spencer
Well, Jeff, you know I am going to have to get going. But if you want to cheer for those leaves, I am with you because I was. I do. I tell you; we got it. We are going to have to put our heads down. We have two games to go, but. Leaves go.
Jeff
Yes. I mean, I really want them to win, but uh, I am. I am not going to.
Spencer
Hold my breath. Well, good talking to you and we will. Uh, we will touch base soon.
Jeff
OK, see you.
Spencer
OK. Bye, Jeff.