Testing Windows Forms (or, code road from dev to test)

As we near the end of a product cycle QA (Quality Assurance or just 'Test') picks up the reigns and drives the product home. It's their job to give the thumbs up or thumbs down of when we're ready to ship.

But the way that dev interacts with test is an interative process and there are many points where we test the product. That said, it's my opinion that we've got a long way to go on this one, and I'm not alone. But there's kind of a hierarchy of testing that happens that looks something like this. Think of it as a progression in depth and breadth.

Developer unit testing is the first line of defense. This is the developer trying scenarios on the desktop to find issues as early as possible, is generally ad-hoc and focused on the area being worked on.

The next level are check in tests, which we sometimes call BVTs (Build Verification Tests, an old term that's stuck) or check in suites. This is a set of automated tests that span the product and try to exercise mainline scenarios to make sure nothing is majorly broken. Developers write these tests and are encourage to augment them as features are added or changed, or are found to be prone to breaking. Assuming we can't fix the fragility itself for whatever reason. These may also include tests from other teams to ensure we didn't break them, just like some other teams run some of our tests for the same reason.

Once the code is checked in, it's picked up by the Build Lab, which is a centralized team who runs doing the builds every day (we've got like 60 SKUs to build or something crazy like that), then run an expanded set of BVTs that span the product, we call these DDINTs (pronounced 'dee-dee-ints', which ). A full build of the VS+.NET stack takes about 8.5 hours on a really fast machine, and then the suites take another 3-6 hours depending on if they are debug or retail. Oh, and they have to run them on Windows 2000, Windows XP, Windows Server 2003, and some of them on Windows 98, AMD64, and IA 64. The lab has a lot of machines. My suspicion is that there are small towns that use less electricity then they do. But the infrastructure they build is pretty amazing; when a suite fail a bug is automatically logged with a link to a Remote Desktop connection to the machine it failed on that automatically gives you privileges and the whole bit.

After the Build Lab cranks out a build that passes the suites, QA picks it up and does their testing on it. They have their standard level of testing called "nightlies" that really don't usually end up running nightly but more like weekly in most cases.

And the final level of internal testing is when QA runs their Full Test Pass (FTP) towards the end of each milestone. An FTP is a mammoth undertaking where they run all the automation, all the manual tests, all the compat tests, all the platform-specific tests (unfortunately our team gets the worst of this: Win98, WinME, Win2k, WinXP, Win2k3, AMD64, IA64), side by side testing (things like install VS 2003, install VS 2005, uninstall VS 2003, make sure everything works), language specific tests (our designer supports VB.NET, C#, J#, Managed C++), and a bunch of other stuff that I can't think of. This generates our major lumps of bugs. Check out an article on Scott Guthrie's blog about testing ASP.NET -- this is an awesome post and the vast majority of the processes and details apply to our team too.

And then there's customer testing, which was historically done via beta programs but now is more tightly integrated into our process via the MSDN Feedback Center, which we call "Ladybug". We've gotten great response and participation from the community (you people!) here and it's really great to find out about, and fix, a bug that we probably would not have found on our own. It turns out customers like to do things with the product we've never even thought of.

As I mentioned above, there are things here I'm working to improve. I want to tighten the feedback loop between dev and test. I'd like to have more of their tests running on our code more often (there are actually some steps between our checkin and the Build Lab turning a build that I skipped over here because they're not really relevant), and I'd like to share more infrastructure between our teams. Right now dev has a couple of home-grown harnesses and QA has some. Unifying them allows teams to share tests (many teams at MS are set up this way), and there are historical reasons why we're not unified. There is also some really cool technology being developed around Code Coverage which will tell you how much of your code your tests are exercising, and people are working on plugging that into the process as well. I'd like devs to be able to access a huge library of QA tests from across the division and run them after making a change to a given area, and to be able to use the code coverage data to figure out which tests will exercise that. That would be cool.

Print | posted @ Tuesday, December 14, 2004 5:56 PM

Comments have been closed on this topic.