10 critical factors to successfully perform remote testing with the crowd
As the world of technology continues to integrate with every aspect of day-to-day life, it’s imperative that comprehensive and extensive testing of software and devices takes place. However, due to resource and time restrictions, organisational blindness, platform fragmentation among many other issues, testing often becomes an afterthought to the development process. Crowdsourced testing (crowdtesting), an original approach to testing that utilises a crowd that represents a digital product’s target audience to perform testing is able to address many of these issues. Today, crowdtesting has become an established solution as a complementary service to traditional testing methods in virtually all industries.
[easy-tweet tweet=”Crowdtesting has become a complementary service to traditional testing methods” hashtags=”cloud, tech”]
Its perks vary from being able to produce quick results due and around the clock availability of testers (thanks to crowdsourcing), to the availability of virtually every device and operating system imaginable. Compared to laboratory testing, crowdtesting tends to be more cost efficient and thanks to a diverse, global crowd; websites, apps and all manner of technology can function and be relevant on an international basis. However, to benefit from its many advantages, the characteristics of the method need to be taken into consideration. In many ways, crowdtesting differs from traditional methods when it comes to checking functionality and usability of digital products. The following 10 factors are a must to successfully perform crowdtesting and integrate it into the development process including practical recommendations from Philipp Benkler, Managing Director of Testbirds a crowdtesting service provider:
1. Employer Side Responsibility
Testing is an essential part of the development lifecycle and crowdtesting can be applied to both waterfall and agile testing methods. Therefore, it’s a must that the method is embedded into the overall strategy in a way that works on an individual basis. When working with a crowd, it’s vital that tasks are clearly outlined and crowdsourced workers have a strong grasp on the responsibilities that are expected of them. This is especially true when it comes to crowdtesting due to the fact that the crowd is often located internationally and communicated with remotely. Similarly, it’s important that the entire process is well organised and thought through in detail before testing begins. Who oversees the testers and who creates the test design? Is it the service provider or the company itself? Who implements the results of the testing process and responds to questions in case the software doesn’t work after fixes are applied?
Solution: Including crowdtesting into early planning allows it to become a fixed part of the development process. If responsibilities are clearly defined early on and the processes are well-established, service providers, clients and the crowd will all be able to have an efficient and successful testing experience.
[easy-tweet tweet=”Prior to launch, a crowd test can also help customers use your app.” hashtags=”tech, cloud”]
Personal Recommendation: “Most of the time you only have one shot at success, especially if it’s an online shop or gaming app. There are so many alternatives available on the market that a customer can choose instead of your product. Therefore, involving the crowd, YOUR crowd or target group as soon as possible in order to get feedback to be able to deliver what they want is extremely important. Prior to launch, a crowd test featuring a number of different devices to test functionality can also help customers use your app instead of deleting it because it doesn’t work as expected.”
2. Realistic Planning and the Right Testing Method
When developing technology today, time pressures are extremely high as companies attempt to deliver their digital products to the market as quickly as possible. Due to delays or a tough schedule, there’s often not enough time for testing. Even though the crowd works quickly and are extremely flexible, it’s still important that the testing phase is planned realistically and the right testing method is chosen based on the individual necessities of the digital product in question. It’s equally important that time is set aside for fixing issues found during testing. It doesn’t make sense to find problems if there is no time to fix them.
Solution: Depending on the digital product in question, it could make sense to test single components or functions before the product is ready. For example, using the crowd to test prototypes or mock-ups allows problems to be fixed before they arise. This in turn allows the development process to stick to tough time schedules despite the inclusion of extensive testing.
Practical Recommendation: “We often get calls from people who want to test over the weekend and go live the Monday after. This is completely manageable using our crowd. But how do clients expect to fix the bugs in such a short time period? We always recommend calculating a testing phase of up to two weeks and an extra two weeks for fixing the problems that are found.”
3. Careful Preparation
Planning is an absolute must no matter what testing method is used. With crowdsourced testing it’s especially important as testers work remotely, meaning that they are unable to receive feedback immediately. Before testing it’s important that the goal is kept in mind, questionnaires with detailed instructions on what to do are designed and the steps that will guide testers through the process are well defined. It’s important that the people designing the test try to visualise the problems that might arise or aspects that might be unclear to testers when setting them up. Asking the right questions means getting the right results.
Solution: Testing in a laboratory environment and remotely are two different things, therefore the requirements of those conducting the tests are also different. Due to the specific needs of remote testing, working with people who know how to set up clear instructions and have experience in how the testers will act and carry them out is a must.
Practical Recommendation: “Being well prepared is an absolute must, especially for tests where you need to gather feedback from a tester. Questions need to be defined carefully as we want to refrain from receiving the same feedback for two different questions. Take your time when defining them. At Testbirds, a project manager is always there to help and advise you with expert input.”
4. Best Possible Conditions
In order to complete their tasks quickly and efficiently, it is a must that a high functioning and user-friendly online platform and infrastructure is available for testing. It’s equally important to keep in mind the interests of all stakeholders involved. Besides flexibility, when it comes to different testing methods, the platform should also feature helpful aspects such as bug exporting tools, tracking tools, and direct communication channels between testers and the managers of the test. It’s important that these factors are taken into consideration when choosing a service provider.
Solution: Continuous servicing and development are essential aspects of a productive and successful infrastructure. Regularly gathering feedback from stakeholders grants a better understanding of what is needed or missing. The rule of thumb here is to never stop improving.
Practical Recommendation: “Our testing platform is exclusively developed and maintained by Testbirds. This means that whenever there is something wrong we can immediately react and try to fix it. For example, if a tester reports that the button to submit a bug doesn’t work on their device, we can instantly fix it. Similarly, every client has different needs. In order to create the best possible conditions for them and their team on our platform we try to gather feedback for improvements after every test run.”
5. Reliable Testing Environment
Testers need easy and reliable access to test, no matter the kind of digital product that is being tested (beta apps, live online shops, connected devices, etc). Applications have to be stable enough to be tested. It’s difficult to test when updates are running as this affects the process and leads to unreliable results. Furthermore, it’s important to document which version has been tested and on which one bugs have occurred. For unreleased software, security issues have to be considered due to confidentiality; therefore a safe method of access needs to be provided.
[easy-tweet tweet=”Testers need easy and reliable access, no matter the kind of digital product being tested” hashtags=”cloud, tech”]
Solution: A VPN for secure and easy access is a must during the testing or staging environment. For fast beta app distribution, online cloud based tools are often a great solution.
Practical Recommendation: “Whenever we test products that have not yet gone live, we face clients that fear the app will be distributed to all 150,000 members of our crowd. In reality, though, only testers who are specifically chosen have access to the product. We also offer our own app distribution tool, BirdFlight, where we offer full control over who has access to a beta app. After a test is completed, we remove it internally from our servers hosted in Germany so that no one can access it again.”
6. Entry Tests and Trainings
In the lab, test managers can help testers who encounter problems. Remotely, however, testers have to partly take over the tasks of the managers themselves. In essence, they have more responsibility. For this reason, it’s vital to train people appropriately and make them aware of requirements. It’s equally important to show them how best to use the platform. Entry tests can ensure that testers know how to handle their devices and the testing platform, which in turn can yield to results of a higher quality.
Solution: Tutorials and videos that show how the platform works and how to report bugs can ensure that testers are well prepared to handle the responsibilities expected of them.
Practical Recommendation: “All testers who sign up to our platform have to complete an entry test before they can participate in real tests. We do that so that our testers have a chance to become familiar with our platform and the way in which we conduct tests. This way, they can later on concentrate on the actual test instead of trying to figure out how our platform works. In addition, we offer training in our “Bird School” where testers can learn more about, for example, functionality testing or why good usability is important among many other topics.”
7. Choose the right testers
The crowd is a reflection of society. It can represent a number of age groups, jobs, education levels, family members among many more demographics. These are often normal people without extensive testing knowledge or expertise. In addition, what is required of them differs from project to project. To select the right people you need as much information about them as possible. For usability testing, people that represent the target group of the digital product that is to be tested is a must as in the end, they will be the ones to use and review it. For functionality testing, this isn’t as big of an issue, instead it’s important that testers are experienced and have the necessary devices.
[easy-tweet tweet=”The crowd is a reflection of society. It can represent many demographics” hashtags=”cloud, tech”]
Solution: Inexperienced testers search for bugs and usability issues in a different manner than testing experts. When hunting for bugs or figuring out shortcomings in user experience, it’s important to have a group of testers with varying degrees of experience.
Practical Recommendation: “Let’s say a client asks us to test their app with their target group consisting of women over 50 years old. While we can easily accomplish this, an important question to consider is if it makes sense to include their target group if they are only looking for technical issues in the app. Sometimes, the target group of a digital product isn’t the type of person who knows how to find bugs. For these situations, we have special bug finders on our platform. They know where apps often have flaws and will try everything they can to break the app. This will get you the best results.”
8. How to Handle Further Enquiries
Test managers are unable to provide direct feedback when remote testing. Therefore, it’s even more important that the feedback they do provide is perfect. It’s important to try to solve possible technical issues that testers may encounter with strong and clear instructions. When there are issues it’s vital that test managers react quickly to get high quality results. Checking the feedback from testers carefully and changing instructions based on misunderstandings is also a must.
Solution: Always provide testers with a personal contact and have their information readily available. Communicate the availability of this person in case testers need to ask something. A forum for communication between the testers themselves can also be helpful and allows them to fix issues on their own.
Practical Recommendation: “In every test there are some testers who have questions such as about the setup or an app that can’t be installed on a particular device, etc. At Testbirds, project managers are the tester’s person of contact. During the setup and briefing, they will receive a lot of information on the product and the internal processes so that they can answer almost any question that testers might have. Some tests also have a dedicated forum where testers can communicate and help each other. Finally, if necessary, the client can also communicate with them directly.”
9. Detailed Documentation and Quality Assurance
One of the critical aspects of successful remote testing is comprehensive documentation. Testers have to report the steps they take in a detailed and precise manner. Every step needs to be documented and supported with evidence such as screenshots. Reports have to be reviewed in regards to how complete they are, if they are traceable, of a high quality and whether they are reproducible or not. Have the testers done what they were supposed to do? Did they test on the right devices and software? How important or critical is the bug in question?
Solution: Testers learn and develop with time and projects. Give feedback on what they should be aware of for the next project, what can be done better and clearly explain situations that may arise such as why a bug was not accepted.
Practical Recommendation: “All clients have their own way of documenting bugs in their internal systems. This is why we are working with a flexible tool that can be adapted to all kinds of bug documentation systems. Furthermore, all bugs run through an internal quality assurance process in order to make sure they are valid and well documented for clients. This way we can ensure that clients only receive relevant bugs and no duplicates that clog the system. “
10. Fair Payment and Incentives
When it comes to remote testing, testers are the key to success. Happy testers are good testers. It is therefore a must to provide adequate incentives and motivating tasks. Other than fair payment, a reward system can also be used to motivate people, based on, for example the quality of the bugs or gamification of the testing process. Additional payment for bugs found when performing functionality testing can lead to an increase in creativity and engagement when hunting for bugs.
Solution: Experience points for the participation of tests helps to qualify testers and can also serve as motivation for the testers themselves. Rankings and awards for extraordinary achievements can also be used to further motivate the crowd.
Practical Recommendation: “A lot of testers test with us because they are interested in trying out new products. However, others also test with us for earning some money while testing new or soon to be released products. It’s a win-win situation. Other than those aspects, Community Points are extremely popular with our testers, which they receive for performing tasks such as participating as backup testers or writing an excellent test report. This in turn allows them to climb the ranks, which has now become a popular competition on our platform.”
Sahil Deva is Testbirds’ UK Marketing & PR Manager.
Sahil Deva uses his background in Business and IT Management as well as Creative Writing to make Testbirds UK’s leading innovative testing service provider.
Company Bio
Testbirds specialises in the testing of digital products such as apps, websites and connected devices by using innovative technologies and methods. Under the slogan “Testing 4.0 – The Next Generation of Quality”, Testbirds aims to become the world’s leading company for innovative testing solutions. With over 150,000 testers in 193 countries, Testbirds is now one of the world’s leading crowdtesting providers. In addition, the company offers cloud based technologies for the optimization of digital products. The combination of the two form Testbirds’ unique portfolio that promises to take the quality of technology to the next level.