You’re doing forms all wrong: How to ask correct questions to collect precise data-points
Pranav came to me to discuss an idea of him running masterclasses in his new venture Material360. He wanted to share the hard truths and efficient processes he created through his years of building architecture and interior design businesses from the ground up.
He had already discussed the idea with a few of his peers and colleagues, and had a few early sign ups and soft commits.
As a long time mentor to founders Pranav and Eshwar and an advisor to Material360, I was enlisted to hear him out and his plans. The clarity of why and the need were present in the first few sentences and we proceeded to discuss the details.
His team has incredible belief in his competence, and so do I. This is why in our initial conversation, we quickly proceeded to discuss the details rather than whether he’s the right person or his capacity to run this.
Of the very few things he missed, collecting key data points from his peers, from his first sign ups and soft commits was missing. I implored him and his team to sit down, brainstorm, and create a form to gather information that could help him run a much more effective masterclass that’s tailored to the audience. Having realized its importance, he took to brainstorming with his team and returned a day later with a form.
Having identified its strong suits and a few areas where the form could do better, I then sat down with Pranav and his team to guide them through creating effective forms. A key area they needed to work on was in structuring better options and reducing open-ended questions.
The conversation that ensued that day resulted in this guide — a sector agnostic actionable handbook to creating effective forms. Some sections are even accompanied with excerpts from those conversations.
Two reasons to run a questionnaire and prerequisites of creating an effective form
Get on the same page
Ever filled a form before an event like a webinar? Answers from that form would have helped the show runners get on the same page with their audience's depth of knowledge about the topic and their expectations.
Instead of opening the event with a broad open-ended question and waiting for people to open up, you should have your audience answer a few precise questions before the event.
This clear goal around the event and getting on the same page helps you structure your questions effectively to deliver greater value to your audience as opposed to a YouTube tutorial that caters to a broad audience.
Discover insights
The second reason is a more inquisitive one — a questionnaire that aims to discover insights, uncover hidden realities, and understand ground truths.
For example, we’re all familiar with the infamous forms Superhuman uses to continuously assess PMF or the surveys SaaS companies run to gauge customer satisfaction.
There are infinitely more surveys run elsewhere, like governments or research institutes running broader surveys.
The science to asking better questions
Learning how to ask questions better can help you extract quality data-points from both your forms and everyday conversations with people. Here are a few pitfalls in asking questions.
Boring or looooooong questions
Questions that aren’t relevant or take a long winding road to get to the point tend to go unanswered, or are answered with the least interest or thought. This can skew your data with answers that were filled up haphazardly.
This means, you either don’t get enough fill up rates which results in not having enough statistical data to draw conclusions or run the bigger risk of misleading conclusions.
Complexity of questions
Make it extremely simple. It should be easy enough to understand and answer quickly. Time to answer after finishing the question must not be measured in minutes, or even dozens of seconds. While you want your audience to put thought into what they’re answering, you’re not asking them to explain quantum mechanics.
Less to no open-ended questions, All or mostly close-ended questions:
Aside from showing how you couldn’t think of all scenarios, there might be a time and place for open-ended questions — it’s usually at the end and to collect information about what you missed in the questionnaire.
In all seriousness, not all questions can be captured in short answers or choices or checkboxes. But try your best to do so and reserve open-ended questions and answers to 1 question per questionnaire.
This is because it requires a lot of effort on your audience's part to think, structure, write, and submit this (mini-)essay.
This leads to low answer rates for such questions and if you leave this to an important topic, you’re lacking data. Of the answers you end up getting for your open-ended questions, it becomes a challenge to assess its quality and pick data points that can help you draw qualitative conclusions
Here’s a screenshot of the form the team had created, with one open-ended questions after each question in an attempt to “capture everything”
To drive home the point, I encouraged the team to do a simple A/B test with their peers between the original form and the new form created after the discussions. Here are a few responses from their original form (don’t worry, I’ve obtained all permissions from the authors of each answer you see in the screenshot below)
// image
Staying on the topic
Don’t discuss more than one topic in the question, otherwise you run the risk of confusing your audience, or worse, misleading your audience making them choose a less ideal or wrong part or meaning to your question
Misleading questions and acquiescence bias (agreeing without much protest): yes-no, agree-disagree, instead offer example statements.
questions shouldn’t offer conclusions or leading questions that follow with a yes-no, agree-disagree option. For example:
Do you think a workshop will be effective to help you get better? vs What would you choose to get better? a workshop? a tag-along for a week to learn the ropes.
Questions order:
Your goal should be to swiftly walk your audience through a story in a logical order.
If you make them switch topics, only to return at a later question, you run the risk of confusing your audience and mental overhead with all that context switching.
This results in poor answers especially with the questions towards the end of the questionnaire, thus weakening your data.
Lack of logic or disjoint stories and topics result in influence between questions. Pew Research study: “Researchers have demonstrated that the order in which questions are asked can influence how people respond; earlier questions can unintentionally provide context for the questions that follow (these effects are called “order effects”).”
This not only skews your data for this questionnaire, it also means trends over time are less reliable.
This takes us back to context switching between questions — to avoid influencing opinions of the previous question, try to be on the same topic and have clear distinctions when moving to another
Create meaningful options to choose from.
Ambiguous answers are open for interpretation. Most often observed with options like beginner, intermediate, and advanced — is there a definition of who and what qualifies a person as a beginner vs intermediate vs advanced? Even designations like Sr XYZ are not sufficient.
Here’s an easy solution: Pick an instance that qualifies a person as a beginner.
Getting to the root of what data-point you’re trying to capture with the question should help you create options for the question. After pointing out the ambiguous nature, here’s a version we created to capture accurate information about the experience of the audience who were attending the masterclass:
Clear and mutually exclusive options that have no common topics between the options. This avoids confusing your audience and gives you clear data-points.
How well did it work?
When compared to the responses from their peers, the new form had higher completion rates, more accurate responses, and best of all, time spent on the form was halved.
While the team at Material360 were grateful for having learned a lot of “something new”, I on the other had a fantastic time working with the brilliant and very entrepreneurial team at Material360 and I look forward to more such spirited discussion