Making a lead qualification quiz shorter: a data-driven approach¶
Xoel López Barata - @xoelipedes¶
The problem¶
The quiz¶
- Isn't completed by ~50% of people that register
- Takes ~1h to complete
- Random set of 8 questions taken out of a bank of 12
Goals¶
- Get more people to take the quiz?
- Get less people to not abandon the quiz?
- Make it shorter?
The data¶
- Only students that started the quiz are included (so ~50% of users on Apply are excluded because there's no data for them I could use)
- The dataset has ~2700 students
The dataset has 2707 rows and 54 columns.
| answer | completed_at | result | time_to_complete | status | id | user_id | num_questions_right | passed | quiz_questions | |||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| red_dad | hurt_tea | left_bread | lean_throat | frail_food | chief_scene | huge_depth | long_road | sweet_gate | parched_night | scarce_hair | smooth_love | red_dad | hurt_tea | left_bread | lean_throat | frail_food | chief_scene | huge_depth | long_road | sweet_gate | parched_night | scarce_hair | smooth_love | red_dad | hurt_tea | left_bread | lean_throat | frail_food | chief_scene | huge_depth | long_road | sweet_gate | parched_night | scarce_hair | smooth_love | red_dad | hurt_tea | left_bread | lean_throat | frail_food | chief_scene | huge_depth | long_road | sweet_gate | parched_night | scarce_hair | smooth_love | |||||||
| 0 | Davon is 3, Josefa is 5, Blanca is 10, Gerda i... | Clara purchased book 6. | Alford has a meeting first thing in the morning. | NaN | Gladyce juggles and practices figureskating | 1: Tomasa, 2: Paxton, 3: Jed, 4: Ford, 5: Juan... | NaN | Rodolfo is on the Gene Sequencing team. | NaN | Attending: Colton, Zachary, Jonathan, Joe, She... | NaN | Nobody chose a number between Alanna and Alexys. | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | True | True | True | NaN | True | True | NaN | True | NaN | True | NaN | True | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | rejected_at_interview | 3470 | 4581 | 8.0 | True | [extracurricular_activities, business_meeting,... |
| 1 | NaN | Johathan purchased book 5. | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 2019/07/29 09:08:13 +0000 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | False | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 128.470957 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | pending | 7813 | 9165 | 0.0 | False | [book_club, hackday, game_night, extracurricul... |
| 2 | NaN | Brooke purchased book 6. | Brendan is having lunch with Israel. | A beanie can only be green. | NaN | NaN | Maya is 19 years old. | Quinn is on the Gene Sequencing team. | Rolando: #1, Jacinto: #2, Vernon: #3, Declan: ... | NaN | Maybelle spilled the milk. | Nobody chose a number between Gerhard and Cary. | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | True | True | True | NaN | NaN | True | True | True | NaN | True | True | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | accepted_invitation | 455 | 650 | 8.0 | True | [game_night, conference_swag, business_meeting... |
| 3 | Judge is 1, Juliana is 6, Gino is 7, Natasha i... | Sam purchased book 7. | NaN | A hoodie can only be white. | NaN | NaN | Antwan is 19 years old. | Crawford is on the Video Game team. | Jasen: #1, Werner: #2, Elta: #3, Jailyn: #4, H... | NaN | Lenny spilled the milk. | Craig chose number 22. | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | True | False | NaN | True | NaN | NaN | True | True | False | NaN | False | False | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | rejected_at_logic_evaluation | 54 | 84 | 4.0 | False | [book_club, massage_therapist, sports_team, ga... |
| 4 | Neha is 1, Jaylen is 3, Vesta is 7, Clarissa i... | NaN | NaN | A hoodie can be turquoise, pink or yellow. | Elouise attends a painting workshop and goes r... | 1: Kenyatta, 2: Solon, 3: Leo, 4: Jackeline, 5... | NaN | Hassie is on the Botnet team. | Layne: #1, Lucienne: #2, Elfrieda: #3, Clint: ... | Attending: Vida, Zella, Nelda, Elmira, Tito, C... | Rogelio spilled the milk. | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | True | NaN | NaN | False | True | False | NaN | True | True | True | True | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | accepted_invitation | 1767 | 2459 | 6.0 | True | [adoption, extracurricular_activities, negotia... |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 2702 | Chanel is 2, Deshawn is 3, Brandyn is 10, Odie... | Joelle purchased book 7. | NaN | A t-shirt can be either pink or blue. | NaN | 1: Carole, 2: Maxie, 3: Willy, 4: Lauryn, 5: C... | Jamison came in #1 in Chess. | Madisen is on the Facial Recognition team. | NaN | NaN | Kamille spilled the milk. | Barton chose number 89. | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | True | False | NaN | False | NaN | False | True | False | NaN | NaN | False | False | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | pending | 6281 | 7589 | 2.0 | False | [adoption, family_reunion, conference_swag, ha... |
| 2703 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | pending | 6951 | 8260 | 0.0 | False | [game_night, book_club, adoption, spilled_milk... |
| 2704 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | pending | 5740 | 7046 | 0.0 | False | [adoption, sports_team, conference_swag, massa... |
| 2705 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | pending | 3589 | 4736 | 0.0 | False | [business_meeting, game_night, sports_team, sp... |
| 2706 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | pending | 4336 | 5646 | 0.0 | False | [conference_swag, negotiations, spilled_milk, ... |
2707 rows × 54 columns
What does 'passed' mean?¶
- A student got 6 or more questions right
- A student got 5 questions right and its status is not
rejected_at_initial_evaluationorrejected_at_logic_evaluation
I'll try to find out which questions are the best predictors of good logical thinking.
Exploratory Data Analysis¶
In any data project, the first part will always be an exploration of the data to understand how the variables are distributed and related to each other.
This will let us know which questions to ask and in which direction to go with our analysis.
Number of people that got N questions right¶
- 52% of the people get 7 or 8 questions right
- 21% get 5 or 6 right
- 27% get 4 or less right.
- ~73% of the people pass the test.
Time to complete questions¶
There are some questions that some people took way to long to answer, so we have outliers that distort the graph and don't let us see clearly the distribution of time each question took.
It's necessary to remove those outliers to see it better.
After deleting the times of the questions that took longer than the 99% percentile, the data looks much clearer. We can see that people take about 6 minutes per question.
If we remove the longest questions, this number can go down too.
Time to complete the quiz in total¶
In total, most of the people are taking between 30 minutes and 1h15 to complete the quiz
The people that take the least amount of time normally don't pass, while after 15-20 mins spent on the test we start seeing people that passed.
Time per question¶
huge_depth 10.105253 chief_scene 7.218838 sweet_gate 6.753527 hurt_tea 5.800526 smooth_love 5.516701 parched_night 5.408823 scarce_hair 5.342117 long_road 5.276620 frail_food 5.273024 red_dad 4.084204 left_bread 3.612293 lean_throat 2.634215 Name: Median time (minutes), dtype: float64
Some questions take much longer than others, like huge_depth (~10 mins), while others like lean_throat take less than 3 minutes.
Here we can see the distribution of time for each question. The best we can get from this graph is that for some questions, like spilled_milk, even though the median time they take is not very long, there are some people that take much longer to answer it than for other questions.
The same thing happens with extracurricular_activities or massage_therapist
Difficulty - % of the people that get right each question¶
Some questions like hackday or adoption are gotten right by most of the people. In theory, those shouldn't be very useful to tell us how likely a student is to pass the quiz or not, as most of them are answering them correctly.
We'll see later if this is true.
Time VS difficulty¶
Now that we have median times and the percentage of people that get each question right, we can more easily compare the data for each question:
| % answers right | Median time (minutes) | |
|---|---|---|
| long_road | 88.8% | 5.3 |
| red_dad | 87.7% | 4.1 |
| chief_scene | 82.0% | 7.2 |
| parched_night | 81.6% | 5.4 |
| sweet_gate | 79.0% | 6.8 |
| huge_depth | 77.4% | 10.1 |
| left_bread | 77.4% | 3.6 |
| hurt_tea | 77.2% | 5.8 |
| lean_throat | 68.6% | 2.6 |
| smooth_love | 65.1% | 5.5 |
| frail_food | 64.7% | 5.3 |
| scarce_hair | 60.6% | 5.3 |
We can also plot is with a scatterplot, but this is still not enough information to take a decision on which questions are the best to take out of the quiz.
Who to focus on?¶
- People that leave before finishing?
- People that never get to take the quiz?
To answer this question, we should look at how many people are leaving the quiz half-trough. If it's a significant amount of people, we should also focus on them.
But if they're not, the focus should be on making the quiz shorter so that that can be announced and more people potentially take the quiz and join the school.
As we see, most of the people (91%) answer the 8 questions of the quiz, and about 6.5% of them only answer 1 or no question at all.
About 8% of poeple answer only 3 questions or less.
And here we see the % of right questions that people get, among the people that answer few questions, is very low.
So those people probably are not good enough and the focus shouldn't be on them.
Therefore, what should tried to be done is making the quiz shorter so that more people take it in the first place. This will be done by taking out questions, and to decide that, I'll calculate the importance of each one, to take out the least important ones and the ones that take the longest.
Correlation of answers¶
This is something I tried but wasn't so useful: Getting the correlation between the answers. If there were some pairs of answers with a correlation very close to 1, it would mean that people that got one right would also get the other one right most of the time, and viceversa, and one of those two questions could be removed. But it wasn't the case
Question importances¶
The basic idea: we're training a maching learning algorithm¶
To calculate importances, I'll first train a machine learning algorithm to try to predict whether a student passed or not, based on his answers.
Then, I'll remove the answers for each one of the questions of the dataset. The algorithm's performance will decrease.
The questions where the algorithm performance decreases the most will be the most important ones.
Now we can compare all the 3 relevant metrics we've calculated for each question
| % answers right | Median time (minutes) | Importance | |
|---|---|---|---|
| hurt_tea | 77.2% | 5.8 | 3.88% |
| left_bread | 77.4% | 3.6 | 3.33% |
| scarce_hair | 60.6% | 5.3 | 2.72% |
| chief_scene | 82.0% | 7.2 | 2.09% |
| lean_throat | 68.6% | 2.6 | 2.01% |
| frail_food | 64.7% | 5.3 | 1.98% |
| parched_night | 81.6% | 5.4 | 1.96% |
| huge_depth | 77.4% | 10.1 | 1.79% |
| smooth_love | 65.1% | 5.5 | 1.75% |
| sweet_gate | 79.0% | 6.8 | 1.55% |
| red_dad | 87.7% | 4.1 | 1.46% |
| long_road | 88.8% | 5.3 | 1.21% |
The most important plot...¶
Another way to represent in a more visual form the 3 metrics for each question
Recapping...¶
- We want to have the questions that give us the most information
- We want to make quiz shorter
- Because the goal is to get more people to take the quiz
Finally: removing questions¶
Now, I'll remove questions from the quiz, starting from the least important ones, and see how many people would pass and how many people would not, from the original people that completed the quiz.
I'll always remove huge_depth, as it takes a very long time to complete, compared to the other ones.
These results could help to decide where to put the new threshold for the minimum number of right questions, after removing questions. Ideally, the threshold would let everyone that passed before pass now, and reject everyone that didn't pass before.
- Removing questions from the initial bank of questions (currently at 12 questions)
- Showing people fewer questions than 8
Results for showing 8 questions¶
Results for the people that received 8 questions out of 11¶
Questions to remove: huge_depth
Times to complete the quiz (minutes): Q1: 31.0. Median: 45.1. Q3: 61.5.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 12 | 0 |
| 1.0 | 28 | 0 |
| 2.0 | 36 | 0 |
| 3.0 | 42 | 0 |
| 4.0 | 47 | 0 |
| 5.0 | 8 | 59 |
| 6.0 | 0 | 119 |
| 7.0 | 0 | 205 |
| 8.0 | 0 | 261 |
Total number of people: 817. People that did not pass: 173. People that passed: 644
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 6.9% | 100.0% | 3.8% | 0.0% |
| 2.0 | 23.1% | 100.0% | 6.3% | 0.0% |
| 3.0 | 43.9% | 100.0% | 7.4% | 0.0% |
| 4.0 | 68.2% | 100.0% | 6.9% | 0.0% |
| 5.0 | 95.4% | 100.0% | 3.1% | 0.0% |
| 6.0 | 100.0% | 90.8% | 0.0% | 2.2% |
| 7.0 | 100.0% | 72.4% | 0.0% | 3.5% |
| 8.0 | 100.0% | 40.5% | 0.0% | 3.8% |
Results for the people that received 8 questions out of 10¶
Questions to remove: huge_depth, red_dad
Times to complete the quiz (minutes): Q1: 36.7. Median: 49.4. Q3: 67.3.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 2 | 0 |
| 1.0 | 4 | 0 |
| 2.0 | 12 | 0 |
| 3.0 | 10 | 0 |
| 4.0 | 14 | 0 |
| 5.0 | 2 | 19 |
| 6.0 | 0 | 34 |
| 7.0 | 0 | 62 |
| 8.0 | 0 | 65 |
Total number of people: 224. People that did not pass: 44. People that passed: 180
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 4.5% | 100.0% | 6.2% | 0.0% |
| 2.0 | 13.6% | 100.0% | 10.1% | 0.0% |
| 3.0 | 40.9% | 100.0% | 14.5% | 0.0% |
| 4.0 | 63.6% | 100.0% | 14.2% | 0.0% |
| 5.0 | 95.5% | 100.0% | 6.2% | 0.0% |
| 6.0 | 100.0% | 89.4% | 0.0% | 4.5% |
| 7.0 | 100.0% | 70.6% | 0.0% | 6.7% |
| 8.0 | 100.0% | 36.1% | 0.0% | 7.0% |
Results for the people that received 8 questions out of 9¶
Questions to remove: huge_depth, red_dad, long_road
Less than 50 people, probably not enough data to make significant conclusions
Results for the people that received 8 questions out of 8¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate
Less than 50 people, probably not enough data to make significant conclusions
Results for showing 7 questions¶
Results for the people that received 7 questions out of 11¶
Questions to remove: huge_depth
Times to complete the quiz (minutes): Q1: 28.9. Median: 38.8. Q3: 55.2.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 28 | 0 |
| 1.0 | 61 | 0 |
| 2.0 | 89 | 0 |
| 3.0 | 105 | 0 |
| 4.0 | 55 | 94 |
| 5.0 | 6 | 224 |
| 6.0 | 0 | 417 |
| 7.0 | 0 | 568 |
Total number of people: 1647. People that did not pass: 344. People that passed: 1303
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 8.1% | 100.0% | 2.9% | 0.0% |
| 2.0 | 25.9% | 100.0% | 4.6% | 0.0% |
| 3.0 | 51.7% | 100.0% | 5.3% | 0.0% |
| 4.0 | 82.3% | 100.0% | 4.0% | 0.0% |
| 5.0 | 98.3% | 92.8% | 1.4% | 1.4% |
| 6.0 | 100.0% | 75.6% | 0.0% | 2.3% |
| 7.0 | 100.0% | 43.6% | 0.0% | 2.7% |
Results for the people that received 7 questions out of 10¶
Questions to remove: huge_depth, red_dad
Times to complete the quiz (minutes): Q1: 27.9. Median: 39.3. Q3: 55.4.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 24 | 0 |
| 1.0 | 50 | 0 |
| 2.0 | 64 | 0 |
| 3.0 | 75 | 0 |
| 4.0 | 31 | 71 |
| 5.0 | 5 | 172 |
| 6.0 | 0 | 286 |
| 7.0 | 0 | 383 |
Total number of people: 1161. People that did not pass: 249. People that passed: 912
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 9.6% | 100.0% | 3.7% | 0.0% |
| 2.0 | 29.7% | 100.0% | 5.7% | 0.0% |
| 3.0 | 55.4% | 100.0% | 6.2% | 0.0% |
| 4.0 | 85.5% | 100.0% | 4.4% | 0.0% |
| 5.0 | 98.0% | 92.2% | 1.7% | 1.7% |
| 6.0 | 100.0% | 73.4% | 0.0% | 2.9% |
| 7.0 | 100.0% | 42.0% | 0.0% | 3.2% |
Results for the people that received 7 questions out of 9¶
Questions to remove: huge_depth, red_dad, long_road
Times to complete the quiz (minutes): Q1: 29.8. Median: 40.4. Q3: 57.1.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 11 | 0 |
| 1.0 | 19 | 0 |
| 2.0 | 31 | 0 |
| 3.0 | 28 | 0 |
| 4.0 | 11 | 33 |
| 5.0 | 2 | 77 |
| 6.0 | 0 | 153 |
| 7.0 | 0 | 162 |
Total number of people: 527. People that did not pass: 102. People that passed: 425
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 10.8% | 100.0% | 6.0% | 0.0% |
| 2.0 | 29.4% | 100.0% | 8.8% | 0.0% |
| 3.0 | 59.8% | 100.0% | 9.5% | 0.0% |
| 4.0 | 87.3% | 100.0% | 6.5% | 0.0% |
| 5.0 | 98.0% | 92.2% | 2.7% | 2.5% |
| 6.0 | 100.0% | 74.1% | 0.0% | 4.2% |
| 7.0 | 100.0% | 38.1% | 0.0% | 4.6% |
Results for the people that received 7 questions out of 8¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate
Times to complete the quiz (minutes): Q1: 28.8. Median: 37.7. Q3: 50.7.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 3 | 0 |
| 1.0 | 7 | 0 |
| 2.0 | 14 | 0 |
| 3.0 | 8 | 0 |
| 4.0 | 2 | 9 |
| 5.0 | 2 | 17 |
| 6.0 | 0 | 43 |
| 7.0 | 0 | 51 |
Total number of people: 156. People that did not pass: 36. People that passed: 120
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 8.3% | 100.0% | 9.0% | 0.0% |
| 2.0 | 27.8% | 100.0% | 14.6% | 0.0% |
| 3.0 | 66.7% | 100.0% | 15.4% | 0.0% |
| 4.0 | 88.9% | 100.0% | 10.3% | 0.0% |
| 5.0 | 94.4% | 92.5% | 7.5% | 4.7% |
| 6.0 | 100.0% | 78.3% | 0.0% | 7.4% |
| 7.0 | 100.0% | 42.5% | 0.0% | 8.8% |
Results for the people that received 7 questions out of 7¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food
Less than 50 people, probably not enough data to make significant conclusions
Results for showing 6 questions¶
Results for the people that received 6 questions out of 10¶
Questions to remove: huge_depth, red_dad
Times to complete the quiz (minutes): Q1: 24.7. Median: 33.4. Q3: 46.6.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 28 | 0 |
| 1.0 | 64 | 0 |
| 2.0 | 81 | 0 |
| 3.0 | 46 | 47 |
| 4.0 | 5 | 138 |
| 5.0 | 0 | 277 |
| 6.0 | 0 | 393 |
Total number of people: 1079. People that did not pass: 224. People that passed: 855
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 12.5% | 100.0% | 4.3% | 0.0% |
| 2.0 | 41.1% | 100.0% | 6.4% | 0.0% |
| 3.0 | 77.2% | 100.0% | 5.5% | 0.0% |
| 4.0 | 97.8% | 94.5% | 1.9% | 1.5% |
| 5.0 | 100.0% | 78.4% | 0.0% | 2.8% |
| 6.0 | 100.0% | 46.0% | 0.0% | 3.3% |
Results for the people that received 6 questions out of 9¶
Questions to remove: huge_depth, red_dad, long_road
Times to complete the quiz (minutes): Q1: 23.0. Median: 33.4. Q3: 47.6.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 42 | 0 |
| 1.0 | 82 | 0 |
| 2.0 | 91 | 0 |
| 3.0 | 54 | 63 |
| 4.0 | 7 | 184 |
| 5.0 | 0 | 307 |
| 6.0 | 0 | 437 |
Total number of people: 1267. People that did not pass: 276. People that passed: 991
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 15.2% | 100.0% | 4.2% | 0.0% |
| 2.0 | 44.9% | 100.0% | 5.9% | 0.0% |
| 3.0 | 77.9% | 100.0% | 4.9% | 0.0% |
| 4.0 | 97.5% | 93.6% | 1.9% | 1.5% |
| 5.0 | 100.0% | 75.1% | 0.0% | 2.7% |
| 6.0 | 100.0% | 44.1% | 0.0% | 3.1% |
Results for the people that received 6 questions out of 8¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate
Times to complete the quiz (minutes): Q1: 24.0. Median: 33.3. Q3: 47.5.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 28 | 0 |
| 1.0 | 47 | 0 |
| 2.0 | 57 | 0 |
| 3.0 | 43 | 46 |
| 4.0 | 1 | 122 |
| 5.0 | 0 | 223 |
| 6.0 | 0 | 262 |
Total number of people: 829. People that did not pass: 176. People that passed: 653
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 15.9% | 100.0% | 5.4% | 0.0% |
| 2.0 | 42.6% | 100.0% | 7.3% | 0.0% |
| 3.0 | 75.0% | 100.0% | 6.4% | 0.0% |
| 4.0 | 99.4% | 93.0% | 1.1% | 2.0% |
| 5.0 | 100.0% | 74.3% | 0.0% | 3.4% |
| 6.0 | 100.0% | 40.1% | 0.0% | 3.8% |
Results for the people that received 6 questions out of 7¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food
Times to complete the quiz (minutes): Q1: 24.4. Median: 33.0. Q3: 47.5.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 8 | 0 |
| 1.0 | 26 | 0 |
| 2.0 | 28 | 0 |
| 3.0 | 13 | 12 |
| 4.0 | 2 | 43 |
| 5.0 | 1 | 89 |
| 6.0 | 0 | 112 |
Total number of people: 334. People that did not pass: 78. People that passed: 256
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 10.3% | 100.0% | 6.7% | 0.0% |
| 2.0 | 43.6% | 100.0% | 11.0% | 0.0% |
| 3.0 | 79.5% | 100.0% | 9.0% | 0.0% |
| 4.0 | 96.2% | 95.3% | 4.3% | 2.6% |
| 5.0 | 98.7% | 78.5% | 2.5% | 5.0% |
| 6.0 | 100.0% | 43.8% | 0.0% | 6.1% |
Results for the people that received 6 questions out of 6¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene
Times to complete the quiz (minutes): Q1: 28.1. Median: 33.4. Q3: 41.9.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 2 | 0 |
| 1.0 | 3 | 0 |
| 2.0 | 5 | 0 |
| 3.0 | 1 | 1 |
| 4.0 | 1 | 9 |
| 5.0 | 0 | 26 |
| 6.0 | 0 | 24 |
Total number of people: 72. People that did not pass: 12. People that passed: 60
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 16.7% | 100.0% | 21.1% | 0.0% |
| 2.0 | 41.7% | 100.0% | 27.9% | 0.0% |
| 3.0 | 83.3% | 100.0% | 21.1% | 0.0% |
| 4.0 | 91.7% | 98.3% | 15.6% | 3.2% |
| 5.0 | 100.0% | 83.3% | 0.0% | 9.4% |
| 6.0 | 100.0% | 40.0% | 0.0% | 12.4% |
Results for showing 5 questions¶
Results for the people that received 5 questions out of 9¶
Questions to remove: huge_depth, red_dad, long_road
Times to complete the quiz (minutes): Q1: 20.3. Median: 30.3. Q3: 41.7.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 24 | 0 |
| 1.0 | 61 | 0 |
| 2.0 | 41 | 25 |
| 3.0 | 8 | 76 |
| 4.0 | 1 | 156 |
| 5.0 | 0 | 240 |
Total number of people: 632. People that did not pass: 135. People that passed: 497
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 17.8% | 100.0% | 6.4% | 0.0% |
| 2.0 | 63.0% | 100.0% | 8.1% | 0.0% |
| 3.0 | 93.3% | 95.0% | 4.2% | 1.9% |
| 4.0 | 99.3% | 79.7% | 1.4% | 3.5% |
| 5.0 | 100.0% | 48.3% | 0.0% | 4.4% |
Results for the people that received 5 questions out of 8¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate
Times to complete the quiz (minutes): Q1: 19.4. Median: 27.1. Q3: 37.6.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 47 | 0 |
| 1.0 | 88 | 0 |
| 2.0 | 67 | 37 |
| 3.0 | 23 | 141 |
| 4.0 | 1 | 298 |
| 5.0 | 0 | 419 |
Total number of people: 1121. People that did not pass: 226. People that passed: 895
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 20.8% | 100.0% | 5.3% | 0.0% |
| 2.0 | 59.7% | 100.0% | 6.4% | 0.0% |
| 3.0 | 89.4% | 95.9% | 4.0% | 1.3% |
| 4.0 | 99.6% | 80.1% | 0.9% | 2.6% |
| 5.0 | 100.0% | 46.8% | 0.0% | 3.3% |
Results for the people that received 5 questions out of 7¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food
Times to complete the quiz (minutes): Q1: 19.2. Median: 28.5. Q3: 38.4.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 41 | 0 |
| 1.0 | 69 | 0 |
| 2.0 | 67 | 20 |
| 3.0 | 35 | 116 |
| 4.0 | 0 | 286 |
| 5.0 | 0 | 398 |
Total number of people: 1032. People that did not pass: 212. People that passed: 820
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 19.3% | 100.0% | 5.3% | 0.0% |
| 2.0 | 51.9% | 100.0% | 6.7% | 0.0% |
| 3.0 | 83.5% | 97.6% | 5.0% | 1.1% |
| 4.0 | 100.0% | 83.4% | 0.0% | 2.5% |
| 5.0 | 100.0% | 48.5% | 0.0% | 3.4% |
Results for the people that received 5 questions out of 6¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene
Times to complete the quiz (minutes): Q1: 19.3. Median: 25.9. Q3: 35.8.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 28 | 0 |
| 1.0 | 58 | 0 |
| 2.0 | 45 | 12 |
| 3.0 | 15 | 75 |
| 4.0 | 2 | 165 |
| 5.0 | 0 | 205 |
Total number of people: 605. People that did not pass: 148. People that passed: 457
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 18.9% | 100.0% | 6.3% | 0.0% |
| 2.0 | 58.1% | 100.0% | 7.9% | 0.0% |
| 3.0 | 88.5% | 97.4% | 5.1% | 1.5% |
| 4.0 | 98.6% | 81.0% | 1.9% | 3.6% |
| 5.0 | 100.0% | 44.9% | 0.0% | 4.6% |
Results for the people that received 5 questions out of 5¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat
Times to complete the quiz (minutes): Q1: 20.9. Median: 30.4. Q3: 40.2.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 2 | 0 |
| 1.0 | 12 | 0 |
| 2.0 | 13 | 5 |
| 3.0 | 6 | 19 |
| 4.0 | 1 | 54 |
| 5.0 | 0 | 64 |
Total number of people: 176. People that did not pass: 34. People that passed: 142
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 5.9% | 100.0% | 7.9% | 0.0% |
| 2.0 | 41.2% | 100.0% | 16.5% | 0.0% |
| 3.0 | 79.4% | 96.5% | 13.6% | 3.0% |
| 4.0 | 97.1% | 83.1% | 5.7% | 6.2% |
| 5.0 | 100.0% | 45.1% | 0.0% | 8.2% |
Results for showing 4 questions¶
Results for the people that received 4 questions out of 8¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate
Times to complete the quiz (minutes): Q1: 15.7. Median: 22.3. Q3: 35.4.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 23 | 0 |
| 1.0 | 41 | 5 |
| 2.0 | 15 | 28 |
| 3.0 | 2 | 92 |
| 4.0 | 2 | 152 |
Total number of people: 360. People that did not pass: 83. People that passed: 277
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 27.7% | 100.0% | 9.6% | 0.0% |
| 2.0 | 77.1% | 98.2% | 9.0% | 1.6% |
| 3.0 | 95.2% | 88.1% | 4.6% | 3.8% |
| 4.0 | 97.6% | 54.9% | 3.3% | 5.9% |
Results for the people that received 4 questions out of 7¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food
Times to complete the quiz (minutes): Q1: 15.0. Median: 21.2. Q3: 30.0.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 57 | 0 |
| 1.0 | 73 | 7 |
| 2.0 | 56 | 79 |
| 3.0 | 7 | 233 |
| 4.0 | 1 | 397 |
Total number of people: 910. People that did not pass: 194. People that passed: 716
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 29.4% | 100.0% | 6.4% | 0.0% |
| 2.0 | 67.0% | 99.0% | 6.6% | 0.7% |
| 3.0 | 95.9% | 88.0% | 2.8% | 2.4% |
| 4.0 | 99.5% | 55.4% | 1.0% | 3.6% |
Results for the people that received 4 questions out of 6¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene
Times to complete the quiz (minutes): Q1: 13.9. Median: 19.7. Q3: 29.7.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 60 | 0 |
| 1.0 | 93 | 15 |
| 2.0 | 63 | 104 |
| 3.0 | 4 | 303 |
| 4.0 | 1 | 472 |
Total number of people: 1115. People that did not pass: 221. People that passed: 894
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 27.1% | 100.0% | 5.9% | 0.0% |
| 2.0 | 69.2% | 98.3% | 6.1% | 0.8% |
| 3.0 | 97.7% | 86.7% | 2.0% | 2.2% |
| 4.0 | 99.5% | 52.8% | 0.9% | 3.3% |
Results for the people that received 4 questions out of 5¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat
Times to complete the quiz (minutes): Q1: 16.4. Median: 22.8. Q3: 32.4.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 50 | 0 |
| 1.0 | 90 | 4 |
| 2.0 | 37 | 68 |
| 3.0 | 5 | 228 |
| 4.0 | 1 | 356 |
Total number of people: 839. People that did not pass: 183. People that passed: 656
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 27.3% | 100.0% | 6.5% | 0.0% |
| 2.0 | 76.5% | 99.4% | 6.1% | 0.6% |
| 3.0 | 96.7% | 89.0% | 2.6% | 2.4% |
| 4.0 | 99.5% | 54.3% | 1.1% | 3.8% |
Results for the people that received 4 questions out of 4¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love
Times to complete the quiz (minutes): Q1: 14.5. Median: 22.7. Q3: 29.5.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 9 | 0 |
| 1.0 | 39 | 0 |
| 2.0 | 20 | 20 |
| 3.0 | 4 | 104 |
| 4.0 | 0 | 152 |
Total number of people: 348. People that did not pass: 72. People that passed: 276
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 12.5% | 100.0% | 7.6% | 0.0% |
| 2.0 | 66.7% | 100.0% | 10.9% | 0.0% |
| 3.0 | 94.4% | 92.8% | 5.3% | 3.1% |
| 4.0 | 100.0% | 55.1% | 0.0% | 5.9% |
Results for showing 3 questions¶
Results for the people that received 3 questions out of 7¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food
Times to complete the quiz (minutes): Q1: 11.1. Median: 16.6. Q3: 29.4.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 15 | 0 |
| 1.0 | 24 | 8 |
| 2.0 | 6 | 39 |
| 3.0 | 2 | 88 |
Total number of people: 182. People that did not pass: 47. People that passed: 135
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 31.9% | 100.0% | 13.3% | 0.0% |
| 2.0 | 83.0% | 94.1% | 10.7% | 4.0% |
| 3.0 | 95.7% | 65.2% | 5.8% | 8.0% |
Results for the people that received 3 questions out of 6¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene
Times to complete the quiz (minutes): Q1: 9.9. Median: 15.2. Q3: 21.1.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 56 | 0 |
| 1.0 | 54 | 36 |
| 2.0 | 25 | 150 |
| 3.0 | 1 | 300 |
Total number of people: 622. People that did not pass: 136. People that passed: 486
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 41.2% | 100.0% | 8.3% | 0.0% |
| 2.0 | 80.9% | 92.6% | 6.6% | 2.3% |
| 3.0 | 99.3% | 61.7% | 1.4% | 4.3% |
Results for the people that received 3 questions out of 5¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat
Times to complete the quiz (minutes): Q1: 11.5. Median: 16.3. Q3: 22.1.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 97 | 5 |
| 1.0 | 98 | 55 |
| 2.0 | 38 | 245 |
| 3.0 | 2 | 553 |
Total number of people: 1093. People that did not pass: 235. People that passed: 858
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 41.3% | 99.4% | 6.3% | 0.5% |
| 2.0 | 83.0% | 93.0% | 4.8% | 1.7% |
| 3.0 | 99.1% | 64.5% | 1.2% | 3.2% |
Results for the people that received 3 questions out of 4¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love
Times to complete the quiz (minutes): Q1: 12.3. Median: 16.6. Q3: 22.7.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 90 | 1 |
| 1.0 | 107 | 39 |
| 2.0 | 33 | 236 |
| 3.0 | 1 | 599 |
Total number of people: 1106. People that did not pass: 231. People that passed: 875
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 39.0% | 99.9% | 6.3% | 0.2% |
| 2.0 | 85.3% | 95.4% | 4.6% | 1.4% |
| 3.0 | 99.6% | 68.5% | 0.8% | 3.1% |
Results for the people that received 3 questions out of 3¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love, scarce_hair
Times to complete the quiz (minutes): Q1: 11.9. Median: 15.6. Q3: 20.9.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 24 | 0 |
| 1.0 | 66 | 15 |
| 2.0 | 26 | 112 |
| 3.0 | 0 | 377 |
Total number of people: 620. People that did not pass: 116. People that passed: 504
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 20.7% | 100.0% | 7.4% | 0.0% |
| 2.0 | 77.6% | 97.0% | 7.6% | 1.5% |
| 3.0 | 100.0% | 74.8% | 0.0% | 3.8% |
Results for showing 2 questions¶
Results for the people that received 2 questions out of 6¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene
Times to complete the quiz (minutes): Q1: 6.2. Median: 9.5. Q3: 14.6.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 27 | 1 |
| 1.0 | 15 | 12 |
| 2.0 | 2 | 39 |
Total number of people: 96. People that did not pass: 44. People that passed: 52
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 61.4% | 98.1% | 14.4% | 3.7% |
| 2.0 | 95.5% | 75.0% | 6.2% | 11.8% |
Results for the people that received 2 questions out of 5¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat
Times to complete the quiz (minutes): Q1: 6.8. Median: 10.1. Q3: 15.9.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 52 | 10 |
| 1.0 | 30 | 61 |
| 2.0 | 7 | 207 |
Total number of people: 367. People that did not pass: 89. People that passed: 278
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 58.4% | 96.4% | 10.2% | 2.2% |
| 2.0 | 92.1% | 74.5% | 5.6% | 5.1% |
Results for the people that received 2 questions out of 4¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love
Times to complete the quiz (minutes): Q1: 7.2. Median: 10.5. Q3: 14.8.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 110 | 11 |
| 1.0 | 82 | 140 |
| 2.0 | 13 | 517 |
Total number of people: 873. People that did not pass: 205. People that passed: 668
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 53.7% | 98.4% | 6.8% | 1.0% |
| 2.0 | 93.7% | 77.4% | 3.3% | 3.2% |
Results for the people that received 2 questions out of 3¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love, scarce_hair
Times to complete the quiz (minutes): Q1: 7.0. Median: 10.3. Q3: 14.3.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 142 | 4 |
| 1.0 | 114 | 136 |
| 2.0 | 28 | 852 |
Total number of people: 1276. People that did not pass: 284. People that passed: 992
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 50.0% | 99.6% | 5.8% | 0.4% |
| 2.0 | 90.1% | 85.9% | 3.5% | 2.2% |
Results for the people that received 2 questions out of 2¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love, scarce_hair, parched_night
Times to complete the quiz (minutes): Q1: 7.3. Median: 9.9. Q3: 13.9.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 112 | 8 |
| 1.0 | 81 | 121 |
| 2.0 | 8 | 702 |
Total number of people: 1032. People that did not pass: 201. People that passed: 831
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 55.7% | 99.0% | 6.9% | 0.7% |
| 2.0 | 96.0% | 84.5% | 2.7% | 2.5% |
Results for showing 1 questions¶
Results for the people that received 1 questions out of 5¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat
Times to complete the quiz (minutes): Q1: 1.9. Median: 3.1. Q3: 8.5.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 43 | 1 |
| 1.0 | 8 | 14 |
Total number of people: 66. People that did not pass: 51. People that passed: 15
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 84.3% | 93.3% | 10.0% | 12.6% |
Results for the people that received 1 questions out of 4¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love
Times to complete the quiz (minutes): Q1: 2.1. Median: 5.4. Q3: 8.8.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 62 | 17 |
| 1.0 | 16 | 107 |
Total number of people: 202. People that did not pass: 78. People that passed: 124
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 79.5% | 86.3% | 9.0% | 6.1% |
Results for the people that received 1 questions out of 3¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love, scarce_hair
Times to complete the quiz (minutes): Q1: 3.3. Median: 5.4. Q3: 8.4.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 115 | 33 |
| 1.0 | 49 | 382 |
Total number of people: 579. People that did not pass: 164. People that passed: 415
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 70.1% | 92.0% | 7.0% | 2.6% |
Results for the people that received 1 questions out of 2¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love, scarce_hair, parched_night
Times to complete the quiz (minutes): Q1: 2.8. Median: 4.7. Q3: 7.4.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 246 | 68 |
| 1.0 | 77 | 877 |
Total number of people: 1268. People that did not pass: 323. People that passed: 945
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 76.2% | 92.8% | 4.6% | 1.6% |
Results for the people that received 1 questions out of 1¶
Questions to remove: huge_depth, red_dad, long_road, sweet_gate, frail_food, chief_scene, lean_throat, smooth_love, scarce_hair, parched_night, left_bread
Times to complete the quiz (minutes): Q1: 4.0. Median: 5.8. Q3: 8.8.
| passed | False | True |
|---|---|---|
| num_questions_right_after | ||
| 0.0 | 284 | 97 |
| 1.0 | 89 | 1203 |
Total number of people: 1673. People that did not pass: 373. People that passed: 1300
| Didn't pass before, rejected now | Passed before, accepted now | Didn't pass before, rejected now: 95% conf. interval | Passed before, accepted now: 95% conf. interval | |
|---|---|---|---|---|
| Threshold (minimum questions right) | ||||
| 0.0 | 0.0% | 100.0% | 0.0% | 0.0% |
| 1.0 | 76.1% | 92.5% | 4.3% | 1.4% |
Conclusion¶
We can see that the calculated importance seems to work, as when more questions are removed, the threshold seems to be clearer, for each number of questions received.
Also, the threshold seems to be clearer when a students receives more questions, which also makes sense. So there is a trade-off between how confident you'd want to be that a student would have passed and the time it'd take to complete the quiz.
Finally, I wouldn't advice on trying to get the people that start the quiz and don't finish, as it's clearer that the students that get accepted commit to the 8 questions, and the ones that leave without finishing have a low percent of right answers so they wouldn't have been accepted even if they answered more of them.