This study explores the equivalency of seven picture-based speaking tasks by comparing the oral performance they elicit from Japanese L2 learners. Despite controlling for variables across story length, sequential structure, and storyline complexity, the results were similar only in terms of fluency and not complexity, accuracy, or lexis. The study highlights the importance of piloting testing materials for conducting experimental research using a pretest-posttest design.