Robotics Academy Blog  

Part 2 of 4: The Range of Strategies

Part 1 of this series set the stage for an investigation into student mathematics usage at a local LEGO robotics competition. In Part 2, we'll take a look at the types of strategies that teams came up with for solving the challenge, and how those different approaches fared in the competition.

Interviews with the Teams

22 teams from the greater Pittsburgh area participated in the 2010 May Madness robotics competition. Investigators from the University of Pittsburgh's Learning Research and Development Center (LRDC) and Carnegie Mellon University's Robotics Academy (RA) were able to interview 16 of them about their team sizes, grade levels, and experience levels for both students and mentors. They also asked teams to describe their solutions to the challenge and how they came up with those solutions.

The Different Strategies

As expected, different teams came up with very different solutions. In fact, they were so different that apples-to-apples comparisons became nearly impossible at the whole-strategy level. Fortunately, every strategy did include one common component: moving the robot to the center of the board to begin scoring points.

Table 1 breaks down the different strategies that teams used, and the number of teams that used each approach.[1]

That only 3 teams used a (non-rotation) Sensor-Based strategy is likely a direct consequence of the nature of the Botball Hybrid II challenge. In particular, the toilet paper tubes were not steady enough for a robot’s touch sensor to contact them without tipping the tubes over. As a result, teams seeking to score using the tubes had to choose non-contact means of controlling their robot's movement. The 3 teams that did use a Sensor-Based strategy on their first move were all going for the nests, which are much heavier than the toilet paper tubes. However, for various reasons, even these teams abandoned use of their sensors in their moves later in the challenge. In addition, the board surface featured few marked lines, making line-following and line-tracking less attractive.

A Math-Based Strategy for Calculating Motor Rotations

The remaining 13 teams programmed their initial move using the rotation sensor, effectively moving a set distance forward. However, those 13 teams used decidedly different methods to choose their motor rotation values, especially the initial value. Some teams guessed; others used the view mode; but four teams chose to start with a math-based prediction.

These groups all ended up using variants of a three-phase strategy called Calculate-Test-Adjust:

1. Measure the how far the robot has to move and use mathematical means to calculate a (theoretically correct) rotation value for the movement
2. Run the robot with the predicted value
3. Compensate for any observed overshoot or shortfall by making small "tweaks" to the rotation value

Students used several different mathematical relationships to arrive at their predictions. For example, one group measured how far the robot moved forward with each motor rotation, then calculated how many of those 1-motor-rotation distances the robot needed to move the total distance to the target. The students then entered this value into their program, tested it, and fine-tuned the value to get the robot to exactly the right spot.

One notable quality of this strategy is that it is not purely mathematical – all 4 teams that used Calculate-Test-Adjust for their initial motor rotations value ended up having to refine their value with guessing or with the view mode afterward. A math-based calculation does not appear to be sufficient on its own for this type of problem.

The Relative Success of the Different Strategies

So how well did these math-using teams fare compared to their sensor-using, guess-and-testing, and view mode-ing peers? The 22 teams were ranked based on their best point score after 3 rounds of the competition. Figure 1 below shows the average rank of the teams who used each strategy. Bigger bars indicate a higher average ranking for the teams using that strategy — meaning teams who used that strategy had better scores in the competition.

Figure 1: Level of Success in the Competition based on the Strategy Used

Looking at the data in this way, the View-Mode strategy was the most effective and the Sensor-Based strategy was the least effective. The Guess-Test-Adjust and the Calculate-Test-Adjust strategies seem to be in the middle and similar to each other. Given that this particular challenge was somewhat biased against the use of sensors, it probably makes sense that teams who used the Sensor-Based strategy did not fare well. But what of the others?

The View-Mode strategy did seem to do particularly well. The investigators theorize that this strategy leads to success for two reasons. First, teams that use this strategy can program their movements quickly. Figuring out the correct motor rotations value is straightforward and fast, so that frees the team up to spend their limited time improving other parts of their solution (e.g., making their robot base solid and their attachments functional). Second, the View-Mode strategy is very reliable, so once teams get a motor rotation value by using this strategy, they then have a lot of confidence that that value is the right one and will work well. In essence, the View-Mode strategy is easy to implement quickly and gives very reliable results, which explains why teams who chose that strategy tended to do well in the competition.

The Success of the Math-Based Strategy

Compared to rolling the robot on the ground and reading a number, both Guess-Test-Adjust and Calculate-Test-Adjust are slow to implement and potentially less reliable as well. And in the results, teams who used these strategies did okay in the competition, but not as well as teams who used the View-Mode strategy… case closed. Right?

Averages, it turns out, don't tell the whole story. Calculating the standard deviation of the ranks gives us a sense of how tightly clustered these different success levels are for each strategy. If everything were cut-and-dry, we'd see all the View Mode teams clustered at the top, followed by the test-and-adjust teams, and sensor-based teams at the bottom.

Instead, when we add the standard deviation as error bars on the previous bar plot of the average ranks (see Figure 2), some things fall into place, and others fly loose. The View-Mode strategy was the least variable – teams using it were tightly clumped in the rankings – further supporting the idea that that strategy is straightforward and reliable. But the Calculate-Test-Adjust strategy has a huge variability (the error bars span almost the entire range of possible ranks)! Something important remains untold.

Figure 2: Level of Success in the Competition based on the Strategy Used including the Variability of Success

In fact, a closer look at the 4 teams that used the Calculate-Test-Adjust strategy shows that 2 of them were the top ranked teams in the entire competition (ranked #1 and #2 out of 22 teams). This suggests that using a math-based calculation strategy can be very powerful. At the same time, the other two Calculate-Test-Adjust teams were #17 and #21 out of 22 in the rankings – the complete opposite end of the scoring spectrum.

This dramatic separation in performance suggests something powerful (see Figure 3). Perhaps it is not enough to simply use a strategy; the result may hinge dramatically upon the strategy being used right. Perhaps when the Calculate-Test-Adjust strategy is implemented well, it is just as quick and just as reliable as the View-Mode strategy, if not even better. Done without a full understanding, however, the calculations could turn into distractors.

Figure 3: Score in the Competition based on Math Use

The research team theorizes that teams who are fluent with mathematics can use math-based calculations to their advantage by determining the correct motor rotation values for different moves relatively quickly. As with the View-Mode strategy, this time savings frees resources for use on building tasks and fine-tuning overall strategy. Teams that are less fluent in mathematics, however, would take longer to perform the math-based calculations, and make more errors, thus taking time away from working on other important parts of the task.

Conclusions

Conclusion #1 – Not many, but some teams do use math. And of the teams that do use math, there is widely varying success, from some of the most successful to some of the least successful.

Overall, teams found a range of ways to approach the challenge. Different challenges may favor different types of strategies, but the May Madness event saw a variety of approaches employed. Some strategies did seem to lead to better success in the competition. In particular, the View-Mode strategy seemed to be very successful for teams, presumably because it is quick and reliable. Not many teams chose to use the math-based Calculate-Test-Adjust strategy, but those who did ended up with both the highest scores in the competition, and some of the lowest scores. This suggests that for the math-based strategy more than any other, it matters not just that a team used that strategy, but how they used it.

Fortunately, in addition to the day-of-competition interviews, the research team also met with a few of the teams outside of the competition to understand their solution strategies in more depth. The winning team in the competition was one of these. Did using math really help this team be successful? And if it did, then how? Continue on to Part 3 to find out about the winning team's strategy and their use of math.

Part 3: A Winning Strategy » (coming soon)

Notes

1. There was one other strategy that teams used to determine how many motor rotations to use in their program. We call this strategy the “Overshooting” strategy because it works in situations where it isn't critical that the robot moves a particular amount as long as the robot moves far enough. For example, when approaching the nests it was okay if the robot went too far because it would just push the nest forward a bit, but the nest would still be in a position where it was easy to grab. This strategy didn't work with the toilet paper tubes, because if the robot went too far and bumped into them, they would fall down and would then be much harder to grab. In cases when overshooting was acceptable, teams were able to choose a motor rotations value that was safely big enough without having to worry if it was exactly right. No team used this strategy on their initial robot movement and teams were more likely to use it when programming the manipulators, so we didn't include it in our primary list of strategies.

2. One could argue that the rotation sensor is a sensor like all the others. In particular, the programming logic is the same, so a strategy that used the rotation sensor could also be labeled Sensor-Based. But here we think the distinction between the rotation sensor and the other sensors (e.g., touch, ultrasonic, and light sensors) is meaningful as the rotation sensor is the only one that will make the robot move with little regard to what is out in the world. Strategies using the other sensors will move varying distances depending on the way the objects in the world are configured, but the rotation sensor strategy (within some error) will always move a consistent amount.

Part 1 of 4: Introduction and Background

Every September, thousands of FIRST® LEGO® League (FLL) coaches and mentors around the world crack their knuckles, dust off their parts bins, and prepare to dive into an intensive three-month odyssey of technical twists and twenty-first century tutelage as they guide their teams to success in the annual FLL competition. But what exactly is success in a world of gameboards and gracious professionalism? Is the highest scorer really the biggest winner? What do students actually gain through their participation? How does it happen? And so, how should the enlightened coach choose from the multitude of competition strategies that lie open as the new season dawns?

Perhaps we can learn something from the recent past.

Since 1999, the Robotics Academy (RA) has been helping teachers, mentors, coaches, and students have positive educational experiences with robotics. Among other things, the Academy develops curricula, offers teacher professional development, and hosts robotics competitions. Recently, the Robotics Academy — in cooperation with the University of Pittsburgh’s Learning Research and Development Center (LRDC) — has been investigating ways to deepen students' experiences with robotics by incorporating math in their activities.

This blog series describes what RA and LRDC researchers found when they interviewed teams at a local LEGO robotics competition, looking to answer a few key questions:

• Are there opportunities to use math in a typical robotics competition problem?
• Does using math have any impact on a team's score?
• Can using math deliver “success” in any other sense?

In short, the investigators found that the answer to the all three questions was an overwhelming yes — there are opportunites to use math in a LEGO robotics competition setting, and when teams do use math it seems to be helpful in ways not limited to points and trophies. Ultimately, the research team arrived at 3 major conclusions:

1. Not many, but some teams do use math. And of the teams that do use math, there is widely varying success, from some of the most successful to some of the least successful.
2. The most successful teams do use math purposefully and efficiently, and their math use is a prominent factor separating their solutions from the solutions of the rest of the teams.
3. Even when a team's use of math doesn't lead to success on the challenge, just attempting to use math can have other benefits in terms of improving students' understanding and developing more positive attitudes about math and robots.

Each remaining article in this series will examine one of these conclusions in detail and describe how we arrived at each one. But first, let’s set the scene.

An Initial Investigation

Since 2000, the Robotics Academy has hosted the FIRST LEGO League (FLL) Pittsburgh Regional Championship. Last Fall, the Academy decided to see what it could learn by interviewing participating teams on the day of the competition. Robotics Academy researcher Ross Higashi interviewed a sample of the more than 70 teams that competed in the 2009 regional competition and put together two Robotics Academy Blog posts that identified who an FLL team is and the connections to Science, Technology, Engineering, and Mathematics that they make. Although praising the article overall, a comment to one of the posts challenged researchers to go into more depth:

If “a few highly successful teams have shown great adherence to principles of good design”, can Carnegie-Mellon or FIRST make their stories, plans, approaches available more widely to the community? Otherwise, without good examples, it will remain hit-or-miss for the vast majority of the teams.

Richard Ho, comment posted January 30, 2010 on the Robotics Academy Blog

And so a followup plan was devised. Surely another round of interviews could find good examples from which the whole FLL community could benefit! The research team’s first opportunity to conduct interviews was at a local competition called May Madness, on Saturday, May 8, 2010 at the Sarah Heinz House in Pittsburgh's North Side neighborhood. Although not as large as the FLL regional championship, the May Madness event attracts the same types of teams and uses similar challenges.

The Challenge

The 2010 May Madness competition included a number of different events, including separate challenges for different age divisions, different robot platforms such as VEX and TETRIX, and even a non-robotic Alice storytelling competition. To provide the most FLL-relevant information, the interview team focused on the “Botball Hybrid II” LEGO MINDSTORMS NXT challenge, geared toward elementary and middle school age students.

Although not quite as complex as typical FLL challenges in terms of the number of missions or the variety of objects on the board, the Botball Hybrid II challenge includes a number of elements that require sophisticated solutions. Two teams occupy the board at the same time, a black team and a white team. Each team can have one robot on the board at a time and the teams start at opposite ends of the board. The object is to get the most points possible in a 90-second round. Points are obtained by collecting ping pong balls and toilet paper tubes of the team's color and also common nests and foam balls. Knocking the ping pong balls loose gets some points, but the most points are obtained by bringing the objects back to a team's end zone. Even more points are obtained by lifting the objects into the gutters on the side of the table See gallery of images below for pictures of the game board, the items on the board and the specifications, a list of the rules of the challenge, and the points system.

The Findings and the Future

How did teams try to solve this challenge? Did math come into the picture at any point… and if so, did it help? Should coaches bother encouraging students to try using math in a challenge like this?

Each remaining article in this series will focus on answering one of these questions. Part 2 of the series describes the range of strategies that teams employed, Part 3 details the winning strategy, and the Part 4 discusses some alternative versions of success that were observed (Parts 3 & 4 will be released soon).

As you read through the research team’s findings and interpretations, please let us know what you think by leaving a comment on the blog! And if you are planning to attend this year's Pittsburgh Regional FLL Competition, the Academy would love to have your team be a part of the next round of investigation (send an email to Eli Silk if you are interested). We hope you find these articles helpful and wish you the best of luck in the upcoming competition season!

Part 2: The Range of Strategies »

Robotics Academy VEX Teacher Training classes are web-based. Learn to program in ROBOTC using Innovation First VEX robot hardware. This video provides a quick overview.

These courses focus on learning how to program VEX-based robots, and how to use robotics as an organizer to teach STEM (Science, Engineering, Technology, and Mathematics) concepts. Included with the course is the Robotics Academy’s Teaching ROBOTC for IFI VEX CD-Rom, along with a license to use it in your classroom afterward.

You will also have the opportunity to earn professional development credits and a Certificate of Completion by completing various assignments throughout the course.

For current class listings, click here.

In Part 2, we’re going to take a look at what an FLL team gets out of the experience. Every student goes home with a medal for participation, and some earn trophies as well. But what do students really take home with them in terms of learning and experience? Read the rest of this entry »

ITEA’s flagship publication, The Technology Teacher, has published an article on designing technology lessons that teach mathematics. Written in collaboration with our partners Eli Silk and Dr. Christian Schunn at the University of Pittsburgh’s Learning Research and Development Center, the article Designing Technology Activities that Teach Mathematics is an insightful look into some of the strategies that help to bring out mathematical value when setting up a robotics problem.

In the article, the authors lay out four principles for producing a technology activity that leads to math learning. The paper also gives several examples of these principles in the redesign of the Robot Algebra Robot Synchronized Dance activity.

Designing Technology Activities that Teach Mathematics can be found featured on the cover of the December/January 2010 volume of The Technology Teacher.

Silk, E. M., Higashi, R., Shoop, R., & Schunn, C. D. (2010). Designing Technology Activities that Teach Mathematics. The Technology Teacher, 60(4), 21-27.

Robin Shoop of the Carnegie Mellon University Robotics Academy, Michael Amrhein of California University of Pennsylvania and Dr. Stephen Catt of Butler County Community College discuss a new robotics technician training program in a television interview conducted by host Bill Flanagan on the program ‘Our Region’s Business’.

This coming summer, you can attend Carnegie Mellon Robotics Academy Teacher Training classes in Pittsburgh, PA. A full slate of classes is available, covering different robotic platforms and software. This video provides a quick overview.

Robotics Academy teacher training courses are designed for robotics educators who use LEGO MIndstorms, TETRIX, or VEX robots in their middle school and high school classrooms. All training is conducted at the National Robotics Engineering Center (NREC) in Pittsburgh, PA. The NREC is part of the Carnegie Mellon University Robotics Institute, a world-renowned robotics organization, where you’ll be surrounded by real-world robot research and commercialization. You also can take advantage of Pittsburgh’s attractions, from world-class museums and entertainment, to shopping, excursions, sports, and more.

For current class listings, click here.

We interviewed coaches at the 10th annual Pittsburgh Regional FLL competition. In Part 1 of our analysis, we look at what an FLL team REALLY is.

Read the rest of this entry »