Understanding the influence of text complexity and question type on reading outcomes

Abstract

In the current study, we examined how student characteristics and cognitive skills, differing levels of text complexity (cohesion, decoding, vocabulary, and syntax), and reading comprehension question types (literal, inferential, critical analysis, and reading strategy) affected different types of reading outcomes (multiple-choice reading comprehension questions, free recall, and oral reading fluency) in a sample of 181 native English-speaking adolescents (9 to 14.83 years). Results from item response theory one-parameter models and multilevel models suggested that different cognitive skills predicted performance across the three reading outcomes. After controlling for student characteristics and cognitive skills, text complexity negatively impacted reading outcomes, particularly oral reading fluency and free recall. Critical analysis and inferential questions emerged as the most difficult types of comprehension questions. The implications of these findings are discussed.

Appendices

Appendix A

See Table 6.

Table 6 Passage topics and manipulations

Appendix B: examples of cohesion-, decoding-, syntax-, and vocabulary-manipulated passages

Baseline versus cohesion-manipulated passages

Baseline passage

Humans are born without the well-designed wings that birds have. But humans all over have looked up at the sky and dreamed of flying. The goal of flight was met first by two brothers. They created the world’s first hot air balloon. They lived in France 200 years ago. These brothers knew that air gets lighter when it gets hot, and so the men ran a vital test. The brothers put an upside-down paper bag over heat. The hot air was trapped inside the bag, like a balloon. Then they used a small bit of string for tying the bag shut. The bag floated up toward the sky when they let it go, just as they expected it would. So, the brothers then set out to change the world.

The brothers then made a much bigger hot air balloon from silk fabric. At first, the brothers were not sure if the hot air balloon could carry adults safely. As a result, they chose to use a lamb and a duck from a nearby farm. An amazed group met to see the first flight. Even the king of France decided to attend! The hot air balloon soared in the sky for a full 6 min. It landed gently a few miles away. The hot air balloon’s passengers were very scared, but they were safe. The brothers then redid the test with humans for passengers.

Since the day that the first humans flew, the dream of flight has taken off. Today, crowds meet from all over the world to fly hot air balloons together. Most of them fly in hot air balloons just for fun. We fly in airplanes to get from one place to another. However, scientists still use balloons to understand the climate. They are also used to collect vital information from inside hurricanes, tornados, and storms.

Cohesion-manipulated passage

We are born without the well-designed wings that birds have. Humans all over have looked up at the sky and dreamed of what it would be like to fly. Two brothers created the very first hot air balloon in France over 200 years ago. They were the first to meet the goal of sending a person up into the air. Knowing that air gets lighter as it heats up, the men ran a vital test. They hung an upside-down paper bag over an open flame. The hot air got trapped inside. Using a small bit of string, it was tied shut. The bag floated up toward the sky when it was let go, just as expected. The brothers set out to change the world.

They made a much bigger hot air balloon from some light fabrics, such as silk. At first, nobody was sure that the new one could carry adults safely. They chose to use a lamb and a duck from a nearby farm instead. An amazed group met to watch the first trip. Even the king of France decided to attend! It soared for a full 6 min before coming down, landing gently a few miles away. The passengers were very scared, but also safe. The innovators redid the same test with humans. Those were the first brave guys to ever lift off of the ground.

That first day of flight brings the dream of flying within reach. Today, crowds gather all over the world to fly hot air balloons together. There are festivals every few months. Most enjoy ballooning as a hobby that is practiced during spare time. In general, airplanes get us from one place to another. With balloons, scientists can better understand severe climates. They are used to collect vital information from inside hurricanes, tornados, and storms.

Baseline versus decoding-manipulated passages

Baseline passage

Most folks do not grasp the difference between moths and butterflies. Both insects have six legs and are the same size. Both come out of cocoons. You may think that butterflies are more appealing, but that is not always correct. In fact, several butterflies are a tawny or pale color. By contrast, the wings of some moths have nice patterns or bright colors. A more dependable method to identify them is to note when they appear. Butterflies are awake during the daytime. Most moths only wake up after sunset. There is also a visible difference in their feelers, which are used for sensing their surroundings. Moths have thick feelers that look like feathers. A butterfly’s feelers are skinny with a minor bump at the tip.

Most moths use their noses like straws for drinking nectar from flowers. In fact, one has a huge nose that is three times as long as the rest of its body! Another moth has a nose that is as sharp as a blade. It uses its nose to stab an animal’s skin and suck out the blood. It is called a “vampire” moth because it likes to drink blood. Some odd names have been given to others, too. There are “silk,” “wax,” and “hawk” moths.

Everyone is aware of the fact that flying insects are enticed by lamps. Yet for hundreds of years, nobody knew why. However, scientists now think they can explain this fact. They have seen that the light from the moon and stars enable moths to find their way in the dark. When moths see a light bulb’s glow, they get confused because they think the light is up in the sky. The moths you see swirling around lamps at night are lost. By turning off the lights, you will assist them to find their way again.

Decoding-manipulated passage

Most folks do not realize the difference between moths and butterflies. Both insects have six legs and are the same size. Both come out of cocoons. You may think that butterflies are more colorful, but that is not always accurate. In fact, various butterflies are a beige or neutral color. By contrast, the wings of some moths have pretty designs or beautiful colors. A more reliable way to distinguish them is to observe when they appear. Butterflies are awake during the daytime. Most moths only wake up after the sun sets. There is also an obvious difference in their feelers, which are used for sensing their surroundings. Moths have thick feelers that look like feathers. A butterfly’s feelers are wiry with a miniature knob at the tip.

Most moths use their noses like straws for drinking nectar from flowers. In fact, one has a giant nose that is three times as long as the rest of its body! Another moth has a nose that is as sharp as a knife. It uses its nose to pierce an animal’s skin and suck out the blood. It is called a “vampire” moth because it likes to drink blood. Some unusual names have been given to other ones, too. There are “emperor,” “rough,” and “gypsy” moths.

Everyone is familiar with the knowledge that flying insects are beguiled by lamps. Yet for hundreds of years, nobody knew why. However, scientists now know they can explain this curiosity. They have learned that the light from the moon and stars enable moths to find their way in the dark. When moths see a light bulb’s glow, they get confused because they believe the light is up in the sky. The moths you see circling around lamps at night are lost. By turning off the lights, you will encourage them to find their way again.

Baseline versus syntax-manipulated passages

Baseline passage

In the winter, the Arctic landscape consists of snow, rocks, and icy waters. Shelter keeps a person from freezing to death in the icy climate. Eskimos have lived in the Arctic for hundreds of years. They survived harsh winters by building homes out of snow. These homes are shaped like domes and are actually very warm. They are called “igloos.” The word comes from the Eskimo word for “ice home.” An arctic wind can push down the walls or topple the roof of a regular home. But even in the worst storm, the harsh wind sweeps gently over the round igloo.

Eskimos use snow from a single snowfall when building an igloo. Since each block comes from the same snow, they are all just as strong. Eskimos have built homes out of snow the same way for hundreds of years. First, big blocks of snow are placed in a ring. Next, layers are added in a spiral. The top of each block is shaped to lean inward. The final block goes in the middle of the dome. It seals the igloo shut. Finally, snow is stuffed between cracks, and a doorway is dug out. An Eskimo family can connect some of these “ice homes” with tunnels. One of these domes may serve as a family room. Other domes could be used as a kitchen or living room.

If Eskimos get caught in a sudden storm while traveling, they make a small igloo for shelter. After they get inside, they will block the entryway with a final block of snow. Finally, they only have to wait for the storm to stop. Today, very few Arctic families live in homes made of snow all year round. But lots of Eskimos still know how to build one. They practice this very old skill by building igloos when they go ice fishing.

Syntax-manipulated passage

In the winter, snow, rocks, and icy waters make up the Arctic landscape. Without shelter in this icy climate, humans would freeze to death in minutes. For hundreds of years, the Arctic has been the home of the Eskimos. Through the long, harsh winters, Eskimos stayed warm by building homes out of snow. These remarkably warm homes are shaped like domes. The Eskimo word for “ice home” is actually “igloo”. The walls or roof of a regular home would quickly be crushed by strong arctic winds. But even in the worst storm, the harsh wind sweeps gently over the round igloo.

The blocks of snow in an igloo consist of snow from a single snowfall. Since each block comes from the same snow, they are all just as strong. For hundreds of years, Eskimos have built homes the same way. First, big blocks of snow are placed in a ring. Next, layer after layer is added in a spiral. The top of each block is shaped to lean inward. Then, the final block in the middle of the dome seals the igloo shut. At the end of the process, snow is stuffed between cracks and a doorway is dug out. Sometimes, an Eskimo family with more than one dome will connect them with tunnels. One of these domes may serve as a family room. The other domes can be used as kitchens or living rooms.

If Eskimos get caught in a sudden storm while traveling, they make a small igloo for shelter. A final block of snow will block the entryway after everyone is safe inside the igloo. Finally, they only have to wait for the storm to stop. Today, very few Arctic families live in homes made of snow all year round. However, this very old skill is still practiced when Eskimos go ice fishing.

Baseline versus vocabulary-manipulated passages

Baseline passage

Maple trees need a lot of water. They simply cannot live without it. On rainy days, the roots take up as much water as is needed. Then the sap carries the water up to the treetop. The leaves use the water and turn sunlight into food so that the tree can grow. Next, the sap brings extra food from the leaves back down to the roots. The roots store food for another day.

The sap of “sugar maple” trees is distinctive. It has a sweet taste, unlike the sap of other trees. It is the kind of sap that farmers use to cook the maple syrup that we put on pancakes. Cooking syrup is a custom that is 200 years old. First, a farmer drills holes in the trunks of his maple trees. The sap runs right under the bark, so the holes are not too deep. Then the farmer pounds a spout into the hole and hangs a bucket below. On a warm day the sap drips slowly into the bucket. When it is full, the farmer moves the sap into a cooking pot. Finally, the pot is set on a hot stove where it boils until the water is gone. Making just one gallon of maple syrup takes forty gallons of sap.

Sometimes the farmer decides to make some candy by cooking the syrup a little longer on the stove. He waits and waits for the boiling liquid to get very thick. Then the farmer takes it off the stove and puts it in the freezer. The syrup cools until it is finally hard. The end result is a tasty candy. It is a nice treat for the farmer’s children. Anybody can make maple candy this way. If you do not have a sugar maple tree, you can get some syrup at a store.

Vocabulary-manipulated passage

Maple trees need a heap of water. They simply cannot subsist without it. On sodden days, the roots sop up as much water as is vital. Then the sap relays the water up to the treetop. The leaves use the water to turn sunlight into food so that the tree can grow. Next the sap transports leftover food from the leaves back down to the roots. The roots amass food for another day.

The sap of “sugar maple” trees is matchless. It boasts a sweet taste, unlike the sap of other trees. It is the kind of sap that farmers employ to cook the maple syrup that we pour on pancakes. Cooking syrup is a custom that is 200 years old. First, a farmer drills holes in the trunks of his maple trees. The sap runs right under the bark, so the holes are not too deep. Then the farmer hammers a spout into the hole and suspends a bucket below. On a balmy day the sap drips sluggishly into the bucket. When it is finally full, the farmer moves the sap into a cooking pot. Finally, the pot is set on a hot stove where it simmers until the water is gone. Cooking just one gallon of maple syrup entails forty gallons of sap.

Sometimes the farmer opts to make some candy by cooking the syrup a tad longer on the stove. He waits and waits for the boiling liquid to get very dense. Then the farmer takes it off the stove and puts it in the freezer. The syrup cools until it is finally stiff. The finishing result is a yummy candy. It is a nice tidbit for the farmer’s children. Anybody can make maple candy this way. If you do not have a sugar maple tree, you can get some syrup at a supermarket.

Appendix C: example reading comprehension passage and multiple-choice question types

Toads can be found in almost every part of the world. Freezing temperatures cause unsafe conditions for these cold-blooded animals. In northern climates, toads hibernate all through winter. They protect themselves by making nests underground. Toads sleep in those nests all day. They are mostly awake after dark. That is when the hunt for food takes place. Toads eat mostly bugs and worms. In some places, they have learned to try new foods. Toads capture prey with their sticky tongue. They hunt on land and in water.

Toads have odd eyes that help them to swim and hunt in low light. Their eyeballs have a see-through lid, which enables them to see underwater. Their vision is really good at spotting movement, even in a murky pond or on a cold, moonless night. But a toad’s eyesight is not very sharp. An insect can be standing right in front of the toad and as long as that bug stays very still, the toad will not see anything. But as soon as the insect moves, the toad will suddenly react. In an instant, the long sticky tongue will dart out and eat the victim. The hunter’s aim is spot-on, so it catches many snails, slugs, and flies.

In many parts of the world, toads are considered to be pests. A long time ago, humans used to bring toads on ships to new lands. In those days, humans used to believe that toads would eat the bugs that like to eat crops. But without natural predators, toads quickly grew in numbers. Now, toads harm local habitats by eating up all the food. They also harm many animals who try to eat them. The animals do not know that toads have poison in their skin. So now, humans are trying to get rid of toads in gardens and parks.

Factual Humans used to believe that toads:

• (a) were a symbol of wealth.

• (b) would eat the bugs that like to eat crops.

• (c) could cure disease.

Inferential According to the information in the passage, if a fox eats a toad, it would probably:

• (a) be very sick.

• (b) be unable to digest the toad’s eyes.

• (c) choke on it.

Process Strategy You will most likely find this passage in:

• (a) an email.

• (b) a diary.

• (c) an encyclopedia.

Critical Analysis In the second paragraph, the author shows that toads:

• (a) may not see a moving firefly.

• (b) can see a bug swimming in a murky pond.

• (c) cannot see through their eyelids.