This post is by MSU postdoc Jason Keagy
How do species form?
Stated more precisely, how does one species become two? This turns out to be an immensely difficult question to answer, because 1) species are not always distinct entities (species definitions are argued about ad naseum ) and 2) the formation of species (speciation) is a process that often takes a long time to complete.
One way in which species could form is if selection is divergent and a population responds to that selection  – for example, Anolis lizards that have adapted such that each species has limbs that are optimal for living in different types of vegetation , or insects that have specialized on feeding on different plants . One way to represent the relationship between phenotypes (traits such as limbs, coloration, or digestive enzymes) and fitness is with a “fitness landscape” , so called because in three-dimensional representations (e.g., two traits as the x and y axes and fitness as the z axis), it can resemble a landscape of peaks and valleys. However, we don’t have a lot of great examples of these because it is often difficult to measure fitness and fitness often depends on multiple independent phenotypic traits in complicated ways.
The power of sticklebacks
In some freshwater lakes in British Columbia, you can find two different types of stickleback, called “benthics” and “limnetics” that are reproductively isolated, and therefore, typically referred to as species. These benthic and limnetic sticklebacks are descended from marine sticklebacks who bred in glacially fed streams. After the glaciers melted ~12,000 years ago, the weight of the ice being removed caused the land to rebound, and the uplifted streams became isolated lakes. Because of this relatively short timescale, these fish have become a model system for studying adaptation and speciation.
What is the difference between benthic and limnetic sticklebacks? Limnetics live in open water, eat plankton, and are more visually oriented, whereas benthics eat invertebrates off of plants or the lake bottom, live in complex spatially structured vegetated habitats, and are more dependent on smell. Limnetic and benthic sticklebacks also differ in body size, shape, and mating traits. In other words, they are really different! Critical for maintaining these differences is strong reproductive isolation and so the Boughman lab has long been interested in understanding what influences this isolation.
The role of male competition
Typically, the focus in speciation research has been on natural selection (even in sticklebacks). Much less studied and controversial is whether sexual selection can drive speciation. Especially unstudied is intrasexual (often male-male) competition’s role. That seems like a pretty big oversight to me. Flip on any nature show and you’re sure to see at least one scene of males bashing each other to pieces. It turns out Jenny Boughman, Liliana Lettieri, and I were already working on a project which was perfect for studying how male competition might impact speciation.
Male sticklebacks compete for territories on which they build their nests (Fig. 1). They’ll even destroy each others’ nests and steal pieces such as the choicest algae. Eventually, these males will try to attract females via courtship behavior to convince them to lay eggs in their nests. Male competition is extremely important to determining male fitness: if males can’t successfully obtain and keep a territory, and build and keep a nest, they are unable to reproduce (we rarely see sneak spawning). Male competition could have important impacts on speciation because males of each species build nests very close to each other in nature and are therefore direct competitors for space and resources.
Our main research questions included: How do male phenotypes relate to male competitive fitness? Do the resulting fitness landscapes have multiple peaks? Would these peaks promote speciation? We created hundreds of hybrid males in the laboratory through artificial crosses. This greatly expanded the combinations of phenotypes from that seen in the wild. Then we put these males in large outdoor tanks at Kellogg Biological Station that had sand and algae and food caught from nearby ponds. We measured lots of physical traits on the males and spent hundreds of hours recording their male competition behavior (with the help of an awesome army of undergrads).
Our research revealed some surprises . First, there were indeed two fitness peaks corresponding to pure benthic and pure limnetic multivariate phenotypes. But there was another region of high fitness (a bridge connecting the peaks) that implies certain intermediate hybrids were also good competitors. Interestingly, these hybrids had phenotypes like fish now seen in Enos Lake, where after anthropogenic disturbance (someone released crayfish into the water, Fig. 2) formerly distinct benthic and limnetic species are now a hybrid swarm (a depressing example of evolution in action). Previously the hybridization had been attributed to the crayfish’s introduction resulting in generalist rather than specialist sticklebacks having higher survival, a change in natural selection . These generalists would have been produced by hybridization, which before happened at inconsequential numbers, but this trickle would have become larger as hybrids were now surviving to adulthood. However, our results show that sexual selection through male competition may also have been a contributing factor that sped up the species collapse. The hybrid males with phenotypes corresponding to the bridge within our fitness landscape would have likely been very successful at getting nests, increasing the likelihood of further hybridization. Our data strongly suggest male competition could be very important in the speciation process and impact speciation in complex ways.
 As one example of this disagreement, see Wu, C-I. 2001. The genic view of the process of speciation. Journal of Evolutionary Biology. 14: 851-865 and the ten responses.
 For a book dedicated to this topic, see Nosil, P. 2012. Ecological speciation. Oxford: Oxford University Press.
 A nice HHMI video description of this research is here: http://www.hhmi.org/biointeractive/origin-species-lizards-evolutionary-tree
 There are many nice examples of this including 1) pea aphids that have diverged to specialize on red clover and alfalfa, 2) fruit flies feeding on different species of cactus, 3) the races of apple maggot fly that feed on either hawthorn or apples, and 4) stick insects adapted to wildly different plants in California.
 There is some disagreement over the what specifically “fitness landscape” refers to and what the proper term is for what I refer to as a “fitness landscape” here (especially among philosophers of science). You can read about it in the first section of this book: Svensson, E., Calsbeek, R. (eds) 2012. The Adaptive Landscape in Evolutionary Biology. Oxford: Oxford University Press.
 Keagy, J., Lettieri, L., Boughman, J.W. 2016. Male competition fitness landscapes predict both forward and reverse speciation. Ecology letters. 19: 71-80.
 Behm, J.E., Ives, A.R., Boughman, J.W. 2010. Breakdown in postmating isolation and the collapse of a species pair through hybridization. American Naturalist. 175: 11–26.