Decoction mashing is removing part of the mash, boiling it, and then returning it to the main mash, which is held at a constant temperature.
For those unfamiliar with brewing, the mash is the porridge-type mixture that occurs when the crushed malt grains are mixed with water. The crushed malt and other starches are transformed into sugars and proteins as well as other materials become soluble, all of which creates the wort, which is then fermented into beer.
The goals of boiling one part of the mash (i.e. decoction mashing) are to help destroy the cell walls to make the starches more accessible to the malt enzymes and to raise the temperature of the bulk mash to a set higher temperature after mixing both parts. In the days before thermometers, decoction mashing was used to achieve a mash with more than one temperature.
If it sounds confusing, it kind of is. Luckily, technology has advanced such that decoction brewing is no longer a necessity, but some brewers still advocate for it. Here is a brief 5 minute video that demonstrates decoction brewing in action (be warned: you may want to mute it if you're not a fan of 5 minutes of polka music):