I'll take a shot. "Detritus" is the easier part: it's decaying plant and animal matter. So the worms are eating leaves and stuff after it's started breaking down.

"Dirt" is a little more difficult because it doesn't have as crisp of a definition. Usually when people say "dirt" in this context, they mean "soil," but that's only a little better. The relevant definition for soil is, "the upper layer of earth that may be dug or plowed and in which plants grow."

That detritus gets broken down by bacteria and becomes soil even without worms, but worms do basically the same thing faster. Plus their moving around helps loosen the soil, which also is helpful for growing plants.