Well, now that I’ve reviewed up the definition of detritus, not a whole lot.
😂 I would say that detritus is coarser and implies the sense being recognizeable as having been part of a larger whole.
I meant it in the sense of organic debris: leaf litter and such. I think of dirt as being finer and relatively uniform. Water + dirt = mud. Water + detritus = clean detritus.

What I meant was that worms sustain themselves on organic material, and, after they break it down, it is more incorporated in to the soil.

Theres is a whole other discussion to be had whether dirt ≈ soil…