Is this in filter pad or floss? If your tank is healthy don't worry about it, just swish it in your used tank water and put it in back your filter. It is probably waste that is gathering there and breaking down. That's normal. When you throw out your filter material you are throwing out your good bacteria.
I have one layer of pad in my Aquaclear HOB
and it looks pretty brown. The white foam looks light tan. The biomax has brown on it. I just swish it all out. I did my water change and other maintenance today, and after swishing, my discard water smelled nice and fishy-earthy which is perfect. Your brown stuff doesn't smell foul does it? I describe the good smell as potting soil with a hint of fresh fish. If you can take a pic of it, post it so others can weigh in. It is most likely normal and most likely you want it.