We allocate a lot of single-use buffers that a filter uses for one shader then doesn't need; these can probably be shared.
For example, PAMEdgeDetector uses m_edgeIndexes / m_edgeStates / m_edgeRising during the first pass and m_edgeOffsetsScratch during the second, but they are never used by the same shader so they can theoretically share the same memory allocation.