If you increase the number of buffers to put any memory pressure on the system, various allocations will start to fail. Even allocating 4K when 50K is free, and 34K is the largest block, can fail.

I've mostly experienced this on builds like M5DEMO where there is no PSRAM.