I have the same problem. With Core or Core2 the CAN Unit is not working. I tried Port B and C (with Bottom/Bottom2). It even blocks CAN communication at my test setup (2 ESP32 development modules with additional SN65HVD230). If I use the same code with an Atom it works. Also the Atom CAN (basically the same as the CAN Unit) works with no problem. I also tried the sample code from m5Stack, doesn't work...
The CAN Unit is not the problem. If I hook up a logic analyzer at the connection between Core and CAN Unit the data is fine.