Never mind, this is a model problem, I downloaded another model from hugging's face and it worked. Most of the models I tried from there do not work, I only got two of them to process. It was qwen2.5-7B and the Deepseek one. Both were a little slow, so I compared them using the CPU and the AX8850 and the CPU was slower, but not by much, This is on a pi 5 16GB board.
I also have a Kinara ara-2 to work on with 16GB memory and probably would be faster, but I am having problems converting models to its format. I got the pi 5 to recognize it via driver, I just need a converted model to try it. I also have a Hailo-10H coming, can't wait to test it for a project, its supposed to work better with Home Assistant.