From this post, I got the idea to remove and replace the wheels altogether using a hacksaw.
Here’s what it looks like right now. My problem now is that I can’t remove the leftover metal shaft from the caster wheel.
I’ve tried using some lubricant and hammering it with a screwdriver but it didn’t budge.
I assume the metal axis is probably molten firmly into the plastics with some kind of knurling. So probably you are going to break the plastics if you are trying to push it through. As suggested by others, a solution may be to try to drill it out. The other option I see, is to remove the whole black part and replace it by new rollers which have standardized shafts are when dedicated for furniture.