They need to be in close proximity. The magnet is on the outside of the case so it should be possible to get it off without opening it. On the inside there would just be the back of the plastic indentation.
I will try grinding a part of the edge of the magnet away with a dremel tool and then prying the magnet out with a screwdriver, but I am concerned about breaking the case, but I hoped someone else may have done it.