N point perspective problem

Hi, I have a N point perspective problem, that I could use some pointers to solve. I have a calibrated camera, where I observe 4-6 known world points. Each observation is uniquely tied to a world point and has an 2d coordinate in my image. I know my camera properties so I can convert them to a direction in the camera frame. I have been using openCV's solvePNP method and it works like a charm. My problem is that I dont have that luxury anymore as I need to do the computation in the frontend of a browser alongside the ThreeJS engine. I have tried sending points to a server to return me results via pythons opencv bnindings, but its too slow and requires alot of infrastructure. The opencv implementation is opensource, but it uses a solver and has a complexity that I dont find it feasible to port it line by line. Instead I started making an attempt (how hard can it be?) for a solution by defining a function that could give me an extrinsic matrix from three euler angles and a 3d position. (6DOF). I also made an error function that, given a proposed matrix, the world points and the directions from the camera could return me an error. I used the sum of dot products between the direction vectors, so large values are good. The idea from here would be to nummerically differentiate the error function in relation to the 6DOF variables and gradually take tiny steps up hill on the error function. I am sure the solution will be slow, but I dont mind it takes a while. My problem is that it does not converge at all. Any ideas why? Or pointers to what I could change. I suspect my error function could be better, but I am unsure of what to use instead. Kind regards Jesper