Structure from Motion (SfM) has been a cornerstone of computer vision for decades, aiming to reconstruct 3D scene structure and camera parameters from 2D images. Traditionally, this problem is broken into several subproblems - image matching, triangulation, and global optimization - typically relying on 2D keypoints and reprojection error. Recently, 3D reconstruction models like DUSt3R have proven highly effective for a variety of 3D vision tasks. These models, based on feed-forward neural networks, regress dense 3D pointmaps from pairs of images in a shared coordinate system. In this work, we integrate modern 3D reconstruction priors into the incremental SfM pipeline. We propose a novel optimization method that aligns 3D pointmaps with scene structure and incorporates them as additional constraints in the optimization process. This allows us to enhance global optimization by combining both 2D keypoints and 3D pointmaps, resulting in improved robustness. We evaluate our approach on indoor scenes and demonstrate that it outperforms the baseline pipeline that relies solely on 2D constraints from reprojection error in a shared coordinate system. In this work, we integrate modern 3D reconstruction priors into the incremental SfM pipeline. We propose a novel optimization method that aligns 3D pointmaps with scene structure and incorporates them as additional constraints in the optimization process. This allows us to enhance global optimization by combining both 2D keypoints and 3D pointmaps, resulting in improved robustness. We evaluate our approach on indoor scenes and demonstrate that it outperforms the baseline pipeline that relies solely on 2D constraints from reprojection error.