Building extraction from high resolution remote sensing image is a key technology of digital city construction[14]. In order to solve the problems of low efficiency and low precision of traditional remote sensing image segmentation, an improved U-Net network structure is adopted in this paper. Firstly, in order to extract efficient building characteristic information, FPN structure was introduced to improve the ability of integrating multi-scale information in U-Net model; Secondly, to solve the problem that feature information weakens with the deepening of network depth, an efficient residual block network is introduced; Finally, In order to better distinguish the target area and background area in the image and improve the precision of building target edge detection, the cross entropy loss and Dice loss were linearly combined and weighted. Experimental results show that the algorithm can improve the image segmentation effect and improve the image accuracy by 18%.