Abstract: Modeling analysis of a fractal rough surface at nanoscale is established by the method of removing atoms. It is found that utilizing the standard data file which contained the information of atom and lattice and removing atoms according to the boundary conditions, a rough surface model with fractal characteristics at nanoscale can be established by considering fractal theory. The rough surface determined by the W-M fractal function was prone to unreasonable structure with isolating atom columns when the atomic lattice constant was used as scanning interval. For a better frictional simulation of rough surfaces in nanoscale by the lammps software, Fourier transform and filtering process were adopted to remove these isolated atom columns. During this process, the fractal characteristics of the rough nanoscale surfaces can be preserved by increasing the pass radius of the filter and optimizing initial structure with the method of minimizing energy. Experiments showed that this method effectively established the fractal rough nanoscale surface which is consistent with the practical surface.