Due to the rapid development in the petroleum industry, the leakage detection of crude oil transmission pipes has become an increasingly crucial issue. At present, oil plants at home and abroad mostly use manual inspection method for detection. This traditional method is not only inefficient but also labor-intensive. The present paper proposes a novel convolutional neural network (CNN) architecture for automatic leakage level assessment of crude oil transmission pipes. An experimental setup is developed, where a visible camera and a thermal imaging camera are used to collect image data and analyze various leakage conditions. Specifically, images are collected from various pipes with no leaking and different leaking states. Apart from images from existing pipelines, images are collected from the experimental setup with different types of joints to simulate leakage conditions in the real world. The main contributions of the present paper are, developing a convolutional neural network to classify the information in red-green-blue (RGB) and thermal images, development of the experimental setup, conducting leakage experiments, and analyzing the data using the developed approach. By successfully combining the two types of images, the proposed method is able to achieve a higher classification accuracy, compared to other methods that use RGB images or thermal images alone. Especially, compared with the method that uses thermal images only, the accuracy increases from about 91% to over 96%.