Deep Neural Networks (DNNs) suffer from energy-hungry implementation due to their computation and memory intensive processing patterns. Health wearable devices and embedded systems have small and low power processors which are much slower compared to desktop and server processors. To overcome this problem, in our research project we propose an automatic framework that designs a highly optimized and energy efficient set of deep learning architectures consist of Convolutional Neural Network (CNNs) and Recurrent Neural Networks (RNNs) for health and vision applications. The proposed framework provides efficient methods to explore and prune the design space to find improved set of neural network architectures. Application of these methods cover different health applications, like cardiac arrhythmia monitoring, which is regarded as a key point of the distributed healthcare devices. Other applications can be activity monitoring by wearable sensor devices for elderly people, and fall detection system.