Train your AI, feed your head
Machine learning. Deep learning. AI, or artificial intelligence. Since I read the book The Computer and the Brain written by the mathematician John von Neumann, comparisons between the human brain and computers have always fascinated me. Hardly a day goes by that we don’t hear about the promise of AI, and how it will help to alleviate mundane tasks by making sense of the massive oceans of data bursting storage at the seams worldwide.
Much like a child learning to ride a bicycle using training wheels, deep learning networks too are trainable. As I recently discovered while tinkering with a well-known deep learning framework, training is very compute intensive and is certainly the type of task that you’d want to schedule intelligently across your HPC infrastructure.
Train smart
So why not apply HPC scheduling best practices to the training of deep learning networks? IBM Spectrum LSF is a workload scheduler for demanding HPC environments. Since its inception more than 20 years ago, its scheduling brain has been accelerating workloads throughout the HPC world with intelligent policies to make the best possible use of available resources. By continuously monitoring the load across the computing environment, IBM Spectrum LSF makes lightning-fast decisions to place the right work in the right place at the right time. So why not apply these scheduling smarts to the task of training deep learning networks?
IBM Spectrum LSF supports training deep learning frameworks BVLC Caffe and TensorFlow alongside traditional HPC workloads with ease. Find out more about IBM Spectrum LSF and deep learning framework integrations here.
Not only can IBM Spectrum LSF be used to help accelerate training deep learning networks, but machine learning can also be used to help simplify HPC by specifying memory requirements for IBM Spectrum LSF jobs. Learn more in this IBM Research paper here.
And to paraphrase Jefferson Airplane in White Rabbit, “Remember what the dormouse said, “Feed your [AI] head, feed your [AI] head.” Do it with some help from IBM Spectrum LSF.
The post Train your AI, feed your head appeared first on IBM Systems Blog: In the Making.