Courses of Data Mining & Machine Learning & Pattern Recognition

Data Mining

The subject of Knowledge Discovery and Data Mining (KDD) concerns the extraction of useful information from data. Since this is also the essence of many sub-areas of computer science, as well as the field of statistics, KDD can be said to lie at the intersection of statistics, machine learning, data bases, pattern recognition, information retrieval and artificial intelligence. The subject matter of data mining is vast, making the task of task of learning about the subject itself a task of data mining! The one-semester course that I teach emphasizes the theory and algorithms of data mining. Such algorithms are concerned with deriving global models/ local patterns, visualization, and retrieval by content. Related courses that I teach are pattern recognition and machine learning which cover many other topics related to data mining. Textbook: Principles of Data Mining by D. Hand, H. Mannila and P. Smith, MIT Press, 2001.

Lectures The following are the presentation slides used in a classroom setting. They are given here as links to pdf files. Since these slides are continually updated you may wish to revisit them. The course is being taught in Spring 2010. Introduction to Data Mining Introduction to Data Mining Measurement and Data Measurements and Distances Visualization of Data Summarizing Data, Histograms, Scatter Plots Principal Components Analysis and Multidimensional Scaling Data Analysis and Uncertainty Random Variables Estimation Hypothesis Testing/Sampling Systematic Overview of Data Mining Algorithms Decision Trees and MLP Association Rules and Text Retrieval Models and Patterns Prediction Models Probability Models and Graphical Models Structured Data: Markov Models Pattern Structures Content-Based Information Retrieval Precision and Recall Text Retrieval: Term Frequency and Inverse Document Frequency Text Retrieval: Latent Semantic Indexing and Probabilistic Retrieval Content-Based Image Retrieval

Data Mining

============================================================================================

Machine Learning

Machine learning is an exciting topic about designing machines that can learn from examples. The course covers the necessary theory, principles and algorithms for machine learning. The methods are based on statistics and probability-- which have now become essential to designing systems exhibiting artificial intelligence. The course emphasisizes Bayesian techniques and probabilistic graphical models (PGMs). The material is complementary to a course on Data Mining where statistical concepts are used to analyze data for human, rather than machine, use. The textbooks for different parts of the course are "Pattern Recognition and Machine Learning" by Chris Bishop (Springer 2006) and "Probabilistic Graphical Models" by Daphne Koller and Nir Friedman (MIT Press 2009).

Lecture Slides for Machine Learning and Probabilistic Graphical Models

Following are course topics with pointers to lecture overhead slides and some lecture video files. Previously taught as a single semester course it is now divided into two successive courses taught during Fall and Spring semesters. 

Note about slides and video: The slides for Sections 1-7 are from Fall 2010. Slides for Sections 8, 9 and 11 are from Spring 2011. The videos are from Fall 2008 for Sections 1-7 and the videos for Sections 8,9, and 11 are from Spring 2011. Thus the correspondence may be somewhat off. Lecture slides are frequently updated as the course progresses. Chapters 1-14 (Topic titles in Red) are more recently taught versions.

Introduction Machine Learning-Overview(3MB) Text Classification Example(225KB) Regression Example(562KB) Probability Theory(950KB) Decision-Theory(212KB) Information Theory(160KB) MATLAB Introduction(347KB) Probability Distributions Discrete Distributions(291KB) Gaussian Distribution(833KB) Linear Models for Regression Regression with Basis Functions(1.2MB) Bias-Variance(616KB) Bayesian Regression(1MB) Bayesian Model Comparison(300KB) Evidence Approximation(650KB) Linear Models for Classification Introduction(88KB) Discriminant Functions(2.4MB) Generative Models(1.4MB) Logistic Regression(1.4MB) Laplace Approximation (519KB) Bayesian Logistic Regression(845KB) Neural Networks Introduction(551KB) Training(848KB) Error Backpropagation(821KB) The Hessian Matrix(288KB) Regularization in Neu Networks(1.2MB) Mixture Density Networks (634KB) Bayesian Neural Networks(716KB) Kernel Methods Kernel Methods(1MB) Radial Basis Function Networks(549KB) Gaussian Processes(1.4 MB) Sparse Kernel Machines Support Vector Machines(958KB) Probabilistic Graphical Models (Directed) Bayesian Networks(1.4MB) Querying Probability Distributions(460KB) Genetic Inheritance Example(207KB) Graphs and Distributions(305KB) Reasoning Patterns & D-Separation(393KB) Conditional Independence(830KB) Semantics of Bayesian Networks(385KB) Probabilistic Graphical Models (Undirected) Undirected Graphical Models(690KB) Independencies in Markov Networks(526KB) Constructing Markov Networks(251KB) Alternate Parameterizations of MNs(2.1MB) MRFs in Computer Vision(1.2MB) From BNs to MNs(144KB) Partially Directed Models & CRFs(916KB) Inference in Graphical Models Introduction(1MB) Factor Graphs(1.1MB) Max Sum Algorithm(734KB) Loopy Belief Propagation(78KB) Learning Graphical Models Learning PGMs: Overview(695KB) Learning as Optimization(398KB) Parameter Estimation(1MB) Bayesian Estimation in Bay.Nets(980KB) Mixture Models and EM K-means Clustering(1.1MB) Mixtures of Gaussians(1MB) Latent Variable View of EM(516KB) Approximate Inference Approximate Inference(3.2MB) Sampling Methods Basic Sampling Methods(375KB) Monte Carlo Methods(426KB) Continuous Latent Variables Principal Components Analysis Nonlinear Latent Variable Models Sequential Data Markov Models(433KB) Hidden Markov Models(1.3MB) Extensions to HMMs(287KB) Linear Dynamical Systems(217KB) Conditional Random Fields(1.6MB) Combining Models Boosting(pdf, 156KB) Concept Learning Hypothesis Space (pdf, 111KB) Candidate Elimination (pdf,236KB) Decision Trees Information Gain and ID3(pdf, 286KB) Data Sets and Data Mining(pdf, 332KB) Overfitting and Pruning(pdf, 536KB) Computational Learning Theory PAC Learning(pdf, 98KB) VC Dimension(pdf, 321KB) Mistake Bound(pdf, 51KB)

Lec 1.1 video(zip, 138MB) Lec 1.3 video(zip, 134MB) Lec 1.4 video(zip, 133MB) Lec 1.5 video(zip, 135MB) Lec 1.6 video(zip, 135MB) Lec 1.7 video(zip, 277MB) Lec 3 video(zip, 315MB) Lec 4 vid: 1 2(zip,268 286MB)  Lec 5.1 video(zip, 140MB) Lec 5.3 video(zip, 144MB) Lec 5.5 video(zip, 138MB)  Lec 6.1 video(zip, 145MB) Lec 7.1 video(zip, 311MB) Lec 8.1 video(zip, 134MB)  Lec 8.2 video(zip, 127MB)  Lec 8.3/8.4 video(zip, 140MB)  Lec 8.3/8.4 video(zip, 140MB)  Lec 8.5 video(zip, 135MB)  Lec 8.6 video(zip, 150MB) Lec 10.1 video(zip, 132MB) Lec 12.3 video(zip, 152MB)  Lec 13.1 video(zip, 108MB) Lec 14.1 video(zip, 145MB) See Data Mining Course Slides

Machine Learning

============================================================================================

Pattern Recognition

This is the website for a course on pattern recognition as taught in a first year graduate course (CSE555). The material presented here is complete enough so that it can also serve as a tutorial on the topic. Pattern recognition techniques are concerned with the theory and algorithms of putting abstract objects, e.g., measurements made on physical objects, into categories. Typically the categories are assumed to be known in advance, although there are techniques to learn the categories (clustering). Methods of pattern recognition are useful in many applications such as information retrieval, data mining, document image analysis and recognition, computational linguistics, forensics, biometrics and bioinformatics. You may find the websites of related courses that I teach on Data Mining and Machine Learning useful as supplementary material. Much of the topics concern statistical classification methods. They include generative methods such as those based on Bayes decision theory and related techniques of parameter estimation and density estimation. Next come discriminative methods such as nearest-neighbor classification, support vector machines. Artificial neural networks, classifier combination and clustering are other major components of pattern recognition. A course in probability is helpful as a pre-requisite. Applications of pattern recognition techniques are demonstrated by projects in fingerprint recognition, handwriting recognition and handwriting verification. Reference Textbooks: (i) Pattern Classification (2nd. Edition) by R. O. Duda, P. E. Hart and D. Stork, Wiley 2002,  (ii) Pattern Recognition and Machine Learning by C. Bishop, Springer 2006, and  (iii) Statistics and the Evaluation of Evidence for Forensic Scientists by C. Aitken and F. Taroni, Wiley, 2004.

Lectures Following are the lecture overheads used in class as pdf files. The lectures slides are frequently updated. This course was last taught in Spring 2007. Introduction Bayes Decision Theory Bayes Decision Rule Minimum Error Rate Classification Normal Density and Discriminant Functions Error Integrals and Bounds Bayesian Networks, Compound Decision Theory Generative Methods Maximum-Likelihood and Bayesian Parameter Estimation Maximum-Likelihood Estimation Bayesian Parameter Estimation Sufficient Statistics Some Common Statistical Distributions Dimensionality and Computational Complexity Principal Components Analysis Fisher Linear Discriminant Expectation Maximization Sequential Data and Hidden Markov Models Nonparametric Techniques Density Estimation Discriminative Methods Distance-based Methods Nearest neighbor Classification Metrics and Tangent Distance Fuzzy Classification Linear Discriminant Functions Hyperplane Geometry Gradient Descent and Perceptrons Minimum Squared Error Procedures Support Vector Machines Artificial Neural Networks Biological Motivation and Back-Propagation Non-Metric Methods Recognition with Strings String Matching Algorithm-Independent Machine Learning No-Free Lunch Theorem Bias and Variance Resampling for Estimation Bagging and Boosting Estimation of Misclassification Classifier Combination Unsupervised Learning and Clustering Unsupervised Learning and Clustering Appendix Normal Distribution Statistical Tests

Patter Recognition

http://www.cedar.buffalo.edu/~srihari/

学海无涯，回头是岸，O(∩_∩)O~