Linux开发环境及应用作业 20191031

作业要求

从因特网上搜索相关Web网页，处理网页html数据，从中提取出当前时间点北京各监测站的 PM2.5浓度，输出格式如下。要求：写出各个处 理步骤，并给出解释。
2018-03-15 13:00:00,海淀区万柳,73
2018-03-15 13:00:00,昌平镇,67
2018-03-15 13:00:00,奥体中心,66
2018-03-15 14:00:00,海淀区万柳,73
2018-03-15 14:00:00,昌平镇,73
2018-03-15 14:00:00,奥体中心,75

神经网络作业报告

[TOC]
本次作业部分源码的参考资料已注明

一、实验要求

1. 推导具有单隐层的神经网络的前向传播和反向传播算法，并进行编程（可以使用sklearn中的神经网络）。
   • 探讨10，30，100，300，1000，不同隐藏节点数对网络性能的影响。
   • 探讨不同学习率和迭代次数对网络性能的影响。
   • 改变数据的标准化方法，探讨对训练的影响。
2. 查阅资料说明什么是Hebb学习规则

FINDS 算法与 ID3 算法

作业要求

1. 实现FINDS算法
2. 实现ID3算法

• 不要调库自己写。如果有能力可以继续用课件里的数据集测试两个算法（用天气的4条记录测试FINDS，用贷款的15条记录测试ID3）给出训练误差测试误差等；
• 再有能力可以使用更大的数据集测试算法。

Linux操作系统概述和实验环境介绍

操作系统概念

OS是管理和控制计算机硬件与软件资源的计算机程序，是直接在“裸机”上的最基本的系统软件。

Linux的应用

1. 服务器端：Linux非常稳定，特别适合大型企业生产环境。
2. 作为网络平台的后端服务器被使用。
3. 作为应用服务器、数据库服务器被使用：解决海量数据、高并发的问题；
4. 作为嵌入式操作系统被使用：智能控制、自动化、物联网等领域。

1-1 Welcome to machine learning!

1. Machine learning is the science of getting computers to learn without being explicitly programmed.
2. Machine Learning Honor Code

1-2 Welcome

machine Learning

- Grew out of work in AI
- New capability for computers

Exmaple:

- Database mining
Large datasets from growth of automation/web.
E.g. Web click data, medical records, biology, engineering
- Applicaitons can't program by hand.
E.g., Autonomous helicopter, handwriting recognition, most of Natural Language Processing(NLP), Computer Vision.
- Self customizing programs
E.g., Amazon, Netflix product recommendations
- Understanding human learning(brain, real AI).

1-3 What is machine learning?

To know what is machine learning and give u a sense of when u want to use machine learning.

Machine learning defination

1. Arthur Samuel(1959). Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.
2. Tom Mitchell(1998) Well-posed Learning Problem: A computer Program is said to learn from experience $E$ with respect to some task $T$ and some performance measure $P$, if its performance on $T$ as measured by $P$ improves with experience $E$.

Machine learning algorithms

• Supervised learning
• Unsupervised learning

Others: Reinforcement learning(强化学习), recommender systems(推荐系统).

Also talk about practical advice for applying learning algorithms.

1-4 Supevised learning

Supervised learning:
“right answers” given.

1. Regression: Predict continuous valued output.(Used: house price)
2. Classification: Discrete valued output(0 or 1)(Used: Breast cancer)

How do you deal with an infinite number of features, and how do you even store an infinite number of things on the computer?
SVM will be a neat mathematical trick that will allow a computer to deal with an infinite number of features.

1-5 Unsupervised Learning

Data does not have any labels.
Used:

1. Google News.(Cluster the news of the same topic)
2. DNA microarray data.
3. Organize large computer clusters.
4. Social network analysis
5. Market segmentation
6. Astronomical data analysis(how galaxies are formed.)

Clustering is just one type of Unsupervised Learning.

Cocktail Party problem

recognize and separate the audio.
when you use the right programming enviroment, many learning algorithms can be really shot programs.

If you use Octave as your learning tool and as your prorotyping tool(原型工具), it will let you learn and prototype learning algorithms much quickly.

Fisher算法&SVM&K-Means及其优化

部分源码的参考资料已注明

fisher算法及其实现

1. 请实现fisher算法，并采用自己随机生成2类数据（每类100个）的方式，验证自己的算法。
   参考资料:https://blog.csdn.net/pengjian444/article/details/71138003

Chap1 开始使用Linux

Linux发展过程

系统登陆与退出

字符终端

UNIX和Linux是多用户系统：主机连接多台字符终端，字符终端作为交互式输入输出设备。

Tips：

1. 先修课：C语言、操作系统
2. 上机实验：自己安装VMware虚拟机和Ubuntu
3. 成绩考核：平时成绩+上机作业（30%），期末考试（70%）

hadoop存储格式

行式存储格式

未压缩的Sequence File

通过设置同步的标记，把key value划分成块，当后面出错的时候只需要重传一个块。相当于debug时的断点。
record length-key length-key-value
block的结构要清楚。