新版hadoop3.1相较于老版hadoop2.8在配置文件这块有了变化。
这个是我早些时候配置的
在hadoop2.8中配置java路径和项目路径统一在/path/etc/hadoop/yarn-env.sh中。而hadoop3.1中则是在hadoop-env.sh中开启配置
原来slaves的节点配置改成works文件。
昨晚一不留神掉坑里了。这里附上配置。
最新文章
Compared with the older hadoop2.8, the new hadoop3.1 has changes when it comes to configuration files.
This is what I configured earlier
hadoop2.8 installation tutorial
In hadoop2.8, configuring the java path and the project path was all done in /path/etc/hadoop/yarn-env.sh. In hadoop3.1, however, the configuration is enabled in hadoop-env.sh
The original slaves node configuration has been changed to the works file.
Last night I carelessly fell into a pit. The configuration is attached here.
闲来无事就做个图片抓取。
把某网站的图片抓取后，保存在本地指定目录，同时生成写入数据库的sql语句。
思路如下：
- 获取页面图的url
- 根据图片url下载到本地
- 把下载好的图片存储到指定目录，同时生成写入数据库的sql
With nothing better to do, I put together an image scraper.
After scraping the images from a certain website, save them to a specified local directory, and at the same time generate the SQL statements that write them into the database.
The approach is as follows:
- Get the URLs of the images on the page
- Download them locally based on the image URLs
- Store the downloaded images in the specified directory, and generate the SQL that writes them into the database
hbase分布式数据库，很早就知道了，今天尝试在docker中搭建一下。
首先，安装配置hadoop环境。
这教程是我之前在百度经验分享的，版本是2.8，现在看了看了最新版hadoop，教程还是可以用的。
安装好hadoop后别忘了配置ssh、java环境等。
顺便创建用户和组为hadoop（也可以自己定义，这块不会的去预习一下linux用户和组的创建）
I heard about the HBase distributed database a long time ago; today I’m going to try setting one up in Docker.
First, install and configure the Hadoop environment.
I shared that tutorial on Baidu Experience before; it’s for version 2.8, and now that I’ve looked at the latest Hadoop, the tutorial still works.
After installing Hadoop, don’t forget to configure SSH, the Java environment, and so on.
While you’re at it, create a user and group named hadoop (you can define your own too; if you don’t know how, go read up on creating Linux users and groups)
想要从商业 CA 获取数字证书需要向其支付一定的金钱，不过我们不用那么破费，可以自己成为 root CA，为自己发布证书。
在本实验中，我们将成为 root CA，并为该 CA 生成证书。不像其他 CA 需要被另外的 CA 认证， root CA 的证书是自己为自己认证的，一般 Root CA 的证书都已事先加载在大多数操作系统，浏览器或者依赖 PKI 的软件之中了。Root CA 的证书是被无条件信任的。
Getting a digital certificate from a commercial CA costs money, but we don’t have to go to that expense — we can become a root CA ourselves and issue certificates for ourselves.
In this lab we will become a root CA and generate a certificate for that CA. Unlike other CAs, which need to be certified by another CA, a root CA’s certificate is certified by itself. Root CA certificates are generally already loaded in most operating systems, browsers, or software that depends on PKI. A root CA’s certificate is trusted unconditionally.
这两天想用用zookeeper，顺便做个提升。查了下官方给出的3.5.5和3.4.14。而php手册也有介绍配置方法，不过扩展好久没更新，就是试试用了。
下载php-zookeeper扩展，放到本地编译、安装。在执行make命令的时候总是报错缺少lzookeeper_mt.so库。
官方搜索教程都是老版本的，说是在文件根目录src/c下生成库。可是我用的是3.5.5版本的，目录下根本没有。
根据官方文档可知，新版本分为bin和源码版，也就是说bin是编译好的，下载后配置一下cfg文件就可以用了。
I’ve been wanting to try zookeeper for the past couple of days, and figured I’d level up a bit along the way. I looked up the official releases, 3.5.5 and 3.4.14. The PHP manual also explains how to configure it, though the extension hasn’t been updated in ages — I just gave it a try anyway.
I downloaded the php-zookeeper extension, dropped it locally, compiled and installed it. Every time I ran make it errored out about a missing lzookeeper_mt.so library.
The tutorials I found through official channels were all for old versions, saying the library is generated under src/c in the root directory of the project. But I’m using version 3.5.5, and there’s no such directory there at all.
According to the official docs, new versions come in a bin build and a source build, meaning the bin one is already compiled — you download it and just configure a cfg file to use it.
今天去一家做电商的公司面试，笔试题有个算法，如下：
输入任意金额，计算分配出10元、5元、2元是否可以分配，如果可以分别分配几张。
根据需求可知优先分配最高金额，根据最高金额计算出分配值，然后取整数。
Today I went to interview at an e-commerce company, and there was an algorithm question on the written test, as follows:
Input any amount, calculate whether it can be split into 10 yuan, 5 yuan and 2 yuan, and if so how many notes of each.
From the requirement we know we should allocate the highest denomination first, calculate the allocation value from the highest denomination, then take the integer.
Http1.1
Cache handling
It introduces more cache control strategies, such as Entity tag, If-Unmodified-Since, If-Match, If-None-Match and other cache headers you can choose from to control caching policy.
Bandwidth optimization and network connection usage
A range header field was introduced into requests, allowing you to request only a certain part of a resource, i.e. returning status code 206 (Partial Content), which makes it convenient for developers to choose freely and make full use of bandwidth and connections.
开发的大多数项目并未接触到百万级并发，但是一直以来有个id生成的纠结。我一般就是年月日+随机数生成唯一id,一般来说生成一个备用随机数据，不重复每次一次取值隔天重新从第一个开始取。虽然是可以解决，总感觉这个有点繁琐，仔细查查资料研究了一下，
Most of the projects I have developed never ran into million-level concurrency, but I have always been torn about generating ids. I usually generate a unique id from year-month-day + a random number; generally I generate a batch of spare random data without duplicates, take one value at a time, and the next day start again from the first one. It does solve the problem, but it always felt a bit cumbersome. After carefully digging into the material and studying it,
