-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
456 lines (272 loc) · 308 KB
/
atom.xml
File metadata and controls
456 lines (272 loc) · 308 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>码农想种地</title>
<subtitle>努力的Coder</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://trytofix.com/"/>
<updated>2016-07-08T02:35:56.826Z</updated>
<id>http://trytofix.com/</id>
<author>
<name>brian.yang</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>Celery,Tornado,Supervisor构建和谐的分布式系统</title>
<link href="http://trytofix.com/2016/07/07/Celery-Tornado-Supervisor%E6%9E%84%E5%BB%BA%E5%92%8C%E8%B0%90%E7%9A%84%E5%88%86%E5%B8%83%E5%BC%8F%E6%A1%86%E6%9E%B6/"/>
<id>http://trytofix.com/2016/07/07/Celery-Tornado-Supervisor构建和谐的分布式框架/</id>
<published>2016-07-07T08:29:18.000Z</published>
<updated>2016-07-08T02:35:56.826Z</updated>
<content type="html"><h1 id="Celery-分布式的任务队列"><a href="#Celery-分布式的任务队列" class="headerlink" title="Celery 分布式的任务队列"></a><a href="http://docs.celeryproject.org/en/latest/index.html#" rel="external nofollow" target="_blank">Celery</a> 分布式的任务队列</h1><h3 id="与rabbitmq消息队列的区别与联系:"><a href="#与rabbitmq消息队列的区别与联系:" class="headerlink" title="与rabbitmq消息队列的区别与联系:"></a>与rabbitmq消息队列的区别与联系:</h3><ul><li>rabbitmq 调度的是消息,而Celery调度的是任务.</li><li>Celery调度任务时,需要传递参数信息,传输载体可以选择rabbitmq.</li><li>利用rabbitmq的持久化和ack特性,Celery可以保证任务的可靠性.</li></ul><h3 id="优点"><a href="#优点" class="headerlink" title="优点:"></a>优点:</h3><ul><li>轻松构建分布式的Service Provider。</li><li>高可扩展性,增加worker也就是增加了队列的consumer。</li><li>可靠性,利用消息队列的durable和ack,可以尽可能降低消息丢失的概率,当worker崩溃后,未处理的消息会重新进入消费队列。</li><li>用户友好,利用flower提供的管理工具可以轻松的管理worker。<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5lbun7w3sj211p0fk41t" alt="flower"></li><li>使用tornado-celery,结合tornado异步非阻塞结构,可以提高吞吐量,轻松创建分布式服务框架。</li><li>学习成本低,可快速入门</li></ul><h3 id="快速入门"><a href="#快速入门" class="headerlink" title="快速入门"></a>快速入门</h3><p>定义一个celery实例main.py:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">from celery import Celery</span><br><span class="line">app = Celery(&apos;route_check&apos;, include=[&apos;check_worker_path&apos;], </span><br><span class="line"> broker=&apos;amqp://user:password@rabbitmq_host:port//&apos;)</span><br><span class="line">app.config_from_object(&apos;celeryconfig&apos;)</span><br></pre></td></tr></table></figure><p></p><p>include指的是需要celery扫描是否有任务定义的模块路径。例如<code>add_task</code> 就是扫描add_task.py中的任务</p><p>celery的配置文件可以从文件、模块中读取,这里是从模块中读取,celeryconfig.py为:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line">from multiprocessing import cpu_count</span><br><span class="line"></span><br><span class="line">from celery import platforms</span><br><span class="line">from kombu import Exchange, Queue</span><br><span class="line"></span><br><span class="line">CELERYD_POOL_RESTARTS = False</span><br><span class="line">CELERY_RESULT_BACKEND = &apos;redis://:password@redis_host:port/db&apos;</span><br><span class="line">CELERY_QUEUES = (</span><br><span class="line"> Queue(&apos;default&apos;, Exchange(&apos;default&apos;), routing_key=&apos;default&apos;),</span><br><span class="line"> Queue(&apos;common_check&apos;, Exchange(&apos;route_check&apos;), routing_key=&apos;common_check&apos;),</span><br><span class="line"> Queue(&apos;route_check&apos;, Exchange(&apos;route_check&apos;), routing_key=&apos;route_check&apos;, delivery_mode=2),</span><br><span class="line"> Queue(&apos;route_check_ignore_result&apos;, Exchange(&apos;route_check&apos;), routing_key=&apos;route_check_ignore_result&apos;,</span><br><span class="line"> delivery_mode=2)</span><br><span class="line">)</span><br><span class="line">CELERY_ROUTES = &#123;</span><br><span class="line"> &apos;route_check_task.check_worker.common_check&apos;: &#123;&apos;queue&apos;: &apos;common_check&apos;&#125;,</span><br><span class="line"> &apos;route_check_task.check_worker.check&apos;: &#123;&apos;queue&apos;: &apos;route_check&apos;&#125;,</span><br><span class="line"> &apos;route_check_task.check_worker.check_ignore_result&apos;: &#123;&apos;queue&apos;: &apos;route_check_ignore_result&apos;&#125;</span><br><span class="line">&#125;</span><br><span class="line">CELERY_DEFAULT_QUEUE = &apos;default&apos;</span><br><span class="line">CELERY_DEFAULT_EXCHANGE = &apos;default&apos;</span><br><span class="line">CELERY_DEFAULT_EXCHANGE_TYPE = &apos;direct&apos;</span><br><span class="line">CELERY_DEFAULT_ROUTING_KEY = &apos;default&apos;</span><br><span class="line"># CELERY_MESSAGE_COMPRESSION = &apos;gzip&apos;</span><br><span class="line">CELERY_ACKS_LATE = True</span><br><span class="line">CELERYD_PREFETCH_MULTIPLIER = 1</span><br><span class="line">CELERY_DISABLE_RATE_LIMITS = True</span><br><span class="line">CELERY_TIMEZONE = &apos;Asia/Shanghai&apos;</span><br><span class="line">CELERY_ENABLE_UTC = True</span><br><span class="line">CELERYD_CONCURRENCY = cpu_count() / 2</span><br><span class="line">CELERY_TASK_SERIALIZER = &apos;json&apos;</span><br><span class="line">CELERY_RESULT_SERIALIZER = &apos;json&apos;</span><br><span class="line">CELERY_TASK_PUBLISH_RETRY = True</span><br><span class="line">CELERY_TASK_PUBLISH_RETRY_POLICY = &#123;</span><br><span class="line"> &apos;max_retries&apos;: 3,</span><br><span class="line"> &apos;interval_start&apos;: 10,</span><br><span class="line"> &apos;interval_step&apos;: 5,</span><br><span class="line"> &apos;interval_max&apos;: 20</span><br><span class="line">&#125;</span><br><span class="line">platforms.C_FORCE_ROOT = True</span><br></pre></td></tr></table></figure><p></p><p>这里面是一些celery的<a href="http://docs.celeryproject.org/en/latest/configuration.html" rel="external nofollow" target="_blank">配置参数</a>。</p><p>在上面include的add_task.py定义如下:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">#encoding:utf8</span><br><span class="line"></span><br><span class="line">from main import app</span><br><span class="line"></span><br><span class="line">@app.task</span><br><span class="line">def add(x,y):</span><br><span class="line"> return x+y</span><br></pre></td></tr></table></figure><p></p><p>启动celery<br><code>celery -A main worker -l info -Ofair</code></p><ul><li>-A 后面是包含celery定义的模块,我们在main.py中定义了<code>app = Celery...</code><br>测试celery:</li><li>-l 日志打印的级别,这里是info</li><li>-<a href="http://docs.celeryproject.org/en/latest/whatsnew-3.1.html" rel="external nofollow" target="_blank">Ofair</a> 这个参数可以让Celery更好的调度任务</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"># encoding:utf8</span><br><span class="line">__author__ = &apos;brianyang&apos;</span><br><span class="line"></span><br><span class="line">import add_task</span><br><span class="line"></span><br><span class="line">result = add_task.add.apply_async((1,2))</span><br><span class="line">print type(result)</span><br><span class="line">print result.ready()</span><br><span class="line">print result.get()</span><br><span class="line">print result.ready()</span><br></pre></td></tr></table></figure><p>输出是<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">&lt;class &apos;celery.result.AsyncResult&apos;&gt;</span><br><span class="line">False</span><br><span class="line">3</span><br><span class="line">True</span><br></pre></td></tr></table></figure><p></p><p>当调用result.get()时,如果还没有返回结果,将会阻塞直到结果返回。这里需要注意的是,如果需要返回worker执行的结果,必须在之前的config中配置<code>CELERY_RESULT_BACKEND</code>这个参数,一般推荐使用Redis来保存执行结果,如果不关心worker执行结果,设置<code>CELERY_IGNORE_RESULT=True</code>就可以了,关闭缓存结果可以提高程序的执行速度。<br>在上面的测试程序中,如果修改为:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"># encoding:utf8</span><br><span class="line">__author__ = &apos;brianyang&apos;</span><br><span class="line"></span><br><span class="line">import add_task</span><br><span class="line"></span><br><span class="line">result = add_task.add.(1,2)</span><br><span class="line">print type(result)</span><br><span class="line">print result</span><br></pre></td></tr></table></figure><p></p><p>输出结果为:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">&lt;type &apos;int&apos;&gt;</span><br><span class="line">3</span><br></pre></td></tr></table></figure><p></p><p>相当于直接本地调用了add方法,并没有走Celery的调度。<br>通过flower的dashbord可以方便的监控任务的执行情况:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5ld0lp454j211e07p427" alt="task list"><br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5ld11kqvyj20v20dhn0k" alt="task detail"><br>还可以对worker进行重启,关闭之类的操作<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5ld1nfw5uj210p0a1tb0" alt="taks_op"><br>使用Celery将一个集中式的系统拆分为分布式的系统大概步骤就是:</p><ul><li>根据功能将耗时的模块拆分出来,通过注解的形式让Celery管理</li><li>为拆分的模块设置独立的消息队列</li><li>调用者导入需要的模块或方法,使用apply_async进行异步的调用并根据需求关注结果。</li><li>根据性能需要可以添加机器或增加worker数量,方便弹性管理。</li></ul><p>需要注意的是:</p><blockquote><ul><li>尽量为不同的task分配不同的queue,避免多个功能的请求堆积在同一个queue中。</li><li><code>celery -A main worker -l info -Ofair -Q add_queue</code>启动Celery时,可以通过参数Q加queue_name来指定该worker只接受指定queue中的tasks.这样可以使不同的worker各司其职。</li><li><code>CELERY_ACKS_LATE</code>可以让你的Celery更加可靠,只有当worker执行完任务后,才会告诉MQ,消息被消费。</li><li><code>CELERY_DISABLE_RATE_LIMITS</code> Celery可以对任务消费的速率进行限制,如果你没有这个需求,就关闭掉它吧,有益于会加速你的程序。</li></ul></blockquote><h3 id="tornado-celery"><a href="#tornado-celery" class="headerlink" title="tornado-celery"></a>tornado-celery</h3><p>tornado应该是python中最有名的异步非阻塞模型的web框架,它使用的是单进程轮询的方式处理用户请求,通过epoll来关注文件状态的改变,只扫描文件状态符发生变化的FD(文件描述符)。<br>由于tornado是单进程轮询模型,那么就不适合在接口请求后进行长时间的耗时操作,而是应该接收到请求后,将请求交给背后的worker去干,干完活儿后在通过修改FD告诉tornado我干完了,结果拿走吧。很明显,Celery与tornado很般配,而tornado-celery是celery官方推荐的结合两者的一个模块。<br>整合两者很容易,首先需要安装:</p><ul><li>tornado-celery</li><li>tornado-redis<br>tornado代码如下:</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"># encoding:utf8</span><br><span class="line">__author__ = &apos;brianyang&apos;</span><br><span class="line"></span><br><span class="line">import tcelery</span><br><span class="line">import tornado.gen</span><br><span class="line">import tornado.web</span><br><span class="line"></span><br><span class="line">from main import app</span><br><span class="line">import add_task</span><br><span class="line"></span><br><span class="line">tcelery.setup_nonblocking_producer(celery_app=app)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">class CheckHandler(tornado.web.RequestHandler):</span><br><span class="line"> @tornado.web.asynchronous</span><br><span class="line"> @tornado.gen.coroutine</span><br><span class="line"> def get(self):</span><br><span class="line"> x = int(self.get_argument(&apos;x&apos;, &apos;0&apos;))</span><br><span class="line"> y = int(self.get_argument(&apos;y&apos;, &apos;0&apos;))</span><br><span class="line"> response = yield tornado.gen.Task(add_task.add.apply_async, args=[x, y])</span><br><span class="line"> self.write(&#123;&apos;results&apos;: response.result&#125;)</span><br><span class="line"> self.finish</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">application = tornado.web.Application([</span><br><span class="line"> (r&quot;/add&quot;, CheckHandler),</span><br><span class="line">])</span><br><span class="line"></span><br><span class="line">if __name__ == &quot;__main__&quot;:</span><br><span class="line"> application.listen(8889)</span><br><span class="line"> tornado.ioloop.IOLoop.instance().start()</span><br></pre></td></tr></table></figure><p>在浏览器输入:<code>http://127.0.0.1:8889/add?x=1&amp;y=2</code><br>结果为:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5ldt3fclhj20ap04h3yx" alt=""></p><p>通过tornado+Celery可以显著的提高系统的吞吐量。</p><h3 id="Benchmark"><a href="#Benchmark" class="headerlink" title="Benchmark"></a>Benchmark</h3><p>使用Jmeter进行压测,60个进程不间断地的访问服务器:<br>接口单独访问响应时间一般在200~400ms</p><ul><li>uwsgi + Flask方案:<br>uwsgi关键配置:<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">processes = 10</span><br><span class="line">threads = 3</span><br></pre></td></tr></table></figure></li></ul><p>Flask负责接受并处理请求,压测结果:<br>qps是46,吞吐量大概是2700/min<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5ldw47khbj20zz02i75i" alt="uwsgi+Flask"></p><ul><li>tornado+Celery方案:<br>Celery配置:<br><code>CELERYD_CONCURRENCY = 10</code>也就是10个worker(进程),压测结果:<br>qps是139,吞吐量大概是8300/min<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5le2vh8kpj210d0270tw" alt="tornado+Celery"><br>从吞吐量和接口相应时间各方面来看,使用tornado+Celery都能带来更好的性能。</li></ul><h3 id="Supervisor"><a href="#Supervisor" class="headerlink" title="Supervisor"></a>Supervisor</h3><ul><li>什么是supervisor<br>supervisor俗称Linux后台进程管理器</li><li>适合场景<br>– 需要长期运行程序,除了nohup,我们有更好的supervisor<br>– 程序意外挂掉,需要重启,让supervisor来帮忙<br>– 远程管理程序,不想登陆服务器,来来来,supervisor提供了高大上(屁~)的操作界面.<br>之前启动Celery命令是<code>celery -A main worker -l info -Ofair -Q common_check</code>,当你有10台机器的时候,每次更新代码后,都需要登陆服务器,然后更新代码,最后再杀掉Celery进程重启,恶不恶心,简直恶心死了。<br>让supervisor来,首先需要安装:<br><code>pip install supervisor</code><br>配置文件示例:</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line">[unix_http_server]</span><br><span class="line">file=/tmp/supervisor.sock ; path to your socket file</span><br><span class="line">chmod=0777</span><br><span class="line">username=admin</span><br><span class="line">password=admin</span><br><span class="line"></span><br><span class="line">[inet_http_server]</span><br><span class="line">port=0.0.0.0:2345</span><br><span class="line">username=admin</span><br><span class="line">password=admin</span><br><span class="line"></span><br><span class="line">[supervisord]</span><br><span class="line">logfile=/var/log/supervisord.log ; supervisord log file</span><br><span class="line">logfile_maxbytes=50MB ; maximum size of logfile before rotation</span><br><span class="line">logfile_backups=10 ; number of backed up logfiles</span><br><span class="line">loglevel=info ; info, debug, warn, trace</span><br><span class="line">pidfile=/var/run/supervisord.pid ; pidfile location</span><br><span class="line">nodaemon=false ; run supervisord as a daemon</span><br><span class="line">minfds=1024 ; number of startup file descriptors</span><br><span class="line">minprocs=200 ; number of process descriptors</span><br><span class="line">user=root ; default user</span><br><span class="line">childlogdir=/var/log/ ; where child log files will live</span><br><span class="line"></span><br><span class="line">[rpcinterface:supervisor]</span><br><span class="line">supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface</span><br><span class="line"></span><br><span class="line">[supervisorctl]</span><br><span class="line">serverurl=unix:///tmp/supervisor.sock ; use unix:// schem for a unix sockets.</span><br><span class="line">username=admin</span><br><span class="line">password=admin</span><br><span class="line">[program:celery]</span><br><span class="line">command=celery -A main worker -l info -Ofair</span><br><span class="line"></span><br><span class="line">directory=/home/q/celeryTest</span><br><span class="line">user=root</span><br><span class="line">numprocs=1</span><br><span class="line">stdout_logfile=/var/log/worker.log</span><br><span class="line">stderr_logfile=/var/log/worker.log</span><br><span class="line">autostart=true</span><br><span class="line">autorestart=true</span><br><span class="line">startsecs=10</span><br><span class="line"></span><br><span class="line">; Need to wait for currently executing tasks to finish at shutdown.</span><br><span class="line">; Increase this if you have very long running tasks.</span><br><span class="line">stopwaitsecs = 10</span><br><span class="line"></span><br><span class="line">; When resorting to send SIGKILL to the program to terminate it</span><br><span class="line">; send SIGKILL to its whole process group instead,</span><br><span class="line">; taking care of its children as well.</span><br><span class="line">killasgroup=true</span><br><span class="line"></span><br><span class="line">; Set Celery priority higher than default (999)</span><br><span class="line">; so, if rabbitmq is supervised, it will start first.</span><br><span class="line">priority=1000</span><br></pre></td></tr></table></figure><p>示例文件很长,不要怕,只需要复制下来,改改就可以<br>比较关键的几个地方是:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[inet_http_server]</span><br><span class="line">port=0.0.0.0:2345</span><br><span class="line">username=admin</span><br><span class="line">password=admin</span><br></pre></td></tr></table></figure><p></p><p>这个可以让你通过访问<code>http://yourhost:2345</code> ,验证输入admin/admin的方式远程管理supervisor,效果如下:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5leiy45vuj20vq0a0774" alt="remote supervisor"><br><code>[program:flower]</code>这里就是你要托管给supervisor的程序的一些配置,其中<code>autorestart=true</code>可以在程序崩溃时自动重启进程,不信你用kill试试看。<br>剩下的部分就是一些日志位置的设置,当前工作目录设置等,so esay~</p><p>supervisor优点:</p><ul><li>管理进程简单,再也不用nohup &amp; kill了。</li><li>再也不用担心程序挂掉了</li><li>web管理很方便</li></ul><p>缺点:</p><ul><li>web管理虽然方便,但是每个页面只能管理本机的supervisor,如果我有一百台机器,那就需要打开100个管理页面,很麻烦.</li></ul><p>怎么办~</p><h3 id="supervisor-easy闪亮登场"><a href="#supervisor-easy闪亮登场" class="headerlink" title="supervisor-easy闪亮登场"></a><a href="https://github.com/trytofix/supervisor-easy" rel="external nofollow" target="_blank">supervisor-easy</a>闪亮登场</h3><p>通过rpc调用获取配置中的每一个supervisor程序的状态并进行管理,可以分组,分机器进行批量/单个的管理。方便的不要不要的。来两张截图:</p><ul><li>分组管理:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5lerqg160j20vr0cetbm" alt="group"></li><li>分机器管理:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5letuuixfj20vw0bpgod" alt="server"><br>通过简单的配置,可以方便的进行管理。</li></ul></content>
<summary type="html">
<h1 id="Celery-分布式的任务队列"><a href="#Celery-分布式的任务队列" class="headerlink" title="Celery 分布式的任务队列"></a><a href="http://docs.celeryproject.org/en
</summary>
<category term="python" scheme="http://trytofix.com/categories/python/"/>
<category term="python" scheme="http://trytofix.com/tags/python/"/>
</entry>
<entry>
<title>supervisor-easy使用指南</title>
<link href="http://trytofix.com/2016/06/28/supervisor-easy%E4%BD%BF%E7%94%A8%E6%8C%87%E5%8D%97/"/>
<id>http://trytofix.com/2016/06/28/supervisor-easy使用指南/</id>
<published>2016-06-28T13:43:29.000Z</published>
<updated>2016-06-28T13:44:01.914Z</updated>
<content type="html"><p>supervisor是一个python实现的可以对进程进行管理的程序,可以对托管的程序方便的进行类似重启,查看日志之类的操作,并提供了一个简单的操作界面.<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5b87cx8nij20mm07k3zl" alt=""><br>但是在生产中可能会在上百台不同的服务器上通过supervisor部署程序,而supervisor默认提供的web管理工具是不能跨平台进行管理的,这对于程序的维护将会是致命的。</p><p><a href="https://github.com/trytofix/supervisor-easy" rel="external nofollow" target="_blank">supervisor-easy</a>是一个可以集中管理supervisor程序的项目。</p><p>这个项目用到了:</p><ul><li>Flask web server</li><li>bootstrap + jquery UI</li></ul><p>工具的主要特点包括:</p><ul><li>方便部署:</li></ul><p>不需要额外的数据库,缓存,仅仅需要python环境以及安装了Flask.</p><ul><li>将任何程序进行分组管理。例如可以将不同机器上执行相同功能的应用放在一个组里,可以对真个组进行管理,比如:重启,开始,停止等操作。需要做的仅仅是按照规则修改config.py文件.</li></ul><p><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5b8krfyrcj219b0da77l" alt=""></p><ul><li>显示所有配置的主机以及他们所包含的应用。</li></ul><p>同样的,这些应用也可以批量的进行管理。</p><p><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5b8n8i0t3j21990cb77r" alt=""></p><p>##用法</p><ul><li>git clone <a href="https://github.com/trytofix/supervisor-easy.git" rel="external nofollow" target="_blank">https://github.com/trytofix/supervisor-easy.git</a></li><li>edit config.py</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">from Server import Server</span><br><span class="line"></span><br><span class="line">SERVERS = [</span><br><span class="line"> Server(</span><br><span class="line"> name=&apos;celery1&apos;,</span><br><span class="line"> host=&apos;127.0.0.1&apos;,</span><br><span class="line"> port=12345,</span><br><span class="line"> user=&apos;admin&apos;,</span><br><span class="line"> password=&apos;admin&apos;</span><br><span class="line"> ),</span><br><span class="line"> Server(</span><br><span class="line"> name=&apos;celery2&apos;,</span><br><span class="line"> host=&apos;remote.supervisor.com&apos;,</span><br><span class="line"> port=12345,</span><br><span class="line"> user=&apos;admin&apos;,</span><br><span class="line"> password=&apos;admin&apos;</span><br><span class="line"> )</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line">GROUPS = [</span><br><span class="line"> &#123;</span><br><span class="line"> &apos;name&apos;: &apos;celery&apos;,</span><br><span class="line"> &apos;apps&apos;: [&apos;celery1.test:celery&apos;, &apos;celery2.test:celery&apos;]</span><br><span class="line"> &#125;,</span><br><span class="line"> &#123;</span><br><span class="line"> &apos;name&apos;: &apos;flower&apos;,</span><br><span class="line"> &apos;apps&apos;: [&apos;celery1.flower&apos;]</span><br><span class="line"> &#125;</span><br><span class="line">]</span><br></pre></td></tr></table></figure><p>上述配置中,一个Server对应的就是一个服务器上的supervisor实例,Server中的name是方便人来看的名字。</p><blockquote><p>Notice: name必须是唯一的。不能重复。</p></blockquote><p>下面是一个supervisor程序的配置文件。<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">[program:test]</span><br><span class="line">command=celery -A main worker -l info -Ofair -Q test</span><br><span class="line"></span><br><span class="line">directory=/home/q/celeryTest</span><br><span class="line">user=brianyang</span><br><span class="line">numprocs=1</span><br><span class="line">stdout_logfile=/var/log/common.log</span><br><span class="line">stderr_logfile=/var/log/common_err.log</span><br><span class="line">autostart=true</span><br><span class="line">autorestart=true</span><br><span class="line">startsecs=10</span><br><span class="line"></span><br><span class="line">killasgroup=true</span><br><span class="line"></span><br><span class="line">priority=1000</span><br><span class="line"></span><br><span class="line">[group:group1]</span><br><span class="line">programs=celery,test</span><br></pre></td></tr></table></figure><p></p><p>对照这个配置文件来讲解GROUPS</p><ul><li><em>GROUPS</em> 可以将任何的应用分到一组进行批量管理.</li><li><em>apps</em> 是分组中要包含应用的列表.</li><li>‘apps’中的每个字符串的定义为 <code>server_name.group_name:application_name</code>.</li><li><em>application_name</em> 对应supervisord.conf中’[program:test]’ 里的’test’.</li><li><em>group_name</em>对应supervisord.conf中’[group:group1]’ 中的’group1’.</li><li><em>server_name</em> 就是之前在SERVERS中定义的某个Server的name.</li></ul><h2 id="运行-python-webui-py测试项目。"><a href="#运行-python-webui-py测试项目。" class="headerlink" title="运行: python webui.py测试项目。"></a>运行: <code>python webui.py</code>测试项目。</h2><blockquote><p>不要在生产环境中使用 <code>python webui.py</code> 。 更好的做法是使用一种常用的uwsgi服务器,例如uwsgi.</p></blockquote><h2 id="下一步要做的"><a href="#下一步要做的" class="headerlink" title="下一步要做的"></a>下一步要做的</h2><ul><li>性能提升</li><li>UI提升</li><li>异常处理</li></ul></content>
<summary type="html">
<p>supervisor是一个python实现的可以对进程进行管理的程序,可以对托管的程序方便的进行类似重启,查看日志之类的操作,并提供了一个简单的操作界面.<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f5b87cx
</summary>
<category term="supervisor" scheme="http://trytofix.com/categories/supervisor/"/>
<category term="supervisor" scheme="http://trytofix.com/tags/supervisor/"/>
</entry>
<entry>
<title>RabbitMQ浅读</title>
<link href="http://trytofix.com/2016/05/12/RabbitMQ%E6%B5%85%E8%AF%BB/"/>
<id>http://trytofix.com/2016/05/12/RabbitMQ浅读/</id>
<published>2016-05-12T02:45:29.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><h1 id="消息分发策略"><a href="#消息分发策略" class="headerlink" title="消息分发策略"></a>消息分发策略</h1><p>当有多个消费者时,RabbitMQ将会轮流的将消息发送给消费者.这种分发消息的方式称为’round-robin’<br>看例子<br>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line"></span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(queue=<span class="string">'hello'</span>)</span><br><span class="line"></span><br><span class="line">channel.basic_publish(exchange=<span class="string">''</span>,</span><br><span class="line"> routing_key=<span class="string">'hello'</span>,</span><br><span class="line"> body=sys.argv[<span class="number">1</span>])</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'send done'</span></span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>receiver.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(<span class="string">'hello'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(ch, method, prop, body)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> body</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">channel.basic_consume(callback, queue=<span class="string">'hello'</span>, no_ack=<span class="keyword">True</span>)</span><br><span class="line">channel.start_consuming()</span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>运行两个消费者<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">shell1$ python receiver.py</span><br></pre></td></tr></table></figure><p></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">shell2$ python receiver.py</span><br></pre></td></tr></table></figure><p>发送消息<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">python sender.py msg1</span><br><span class="line">python sender.py msg2</span><br><span class="line">python sender.py msg3</span><br><span class="line">python sender.py msg4</span><br><span class="line">python sender.py msg5</span><br><span class="line">python sender.py msg6</span><br></pre></td></tr></table></figure><p></p><p>生产者收到的内容为<br>receiver1<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">shell1$ msg1</span><br><span class="line">shell1$ msg3</span><br><span class="line">shell1$ msg5</span><br></pre></td></tr></table></figure><p></p><p>receiver2<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">shell1$ msg2</span><br><span class="line">shell1$ msg4</span><br><span class="line">shell1$ msg6</span><br></pre></td></tr></table></figure><p></p><a id="more"></a><p>可以看到消息是以轮询的方式发送给消费者的.</p><h1 id="消息容错机制"><a href="#消息容错机制" class="headerlink" title="消息容错机制"></a>消息容错机制</h1><h2 id="客户端崩溃"><a href="#客户端崩溃" class="headerlink" title="客户端崩溃"></a>客户端崩溃</h2><p>引入ack确认机制.客户端每成功消费一个消息,向服务器发送一个确认消息,通知服务器这条消息已经被成功消费.服务器收到消息后,将这条消息从unacknowledged中移除,否则,服务器就会一直等待客户端发来的ack消息.当客户端出现异常时,未消费的消息被重新放到消费队列中.这样避免了客户端崩溃造成的数据丢失.<br>启用ack机制的消费者代码如下:<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(<span class="string">'hello1'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(ch, method, prop, body)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> body</span><br><span class="line"> time.sleep(<span class="number">3</span>)</span><br><span class="line"> ch.basic_ack(delivery_tag=method.delivery_tag)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">channel.basic_consume(callback, queue=<span class="string">'hello1'</span>, no_ack=<span class="keyword">False</span>)</span><br><span class="line">channel.start_consuming()</span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>就是要保证no_ack为False,这也是no_ack的默认值.<br>这里一定要显式的发送确认消息`ch.basic_ack(delivery_tag=method.delivery_tag)明确的告诉服务器消息被处理了.<br>查看队列中的消息数量,可以使用<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">brianyang@brianyang-Latitude-E5440:/home/q/title/test$ sudo rabbitmqctl list_queues name messages_ready messages_unacknowledged</span><br><span class="line">Listing queues ...</span><br><span class="line">hello1 6 2</span><br></pre></td></tr></table></figure><p></p><p>当消费者在消费时被强行结束时,消息并没有丢失,只要出现可用的消费者时,消息会被重新发送.</p><h2 id="服务端崩溃"><a href="#服务端崩溃" class="headerlink" title="服务端崩溃"></a>服务端崩溃</h2><p>可以通过持久化(durable)来确保通道和消息都被保存到磁盘中进行持久化,但是由于从内存写入磁盘也需要时间,如果这段时间出现故障,则这些消息也是会丢失的.所以durable是一种弱的持久化.<br>持久化需要在queue和message中声明.<br>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line"></span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(queue=<span class="string">'hello1'</span>, durable=<span class="keyword">True</span>)</span><br><span class="line"></span><br><span class="line">channel.basic_publish(exchange=<span class="string">''</span>,</span><br><span class="line"> routing_key=<span class="string">'hello1'</span>,</span><br><span class="line"> body=sys.argv[<span class="number">1</span>],</span><br><span class="line"> properties=pika.BasicProperties(</span><br><span class="line"> delivery_mode=<span class="number">2</span>,</span><br><span class="line"> )</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'send done'</span></span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>receiver.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(<span class="string">'hello1'</span>, durable=<span class="keyword">True</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(ch, method, prop, body)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> body</span><br><span class="line"> time.sleep(<span class="number">3</span>)</span><br><span class="line"> ch.basic_ack(delivery_tag=method.delivery_tag)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">channel.basic_consume(callback, queue=<span class="string">'hello1'</span>, no_ack=<span class="keyword">False</span>)</span><br><span class="line">channel.start_consuming()</span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>hello1被声明为了持久化的通道,这里不能用hello命名,因为之前已经存在了一个非持久化的通道hello,RabbitMQ不允许对一个已经声明过的通道进行重定义.<br>生产者在发送消息时,将消息的类型定义为<code>delivery_mode=2</code>,用来将消息持久化.<br>通过例子来看下效果,消息持久化前,也就是没有添加durable时,重启server后消息会丢失.<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3so8texblg211y0lc1kx" alt="非持久化"><br>添加持久化选项后,重启server后消息没有丢失.<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3soh3wf0xg211y0lcu0x" alt="持久化"><br>RabbitMQ对新入列的消息进行分配,不会考虑消费者的状态,如果两个消费者一个处理能力强,一个处理能力弱,长时间下来就会造成一个消费者消息堆积,另一个消费者相对很闲,为了公平期间,可以设置每次每个消费者只分发N个任务,只有任务收到ack后,才继续分发任务.(当no_ack为True时,这个功能失效)<br>修改后的代码receiver.py内容不变,sender.py修改为<br>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(<span class="string">'hello1'</span>, durable=<span class="keyword">True</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(ch, method, prop, body)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> body</span><br><span class="line"> time.sleep(<span class="number">3</span>)</span><br><span class="line"> ch.basic_ack(delivery_tag=method.delivery_tag)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">channel.basic_qos(prefetch_count=<span class="number">1</span>)</span><br><span class="line">channel.basic_consume(callback, queue=<span class="string">'hello1'</span>, no_ack=<span class="keyword">False</span>)</span><br><span class="line">channel.start_consuming()</span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p><code>prefetch_count=1</code>意味着每次只为消费者分配一条消息,消费者处理完成之后,才分配新的消息</p><h1 id="exchange-发布-订阅模型"><a href="#exchange-发布-订阅模型" class="headerlink" title="exchange 发布/订阅模型"></a>exchange 发布/订阅模型</h1><p>exchange的作用好比:收发邮件时的邮件组.如果A,B在邮件组中,C不在邮件组中,那么当Z向邮件组发邮件时,只有A,B能收到,对于Z而言,并不关心邮件组里有谁,只负责向邮件组里发邮件,如果邮件组里没人,那么邮件就会被丢弃,当Z发了100封邮件后,C加入了邮件组,那么C只能收到从第101封开始的邮件,之前的邮件是看不到的.在这里Z是发布者,A,B,C是订阅着,邮件组是exchange.<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3sq32ge6dj20h906x3yx" alt="盗图一张"><br>其中P就是发布者(Z),C1,C2就是消费者(A,B,C),X就是exchange(邮件组),amq.gen-RQ6..和amq.gen_As8..就是exchange与消费者之间通信的通道(邮箱).<br>上代码<br>receiver.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">result = channel.queue_declare(exclusive=<span class="keyword">True</span>)</span><br><span class="line">queue_name = result.method.queue</span><br><span class="line">channel.queue_bind(exchange=<span class="string">'exg'</span>, queue=queue_name)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(ch, method, prop, body)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> body</span><br><span class="line"> time.sleep(<span class="number">3</span>)</span><br><span class="line"> ch.basic_ack(delivery_tag=method.delivery_tag)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">channel.basic_consume(callback, queue=queue_name, no_ack=<span class="keyword">False</span>)</span><br><span class="line">channel.start_consuming()</span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line"></span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.exchange_declare(exchange=<span class="string">'exg'</span>, type=<span class="string">'fanout'</span>)</span><br><span class="line"></span><br><span class="line">channel.basic_publish(exchange=<span class="string">'exg'</span>,</span><br><span class="line"> routing_key=<span class="string">''</span>,</span><br><span class="line"> body=sys.argv[<span class="number">1</span>])</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'send done'</span></span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>下面是个例子,可以体会下与使用queue有什么不同.<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3sqfm2g46g211y0lcnmo" alt="exchange"></p><p>直接使用channel时,消息是面对消费者的,每条消息都会等待消费者消费,而使用exchange时,消息是面对exchange的,对于是否有消费者通过channel与exchange绑定是未知的,exchange不会将消息保存起来等待消费者.</p><h1 id="路由"><a href="#路由" class="headerlink" title="路由"></a>路由</h1><p>使用type类型为fanout的exchange作为中转时,所有的订阅着都会收到相同的消息,假设我有两个用户,一个是vip,一个是普通用户,有些消息我只想发给vip,不想让普通用户也收到,这时候就要学习下路由功能.</p><p>首先看下代码<br>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line"></span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.exchange_declare(exchange=<span class="string">'exchange'</span>, type=<span class="string">'direct'</span>)</span><br><span class="line"></span><br><span class="line">channel.basic_publish(exchange=<span class="string">'exchange'</span>,</span><br><span class="line"> routing_key=sys.argv[<span class="number">1</span>],</span><br><span class="line"> body=sys.argv[<span class="number">2</span>])</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'send done'</span></span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>receiver.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line">exg_types = sys.argv[<span class="number">1</span>:]</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">result = channel.queue_declare(exclusive=<span class="keyword">True</span>)</span><br><span class="line">queue_name = result.method.queue</span><br><span class="line"><span class="keyword">for</span> exg_type <span class="keyword">in</span> exg_types:</span><br><span class="line"> channel.queue_bind(exchange=<span class="string">'exchange'</span>, queue=queue_name, routing_key=exg_type)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(ch, method, prop, body)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> method.routing_key</span><br><span class="line"> <span class="keyword">print</span> body</span><br><span class="line"> time.sleep(<span class="number">1</span>)</span><br><span class="line"> ch.basic_ack(delivery_tag=method.delivery_tag)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">channel.basic_consume(callback, queue=queue_name, no_ack=<span class="keyword">False</span>)</span><br><span class="line">channel.start_consuming()</span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>在receiver.py中,绑定exchange与queue的操作中多了一个<code>routing_key=exg_type</code>,这里routing_key就是一个路由标识,只有当sender使用basic_publish指定的routing_key等于这个routing_key时,消息才会通过exchange发送到该queue中,同时要注意,声明exchange的type也变为了direct而不是之前的fanout.从字面上也很容易理解,之前的方式是广播,现在的方式是直连.<br>继续盗图<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3srqueggtj20ll07f0tg" alt="routing_pic"><br>通过演示看下效果<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3srl000h1g211y0lcb29" alt="routing"></p><h1 id="更加复杂的路由"><a href="#更加复杂的路由" class="headerlink" title="更加复杂的路由"></a>更加复杂的路由</h1><p>当exchange的type为direct时,通过判断绑定的routing_key与发送的routing_key是否相等来判断应该将消息放入到哪个channel中.这是最简单的一种匹配方式,设想有这样一种场景,公司给员工通过队列发送消息,员工分为程序猿,前端喵和产品狗,同时员工又分为不同的级别:初级,中级和高级,员工又有不同的性别,公司对每种类别的员工的消息也不同,例如对于初级女前端,公司的祝福语为:前端的萌妹子感谢你在公司1年的付出,对于高级男程序猿,公司的祝福语为:后端的屌丝男感谢你在公司3年的付出.<br>对于这种维度更加广的路由,可以使用Topics. 使用Topics也很简单,首先将exchange的type变为topic.<br>topics支持通配符,如下:</p><ul><li><code>*</code> (star) can substitute for exactly one word. 使用*表示一个单词</li><li><code>#</code> (hash) can substitute for zero or more words. 使用#表示0个或多个单词<br>routing_key的定义必须遵循 - 由一系列英文逗号分割的单词组成<br>例如上面的高级男程序猿就可以定义为:high.man.monkey,初级女前端喵定义为:low.women.cat<br>如果一个channel的routing_key为</li><li>high.# : 接收向所有的高级员工发送的信息</li><li>high.*.monkey : 接收向所有的高级程序猿发送的信息</li><li><em>.</em>.dog : 向所有产品狗发送的信息<blockquote><p>原谅我这么粗鲁的称呼,这只是俚语!</p></blockquote></li></ul><p>通过topic的方式,可以更加精准的控制路由<br>继续盗图:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3sskqvgusj20m8080wf2" alt="topic"><br>代码如下:<br>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line"></span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.exchange_declare(exchange=<span class="string">'exchange'</span>, type=<span class="string">'topic'</span>)</span><br><span class="line"></span><br><span class="line">channel.basic_publish(exchange=<span class="string">'exchange'</span>,</span><br><span class="line"> routing_key=sys.argv[<span class="number">1</span>],</span><br><span class="line"> body=sys.argv[<span class="number">2</span>])</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'send done'</span></span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>receiver.py代码不变,看下演示:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3stzuyn3mg211y0lcqv5" alt="topic"></p><h1 id="实现RPC"><a href="#实现RPC" class="headerlink" title="实现RPC"></a>实现RPC</h1><p>RabbitMQ将消息放在消息队列中,可以方便的实现生产者-消费者模型.而RPC(远程过程调用)是一种构建SOA非常关键的技术,即面向服务架构.服务可以分布在集群中,通过增减机器可以方便的扩展服务的处理能力.<br>RabbitMQ实现RPC的原理就是,请求服务的应用将请求参数放入到请求队列中,同时传递一个回调队列和唯一id,回调队列用来存放服务方的计算结果,唯一id用来识别是客户端的哪一次请求,需要保证唯一性.<br>服务端在请求队列中获取到消息后,进行计算,计算结束后将结果放入请求方给的回调队列中,同时传回唯一id.<br>盗图<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3swbwafeoj20no07kwfa" alt="RPC_pic"><br>首先看下代码<br>customer.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(</span><br><span class="line"> host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line">channel.queue_declare(<span class="string">'service_queue'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">Fib</span><span class="params">(n)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> n == <span class="number">1</span> <span class="keyword">or</span> n == <span class="number">2</span>:</span><br><span class="line"> <span class="keyword">return</span> n</span><br><span class="line"> <span class="keyword">return</span> Fib(n - <span class="number">1</span>) + Fib(n - <span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">Cal</span><span class="params">(ch, method, props, body)</span>:</span></span><br><span class="line"> num = int(body)</span><br><span class="line"> <span class="keyword">print</span> <span class="string">'cal &#123;&#125;'</span>.format(num)</span><br><span class="line"> resp = Fib(num)</span><br><span class="line"> ch.basic_publish(exchange=<span class="string">''</span>,</span><br><span class="line"> routing_key=props.reply_to,</span><br><span class="line"> properties=pika.BasicProperties(correlation_id=props.correlation_id),</span><br><span class="line"> body=str(resp)</span><br><span class="line"> )</span><br><span class="line"> ch.basic_ack(delivery_tag=method.delivery_tag)</span><br><span class="line"></span><br><span class="line">channel.basic_qos(prefetch_count=<span class="number">1</span>)</span><br><span class="line">channel.basic_consume(Cal, queue=<span class="string">'service_queue'</span>)</span><br><span class="line">channel.start_consuming()</span><br></pre></td></tr></table></figure><p></p><p>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"><span class="keyword">import</span> uuid</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">FibonacciRpcClient</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.connection = pika.BlockingConnection(pika.ConnectionParameters(</span><br><span class="line"> host=<span class="string">'localhost'</span>))</span><br><span class="line"></span><br><span class="line"> self.channel = self.connection.channel()</span><br><span class="line"></span><br><span class="line"> result = self.channel.queue_declare(exclusive=<span class="keyword">True</span>)</span><br><span class="line"> self.callback_queue = result.method.queue</span><br><span class="line"></span><br><span class="line"> self.channel.basic_consume(self.on_response, no_ack=<span class="keyword">True</span>,</span><br><span class="line"> queue=self.callback_queue)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">on_response</span><span class="params">(self, ch, method, props, body)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> self.corr_id == props.correlation_id:</span><br><span class="line"> self.response = body</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">call</span><span class="params">(self, n)</span>:</span></span><br><span class="line"> self.response = <span class="keyword">None</span></span><br><span class="line"> self.corr_id = str(uuid.uuid4())</span><br><span class="line"> self.channel.basic_publish(exchange=<span class="string">''</span>,</span><br><span class="line"> routing_key=<span class="string">'service_queue'</span>,</span><br><span class="line"> properties=pika.BasicProperties(</span><br><span class="line"> reply_to = self.callback_queue,</span><br><span class="line"> correlation_id = self.corr_id,</span><br><span class="line"> ),</span><br><span class="line"> body=str(n))</span><br><span class="line"> <span class="keyword">while</span> self.response <span class="keyword">is</span> <span class="keyword">None</span>:</span><br><span class="line"> self.connection.process_data_events()</span><br><span class="line"> <span class="keyword">return</span> int(self.response)</span><br><span class="line"></span><br><span class="line">fibonacci_rpc = FibonacciRpcClient()</span><br><span class="line"></span><br><span class="line">num = int(sys.argv[<span class="number">1</span>])</span><br><span class="line">response = fibonacci_rpc.call(num)</span><br><span class="line">print(<span class="string">" [x] Requesting fib(&#123;&#125;)"</span>.format(num))</span><br><span class="line">print(<span class="string">" [.] Result is %r"</span> % response)</span><br></pre></td></tr></table></figure><p></p><p>根据官网的demo稍微修改<br>效果如下:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3svxsqykbg211y0lcqv6" alt="RPC"><br>可以看到当客户端请求计算一个比较大的数的Fib数列值的时候,客户端和服务器都阻塞了,当另一个客户端请求计算时,由于没有消费者可以消费所以也阻塞了,这就好比在生产中单台服务器提供服务遇到了瓶颈,SOA架构可以方便的扩容,在这里就是又启了一个消费者,通过这种方法可以动态的改变集群的处理能力.<br>这些内容都在官方快速入门可以看到,我只是搬运工+汉化,加深印象,<a href="https://www.rabbitmq.com/getstarted.html" rel="external nofollow" target="_blank">RabbitMQ tutorials</a></p></content>
<summary type="html">
<h1 id="消息分发策略"><a href="#消息分发策略" class="headerlink" title="消息分发策略"></a>消息分发策略</h1><p>当有多个消费者时,RabbitMQ将会轮流的将消息发送给消费者.这种分发消息的方式称为’round-robin’<br>看例子<br>sender.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line"></span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(queue=<span class="string">'hello'</span>)</span><br><span class="line"></span><br><span class="line">channel.basic_publish(exchange=<span class="string">''</span>,</span><br><span class="line"> routing_key=<span class="string">'hello'</span>,</span><br><span class="line"> body=sys.argv[<span class="number">1</span>])</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'send done'</span></span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>receiver.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pika</span><br><span class="line"></span><br><span class="line">connection = pika.BlockingConnection(pika.ConnectionParameters(host=<span class="string">'localhost'</span>))</span><br><span class="line">channel = connection.channel()</span><br><span class="line"></span><br><span class="line">channel.queue_declare(<span class="string">'hello'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(ch, method, prop, body)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> body</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">channel.basic_consume(callback, queue=<span class="string">'hello'</span>, no_ack=<span class="keyword">True</span>)</span><br><span class="line">channel.start_consuming()</span><br><span class="line">connection.close()</span><br></pre></td></tr></table></figure><p></p><p>运行两个消费者<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">shell1$ python receiver.py</span><br></pre></td></tr></table></figure><p></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">shell2$ python receiver.py</span><br></pre></td></tr></table></figure><p>发送消息<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">python sender.py msg1</span><br><span class="line">python sender.py msg2</span><br><span class="line">python sender.py msg3</span><br><span class="line">python sender.py msg4</span><br><span class="line">python sender.py msg5</span><br><span class="line">python sender.py msg6</span><br></pre></td></tr></table></figure><p></p><p>生产者收到的内容为<br>receiver1<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">shell1$ msg1</span><br><span class="line">shell1$ msg3</span><br><span class="line">shell1$ msg5</span><br></pre></td></tr></table></figure><p></p><p>receiver2<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">shell1$ msg2</span><br><span class="line">shell1$ msg4</span><br><span class="line">shell1$ msg6</span><br></pre></td></tr></table></figure><p></p>
</summary>
<category term="RabbitMQ" scheme="http://trytofix.com/categories/RabbitMQ/"/>
</entry>
<entry>
<title>Flask中使用装饰器遇到的问题AssertionError: View function mapping is overwriting an existing endpoint function</title>
<link href="http://trytofix.com/2016/05/05/Flask%E4%B8%AD%E4%BD%BF%E7%94%A8%E8%A3%85%E9%A5%B0%E5%99%A8%E9%81%87%E5%88%B0%E7%9A%84%E9%97%AE%E9%A2%98AssertionError-View-function-mapping-is-overwriting-an-existing-endpoint-function/"/>
<id>http://trytofix.com/2016/05/05/Flask中使用装饰器遇到的问题AssertionError-View-function-mapping-is-overwriting-an-existing-endpoint-function/</id>
<published>2016-05-05T07:43:25.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>在生产中有几个api需要用户认证才可以访问,最初的做法是在每个方法最前面写一段相同的代码来验证身份,这种通用的功能更适合提炼出来而不是在每个方法中都粘贴一份.</p><p>python支持的装饰器可以扮演拦截器的角色,将验证身份的代码放入到装饰器中,并为需要的方法添加装饰器是一个更好的做法.</p><p>首先定义一个装饰器auth_util.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_auth_wrapper</span><span class="params">(self, func)</span>:</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line"> uid = checkuserislogin(request)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> uid:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request)</span><br><span class="line"> admin = self.check_auth(uid)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> admin:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request, url=<span class="string">'/'</span>)</span><br><span class="line"> <span class="keyword">return</span> func(*args, **kwargs)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure><p></p><p>使用时<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@web_api.route("/alert/&lt;alert_id&gt;/delete")</span></span><br><span class="line"><span class="meta">@auth_util.check_auth_wrapper</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alert_delete</span><span class="params">(alert_id)</span>:</span></span><br><span class="line"> del_alert(alert_id)</span><br><span class="line"> <span class="keyword">return</span> <span class="string">'delete success'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@web_api.route('/alert/')</span></span><br><span class="line"><span class="meta">@auth_util.check_auth_wrapper</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alert_index</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> render_template(<span class="string">'alert_index.html'</span>)</span><br></pre></td></tr></table></figure><p></p><p>启动程序时会报错<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">Traceback (most recent call last):</span><br><span class="line"> File &quot;/home/q/www/api/all_webui.py&quot;, line 22, in &lt;module&gt;</span><br><span class="line"> app.register_blueprint(routecheck_api)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 62, in wrapper_func</span><br><span class="line"> return f(self, *args, **kwargs)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 889, in register_blueprint</span><br><span class="line"> blueprint.register(self, options, first_registration)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/blueprints.py&quot;, line 153, in register</span><br><span class="line"> deferred(state)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/blueprints.py&quot;, line 172, in &lt;lambda&gt;</span><br><span class="line"> s.add_url_rule(rule, endpoint, view_func, **options))</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/blueprints.py&quot;, line 76, in add_url_rule</span><br><span class="line"> view_func, defaults=defaults, **options)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 62, in wrapper_func</span><br><span class="line"> return f(self, *args, **kwargs)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 984, in add_url_rule</span><br><span class="line"> &apos;existing endpoint function: %s&apos; % endpoint)</span><br><span class="line">AssertionError: View function mapping is overwriting an existing endpoint function: /home/q/www/route_web/apis/routecheck/routecheck_webui.pyc.wrapper</span><br></pre></td></tr></table></figure><p></p><a id="more"></a><p>报错的大概意思就是存在相同的view,在Flask中,如果定义了两个相同函数名的view,就会报这种错误,例如<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@web_api.route('test')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'test'</span></span><br><span class="line"></span><br><span class="line"><span class="meta">@web_api.route('test1')</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'test1'</span></span><br></pre></td></tr></table></figure><p></p><p>但是我们上例中的两个函数名分别为alert_delete和alert_index,是不相同的,为什么会报这种错误呢?</p><p>这是因为python的装饰器本质上就是函数的链式调用,比如下面的写法<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@funA</span></span><br><span class="line"><span class="meta">@funB</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">C</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">pass</span></span><br></pre></td></tr></table></figure><p></p><p>相当于执行<code>funA(funB(C))</code><br>那么<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@web_api.route("/alert/&lt;alert_id&gt;/delete")</span></span><br><span class="line"><span class="meta">@auth_util.check_auth_wrapper</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alert_delete</span><span class="params">(alert_id)</span>:</span></span><br><span class="line"> del_alert(alert_id)</span><br><span class="line"> <span class="keyword">return</span> <span class="string">'delete success'</span></span><br></pre></td></tr></table></figure><p></p><p>相当于执行<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">web_api.route(<span class="string">"/alert/&lt;alert_id&gt;/delete"</span>)(auth_util.check_auth_wrapper(alert_delete, alert_id))</span><br></pre></td></tr></table></figure><p></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@web_api.route('/alert/')</span></span><br><span class="line"><span class="meta">@auth_util.check_auth_wrapper</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alert_index</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> render_template(<span class="string">'alert_index.html'</span>)</span><br></pre></td></tr></table></figure><p>相当于执行<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">web_api.route(<span class="string">"/alert/"</span>)(auth_util.check_auth_wrapper(alert_index))</span><br></pre></td></tr></table></figure><p></p><p>对于web_api.route这个方法而言,传入的方法都是auth_util.check_auth_wrapper,而不是更深一层的alert_delete和alert_index.所以会报上面的错误.</p><h3 id="解决方法:"><a href="#解决方法:" class="headerlink" title="解决方法:"></a>解决方法:</h3><p>python自带的functools提供了解决方案<br><a href="https://docs.python.org/2/library/functools.html" rel="external nofollow" target="_blank">functools.update_wrapper(wrapper, wrapped[, assigned][, updated]</a></p><blockquote><p>Update a wrapper function to look like the wrapped function</p></blockquote><p>update_wrapper的功能就是让装饰器看起来像被装饰的方法,默认会将装饰方法的属性( <strong>name</strong>, <strong>module</strong> and <strong>doc</strong>)替换为被装饰方法的.</p><p>functools.wraps提供了update_wrapper的装饰器功能,</p><p>修改之后的check_auth_wrapper为<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_auth_wrapper</span><span class="params">(self, func)</span>:</span></span><br><span class="line"><span class="meta"> @wraps(func)</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line"> uid = checkuserislogin(request)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> uid:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request)</span><br><span class="line"> admin = self.check_auth(uid)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> admin:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request, url=<span class="string">'/routecheck'</span>)</span><br><span class="line"> <span class="keyword">return</span> func(*args, **kwargs)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure><p></p><p>添加@wraps(func)就可以正确的运行了.</p><p>为了验证之前内容的正确性,在check_auth_wrapper中加入一行<code>print wrapper.__name__</code></p><p>没有加入@wraps之前,报错版本<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_auth_wrapper</span><span class="params">(self, func)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line"> uid = checkuserislogin(request)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> uid:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request)</span><br><span class="line"> admin = self.check_auth(uid)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> admin:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request, url=<span class="string">'/routecheck'</span>)</span><br><span class="line"> <span class="keyword">return</span> func(*args, **kwargs)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">print</span> wrapper.__name__</span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure><p></p><p>输出为<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">wrapper</span><br><span class="line">wrapper</span><br></pre></td></tr></table></figure><p></p><p>两个不同的view经过装饰器后返回的都是相同的view function<br>加入@wraps之后,正确版本<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_auth_wrapper</span><span class="params">(self, func)</span>:</span></span><br><span class="line"><span class="meta"> @wraps(func)</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line"> uid = checkuserislogin(request)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> uid:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request)</span><br><span class="line"> admin = self.check_auth(uid)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> admin:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request, url=<span class="string">'/routecheck'</span>)</span><br><span class="line"> <span class="keyword">return</span> func(*args, **kwargs)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">print</span> wrapper.__name__</span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure><p></p><p>输出为<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">alert_delete</span><br><span class="line">alert_index</span><br></pre></td></tr></table></figure><p></p><p>返回的是被装饰对象的方法名.<br>Bingo~</p></content>
<summary type="html">
<p>在生产中有几个api需要用户认证才可以访问,最初的做法是在每个方法最前面写一段相同的代码来验证身份,这种通用的功能更适合提炼出来而不是在每个方法中都粘贴一份.</p><p>python支持的装饰器可以扮演拦截器的角色,将验证身份的代码放入到装饰器中,并为需要的方法添加装饰器是一个更好的做法.</p><p>首先定义一个装饰器auth_util.py<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_auth_wrapper</span><span class="params">(self, func)</span>:</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line"> uid = checkuserislogin(request)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> uid:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request)</span><br><span class="line"> admin = self.check_auth(uid)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> admin:</span><br><span class="line"> <span class="keyword">return</span> make_redirect(request, url=<span class="string">'/'</span>)</span><br><span class="line"> <span class="keyword">return</span> func(*args, **kwargs)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure><p></p><p>使用时<br></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@web_api.route("/alert/&lt;alert_id&gt;/delete")</span></span><br><span class="line"><span class="meta">@auth_util.check_auth_wrapper</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alert_delete</span><span class="params">(alert_id)</span>:</span></span><br><span class="line"> del_alert(alert_id)</span><br><span class="line"> <span class="keyword">return</span> <span class="string">'delete success'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@web_api.route('/alert/')</span></span><br><span class="line"><span class="meta">@auth_util.check_auth_wrapper</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">alert_index</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> render_template(<span class="string">'alert_index.html'</span>)</span><br></pre></td></tr></table></figure><p></p><p>启动程序时会报错<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">Traceback (most recent call last):</span><br><span class="line"> File &quot;/home/q/www/api/all_webui.py&quot;, line 22, in &lt;module&gt;</span><br><span class="line"> app.register_blueprint(routecheck_api)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 62, in wrapper_func</span><br><span class="line"> return f(self, *args, **kwargs)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 889, in register_blueprint</span><br><span class="line"> blueprint.register(self, options, first_registration)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/blueprints.py&quot;, line 153, in register</span><br><span class="line"> deferred(state)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/blueprints.py&quot;, line 172, in &lt;lambda&gt;</span><br><span class="line"> s.add_url_rule(rule, endpoint, view_func, **options))</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/blueprints.py&quot;, line 76, in add_url_rule</span><br><span class="line"> view_func, defaults=defaults, **options)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 62, in wrapper_func</span><br><span class="line"> return f(self, *args, **kwargs)</span><br><span class="line"> File &quot;/usr/local/lib/python2.7/dist-packages/Flask-0.10.1-py2.7.egg/flask/app.py&quot;, line 984, in add_url_rule</span><br><span class="line"> &apos;existing endpoint function: %s&apos; % endpoint)</span><br><span class="line">AssertionError: View function mapping is overwriting an existing endpoint function: /home/q/www/route_web/apis/routecheck/routecheck_webui.pyc.wrapper</span><br></pre></td></tr></table></figure><p></p>
</summary>
<category term="python" scheme="http://trytofix.com/categories/python/"/>
<category term="Flask" scheme="http://trytofix.com/tags/Flask/"/>
</entry>
<entry>
<title>PG语法汇总</title>
<link href="http://trytofix.com/2016/05/05/PG%E8%AF%AD%E6%B3%95%E6%B1%87%E6%80%BB/"/>
<id>http://trytofix.com/2016/05/05/PG语法汇总/</id>
<published>2016-05-05T06:43:08.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>假如test表中type字段的格式xxxx:yyyy:zzzz,需要取到以’:’为分隔符将type字段进行分割后的第一部分也就是xxxx的内容进行分组并排序<br></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">select</span> split_part(<span class="keyword">type</span>,<span class="string">':'</span>,<span class="number">1</span>) <span class="keyword">as</span> atype, <span class="keyword">count</span>(<span class="number">1</span>) <span class="keyword">as</span> <span class="keyword">count</span> <span class="keyword">from</span> <span class="keyword">test</span> <span class="keyword">group</span> <span class="keyword">by</span> atype <span class="keyword">order</span> <span class="keyword">by</span> <span class="keyword">count</span> <span class="keyword">desc</span></span><br></pre></td></tr></table></figure><p></p><p>替换type中的xxxx:yyyy:zzzz为xxxx,yyyy,zzzz<br></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">select</span> <span class="keyword">replace</span>(<span class="keyword">type</span>,<span class="string">':'</span>, <span class="string">','</span>) <span class="keyword">as</span> <span class="keyword">type</span> <span class="keyword">from</span> <span class="keyword">test</span>;</span><br></pre></td></tr></table></figure><p></p><p>依此计算返回式的值,遇到非null值就返回<br></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">select</span> <span class="keyword">coalesce</span>(<span class="literal">null</span>, <span class="keyword">type</span>) <span class="keyword">from</span> <span class="keyword">type</span> <span class="keyword">from</span> <span class="keyword">test</span>;</span><br></pre></td></tr></table></figure><p></p></content>
<summary type="html">
<p>假如test表中type字段的格式xxxx:yyyy:zzzz,需要取到以’:’为分隔符将type字段进行分割后的第一部分也就是xxxx的内容进行分组并排序<br></p><figure class="highlight sql"><table><tr><td class=
</summary>
<category term="数据库" scheme="http://trytofix.com/categories/%E6%95%B0%E6%8D%AE%E5%BA%93/"/>
<category term="PostgreSQL" scheme="http://trytofix.com/tags/PostgreSQL/"/>
</entry>
<entry>
<title>solr查询语法积累</title>
<link href="http://trytofix.com/2016/05/05/solr%E6%9F%A5%E8%AF%A2%E8%AF%AD%E6%B3%95%E7%A7%AF%E7%B4%AF/"/>
<id>http://trytofix.com/2016/05/05/solr查询语法积累/</id>
<published>2016-05-05T06:31:02.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><h3 id="查询某个字段包含某个关键字"><a href="#查询某个字段包含某个关键字" class="headerlink" title="查询某个字段包含某个关键字"></a>查询某个字段包含某个关键字</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">q=cities:北京</span><br></pre></td></tr></table></figure><h3 id="查询某个字段不包含某些关键字"><a href="#查询某个字段不包含某些关键字" class="headerlink" title="查询某个字段不包含某些关键字"></a>查询某个字段不包含某些关键字</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">q=cities:* NOT 上海 NOT 北京 NOT 黔南 NOT 三亚</span><br></pre></td></tr></table></figure><h3 id="分组查询"><a href="#分组查询" class="headerlink" title="分组查询"></a>分组查询</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">http://localhost/searchapi/search_engine/select?q=name:brian*&amp;fl=id,name&amp;rows=5000&amp;wt=json&amp;indent=true&amp;group=true&amp;group.field=groupid&amp;group.offset=5&amp;group.limit=10&amp;group.ngroups=true&amp;group.sort=feed_time%20desc</span><br></pre></td></tr></table></figure></content>
<summary type="html">
<h3 id="查询某个字段包含某个关键字"><a href="#查询某个字段包含某个关键字" class="headerlink" title="查询某个字段包含某个关键字"></a>查询某个字段包含某个关键字</h3><figure class="highlight plai
</summary>
<category term="Solr" scheme="http://trytofix.com/categories/Solr/"/>
<category term="搜索引擎" scheme="http://trytofix.com/tags/%E6%90%9C%E7%B4%A2%E5%BC%95%E6%93%8E/"/>
</entry>
<entry>
<title>xclip操作剪切板</title>
<link href="http://trytofix.com/2016/05/05/xclip%E6%93%8D%E4%BD%9C%E5%89%AA%E5%88%87%E6%9D%BF/"/>
<id>http://trytofix.com/2016/05/05/xclip操作剪切板/</id>
<published>2016-05-05T05:53:25.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><blockquote><p>Reads from standard in, or from one or more files, and makes the data<br>available as an X selection for pasting into X applications. Prints<br>current X selection to standard out.</p></blockquote><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-get install xclip</span><br></pre></td></tr></table></figure><p>使用xclip可以从标准输入,文件读取内容存入到剪切板中,也可以将剪切板中的内容输出到标准输出中.</p><p>例子:</p><ul><li><p>将当前系统运行状态复制到剪切板(默认是xlip的剪切板,不是系统剪切板,所以使用ctrl+v无效</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">uptime | xclip</span><br></pre></td></tr></table></figure></li><li><p>输出剪切板中的内容</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">xclip -o</span><br></pre></td></tr></table></figure></li><li><p>将当前系统运行状态复制到系统剪切板中</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">uptime | xclip -selection clipboard</span><br><span class="line">uptime | xclip -selection clip</span><br></pre></td></tr></table></figure></li><li><p>从系统剪切板中获取内容ctrl+v或者</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">xclip -o -selection clipboard</span><br><span class="line">xclip -o -selection clip</span><br></pre></td></tr></table></figure></li><li><p>将文件复制到剪切板</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">xclip /etc/profile</span><br></pre></td></tr></table></figure></li><li><p>将剪切板中的图片保存到文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">xclip -selection clip -t image/png -o &gt; /tmp/snapshot.png</span><br></pre></td></tr></table></figure></li><li><p>指定文件只允许被复制十次</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">xclip -l 10 -verbose -selection clip /etc/profile</span><br></pre></td></tr></table></figure></li></ul></content>
<summary type="html">
<blockquote><p>Reads from standard in, or from one or more files, and makes the data<br>available as an X selection for pasting into X appli
</summary>
<category term="Linux" scheme="http://trytofix.com/categories/Linux/"/>
<category term="Command Line" scheme="http://trytofix.com/tags/Command-Line/"/>
</entry>
<entry>
<title>Linux Gnome下录制Gif动图</title>
<link href="http://trytofix.com/2016/05/05/Linux-Gnome%E4%B8%8B%E5%BD%95%E5%88%B6Gif%E5%8A%A8%E5%9B%BE/"/>
<id>http://trytofix.com/2016/05/05/Linux-Gnome下录制Gif动图/</id>
<published>2016-05-05T04:02:07.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-get install byzanz</span><br></pre></td></tr></table></figure><p>使用</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">#录制全屏,时间为35秒,保存到byzanz-demo.gif中</span><br><span class="line">byzanz-record -d 35 -x 0 -y 0 byzanz-demo.gif</span><br></pre></td></tr></table></figure><ul><li>-d 录制时间</li><li>-x 左上角x轴坐标</li><li>-y 左上角y轴坐标</li><li>-w 录制区域的宽度</li><li>-h 录制区域的高度</li><li>-c 记录鼠标指针</li></ul><p>更多用法参照:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">man byzanz-record</span><br></pre></td></tr></table></figure><p></p></content>
<summary type="html">
<p>安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre>
</summary>
<category term="Linux" scheme="http://trytofix.com/categories/Linux/"/>
<category term="Command Line" scheme="http://trytofix.com/tags/Command-Line/"/>
</entry>
<entry>
<title>gnome-screenshot ubuntu gnome截图</title>
<link href="http://trytofix.com/2016/05/05/gnome-screenshot-ubuntu-gnome%E6%88%AA%E5%9B%BE/"/>
<id>http://trytofix.com/2016/05/05/gnome-screenshot-ubuntu-gnome截图/</id>
<published>2016-05-05T03:37:10.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>首先看下官方说明</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">gnome-screenshot is a GNOME utility for taking screenshots of the</span><br><span class="line">entire screen, a window or an user-defined area of the screen, with</span><br><span class="line">optional beautifying border effects</span><br></pre></td></tr></table></figure><p>在gnome桌面的应用程序-&gt;附件-&gt;截图使用的就是gnome-screenshot.</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">gnome-screenshot -a #截取一个区域</span><br><span class="line">gnome-screenshot -a -c #截取一个区域并保存到剪切板</span><br><span class="line">gnome-screenshot -w -c #截取当前活动窗口并保存到剪切板</span><br><span class="line">gnome-screenshot -w -p #截取当前活动窗口并保存鼠标指针</span><br><span class="line">gnome-screenshot -i #弹出交互式的窗体</span><br><span class="line">gnome-screenshot -a -f /tmp/test.png #截取一个区域保存到/tmp/test.png中.</span><br></pre></td></tr></table></figure><p>-c 与 -f 同时使用时只会保存到剪切板中(ubuntu14.04亲测)</p><p>更多用法参见</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">man gnome-screenshot</span><br></pre></td></tr></table></figure></content>
<summary type="html">
<p>首先看下官方说明</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span>
</summary>
<category term="Linux" scheme="http://trytofix.com/categories/Linux/"/>
<category term="Command Line" scheme="http://trytofix.com/tags/Command-Line/"/>
</entry>
<entry>
<title>markdown使用截图以及图床优化写作体验</title>
<link href="http://trytofix.com/2016/05/04/markdown%E4%BD%BF%E7%94%A8%E6%88%AA%E5%9B%BE%E4%BB%A5%E5%8F%8A%E5%9B%BE%E5%BA%8A%E4%BC%98%E5%8C%96%E5%86%99%E4%BD%9C%E4%BD%93%E9%AA%8C/"/>
<id>http://trytofix.com/2016/05/04/markdown使用截图以及图床优化写作体验/</id>
<published>2016-05-04T09:39:07.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>使用方法,截图+粘贴</p><p>在之前,使用markdown插入截图时,通常需要:</p><ul><li>使用截图工具,截图并保存</li><li>将截图保存到工作目录</li><li>使用markdown语法引入图片<br>如果需要图床,需要将图片上传到图床,再将图床的url引入markdown</li></ul><p>使用优化后的方法仅仅需要:</p><ul><li>截图</li><li>随意粘贴图床地址/图片本地路径<a id="more"></a> 看一下优化后使用markdown写作插入截图时的体验:</li></ul><p><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3jj0xb2i8g211y0lc4qs" alt="snapshot"></p><p>在这个例子中,我们是将截图放到了图床中,并返回了图床url到剪切板,经过了一下几个过程:</p><ul><li>调用gnome-screenshot -a -c,使用ubuntu自带的截图软件截图并将结果复制到剪切板</li><li>调用xclip -selection clipboard -t image/png -o &gt; /tmp/snapshot.png将剪切板中的内容重定向到文件</li><li>调用之前微博图床的工具,传入截图路径,获取截图在新浪图床中的url.</li><li>将图床的url复制到系统的剪切板中,同样是使用xclip -selection clipboard</li></ul><p>将以上内容保存到一个脚本中,snapshot.sh内容如下:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">#!/bin/bash</span><br><span class="line"></span><br><span class="line">gnome-screenshot -a -c</span><br><span class="line">xclip -selection clipboard -t image/png -o &gt; /tmp/snapshot.png</span><br><span class="line">python /home/q/hexo/blog/weibo_util.py -f /tmp/snapshot.png -u your_weibo_username -p your_weibo_password|xclip -selection clipboard</span><br></pre></td></tr></table></figure><p>其中的weibo_util.py,链接为<a href="https://github.com/trytofix/hexo_weibo_image" rel="external nofollow" target="_blank">weibo_util</a></p><p>gnome-screenshot这个工具是ubuntu自带的截图工具,默认快捷键是ctrl+alt+a,如果没有这个工具,可以根据自己的系统将shell中的gnome-screenshot -a -c这句修改为自己使用的截图软件即可,注意的是一定要将截图保存到剪切板中才可以.</p><p>xclip将终端与剪切板连接到了一起,可以在终端可剪切板之间复制粘贴.<br>安装方法:<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-get install xclip</span><br></pre></td></tr></table></figure><p></p><p>使用方法<br></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">man xclip</span><br></pre></td></tr></table></figure><p></p><p>weibo_util.py完成了上传图片到微博图床并返回图床链接,具体的原理,请参照之前的<a href="/2016/04/06/使用新浪微博图床/">博文</a></p><p>要好shell最后的一部就是简历系统快捷键与脚本之间的关联,在ubuntu,在设置中找到键盘设置,如下图:<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3kc8vb8cpj20nn0hudij" alt="设置"><br>在键盘设置中选择快捷键,在自定义快捷键中点击+,新增一项,<br><img src="http://ww3.sinaimg.cn/large/b8b708a7gw1f3kcac5lctj207x04i74m" alt="snapshot"><br>名称就是一个别名,这里起名为snapshot<br>命令就是shell的全路径,这里是:<code>/home/q/hexo/blog/snapshot.sh</code><br>应用后,点击snapshot的后半部分可以自定义快捷键,快捷键是可以自己定义的,我选择了ctrl+alt+w,之后注销账户,使快捷键生效,接下来,试试ctrl+alt+w,会弹出截屏的图标,截取后,找个地方粘贴看看效果吧~.</p><blockquote><p>ps: 这个教程是将截取的图片放到了图床中,返回的是图床地址,将snapshot.sh稍微修改下,就可以将截图放到本地路径中,返回本地的文件路径.</p></blockquote></content>
<summary type="html">
<p>使用方法,截图+粘贴</p><p>在之前,使用markdown插入截图时,通常需要:</p><ul><li>使用截图工具,截图并保存</li><li>将截图保存到工作目录</li><li>使用markdown语法引入图片<br>如果需要图床,需要将图片上传到图床,再将图床的url引入markdown</li></ul><p>使用优化后的方法仅仅需要:</p><ul><li>截图</li><li>随意粘贴图床地址/图片本地路径
</summary>
<category term="Linux" scheme="http://trytofix.com/categories/Linux/"/>
<category term="Command Line" scheme="http://trytofix.com/tags/Command-Line/"/>
<category term="markdown" scheme="http://trytofix.com/tags/markdown/"/>
<category term="hexo" scheme="http://trytofix.com/tags/hexo/"/>
</entry>
<entry>
<title>python lambda</title>
<link href="http://trytofix.com/2016/04/20/python-lambda/"/>
<id>http://trytofix.com/2016/04/20/python-lambda/</id>
<published>2016-04-20T09:44:50.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>lambda表达式用来简单的定义函数</p><p>例如</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">add = <span class="keyword">lambda</span> x,y: x+y</span><br><span class="line"><span class="keyword">print</span> add(<span class="number">1</span>,<span class="number">2</span>)</span><br></pre></td></tr></table></figure><p>输出<code>3</code></p><p>lambda表达式将<code>:</code>后的运算结果作为函数返回值</p><a id="more"></a><p>看下面的例子,理解下</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test1</span><span class="params">(x)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> x ** <span class="number">2</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test2</span><span class="params">(x)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> x ** <span class="number">2</span></span><br><span class="line"> <span class="keyword">return</span> x ** <span class="number">2</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">lambda1 = <span class="keyword">lambda</span> x: test1(x)</span><br><span class="line">lambda2 = <span class="keyword">lambda</span> x: test2(x)</span><br><span class="line">lambda3 = <span class="keyword">lambda</span> x: x ** <span class="number">2</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'lambda1'</span></span><br><span class="line"><span class="keyword">print</span> lambda1(<span class="number">10</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'lambda2'</span></span><br><span class="line"><span class="keyword">print</span> lambda2(<span class="number">10</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">print</span> <span class="string">'lambda3'</span></span><br><span class="line"><span class="keyword">print</span> lambda3(<span class="number">10</span>)</span><br></pre></td></tr></table></figure><p>输出</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">lambda1</span><br><span class="line">100</span><br><span class="line">None</span><br><span class="line">lambda2</span><br><span class="line">100</span><br><span class="line">100</span><br><span class="line">lambda3</span><br><span class="line">100</span><br></pre></td></tr></table></figure></content>
<summary type="html">
<p>lambda表达式用来简单的定义函数</p><p>例如</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">add = <span class="keyword">lambda</span> x,y: x+y</span><br><span class="line"><span class="keyword">print</span> add(<span class="number">1</span>,<span class="number">2</span>)</span><br></pre></td></tr></table></figure><p>输出<code>3</code></p><p>lambda表达式将<code>:</code>后的运算结果作为函数返回值</p>
</summary>
<category term="python" scheme="http://trytofix.com/categories/python/"/>
<category term="lambda" scheme="http://trytofix.com/tags/lambda/"/>
</entry>
<entry>
<title>2016/04/20随手记</title>
<link href="http://trytofix.com/2016/04/20/2016-04-20%E9%9A%8F%E6%89%8B%E8%AE%B0/"/>
<id>http://trytofix.com/2016/04/20/2016-04-20随手记/</id>
<published>2016-04-20T09:24:56.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><h2 id="Redis-SET命令"><a href="#Redis-SET命令" class="headerlink" title="Redis SET命令"></a>Redis SET命令</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">redis 127.0.0.1:6379&gt; SET KEY_NAME VALUE [EX seconds]</span><br></pre></td></tr></table></figure><p>示例:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">redis 127.0.0.1:6379&gt; SET url trytofix.com EX 60</span><br></pre></td></tr></table></figure><h2 id="solr-group中的group-limit-group-offset-start-rows"><a href="#solr-group中的group-limit-group-offset-start-rows" class="headerlink" title="solr group中的group.limit,group.offset,start,rows"></a>solr group中的group.limit,group.offset,start,rows</h2><ul><li>group.limit: The number of results (documents) to return for each group. Defaults to 1.</li><li>group.offset: The offset into the document list of each group.</li><li>start: The offset into the list of groups.</li><li>rows: The number of groups to return. Defaults to 10.</li></ul><p>group的limit,offset相当于未分组查询的rows和start,用来定义数量以及偏移量.</p><h2 id="solr-ngroups"><a href="#solr-ngroups" class="headerlink" title="solr ngroups"></a>solr ngroups</h2><p>if true, includes the number of groups that have matched the query. Default is false</p><p>例如下面的查询语句:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">http://127.0.0.1/solr_engine/select?q=name:test*&amp;fl=feed_time&amp;rows=50&amp;wt=json&amp;indent=true&amp;group=true&amp;group.field=name&amp;group.offset=0&amp;group.limit=5&amp;group.ngroups=true&amp;group.sort=feed_time%20desc</span><br></pre></td></tr></table></figure><p>返回的字段中xml中:</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line">grouped: &#123;</span><br><span class="line"> name: &#123;</span><br><span class="line"> matches: 1016709,</span><br><span class="line"> ngroups: 2477,</span><br><span class="line"> groups: [</span><br><span class="line"> &#123;</span><br><span class="line">...</span><br></pre></td></tr></table></figure><p>ngroups:2477就是分组的个数.</p></content>
<summary type="html">
<h2 id="Redis-SET命令"><a href="#Redis-SET命令" class="headerlink" title="Redis SET命令"></a>Redis SET命令</h2><figure class="highlight plain"><tabl
</summary>
<category term="随手记" scheme="http://trytofix.com/categories/%E9%9A%8F%E6%89%8B%E8%AE%B0/"/>
<category term="随手记" scheme="http://trytofix.com/tags/%E9%9A%8F%E6%89%8B%E8%AE%B0/"/>
</entry>
<entry>
<title>PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed</title>
<link href="http://trytofix.com/2016/04/20/PicklingError-Can-t-pickle-type-function-attribute-lookup-builtin-function-failed/"/>
<id>http://trytofix.com/2016/04/20/PicklingError-Can-t-pickle-type-function-attribute-lookup-builtin-function-failed/</id>
<published>2016-04-20T08:37:18.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>在使用python multiprocessing的进程池时,遇到了如下问题,首先看示例代码:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Pool</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fetch_url</span><span class="params">(url)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> url</span><br><span class="line"> <span class="keyword">print</span> <span class="string">'fetching: &#123;&#125;'</span>.format(url)</span><br><span class="line"> <span class="keyword">return</span> url</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">urls = [<span class="string">'a'</span>, <span class="string">'b'</span>, <span class="string">'c'</span>]</span><br><span class="line">results = []</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> pool = Pool(processes=<span class="number">4</span>)</span><br><span class="line"> fetch_url_proxy = <span class="keyword">lambda</span> x: fetch_url(x)</span><br><span class="line"> <span class="keyword">for</span> url <span class="keyword">in</span> urls:</span><br><span class="line"> results.append(pool.apply_async(fetch_url_proxy, (url,)))</span><br><span class="line"> pool.close()</span><br><span class="line"> pool.join()</span><br><span class="line"> <span class="keyword">for</span> result <span class="keyword">in</span> results:</span><br><span class="line"> <span class="keyword">print</span> result.get()</span><br></pre></td></tr></table></figure><a id="more"></a><p>程序报错如下:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">Traceback (most recent call last):</span><br><span class="line"> File &quot;/home/q/title/test.py&quot;, line 23, in &lt;module&gt;</span><br><span class="line"> print result.get()</span><br><span class="line"> File &quot;/usr/lib/python2.7/multiprocessing/pool.py&quot;, line 558, in get</span><br><span class="line"> raise self._value</span><br><span class="line">cPickle.PicklingError: Can&apos;t pickle &lt;type &apos;function&apos;&gt;: attribute lookup __builtin__.function failed</span><br></pre></td></tr></table></figure><p>原因在于lambda函数并不能被pickle数据持久化.</p><p>正确的方法是将方法放在线程池执行之前并定义为全局可访问的函数.例如将上例改为</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Pool</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fetch_url</span><span class="params">(url)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> url</span><br><span class="line"> <span class="keyword">print</span> <span class="string">'fetching: &#123;&#125;'</span>.format(url)</span><br><span class="line"> <span class="keyword">return</span> url</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fetch_url_proxy</span><span class="params">(url)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> fetch_url(url)</span><br><span class="line"></span><br><span class="line">urls = [<span class="string">'a'</span>, <span class="string">'b'</span>, <span class="string">'c'</span>]</span><br><span class="line">results = []</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> pool = Pool(processes=<span class="number">4</span>)</span><br><span class="line"> <span class="keyword">for</span> url <span class="keyword">in</span> urls:</span><br><span class="line"> results.append(pool.apply_async(fetch_url_proxy, (url,)))</span><br><span class="line"> pool.close()</span><br><span class="line"> pool.join()</span><br><span class="line"> <span class="keyword">for</span> result <span class="keyword">in</span> results:</span><br><span class="line"> <span class="keyword">print</span> result.get()</span><br></pre></td></tr></table></figure><p><a href="https://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled" rel="external nofollow" target="_blank">可以被pickle数据持久化的类型</a></p><blockquote><p>The following types can be pickled:<br>None, True, and False<br>integers, long integers, floating point numbers, complex numbers<br>normal and Unicode strings<br>tuples, lists, sets, and dictionaries containing only picklable objects<br>functions defined at the top level of a module<br>built-in functions defined at the top level of a module<br>classes that are defined at the top level of a module<br>instances of such classes whose <strong>dict</strong> or the result of calling <strong>getstate</strong>() is picklable (see section The pickle protocol for details).<br>```</p></blockquote><p>pickle不能持久化的类型包括但不限于lambda函数,内嵌函数等.</p></content>
<summary type="html">
<p>在使用python multiprocessing的进程池时,遇到了如下问题,首先看示例代码:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Pool</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fetch_url</span><span class="params">(url)</span>:</span></span><br><span class="line"> <span class="keyword">print</span> url</span><br><span class="line"> <span class="keyword">print</span> <span class="string">'fetching: &#123;&#125;'</span>.format(url)</span><br><span class="line"> <span class="keyword">return</span> url</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">urls = [<span class="string">'a'</span>, <span class="string">'b'</span>, <span class="string">'c'</span>]</span><br><span class="line">results = []</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> pool = Pool(processes=<span class="number">4</span>)</span><br><span class="line"> fetch_url_proxy = <span class="keyword">lambda</span> x: fetch_url(x)</span><br><span class="line"> <span class="keyword">for</span> url <span class="keyword">in</span> urls:</span><br><span class="line"> results.append(pool.apply_async(fetch_url_proxy, (url,)))</span><br><span class="line"> pool.close()</span><br><span class="line"> pool.join()</span><br><span class="line"> <span class="keyword">for</span> result <span class="keyword">in</span> results:</span><br><span class="line"> <span class="keyword">print</span> result.get()</span><br></pre></td></tr></table></figure>
</summary>
<category term="python" scheme="http://trytofix.com/categories/python/"/>
<category term="多进程" scheme="http://trytofix.com/tags/%E5%A4%9A%E8%BF%9B%E7%A8%8B/"/>
</entry>
<entry>
<title>python multiprocessing</title>
<link href="http://trytofix.com/2016/04/20/python-multiprocessing/"/>
<id>http://trytofix.com/2016/04/20/python-multiprocessing/</id>
<published>2016-04-20T07:32:16.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><blockquote><p>multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.</p><p>The multiprocessing module also introduces APIs which do not have analogs in the threading module. A prime example of this is the Pool object which offers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism). The following example demonstrates the common practice of defining such functions in a module so that child processes can successfully import that module. This basic example of data parallelism using Pool,</p></blockquote><a id="more"></a><blockquote><p>来自官方介绍<a href="https://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool" rel="external nofollow" target="_blank">链接</a></p></blockquote><p>python支持使用threading模块进行多线程编程,但是python使用GIL全局解释器锁保证在同一时刻只有一个线程在运行.只有获得GIL的线程才可以使用python解释器执行代码.</p><p>GIL在保证线程安全的同时,牺牲了效率,在多核处理器普及的时代,GIL使得多线程不能很好的利用多核处理器处理数据.multiprocessing模块的出现,方便了使用python进行多进程编程.多进程既可以有效的利用多核处理器,也可以避免线程安全带来的问题.</p><p>使用线程还是进程,取决于你的程序要处理的业务:</p><ul><li>CPU密集型 : CPU运算时间 &gt;&gt; IO处理时间<br>例如: 图片处理,数值计算<br>使用多进程,充分的利用多核CPU的计算能力.</li><li>IO密集型 : IO处理时间 &gt;&gt; CPU运算时间<br>例如: 爬虫抓取,读写数据库<br>使用多线程,某个线程IO阻塞时会释放GIL,其他线程可以获取GIL.</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> multiprocessing</span><br><span class="line"></span><br><span class="line">pool = multiprocessing.Pool(processes=int(multiprocessing.cpu_count()*<span class="number">0.5</span>))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check</span><span class="params">(param1=None, param2=None, param3=None)</span>:</span></span><br><span class="line"> <span class="comment"># 复杂运算</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> item <span class="keyword">in</span> items:</span><br><span class="line"> pool.apply_async(check, (<span class="string">'param1'</span>, <span class="string">'param2'</span>, <span class="string">'param3'</span>))</span><br><span class="line"></span><br><span class="line">pool.close()</span><br><span class="line">pool.join()</span><br></pre></td></tr></table></figure><p>使用multiprocessing.Pool可以创建一个进程池,方便使用多进程编程.<br>注意先<code>pool.clese()</code> 再<code>pool.join()</code>等待进程都执行完成.<br>因为<code>pool.join()</code>方法体为:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">join</span><span class="params">(self)</span>:</span></span><br><span class="line"> debug(<span class="string">'joining pool'</span>)</span><br><span class="line"> <span class="keyword">assert</span> self._state <span class="keyword">in</span> (CLOSE, TERMINATE)</span><br><span class="line"> self._worker_handler.join()</span><br><span class="line"> self._task_handler.join()</span><br><span class="line"> self._result_handler.join()</span><br><span class="line"> <span class="keyword">for</span> p <span class="keyword">in</span> self._pool:</span><br><span class="line"> p.join()</span><br></pre></td></tr></table></figure><p>在join前会判断线程池是否处于CLOSE(关闭)或者TERMINATE(终止)状态.</p><p>生产中的对大概100w条线路进行质量检测,使用单进程凌晨跑脚本,由于检测逻辑越来越复杂,导致检测时间一度达到26个小时,使用多进程,进程池大小为服务器核数的一半(16核),将脚本检测时间降到了3个半小时.</p><p>多进程与多线程带来的收益曲线都是相似的,随着进程池的核数逐渐增大,带来的收益越来越小,当超过某一阈值时,甚至会降低收益.因此,要根据自己的程序调节多次后找到最优的进程池大小.</p><p>python使用threading进行多线程编程,如果熟悉了multiprocessing的api,可以使用multiprocessing.dummy来进行多线程编程.</p><blockquote><p>Support for the API of the multiprocessing package using threads</p></blockquote><p>以线程池为例(在threading模块中是没有的哦~)</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing.dummy <span class="keyword">import</span> Pool <span class="keyword">as</span> ThreadPool</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">request_api</span><span class="params">(url)</span>:</span></span><br><span class="line"> <span class="comment">#请求api</span></span><br><span class="line"></span><br><span class="line">urls = [<span class="string">'trytofix.com'</span>]</span><br><span class="line"></span><br><span class="line">pool = ThreadPool(processes=<span class="number">8</span>)</span><br><span class="line"><span class="keyword">for</span> url <span class="keyword">in</span> urls:</span><br><span class="line"> pool.apply_async(request_api, (url, ))</span><br><span class="line">pool.close()</span><br><span class="line">pool.join()</span><br></pre></td></tr></table></figure></content>
<summary type="html">
<blockquote><p>multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.</p><p>The multiprocessing module also introduces APIs which do not have analogs in the threading module. A prime example of this is the Pool object which offers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism). The following example demonstrates the common practice of defining such functions in a module so that child processes can successfully import that module. This basic example of data parallelism using Pool,</p></blockquote>
</summary>
<category term="python" scheme="http://trytofix.com/categories/python/"/>
<category term="并行" scheme="http://trytofix.com/tags/%E5%B9%B6%E8%A1%8C/"/>
</entry>
<entry>
<title>linux top</title>
<link href="http://trytofix.com/2016/04/19/linux-top/"/>
<id>http://trytofix.com/2016/04/19/linux-top/</id>
<published>2016-04-19T07:28:47.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>top用来实时的查看系统的状态.</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line">brianyang@brianyang-Latitude-E5440:~$ top</span><br><span class="line"></span><br><span class="line">&lt;--&gt; uptime(uptime and load average)</span><br><span class="line">top - 15:29:58 up 4:49, 10 users, load average: 0.25, 0.22, 0.22 </span><br><span class="line">&lt;--&gt; task</span><br><span class="line">Tasks: 311 total, 1 running, 310 sleeping, 0 stopped, 0 zombie </span><br><span class="line">&lt;--&gt; cpu states</span><br><span class="line">%Cpu(s): 2.9 us, 0.6 sy, 0.0 ni, 96.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st </span><br><span class="line">&lt;--&gt; memory usage</span><br><span class="line">KiB Mem: 8080416 total, 4485820 used, 3594596 free, 73024 buffers </span><br><span class="line">KiB Swap: 8290300 total, 1276 used, 8289024 free. 856840 cached Mem</span><br><span class="line"></span><br><span class="line">&lt;--&gt; ps -aux</span><br><span class="line"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </span><br><span class="line"> 6411 brianya+ 20 0 617396 27444 8740 S 4.3 0.3 3:04.48 gnome-term+ </span><br><span class="line"> 1405 root 20 0 490464 125592 105616 S 2.7 1.6 4:38.17 Xorg </span><br><span class="line"> 6987 brianya+ 20 0 805516 196004 9828 S 2.7 2.4 6:27.70 chrome </span><br><span class="line"> 7359 brianya+ 20 0 4203420 595400 14868 S 2.3 7.4 11:35.58 java </span><br><span class="line"> 7134 root 20 0 3129444 182916 30948 S 2.0 2.3 5:03.40 LocalMan </span><br><span class="line"> 2802 rabbitmq 20 0 2175132 54292 1376 S 1.7 0.7 3:33.01 beam.smp </span><br><span class="line"> 7 root 20 0 0 0 0 S 0.3 0.0 0:47.32 rcu_sched </span><br><span class="line"> 11 root 20 0 0 0 0 S 0.3 0.0 0:27.32 rcuos/3 </span><br><span class="line"> 1127 mysql 20 0 558828 51040 376 S 0.3 0.6 0:11.98 mysqld </span><br><span class="line"> 2130 root 20 0 623892 14408 3328 S 0.3 0.2 0:28.18 daomonit </span><br><span class="line"> 4755 brianya+ 20 0 20228 400 224 S 0.3 0.0 0:07.80 syndaemon </span><br><span class="line"> 5038 brianya+ 20 0 6088 412 164 S 0.3 0.0 0:03.58 xflux </span><br><span class="line"> 5181 brianya+ 20 0 2269936 326424 39096 S 0.3 4.0 10:46.65 chrome </span><br><span class="line"> 5475 brianya+ 20 0 803700 58916 10488 S 0.3 0.7 0:05.16 chrome </span><br><span class="line"> 6738 brianya+ 20 0 873444 102692 34920 S 0.3 1.3 1:34.48 chrome </span><br><span class="line">13492 root 20 0 0 0 0 S 0.3 0.0 0:03.62 kworker/3:2 </span><br><span class="line">15487 brianya+ 20 0 933060 181636 46216 S 0.3 2.2 0:44.92 chrome </span><br><span class="line"></span><br><span class="line">%CPU -&gt; CPU使用率</span><br><span class="line">%MEM -&gt; 内存使用率</span><br><span class="line">COMMAND -&gt; 命令名或命令行</span><br><span class="line">NI -&gt; 任务优先级(负数&gt;0&gt;正数)(用户层面)</span><br><span class="line">PID -&gt; 进程id</span><br><span class="line">PR -&gt; 进程实际优先级 一般情况,PR=NI+20(内核层面)</span><br><span class="line">RES -&gt; 物理内存使用量(kb)</span><br><span class="line">S -&gt; 进程状态(R -&gt; Running, S -&gt; sleeping, Z -&gt; zombie ...)</span><br></pre></td></tr></table></figure><p>快捷键:</p><ul><li>shift+p: 按照cpu使用率排序</li><li>shift+m: 按照内存使用率排序</li><li>shift+t: 按照cpu使用时间排序</li><li>c : 显示command详情</li><li>z 或 -b : 将状态为running的进程突出显示</li><li>k : kill pid 杀死进程</li><li>s : 设置刷新时间</li></ul></content>
<summary type="html">
<p>top用来实时的查看系统的状态.</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">
</summary>
<category term="Linux" scheme="http://trytofix.com/categories/Linux/"/>
<category term="Linux Command" scheme="http://trytofix.com/tags/Linux-Command/"/>
</entry>
<entry>
<title>随手记</title>
<link href="http://trytofix.com/2016/04/13/%E9%9A%8F%E6%89%8B%E8%AE%B0/"/>
<id>http://trytofix.com/2016/04/13/随手记/</id>
<published>2016-04-13T05:16:43.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><h1 id="启动docker图片ocr服务"><a href="#启动docker图片ocr服务" class="headerlink" title="启动docker图片ocr服务"></a>启动docker图片ocr服务</h1><p>sudo docker run –env TESSDATA_PREFIX=/tseract-ocr -it -p 23456:23456 d009 python image_upload_web_tool/ocr_image_web.py</p><h1 id="docker设置环境变量"><a href="#docker设置环境变量" class="headerlink" title="docker设置环境变量"></a>docker设置环境变量</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">--env &lt;key&gt;=&lt;value&gt;</span><br></pre></td></tr></table></figure><h1 id="linux查看cpu核数"><a href="#linux查看cpu核数" class="headerlink" title="linux查看cpu核数"></a>linux查看cpu核数</h1><p>cat /proc/cpuinfo |grep processor|wc|awk ‘{print $1}’</p><h1 id="load-average"><a href="#load-average" class="headerlink" title="load average"></a>load average</h1><p>load average: 0.75, 1.56, 2.76<br>分别对应的时间是:1分钟,5分钟,15分钟.</p><h1 id="python-datetime-格式化时间"><a href="#python-datetime-格式化时间" class="headerlink" title="python datetime 格式化时间"></a>python datetime 格式化时间</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">In [<span class="number">18</span>]: datetime.now().strftime(<span class="string">'%Y_%m_%d %H:%m:%S'</span>)</span><br><span class="line">Out[<span class="number">18</span>]: <span class="string">'2016_04_13 17:04:12'</span></span><br></pre></td></tr></table></figure><h1 id="ssh登陆问题"><a href="#ssh登陆问题" class="headerlink" title="ssh登陆问题"></a>ssh登陆问题</h1><ul><li>ssh_exchange_identification: Connection closed by remote host</li></ul><p>一般可能是登陆机器超过限制,比如同一机房登陆机器超过x台.</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">rm -f ~/.ssh/persist/*</span><br></pre></td></tr></table></figure><h1 id="python-base64编码-解码"><a href="#python-base64编码-解码" class="headerlink" title="python base64编码/解码"></a>python base64编码/解码</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> base64</span><br><span class="line">s = <span class="string">'测试一下,哇哈哈,哇哈哈,哇哈哈,不够长,再长一点.'</span></span><br><span class="line"><span class="keyword">print</span> base64.encodestring(s)</span><br><span class="line"><span class="number">5</span>rWL6K+V5LiA5LiL77yM5ZOH5ZOI5ZOI77yM5ZOH5ZOI5ZOI77yM5ZOH5ZOI5ZOI77yM5LiN5aSf</span><br><span class="line"><span class="number">6</span>ZW/<span class="number">77</span>yM5YaN6ZW/<span class="number">5L</span>iA54K577yO</span><br><span class="line"><span class="keyword">print</span> base64.standard_b64encode(s)</span><br><span class="line"><span class="number">5</span>rWL6K+V5LiA5LiL77yM5ZOH5ZOI5ZOI77yM5ZOH5ZOI5ZOI77yM5ZOH5ZOI5ZOI77yM5LiN5aSf6ZW/<span class="number">77</span>yM5YaN6ZW/<span class="number">5L</span>iA54K577yO</span><br></pre></td></tr></table></figure><p>base64 encodestring和standard_b64encode都可以使用base64编码,其中前者将base64编码后的内容使用\n进行换行分隔,而后者返回的是一个无换行的字符串.</p><p>base64.encode/decode可以对文件进行编码/解码.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">base64.encode(open(<span class="string">'hello.txt'</span>, <span class="string">'r'</span>), open(<span class="string">'base64_hello.txt'</span> ,<span class="string">'w'</span>))</span><br></pre></td></tr></table></figure><p>其中</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">brianyang@brianyang-Latitude-E5440:~/Documents$ cat hello.txt</span><br><span class="line">hello brian</span><br><span class="line">brianyang@brianyang-Latitude-E5440:~/Documents$ cat base64_hello.txt </span><br><span class="line">aGVsbG8gYnJpYW4K</span><br></pre></td></tr></table></figure><p>编码后的文件存入了<code>base64_hello.txt</code>中</p><h1 id="python获取当前进程id"><a href="#python获取当前进程id" class="headerlink" title="python获取当前进程id"></a>python获取当前进程id</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">print</span> os.getpid()</span><br></pre></td></tr></table></figure></content>
<summary type="html">
<h1 id="启动docker图片ocr服务"><a href="#启动docker图片ocr服务" class="headerlink" title="启动docker图片ocr服务"></a>启动docker图片ocr服务</h1><p>sudo docker run –e
</summary>
<category term="随手记" scheme="http://trytofix.com/categories/%E9%9A%8F%E6%89%8B%E8%AE%B0/"/>
</entry>
<entry>
<title>Fabric简化Python发布流程</title>
<link href="http://trytofix.com/2016/04/11/Fabric%E7%AE%80%E5%8C%96Python%E5%8F%91%E5%B8%83%E6%B5%81%E7%A8%8B/"/>
<id>http://trytofix.com/2016/04/11/Fabric简化Python发布流程/</id>
<published>2016-04-11T11:57:56.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>Python通用发布流程:</p><ul><li>git checkout - b workspace</li><li>编写逻辑</li><li>git add &amp;&amp; git commit &amp;&amp; git push</li><li>ssh登陆服务器</li><li>git pull</li><li>启动脚本</li></ul><p>在每次修改代码之后,都需要执行git提交,登陆服务器,更新代码,启动代码.登陆服务器,更新代码是一项重复枯燥无味的工作.</p><p>python使用Fabric可以轻松的在本地用Pythonic的方式完成ssh连接和执行shell命令.快速入门</p><p>安装方法:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo pip install fabric</span><br></pre></td></tr></table></figure><p>以web开发为例,将更新的web代码提交到git,在服务器端通过git更新代码,并重启uwsgi服务器,fabric代码如下:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#fabfile.py</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> fabric.api <span class="keyword">import</span> local, settings, abort, cd, run, env, lcd, task</span><br><span class="line"><span class="keyword">from</span> fabric.contrib.console <span class="keyword">import</span> confirm</span><br><span class="line"><span class="keyword">from</span> fabric.network <span class="keyword">import</span> ssh</span><br><span class="line"></span><br><span class="line">ssh.util.log_to_file(<span class="string">"paramiko.log"</span>, <span class="number">10</span>)</span><br><span class="line"></span><br><span class="line">env.hosts = [<span class="string">'127.0.0.1'</span>] <span class="comment">#服务器地址</span></span><br><span class="line">env.user = <span class="string">'root'</span></span><br><span class="line">env.key_filename = <span class="string">"~/Documents/why.pem"</span> <span class="comment">#ssh私钥文件地址</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">#本地提交代码</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">commit_code</span><span class="params">()</span>:</span></span><br><span class="line"> code_dir = <span class="string">"/home/q/blog"</span></span><br><span class="line"> <span class="keyword">with</span> settings(warn_only=<span class="keyword">True</span>):</span><br><span class="line"> <span class="keyword">with</span> lcd(code_dir):</span><br><span class="line"> result = local(<span class="string">"git pull origin master"</span>)</span><br><span class="line"> <span class="keyword">if</span> result.failed <span class="keyword">and</span> <span class="keyword">not</span> confirm(<span class="string">"Pull failed, continue?"</span>):</span><br><span class="line"> abort(<span class="string">"abort!"</span>)</span><br><span class="line"> local(<span class="string">"git add ."</span>)</span><br><span class="line"> local(<span class="string">"git commit -m 'auto commit'"</span>)</span><br><span class="line"> local(<span class="string">"git push origin master"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">#服务器发布代码</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">deploy_server</span><span class="params">()</span>:</span></span><br><span class="line"> code_dir = <span class="string">"/home/q/blog"</span></span><br><span class="line"> <span class="keyword">with</span> settings(warn_only=<span class="keyword">True</span>):</span><br><span class="line"> <span class="keyword">with</span> cd(code_dir):</span><br><span class="line"> run(<span class="string">"sudo git pull origin master"</span>)</span><br><span class="line"> result = run(<span class="string">"sudo uwsgi --ini conf/uwsgi.conf --reload pidfile"</span>)</span><br><span class="line"> <span class="keyword">if</span> result.failed:</span><br><span class="line"> run(<span class="string">"sudo ps -ef|grep uwsgi"</span>)</span><br><span class="line"> run("""sudo ps -ef|grep <span class="string">"\-\-ini conf/uwsgi.conf"</span>|awk -F <span class="string">' '</span> <span class="string">'&#123;print $2&#125;'</span>|xargs sudo kill <span class="number">-9</span>""")</span><br><span class="line"> result = run(<span class="string">"sudo uwsgi --ini conf/uwsgi.conf"</span>)</span><br><span class="line"> <span class="keyword">if</span> result.failed:</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"launch failed"</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">#发布入口</span><br><span class="line"><span class="meta">@task(alias="install")</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">deploy</span><span class="params">()</span>:</span></span><br><span class="line"> commit_code()</span><br><span class="line"> deploy_server()</span><br></pre></td></tr></table></figure><a id="more"></a><p>执行fab install或者fab deploy结果如下:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line">[127.0.0.1] Executing task &apos;install&apos;</span><br><span class="line">[localhost] local: git pull origin master</span><br><span class="line">来自 github.com:brianfordream/blog</span><br><span class="line"> * branch master -&gt; FETCH_HEAD</span><br><span class="line">Already up-to-date.</span><br><span class="line">[localhost] local: git add .</span><br><span class="line">[localhost] local: git commit -m &apos;auto commit&apos;</span><br><span class="line">位于分支 master</span><br><span class="line">无文件要提交,干净的工作区</span><br><span class="line"></span><br><span class="line">Warning: local() encountered an error (return code 1) while executing &apos;git commit -m &apos;auto commit&apos;&apos;</span><br><span class="line"></span><br><span class="line">[localhost] local: git push origin master</span><br><span class="line">Everything up-to-date</span><br><span class="line">[127.0.0.1] run: sudo git pull origin master</span><br><span class="line">[127.0.0.1] out: From https://github.com/brianfordream/blog</span><br><span class="line">[127.0.0.1] out: * branch master -&gt; FETCH_HEAD</span><br><span class="line">[127.0.0.1] out: Already up-to-date.</span><br><span class="line">[127.0.0.1] out: </span><br><span class="line"></span><br><span class="line">[127.0.0.1] run: sudo uwsgi --ini conf/uwsgi.conf --reload pidfile</span><br><span class="line">[127.0.0.1] out: [uWSGI] getting INI configuration from conf/uwsgi.conf</span><br><span class="line">[127.0.0.1] out: </span><br><span class="line"></span><br><span class="line"></span><br><span class="line">Done.</span><br><span class="line">Disconnecting from 127.0.0.1... done.</span><br></pre></td></tr></table></figure><p>通过fabric可以方便的在本地对本地和服务器执行shell操作,简化了发布过程.</p><p>注:env.use_ssh_config = True #使用系统ssh配置</p></content>
<summary type="html">
<p>Python通用发布流程:</p><ul><li>git checkout - b workspace</li><li>编写逻辑</li><li>git add &amp;&amp; git commit &amp;&amp; git push</li><li>ssh登陆服务器</li><li>git pull</li><li>启动脚本</li></ul><p>在每次修改代码之后,都需要执行git提交,登陆服务器,更新代码,启动代码.登陆服务器,更新代码是一项重复枯燥无味的工作.</p><p>python使用Fabric可以轻松的在本地用Pythonic的方式完成ssh连接和执行shell命令.快速入门</p><p>安装方法:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo pip install fabric</span><br></pre></td></tr></table></figure><p>以web开发为例,将更新的web代码提交到git,在服务器端通过git更新代码,并重启uwsgi服务器,fabric代码如下:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#fabfile.py</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> fabric.api <span class="keyword">import</span> local, settings, abort, cd, run, env, lcd, task</span><br><span class="line"><span class="keyword">from</span> fabric.contrib.console <span class="keyword">import</span> confirm</span><br><span class="line"><span class="keyword">from</span> fabric.network <span class="keyword">import</span> ssh</span><br><span class="line"></span><br><span class="line">ssh.util.log_to_file(<span class="string">"paramiko.log"</span>, <span class="number">10</span>)</span><br><span class="line"></span><br><span class="line">env.hosts = [<span class="string">'127.0.0.1'</span>] <span class="comment">#服务器地址</span></span><br><span class="line">env.user = <span class="string">'root'</span></span><br><span class="line">env.key_filename = <span class="string">"~/Documents/why.pem"</span> <span class="comment">#ssh私钥文件地址</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">#本地提交代码</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">commit_code</span><span class="params">()</span>:</span></span><br><span class="line"> code_dir = <span class="string">"/home/q/blog"</span></span><br><span class="line"> <span class="keyword">with</span> settings(warn_only=<span class="keyword">True</span>):</span><br><span class="line"> <span class="keyword">with</span> lcd(code_dir):</span><br><span class="line"> result = local(<span class="string">"git pull origin master"</span>)</span><br><span class="line"> <span class="keyword">if</span> result.failed <span class="keyword">and</span> <span class="keyword">not</span> confirm(<span class="string">"Pull failed, continue?"</span>):</span><br><span class="line"> abort(<span class="string">"abort!"</span>)</span><br><span class="line"> local(<span class="string">"git add ."</span>)</span><br><span class="line"> local(<span class="string">"git commit -m 'auto commit'"</span>)</span><br><span class="line"> local(<span class="string">"git push origin master"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">#服务器发布代码</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">deploy_server</span><span class="params">()</span>:</span></span><br><span class="line"> code_dir = <span class="string">"/home/q/blog"</span></span><br><span class="line"> <span class="keyword">with</span> settings(warn_only=<span class="keyword">True</span>):</span><br><span class="line"> <span class="keyword">with</span> cd(code_dir):</span><br><span class="line"> run(<span class="string">"sudo git pull origin master"</span>)</span><br><span class="line"> result = run(<span class="string">"sudo uwsgi --ini conf/uwsgi.conf --reload pidfile"</span>)</span><br><span class="line"> <span class="keyword">if</span> result.failed:</span><br><span class="line"> run(<span class="string">"sudo ps -ef|grep uwsgi"</span>)</span><br><span class="line"> run("""sudo ps -ef|grep <span class="string">"\-\-ini conf/uwsgi.conf"</span>|awk -F <span class="string">' '</span> <span class="string">'&#123;print $2&#125;'</span>|xargs sudo kill <span class="number">-9</span>""")</span><br><span class="line"> result = run(<span class="string">"sudo uwsgi --ini conf/uwsgi.conf"</span>)</span><br><span class="line"> <span class="keyword">if</span> result.failed:</span><br><span class="line"> <span class="keyword">print</span> <span class="string">"launch failed"</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">#发布入口</span><br><span class="line"><span class="meta">@task(alias="install")</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">deploy</span><span class="params">()</span>:</span></span><br><span class="line"> commit_code()</span><br><span class="line"> deploy_server()</span><br></pre></td></tr></table></figure>
</summary>
<category term="python" scheme="http://trytofix.com/categories/python/"/>
<category term="fabric" scheme="http://trytofix.com/tags/fabric/"/>
</entry>
<entry>
<title>测试谷歌地图</title>
<link href="http://trytofix.com/2016/04/06/%E6%B5%8B%E8%AF%95%E8%B0%B7%E6%AD%8C%E5%9C%B0%E5%9B%BE/"/>
<id>http://trytofix.com/2016/04/06/测试谷歌地图/</id>
<published>2016-04-06T12:42:09.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>我在:<br><a id="more"></a><br></p><div id="googleMap9102" style="height:500px;width:100%;margin:0;padding:0"></div><style>.gmnoprint img{max-width:none!important}</style><script defer>function makeMaps(){for(var o in window.hexoGmaps.maps)console.log("map #",o),window.hexoGmaps.maps[o]()}function loadScript(){window.hexoGmaps.googleScriptLoaded=!0;var o=document.createElement("script");o.type="text/javascript",o.src="http://www.google.cn/maps/api/js?key=AIzaSyAA2ThKRTD8tK4nl9s-AOP67u3GHsiz2RM&v=3.exp&callback=makeMaps",document.body.appendChild(o)}window.hexoGmaps=window.hexoGmaps||{maps:{}},window.hexoGmaps.maps.initgoogleMap9102=function(){var o={scrollwheel:!0,zoom:17,center:new window.google.maps.LatLng(39.9782805252,116.315163067)},e=document.getElementById("googleMap9102"),a=new window.google.maps.Map(e,o),n={position:new window.google.maps.LatLng(39.9782805252,116.315163067),map:a,title:"新中关购物中心",zIndex:0,icon:""};new window.google.maps.Marker(n)},window.hexoGmaps.googleScriptLoaded||loadScript()</script><p></p></content>
<summary type="html">
<p>我在:<br>
</summary>
<category term="生活" scheme="http://trytofix.com/categories/%E7%94%9F%E6%B4%BB/"/>
</entry>
<entry>
<title>使用七牛为静态资源加速-脚本自动完成</title>
<link href="http://trytofix.com/2016/04/06/%E4%BD%BF%E7%94%A8%E4%B8%83%E7%89%9B%E4%B8%BA%E9%9D%99%E6%80%81%E8%B5%84%E6%BA%90%E5%8A%A0%E9%80%9F-%E8%84%9A%E6%9C%AC%E8%87%AA%E5%8A%A8%E5%AE%8C%E6%88%90/"/>
<id>http://trytofix.com/2016/04/06/使用七牛为静态资源加速-脚本自动完成/</id>
<published>2016-04-06T09:12:58.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>hexo生成的都是静态文件,除了文章内容之外,大部分都是各种主题需要的js,css文件,如果将博客部署在github上,由于某些众所周知的原因,导致github下载速度非常慢,那么这些静态资源文件将会降低用户体验.</p><p>一个优化方法就是将博客部署在github上,而js,css等静态资源文件部署在国内的cdn上.七牛是一款使用广泛的存储空间,普通用户每月都会有免费额度,更关键的是,提供了多种语言的api,方便自动化管理.</p><a id="more"></a><p>以本博客为例,通过<code>hexo g</code>生成的静态文件位于public文件下:</p><ul><li>public/vendor</li><li>public/css</li><li>public/js</li></ul><p><img src="/img/hexo_qiniu_1.png" alt="hexo_qiniu_1.png"></p><p>在主题中引用这些文件的部分代码如下:</p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">head.ejs</span><br><span class="line"></span><br><span class="line"><span class="tag">&lt;<span class="name">%-</span> <span class="attr">css</span>('<span class="attr">vendor</span>/<span class="attr">open-sans</span>/<span class="attr">styles</span>'+'<span class="attr">.css</span>') %&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">%-</span> <span class="attr">css</span>('<span class="attr">vendor</span>/<span class="attr">source-code-pro</span>/<span class="attr">styles</span>'+'<span class="attr">.css</span>') %&gt;</span></span><br><span class="line"></span><br><span class="line"><span class="tag">&lt;<span class="name">%-</span> <span class="attr">css</span>('<span class="attr">css</span>/<span class="attr">style</span>'+'<span class="attr">.css</span>') %&gt;</span></span><br></pre></td></tr></table></figure><p>对静态资源的引用走的依然是当前主机下的流量,当部署在github上时,加载静态资源非常慢.</p><p>以public/js/main.js为例进行说明:</p><ul><li>默认访问地址:<code>http://trytofix.com/js/main.js</code> 由于机器部署在github上,实际是从github上加载main.js.在我这里平均用时180-200ms之间.</li><li>使用cdn地址: <code>http://7xiego.com1.z0.glb.clouddn.com/js/main.js</code>,平均用时40-50ms.<br>使用cdn不仅访问速度明显提升,而且七牛还可以设置静态资源文件的缓存时间,通过在用户浏览器缓存静态资源文件加快网站的访问.</li></ul><p>下面将这些静态资源放到七牛中</p><p>根据七牛提供的api,编写上传静态文件的代码如下:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding:utf8</span></span><br><span class="line">__author__ = <span class="string">'brianyang'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> qiniu <span class="keyword">import</span> Auth, put_file</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"></span><br><span class="line">access_key = <span class="string">'your access key'</span></span><br><span class="line">secret_key = <span class="string">'your secret key'</span></span><br><span class="line"></span><br><span class="line">q = Auth(access_key, secret_key)</span><br><span class="line">bucket_name = <span class="string">'your bucket name'</span></span><br><span class="line"></span><br><span class="line">vendor_dir = <span class="string">'public/vendor'</span></span><br><span class="line">css_dir = <span class="string">'public/css'</span></span><br><span class="line">js_dir = <span class="string">'public/js'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">list_file</span><span class="params">(file_path)</span>:</span></span><br><span class="line"> <span class="keyword">for</span> path <span class="keyword">in</span> os.listdir(file_path):</span><br><span class="line"> full_path = os.path.join(file_path, path)</span><br><span class="line"> <span class="keyword">if</span> os.path.isfile(full_path):</span><br><span class="line"> key = re.sub(<span class="string">r".*?public/"</span>, <span class="string">""</span>, full_path)</span><br><span class="line"> token = q.upload_token(bucket_name, key, <span class="number">3600</span>)</span><br><span class="line"> ret, info = put_file(token, key, full_path)</span><br><span class="line"> <span class="keyword">print</span> ret</span><br><span class="line"> <span class="keyword">print</span> (info)</span><br><span class="line"> <span class="keyword">if</span> os.path.isdir(full_path):</span><br><span class="line"> list_file(full_path)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">list_file(vendor_dir)</span><br><span class="line">list_file(css_dir)</span><br><span class="line">list_file(js_dir)</span><br></pre></td></tr></table></figure><blockquote><p>注意: 本地文件<code>/public/a/b/c/d.css</code>上传到七牛,对应的文件名为<code>a/b/c/d.css</code>,通过七牛给你的个性化地址+七牛文件名进行访问.</p></blockquote><p>下面修改hexo主题的几个地方,以便可以使用cdn.</p><p>修改主题的_config.yml,添加</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cdn_prefix: http://your.qiniu.glb.clouddn.com/</span><br></pre></td></tr></table></figure><p>注意,这里的cdn_prefix请换成自己申请的七牛地址,还有最后一个<code>/</code>一定不要忘记,如果不使用cdn,则置空即可.</p><p>修改主题下的layout中的引入css和js的文件,例如head.ejs中与本文开始对应的三行修改为</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">%-</span> <span class="attr">css</span>(<span class="attr">theme.cdn_prefix</span>+'<span class="attr">vendor</span>/<span class="attr">open-sans</span>/<span class="attr">styles</span>'+'<span class="attr">.css</span>') %&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">%-</span> <span class="attr">css</span>(<span class="attr">theme.cdn_prefix</span>+'<span class="attr">vendor</span>/<span class="attr">source-code-pro</span>/<span class="attr">styles</span>'+'<span class="attr">.css</span>') %&gt;</span></span><br><span class="line"></span><br><span class="line"><span class="tag">&lt;<span class="name">%-</span> <span class="attr">css</span>(<span class="attr">theme.cdn_prefix</span>+'<span class="attr">css</span>/<span class="attr">style</span>'+'<span class="attr">.css</span>') %&gt;</span></span><br></pre></td></tr></table></figure><p>这样做的好处是,当需要更换cdn时,仅需修改_config.yml中的cdn_prefix就可以了.</p><p>如果的你的网站在国内备案了,通过上面的脚本,甚至可以全站上传到七牛,通过cdn来加速全站,效果更佳.</p></content>
<summary type="html">
<p>hexo生成的都是静态文件,除了文章内容之外,大部分都是各种主题需要的js,css文件,如果将博客部署在github上,由于某些众所周知的原因,导致github下载速度非常慢,那么这些静态资源文件将会降低用户体验.</p><p>一个优化方法就是将博客部署在github上,而js,css等静态资源文件部署在国内的cdn上.七牛是一款使用广泛的存储空间,普通用户每月都会有免费额度,更关键的是,提供了多种语言的api,方便自动化管理.</p>
</summary>
<category term="python" scheme="http://trytofix.com/categories/python/"/>
<category term="cdn加速" scheme="http://trytofix.com/tags/cdn%E5%8A%A0%E9%80%9F/"/>
</entry>
<entry>
<title>hexo添加404页面</title>
<link href="http://trytofix.com/2016/04/06/hexo%E6%B7%BB%E5%8A%A0404%E9%A1%B5%E9%9D%A2/"/>
<id>http://trytofix.com/2016/04/06/hexo添加404页面/</id>
<published>2016-04-06T09:01:43.000Z</published>
<updated>2016-05-16T07:18:51.804Z</updated>
<content type="html"><p>为部署在github上的hexo添加自定义的404页面</p><p>在source下新建404.html</p><p>实现404页面到首页自动跳转,404.html内容如下:</p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="meta">&lt;!DOCTYPE html&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">html</span> <span class="attr">lang</span>=<span class="string">"en"</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">head</span>&gt;</span></span><br><span class="line"> <span class="tag">&lt;<span class="name">meta</span> <span class="attr">charset</span>=<span class="string">"UTF-8"</span>&gt;</span></span><br><span class="line"> <span class="tag">&lt;<span class="name">meta</span> <span class="attr">http-equiv</span>=<span class="string">"refresh"</span> <span class="attr">content</span>=<span class="string">"2;url=/"</span>&gt;</span></span><br><span class="line"> <span class="tag">&lt;<span class="name">title</span>&gt;</span>404<span class="tag">&lt;/<span class="name">title</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">head</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">body</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">h1</span>&gt;</span>页面没有找到,跳转到主页...<span class="tag">&lt;/<span class="name">h1</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">img</span> <span class="attr">src</span>=<span class="string">"http://ww1.sinaimg.cn/small/b8b708a7gw1f2n307j0vlj206l064dfs.jpg"</span>/&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">body</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">html</span>&gt;</span></span><br></pre></td></tr></table></figure><p>打开<a href="http://trytofix.com/404">本站404效果</a>查看效果吧!</p></content>
<summary type="html">
<p>为部署在github上的hexo添加自定义的404页面</p><p>在source下新建404.html</p><p>实现404页面到首页自动跳转,404.html内容如下:</p><figure class="highlight html"><table><tr><td
</summary>
<category term="hexo" scheme="http://trytofix.com/categories/hexo/"/>
<category term="404" scheme="http://trytofix.com/tags/404/"/>
</entry>
</feed>