Requests 本身不提供代理池，然而爬数据又要用，所以只能自己搞。其实还挺简单的。我也不知道为什么这么有用的 feature 一直没有被加入。
import requests
class Client:
def __init__(self):
self._session = requests.Session()
self.proxies = None
def set_proxy_pool(self, proxies, auth=None, https=True):
"""Randomly choose a proxy for every GET/POST request
:param proxies: list of proxies, like ["ip1:port1", "ip2:port2"]
:param auth: if proxy needs auth
:param https: default is True, pass False if you don't need https proxy
"""
from random import choice
if https:
self.proxies = [{'http': 'http://' + p, 'https': 'https://' + p} for p in proxies]
else:
self.proxies = [{'http': 'http://' + p} for p in proxies]
def get_with_random_proxy(url, **kwargs):
proxy = choice(self.proxies)
kwargs['proxies'] = proxy
if auth:
kwargs['auth'] = auth
return self._session.original_get(url, **kwargs)
def post_with_random_proxy(url, *args, **kwargs):
proxy = choice(self.proxies)
kwargs['proxies'] = proxy
if auth:
kwargs['auth'] = auth
return self._session.original_post(url, *args, **kwargs)
self._session.original_get = self._session.get
self._session.get = get_with_random_proxy
self._session.original_post = self._session.post
self._session.post = post_with_random_proxy
def remove_proxy_pool(self):
self.proxies = None
self._session.get = self._session.original_get
self._session.post = self._session.original_post
del self._session.original_get
del self._session.original_post
# You can define whatever operations using self._session
替换掉 Session 原本的 get 和 post 方法就行了，不会有什么副作用。class Client 并不必需，直接操作 Session 是一样的。
可以用 httpbin 来做验证
def test_proxy():
# visit http://cn-proxy.com/ to get available proxies if test failed
proxy_ips = ['112.25.41.136', '180.97.29.57']
client = Client()
client.set_proxy_pool(proxy_ips)
for _ in range(5):
result = client._session.get('http://httpbin.org/ip').json()
assert result['origin'] in proxy_ips
result = client._session.post('http://httpbin.org/post',
data={'m':'1'}).json()
assert result['form'] == {'m': '1'}
print(result['origin'])
assert result['origin'] in proxy_ips
client.remove_proxy_pool()
client.set_proxy_pool(proxy_ips, https=False)
for _ in range(5):
result = client._session.get('http://httpbin.org/ip').json()
print(result['origin'])
assert result['origin'] in proxy_ips
I was discussing with someone about Genshiken chapter 121 on reddit the other day, then our discussion turned to Spotted Flower, which is commonly viewed as story of another universe. I disagreed with it, and said Spotted Flower is actually a sequel of Genshiken. Here's my reply:
Ah, I have to say, maybe you know Genshiken well, but you didn't read Spotted Flower very carefully. I happen to get some time now, so I'll explain my theory.
Before that, there's one thing I forgot to say, which is really important: I'm not trying to predict the development of the story(Spotted Flower), what I'm trying to do is reveal Shimoku Kio's intention, or in other words, what does Spotted Flower tell us.
First, let's look at the title "Spotted Flower". What does it mean? I bet you don't know, those who translate this manga probably do. "Spot", in Chinese/Japanese, is "斑", which is the first character of Rame's full name 斑目晴信. "Saki" is "咲", and the meaning of "咲" in Japanese is bloom/flowering. So "Spotted Flower" literally means "Rame and Saki", there's no argument about that. Moreover, the author plays on word to summarize the story:he uses "Spotted Flower" rather than "Spot Flower", why? I can't think of other possibility than saying Saki belongs to Rame.
Then I'll step through the current chapters to show all clues. I'll just call the main characters Rame and Saki.
ch1
Rame wears glasses, Sasahara doesn't. In Genshiken, he wears the same glasses once when Saki told him to change his appearance;
after a CM, Saki asked Rame "Did you hang out with everyone", Rame said "Yeah(skipped some dialog)...That's why you should have come as well", Saki replied "I'm kidding. I wouldn't want tension in the air because I sat next to my ex-boyfriend.";
Saki said "...That I ended up with 2 otaku guys back to back."
ch3
Kaichou cosplay http://www.heymanga.xyz/manga/Spotted_Flower/3/1;
ch4
Rame and saki come to see Saki's grandma. In Genshiken, Saki mentioned that her name is given by her grandma, other people are surprised and say "saki" is like a character's name in Anime/Manga so they can't imagine an old lady can think of such name. In this chapter, the old lady gave many Anime/Manga-ish name too, so obviously it's the same person. Saki also said to her grandma: "You're the one who named me, grandma!", which proves she is indeed Saki.
ch5
Saki said: "You used to be a hopeless otaku...Back then, I thought you were pretty pathetic, too.";
Rame said: "I really would've liked to raise a little girl...", Saki said: "Hey, your dad is really gross, you know that? He's a lolicon." The only lolicon in genshiken is Rame, Sasahara prefers mature women.
ch7
Ohno came to see Saki, and persuaded her to cosplay, like in Genshiken. Ohno has two babies, that is because she married early and became a full-time housewife. This is implied in Genshiken;
ch9
Rame broke his arm, again...his hand is swollen just like the first time.
ch12
This is the first time Hato appeared in SF, and his relationship with Rame is still very subtle. Saki thought:"But every time they meet, he leaves that scent on my husband's clothes, like he's plaunting it. it's perfume. Men can only get pregnant in video games, dammit! I think he still hasn't given up."
This is an important chapter which gives us many information to guess what happened between Hato and Rame. I just came up with a guess: Rame chooses Hato, they are together for a while, then, Saki breaks up with Kousaka, which gives Rame a chance. There's no doubt that Rame would choose Saki among the two. Finally Saki accepts him, here comes Spotted Flower.
ch12.2 & 12.5
Yajima has lost weight successfully. Remember in Genshiken she said she's gonna lose weight?
This chapter also tells us Hato had breast augmentation.
ch14
The cosplay costume Ohno brought is the character Kousaka cosplayed after Rame made love confession to Saki and got rejected.
This is the funniest scene of the manga and I really couldn't stop laughing.
ch17
This is the chapter that really makes me believe SF is a sequel.
Hato said to Rame "Alcohol was what led you to trip me, and push me down onto a bed, even before I was who I am today."
It means, whether or not the two stories split in the future, at least they are the same till that point, which is near the latest chapter of Genshiken so far. Since I'm talking about revealing the author's intention, this is a clear evidence that he wants us to believe two stories won't split. Whether Shimoku will follow this till the end or make a twist ending is not something to be discussed here.
I ignored some details like the relationship between Yajima and Hato. What's going on next in Genshiken will probably gives a conclusion on if SF is a sequel. My point is fairly simple: since SF matches almost every detail of Genshiken, at least we could say that somehow Shimoku wants us to believe it's a sequel. I can't believe so many people misidentify Saki as Ogiue, it's horrible, there's nothing this character has in common with Ogiue except they are both female and have black hair. As for the hair, by the way, apparently Saki no longer dyes her hair after getting married, that doesn't make her Ogiue, does it?
Anyway, Shimoku Kio is such a talented mangaka, I've never seen other people drawing two related manga at the same time like he's been doing. He's done a brilliant work so far, I hope he could bring us more.
update 2016.3.28
Madarame chooses no one in chapter 122, which doesn't surprise me at all. I wrote "What's going on next in Genshiken will probably gives a conclusion on if SF is a sequel. ", now I would say the possibility raises to 90%.
