Net::Telnet with bitflu

18 March 2013


Even Net::Telnet is quite old, it’s still very powerful and simple to use.

use strict;
use warnings;
use Net::Telnet ();
use Data::Dumper;

my $host = '0';
my $port = 4001;

my $telnet = Net::Telnet->new();
unless ($telnet->open( Host => $host, Port => $port, Timeout => 30 )) {
    die "Can't connect to $host:$port\n";
}

$telnet->waitfor('/bitflu> /');

my @messages = map { chomp; $_ } $telnet->cmd(String => 'ls');
pop @messages if $messages[-1] eq 'bitflu';
print Dumper(\@messages);

$telnet->close();

there are some tips:

remove ANSI color

foreach my $msg (@messages) {
    # if you do not remove this, your regex with /^\[/ may break
    $msg =~ s/\e\[[\d;]*[a-zA-Z]//g;
}

Change Window Size

if you use term to do bitflu> ls, you’ll see full torrent name. but with the code above, you can only see few chars. here is the note to change the Window Size:

my $telnet = Net::Telnet->new();

$telnet->option_callback( sub { return; } );
$telnet->option_accept(Do => 31);

unless ($telnet->open( Host => $host, Port => $port, Timeout => 60 )) {
    die "Can't connect to $host:$port\n";
}
$telnet->waitfor('/bitflu> /');

## copied from http://blog.webdir.bg/perl-apache-realtime-output-from-script/
## Many Thanks!
$telnet->telnetmode(0);
$telnet->put(pack("C9",
                  255,                  # TELNET_IAC
                  250,                  # TELNET_SB
                  31, 0, 200, 0, 0,     # TELOPT_NAWS
                  255,                  # TELNET_IAC
                  240));                # TELNET_SE
$telnet->telnetmode(1);

Learn Python Note 1

11 March 2013


I’m starting learning Python a bit.

argparse

if you have args setup in different modules, like you want setup --log on logging, setup --tor --skip-tor on requesting, the combination of add_help=False and parse_known_args is pretty cool.

import argparse
parent_argparse = argparse.ArgumentParser(add_help=False)
parent_argparse.add_argument('--log', action='store', default='ERROR', dest='log', help='logging level: DEBUG, INFO, WARNING, ERROR, CRITICAL')
_args, _args_unknown = parent_argparse.parse_known_args()

afterwards, in real place, do

parser = argparse.ArgumentParser(description='Some real CLI', parents=[parent_argparse])

init.py and as

actually it’s a very cool feature. you can put some common code for all sub-modules in __init__.py like config code (to avoid write another file)

import ConfigParser
try:
    cfg = ConfigParser.ConfigParser()
    cfg.read(['/somwhere/conf/project.ini', '/somwhere/conf/project_local.ini'])
except ConfigParser.Error, e:
    logger.critical('Cannot read configuration file, reason [%s]' % e)

and as is pretty amazing.

from modulename import cfg as project_config

BeautifulSoup is another HTML::TreeBuilder

BeautifulSoup works like Perl HTML::TreeBuilder.

from bs4 import BeautifulSoup
soup = BeautifulSoup(text)

for tag in soup.find_all('span', attrs={'id': re.compile('^ctl00_contentPageContent_lbl(.*?)Value$')}):
    id = tag['id']
    id = id.replace('ctl00_contentPageContent_lbl', '').replace('Value', '')
    data[id] = tag.get_text()

something I do like in Python

def with args and default values. (Perl sub was developed too many years ago.)

you don’t need type so many {} or ().

something I do not like in Python

string interpolation is quite inconvenience. there are %, format and Template. but I really miss the $ with Perl.

I miss Perl regexp a lot. the re.compile, re.sub, re.I is too much typing.

Disclaim

I’m still a newbie. so parden me if anything is wrong. I’ll write more when I learn more. and I admit it’s pretty fun to learn Python.


Blogger local file > Jekyll migrator

07 March 2013


I have an old Blogger before I move to MT5, it contains my posts from 2006 to 2009. some are quite old, broken, useless and even not worth reading. but some are still very useful and it’s a part of my memory and life. (Sorry that comments are dropped that I can’t import it into Disqus.)

so I wrote a simple Perl script to convert the local HTML files into Jekyll here. the source code can be found in github.

it’s quite simple and maybe just fit for my need. but feel free to change it if you have same demand.

Have fun.


add alert note for old posts in Jekyll

07 March 2013


add alert note for old posts

For Jekyll Bootstrap, it’s quite simple as we can do something like below:

edit _includes/themes/twitter/post.html, add

{% assign page_year = page.date | date: "%Y" %}
{% if page_year < '2010' %}
<div class='alert alert-info'>This post may be outdated due to it was written on {{ page_year }}. The links may be broken. The code may be not working anymore. Leave comments if needed.</div>
{% endif %}

20 items in rss/atom

I have another if in the {% for post in site.posts %} for rss.xml and atom.xml

{% if forloop.index < 20 %}
...
{% endif %}

paginate

edit _config.yml

paginate: 10
paginate_path: 'page/:num'

Have fun.


MT > Jekyll perl migrator

06 March 2013


I bought a new domain fayland.me.

I picked Jekyll as the blog engine.

I wrote a simple Perl script to convert my old Movable Type 5 blogs to Jekyll Bootstrap.

Github code

the difference between this one and the Jekyll::MT is that it supports tags and it’s for Jekyll Bootstrap.

at last, using

rsync -arv --delete _site fay:/srv/www/fayland.me/

to rsync the site.

Have fun.


Net-Amazon-DynamoDB

22 November 2012


I have been playing with Net::Amazon::DynamoDB a lot recently. this article is not to comment on that service. it's just a few notes for Perl guys who're using it.

1. Moo based instead of Moose.

well, if you're not using Moose in your system, Moose may be too much for your speed. I am maintain a new branch on that which converted Moose to Moo. changes can be found at https://github.com/fayland/Net-Amazon-DynamoDB/commit/0b3b9a493e92c85461353ef1fc1da422e4d0bb48 and I'll update when the Moose version is updated.

2. custom Cache module supports.

I removed the isa => 'Cache' in the Moo version, instead, I'm using

has cache => ( isa => sub {
     die "thaw/freeze/remove must be supported in cache" unless $_[0]->can('thaw') and $_[0]->can('freeze') and $_[0]->can('remove')
}, is => 'rw', predicate => 'has_cache' );

so that we can use custom layer like

package My::Dummy::Cache::FastMmap;

use Moo;
use Cache::FastMmap;

has 'cache' => (is => 'lazy');
sub _build_cache { Cache::FastMmap->new(share_file => '/tmp/mycache_fastmmap', unlink_on_exit => 0) }

# Cache::FastMmap do not have thaw/freeze sub
sub thaw {
    my $self = shift;
    $self->cache->get(@_);
}
sub freeze {
    my $self = shift;
    $self->cache->set(@_);
}
sub remove {
    my $self = shift;
    $self->cache->remove(@_);
}

1;

I have fixed some bugs for the cache system like not set while return is undef, delete on batch_write etc. but it is still not perfect b/c it's not supported inside batch_get_items yet. I may write patch for it later.

3. set is not array.
well, for SS or NS. it's set. it's not array. it means there is not an order inside.
if you want something like order, we can join it as string and save it. after get, split back.

Thanks.


reCaptcha and lightbox and ajaxPost

14 September 2012


sometimes, you want to do lightbox for Email Us or Contact Us page.
in this case, you don't want to include the recaptcha js in every page, you want to include it in the popup lightbox but here we have an issue that $(document).ready would be called while the google recaptcha js is not loaded. b/c document ready is just checking the original page instead of the lightbox. in this case, here is a common solution for every js loading (wait until that js loaded.)

<script type="text/javascript" src="https://www.google.com/recaptcha/api/js/recaptcha_ajax.js"></script>
<script type="text/javascript">
    $(document).ready(function(){
        var reCaptcha_timer;
        reCaptcha_timer = setInterval(function(){
          if (typeof(Recaptcha) != 'undefined') {
             clearInterval(reCaptcha_timer);
             CreateReCaptcha();
          }
        }, 50);
    });
    function CreateReCaptcha() {
        Recaptcha.create("public_key_blabla", 'captcha-placeholder', {
            theme: "white",
            callback: Recaptcha.focus_response_field
        });
    }
</script>

see we have a tricky that keep check if Recaptcha is inited (which will be done when recatpcha_ajax.js is loaded).
and only after it's loaded, we clear the check, and create the catpcha.

the trick works pretty good. but here is another issue, when the lightbox is loaded and user submits the form, and if some elements are wrong or captcha is wrong, that we need show captcha once again. it will be broken. b/c Recaptcha js variable is already defined, and Recaptcha.create will not working fine b/c it has something stored for previous captcha.

in this case, another tricky is much more simpler.

<script type="text/javascript">
// so that we can reload it
if (typeof(Recaptcha) != 'undefined') Recaptcha = undefined;
</script>
<script type="text/javascript" src="https://www.google.com/recaptcha/api/js/recaptcha_ajax.js"></script>

before load the js, we reset it so it works like it's the first time we are trying to create reCaptcha.

those issue only happens on ajax lightbox load reCaptcha and ajaxPost the reCaptcha form.
stupid but works.

Thanks.

MacOSx homebrew postgresql issue note

12 September 2012


after brew install postgresql

then run "initdb /usr/local/var/postgres -E utf8", may got:

selecting default shared_buffers ... 400kB
creating configuration files ... ok
creating template1 database in /usr/local/var/postgres/base/1 ... FATAL:  could not create shared memory segment: Cannot allocate memory
DETAIL:  Failed system call was shmget(key=1, size=2138112, 03600).
HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMALL.  To reduce the request size (currently 2138112 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
The PostgreSQL documentation contains more information about shared memory configuration.
child process exited with exit code 1
initdb: removing data directory "/usr/local/var/postgres"

fixes as:

 ~  sudo sysctl -w kern.sysv.shmall=1024000
kern.sysv.shmall: 8192 -> 1024000
 ~  initdb /usr/local/var/postgres -E utf8 

then everything will move smoothly.

just for note.


MongoDB and Perl

26 April 2012


it's really a pain to work with MongoDB in Perl. Perl has no 'type' so when you get INT value from DBI, it might be really "1" instead of int 1.

I got some data dumped from MySQL to MongoDB and found all the 'time' field is wrapped as "1335350669" instead of 1335350669.

and when you insert it from code like { time => time() }, you really have 1335350670 instead of "1335350670". it breaks the sort. it breaks the deletion. it breaks everything.

blabla, to fix that, we just need update it in mongodb like below.

PRIMARY> db.jobs.find().forEach(
    function(job) {
        job.time = parseInt(job.time);
        db.jobs.save(job);
    });

but it's still a pain. I didn't check MongoDBx::Class or Mongoose yet, but Moose's type should be able to fix it I think.

Thanks

api and android app

14 April 2012


I'm a fan of Android. even I like my iPad and MBP very much too.

I'd like to write an App for Android. and here is all the story.

first of all, I need write an API which supports JSONP. since it would be very simple, I don't want to mix it up with Dancer. at last, I wrote something based on webmachine-perl. I like the idea behind the webmachine. the chain design looks pretty nice. even I don't use that too much.

the code is at


and live demo as


after API is done. now comes the android app part. Sorry that I don't know much about Java. and I failed to download appmobi/jqmobi (the download never ends here). so at last I picked up phonegap.

the progress went pretty smooth. mixed with jQuery Mobile, I have it out after few hours.

you can download it for fun from http://static.findmjob.com/FindmJob.apk
it's pretty simple, just with one JSONP request and not much different than the demo.

even there, I'd like to share the code with you:


it's very cool and I'm excited! (badly Google costs 25$ for submitting it and I don't want to do it for now)

Thanks.