phantomjs/casperjs and qiandao

14 November 2013


Well, qiandao here means Chinese 签到。

that action is boring that you need open browser, login, then click buttons. afterwards, you get some points that you can use it somehow as a little money.

Programming is born for saving you from daily repeating/boring works. and that’s why programming is fun.

I spent few hours to write a casperjs coffeescript today to do the action for taobao.com/etao.com. the code is quite simple and the result is very interesting. phantomjs or casperjs is born for suck work.

you can take a look of the code at https://github.com/fayland/casperjs-qiandao/blob/master/taobao.coffee, it’s quite self-explaining. open page, click button, wait until frame shows, type user/pass, then submit. afterwards, just wait until the result shows up. and print it. meanwhile, after collect few 淘金币, it also collects one 集分宝.

crontab it as casperjs /path/to/casperjs-qiandao/taobao.coffee –username=fayland_lam –password=secret

and that’s it. programming does the job for you and it’s reliable.

have fun.


phantomjs/casperjs and google plus new share

10 November 2013


phantomjs is great. the wrapper casperjs is even better.

when surfing around for a Google+ poster as what I did for Twitter/Facebook/Linkedin etc., I find amazing altryne/google-plus-api

it use casperjs and it’s very simple to use.

just one tiny issue, it does not use cookie and logins every time. so I made a small patch for that, and send the pull request to the author.

I’ll merge it into findmjob.com soon. you’ll find fresh job posting on Findmjob Google+.

Have fun.


Perl for JS toString parseInt with radix

07 November 2013


it’s almost a copy-paste from module JE. it’s used in one of my code to parse JS code value with Perl.

sub js_toString {
    my ($num, $radix) = @_;

    return $num unless $radix;
    return $num if ($radix == 10 || $radix < 2 || $radix > 36 || $radix =~ /\./);

    if ($radix == 2) {
        return sprintf('%b', $num);
    } elsif($radix == 8) {
        return sprintf('%o', $num);
    } elsif($radix == 16) {
        return sprintf('%x', $num);
    }

    my $result = '';
    my @_digits = (0..9, 'a' .. 'z');
    while ($num >= 1) {
        substr($result,0,0) =
            $_digits[$num % $radix];
        $num /= $radix;
    }

    return $result;
}

sub js_parseInt {
    my ($str, $radix) = @_;

    return unless defined $str;
    $radix ||= 0;

    my $s = qr.[\p{Zs}\s\ck]*.;
    $str =~ s/^$s//;

    my $sign = $str =~ s/^([+-])//
        ? (-1,1)[$1 eq '+']
        :  1;
    $radix = (int $radix) % 2 ** 32;
    $radix -= 2**32 if $radix >= 2**31;
    $radix ||= $str =~ /^0x/i
    ?   16
    :   10
    ;
    $radix == 16 and
        $str =~ s/^0x//i;

    $radix < 2 || $radix > 36 and return;

    my @digits = (0..9, 'a'..'z')[0..$radix-1];
    my $digits = join '', @digits;
    $str =~ /^([$digits]*)/i;
    $str = $1;

    my $ret;
    if(! length $str){
        $ret = '';
    } elsif($radix == 10) {
        $ret = $sign * $str;
    } elsif($radix == 16) {
        $ret = $sign * hex $str;
    } elsif($radix == 8) {
        $ret = $sign * oct $str;
    } elsif($radix == 2) {
        $ret = $sign * eval "0b$str";
    } else {
        my ($num, $place);
        for (reverse split //, $str){
            $num += ($_ =~ /[0-9]/ ? $_
                : ord(uc) - 55)
                * $radix**$place++
        }
        $ret = $num*$sign;
    }

    return $ret;
}

say js_parseInt('z', 36); # 35
say js_toString(35, 36);  # 'z'

sub js_fromCharCode {
    my $str = '';
    my $num;
    for (@_) {
        # % 2**16 is buggy in perl
        $num = $_;
        $num = ($num < 0 ? ceil($num) : floor($num))
            % 2**16 ;
        $str .= chr($num == $num && $num);
            # change nan to 0
    }
    return $str;
}

sub js_charCodeAt {
    my ($str, $pos) = @_;

    if (defined $pos) {
        $pos = int($pos);
        $pos = 0 unless $pos == $pos;
    }

    return ($pos < 0 || $pos >= length $str)
            ? 0
            : ord substr $str, $pos, 1;
}

phantomjs screenshot

05 November 2013


write test.js

var page = require('webpage').create();

page.viewportSize = { width: 1024, height: 800 };
page.clipRect = { top: 0, left: 0, width: 1024, height: 800 };
page.open('http://findmjob.com/', function () {
    page.render('findmjob.png');
    phantom.exit();
});

then do

phantomjs test.js

I found this tool from Wight and it’s pretty amazing.

I’ll play it a bit more and share if something is interesting.


SSL pfx and SOAP

23 August 2013


Sometimes you got a pfx file (with password pharse) that you can import it into Firefox certificates to view the website.

in Perl, it’s a bit different, you first need use openssl to convert it into cert pem files.

openssl pkcs12 -in test.pfx -out mycert.pem -nodes
openssl rsa -in mycert.pem -out mycertkey.pem

with that, you got two files. then with LWP::UserAgent or any other modules based on it:

my $ua = LWP::UserAgent->new(
    ssl_opts => {
        SSL_use_cert => 1,
        verify_hostname => 0,
        SSL_cert_file => "$Bin/mycert.pem",
        SSL_key_file => "$Bin/mycertkey.pem",
    }
)
my $res = $ua->get("https://blabla.com/");

you can always use below to debug SSL:

use IO::Socket::SSL qw(debug4);

and if the pfx is for SOAP::Lite, you can use something like below:

$soap->transport->ssl_opts(
    SSL_use_cert => 1,
    verify_hostname => 0,
    SSL_cert_file => "$Bin/mycert.pem",
    SSL_key_file => "$Bin/mycertkey.pem",
);

basically transport is LWP::UserAgent based on http(s) SOAP url.

Have fun.


set mp3 meta lyrics/artwork

25 June 2013


example code to set mp3 Lyrics and Artwork

use MP3::Tag;

open(my $fh, '<', 'artwork.png');
my $image_data = do { local $/; <$fh>; };
close($fh);

my $mp3 = MP3::Tag->new('test.mp3');

my $id3v2;
$id3v2 = $mp3->{ID3v2} if exists $mp3->{ID3v2};
$id3v2 ||= $mp3->new_tag("ID3v2");

$id3v2->title('test');
$id3v2->add_frame('TALB', 'Album title TEST'); # Album/Movie/Show title
$id3v2->add_frame('TPE1', 'Artist name TEST'); # Lead performer(s)/Soloist(s)
$id3v2->add_frame('TPE2', 'BLBALA Artist name'); # Band/orchestra/accompaniment

$id3v2->add_frame('USLT', 0, 'eng', '', "test blabla\naaa CCCa DDD");
$id3v2->add_frame('APIC', 0, 'image/png', chr(0), '', $image_data);

$id3v2->write_tag();

some code borrowed from


mojo tt2 defaults layout patch

27 April 2013


You wrote code with open source like CPAN. You meet problems with them. You investigate, write tests then provide patches. Your patches are accepeted.

I really enjoy the way it moves.

Here is another patch for Mojolicious::Plugin::TtRenderer.

the issue is that $self->defaults(layout => 'wrapper'); is always dropping the [% content %] without reason. and with debug I see the content is really in $c->stash->{'mojo.content'}. so a simple patch is that just $c->stash->{content} ||= $c->stash->{'mojo.content'}->{content};

recently I’m writing another new app with bitcoin <-> litecoin and other virtual coins exchange. and I have a wrapper.html.tt for the whole site. but I do not want to use it in the email TT2 render.

if I put WRAPPER => 'layouts/wrapper.html.tt', in the TT2 options, it will also apply to the mail render. a better approach is to use defaults layout. then in email render, set the layout as empty or actually we need use partial => 1.

MainApp.pm

$self->plugin('tt_renderer');
$self->defaults(layout => 'wrapper');
$self->renderer->default_handler( 'tt' );

OtherController.pm

my $body = $c->render(
    template => 'emails/forgot_pass', format => 'mail',
    other_stash => $var,
    partial => 1
);

partial => 1 will set layout as null. so it will give us what we want.

Have fun!


Mojolicious TT2 patch

12 April 2013


I am kindy busy with adding new features to my job site

but it’s very boring to restart the app on template updates. TT2 should be able to pick up the changes automatically. it works under Dancer or Catalyst, but not Mojolicious.

I thought I should spend some time on investigating why before writing more stuff. and after a while, I have a patch for Mojolicious::Plugin::TtRenderer.

it dues to the Provider in TtRender always return 1 instead of mtime on _template_modified.

plicease accepts it quite fast with slight changes but I’m waiting for a CPAN release. :-)

here is few changes on http://findmjob.com/ recently.

Happy hacking!


Dancer to Mojolicious

10 April 2013


The progress went smoothly because all of them are just Perl code.

old Dancer code: FindmJob::WWW

new Mojolicious code: FindmJob::WWW, FindmJob::WWW::Root and others

Few notes

hooks

from before_template_render to before_render

Dancer forward VS Mojolicious before_dispatch

we can modify the req->url->path in before_dispatch with assigning stash.

# feed.(rss|atom) and /p.2/
$self->hook( before_dispatch => sub {
    my $self = shift;

    my $p = $self->req->url->path;
    if ($p =~ s{/feed\.(rss|atom)$}{}) {
        $self->stash('is_feed' => $1);
    }
    if ($p =~ s{/p\.(\d+)(/|$)}{$2}) {
        $self->stash('page' => $1);
    }
    $self->req->url->path($p);
});

supervise

Dancer:

plackup -E production -s Starman --workers=3 -l /tmp/findmjob.sock -a /findmjob.com/www/bin/app.pl

Mojolicious:

hypnotoad -f /findmjob.com/bin/www.pl

Note -f is important here, or supervise will keep restarting the script.

above plackup using sock and hypnotoad use port. so we need update the ngnix config a bit.

and it looks like hypnotoad is working better than plackup Starman

Template name conversation

there is no rule in Dancer for the Template name. but in Mojolicious, we have to do it with $name.$format.$handler

in this case we have to rename index.tt2 to index.html.tt

Conclusion

well, I’m not saying that Dancer is worse than Mojolicous or any other words. both of them are great.

but Dancer is a bit messy with Dancer and Dancer2. actually I love Moo a lot, but I don’t want to spend time on upgrading Dancer to Dancer2 (there seems some more difference than expected.)

in the other hand, Mojolicious looks amazing, sri improves it every week and I admire/trust his professional knowledge.

Have fun.


Two new CPAN modules

30 March 2013


I got two new CPAN modules uploaded today.

WWW::SpinnerChief

I wrote this based on the Python code

it’s pretty good that you know how to read the Python.

Business::PayPoint

aka tips to write SOAP module in Perl.

the PHP SOAP lib is pretty good while the Perl one is a little suck.

so usually if I can’t write it at the first try in Perl, I would use PHP to find the sending request/response

$client = new SoapClient('https://www.example.com/Test?wsdl', array(
    'trace' => 1,
) );
$client->__call('TestAction', $params);
echo "RESPONSE:\n" . $client->__getLastResponse() . "\n";
echo "REQUEST HEADER:\n" . $client->__getLastRequestHeaders() . "\n";
echo "REQUEST:\n" . $client->__getLastRequest() . "\n";

After I got the correct sample request, I’ll try to use Perl library to send the same XML with trace on.

use SOAP::Lite +trace => 'all';

well, you even can’t write correct one sometimes (maybe I am a little stupid).

but I have a final trick for it. use XML::Write to genereate the request XML. then use

my $som = $soap->call($method, SOAP::Data->type('xml' => $xml));

you won’t be wrong in this case. :)

Have fun.