Posted on

Configure vim clipboard in fedora (X11)

Apparently, the vim package in fedora comes without the appropriate clipboard flags baked in:

Forunately, the fedora repos provide another package, which has the flags enabled: vim-X11.

 

so whenever you use the native vim hotkeys to control the selection, you can simply copy its contains to the + (clipboard) register:

Now you can paste the copied text outside of the vim terminal session.

To make it complete, you can add the alias to your .bash_profile

Posted on

some postgresql snippets

making polygon from array of points:

if line is not closed:

turning geom_dump setof into array:

create empty array / truncate array:

append to array:

order array:

average of array:

uniq elements in array:

multiline comment

Posted on

Issues downloader for Bitbucket

This is an example of how to create a simple downloader of websites, which are behind login mechanism and further download them – for example bitbucket issues. While the target of your crawing could differ, and could go to xxx, my case was finite, which makes it slighlty easier, since i knew the base of URIs before i even started. Long story short – here is a code which could backup your pages while handling post data & cookies and is purely in Node.js.

What i needed for issues downloader:

  • nodejs runtime(of course)
  • some way to make html requests easy
  • entry point & ending point – basically where to login and when to end
  • https cerificates(explained later)
  • something for parsing html(jquery like cheerio)
  • some output channel for backup – preferably filesystem module(fs)

 

From my point of view, Node.js is for this job suitable as much as python or perl or whatever iterpreted language. The downside of node as “callback-providing-engine” is nested calls when you are lazy. Fortunately, this code won’t have more than 200 lines at most. First thing i needed was to create an authorization request, so let’s check the login page, since i don’t know if bitbucket has an API for that.

 

bitbucket-login

So according to developer tools, i see only next field, csrf field, email and password. The first one is probably not important, but just in case, let’s include it. The second one is token holder, which should be as important as email or password. As it is common, the csrf value is normally doubled in cookies.

 

cookies

 

That means that for authorizing, first i need to scrap the login page for required datas and then forge an HTTP POST request which should be the same as when using the login form on the login page. Lets write some code.

First – include required modules(of course – npm install xxx) – all of them right now, since i know i will need them. Everything should be self explanatory, maybe only rimraf. That is a module for recursive directory removal. Since i will be storing the html pages, i will probably need some directory and with directory i mean CLEAN directory.

 

Second step – retrieve the login page:

I started writing in functions, so it will be easier later when joining the code. As simply as it seems, the first function downloads the login page. Basically everything i need is the csrf token, which is duplicated – one instance is in the form, while the second is in cookies. Since i don’t want to parse the html only for one token, i grab it from the second occurence.

 

Third step – attempt to login. Since i have already checked the login page, i know that the url which handles the POST request from the form has the same url as login page requested through classic HTTP GET request(ie. in browser), so no need to change the url from previous step. Just create the same request, but change the method, add some data.

If you run this code, you will see, that something is not right. Even though the HTTP response status code is 200(ok), the body doesn’t contain the page, which you would normally get after login. When you debug the body variable, you will see the page and thankfully, the site shows you where exactly was the mistake. Well more like mistakes.

That probably means, that the request above didn’t send csrf in cookies, so the server couldn’t do csrf checkup – from request’s perspective. Request module don’t sent cookies between requests automatically, it has flag for it – named ‘jar’. It could be enabled globally – like this:

The second problem require to provide additional header. The idea behind this is generally to stop CSRF attacks. No problem, just change the request to this:

Note: when you run this code and print the output, the html would still contain the CSRF verification problem text. That is because the request would normally redirect the visitor. For the complete response, you can add  followAllRedirects : true(it will redirect the POST HTTP request to dashboard – parameter next – ‘/’)

 

Fourth step – download one issue page and save it. For this we have to store the project url and issue number – url of the issue is working even without the name in it(ie. https://bitbucket.org/<user>/<repo>/issues/<issueNumber>).

And since i am going to save the actual downloaded page into directory, let’s delete it at first and then create it anew.

OK, this is the basics. When you run the script, it will download the page into defined directory like /issues/1/1.html. This downloaded page is fortunately(thank you bitbucket) able to show everything what you normally see when you go through the login process in your browser and navigate to the url of the issue(css + js working). But you may see a little problem – if you aren’t currently logged in while checking the downloaded page, you won’t see the uploaded images for the issue – they are behind another redirect and physically uploaded somewhere in Amazon cloud.

 

So without auth session active, you won’t see uploaded images – and that will be in the part two of this article.

Note: bitbucket.org offers export for your issues, but that works only if you have admin privileges. Otherwise there is not an API for you to casually download those issued like JSON or even raw besides doing it manually. Chrome store has some plugins, but seriously – you have to provide credentials, which i simply don’t want to.

Posted on

ES6 Proxy and catch-all method

As i talked about ecma2015/es6 with a few people, i’ve realized that many features are still a mystery in the community. One of them is Proxy object. Besides classic language changes there is a handful of goodies, which i particularly really like.

 

General Proxy

What is the proxy object? If you haven’t heard of it yet, let’s look at this example schema of classical Proxy server:

Network Proxy in real life

Uses of proxy servers are many, just picture this one: proxy lies between target and source of the request and is masking the source identity, so the target does not know about the real computer beyond the proxy. This concept is similar to NAT(although it is working diferently). As you can see from the schema above – there is a way for the proxy server to see and optionally alter the response or request. And for the goal of this article it’s enough to know, that javascript Proxy object is masking the call to some method/atribute of another object – basically is wrapping the target object. Since it is a wrapper and is “trapping” all calls, it can easily alter them.

In the contrast to some other languages, the javascript lacked this functionality a few years ago. It is said, that the proxy is similar to python’s get-attribute-access methods  or php’s catch-all method, or that it implements meta-programming or is basis for defensive programming. You can definitely use it for many benefits, that’s for sure.
(Note – this article is considering support in Node.js rather than browsers) The existence of Proxy objects is around for several years now(as a planned feature for ecma2016/es6) but wasn’t fully implemented even in Node.js until recently(6.0.0 it looks like). Even so, many enthusiasts has been using shims/transpilers/workarounds to be able to use Proxy objects before. Now they don’t have to complicate things. Back in the good ol’ days of Node.js version 0.12.x developers had another choice – to use old Proxies API, which was somehow not as flexible as new API. Even so, to get this done they had to enable harmony feature(s) to be able to run their code.

 

You can find an archive of this old API on mozilla docs.

Let’s talk about small difference between the old API and new API – the basic usage:

 

The new one is unifying the previous two methods. The target argument in new API accepts array, generic object, even proxy or function. The old API has been more prone to mistakes as could be seen below and this was probably the reason why they updated it. The handler argument alone is basically the same and could be used without changes in both APIs, while it is basically an object which holds functions – or better traps(full list ie. here). I will skip the basics and follow slightly more advanced usage, since i don’t believe that anybody would be using the proxies with only the handler(ie. without meaningful target / proto arguments).

 

This will be basically about implementing the __noSuchMethod__ in objects, which could be aliased as catch-all method. First – the old API:

 

However this code is working, it’s not exactly what i want – i don’t want to call the bundle function and create the instance of the Dummy inside it…let’s put it somewhere else.

 

Ok, that’s better. But now i am contradicting my initial statement, that i want to use both parameters. Moreover, this code is not the best, since the function in get trap would be created everytime when new Dummy would be created. One method of fixing this is to move this function out of constructor.

 

And there is another problem – it uses global variable, which would be overwritten everytime when new Dummy would be created in this scope. Somehow, the reference to created object has to be passes into the get function. But that won’t be the issue with the new API as this below would simply work, since we are not passing non-instantiated Dummy, but complete object.

 

Conclusion

The last code example is crude, yet effective way to implementing catch-all method in your objects which in turn could help you tremendously when creating large project with many models / controllers while fully using inheritance. Personally, i see a big difference while creating code purely in es6(August 2016 – however still using babel with es6). Then again – i would prefer even better usage – if not built-in method for each object, then at least directly extending Proxy prototype(or “class” if we are talking about es6) which is an idea for part 2 of this article.

 

Posted on

Visualizing HERE routing API times using D3.js

HERE.com routing API

In this post I’d like to show you how to visualize traffic situation on a route over various times in a week with HERE routing API.

So first of all, we need some decent API for getting the route which would include traffic data in the route calculation. I chose to use HERE routing API so I registered a new app called ‘test‘. In order to get a suitable data structure, I created a rather simple Python script, which fetches the route with departure time as a changing variable. As I would like to get hourly information for all 7 days a week, it sends 168 requests. Mind, that in order to keep things simple, I only take the first (fastest) route in the returned list. This means, that the route times do not necessarily belong to the same route. I’m only interested in a single attribute in the response: response.route[0].summary.trafficTime . I’ll output the information as JSON which will look like this:

 

Python Script

Okay, here’s the python script.

 

Visualisation

Now for the actual visualization. I chose the ‘heatmap’ chart style despite the fact, that I’m not constructing an actual histogram here.

That should be it. You can review and fork the code from my pen. Feel free to play with the code!

Roman

Posted on

Geocoding with python

There is a lot of services that provide geocoding or reverse geocoding. In this example I would like to show you an easy way how to implement one of this service, datasciencetoolkit (http://www.datasciencetoolkit.org/) in python script. Big advantage of this service is that you dont need to register to get a key.
Continue reading Geocoding with python

Posted on

Content searching on paginated websites

Even if it is not so common, there are websites, which won’t allow you to enter text and search by it. But you can utilize console in your browser.

Moreover, what if they are so called one-page websites. If you still want to search for something and you don’t have time to expose the api, then you have to write some small functions.

 

Here is an example(using vanilla JS – if the web isn’t using mootools or jQuery or anything else):

If you are using some older browser, it doesn’t have to has document.getElementByClassName, so you have to define it by yourself. This function is little different, it returns single element or false, not an array(as default function in newer browsers).

 

Let’s assume you have to check loaded content for any text you are looking for. Then you would need function which governs this:

 

Next you would need some sort of evaluating the return value from last function. I named it testPage:

 

And since you want to run this checking indefinitely, you have to apply some cycling:

I added a little lame stop condition, which stops the next repeat call. I haven’t created any asynchronous support in form of promise. Instead i used only plain timeout. In this time the whole content of the next page should be reloaded. If not – create some promise support* or change the timeout value to bigger number.

By running ‘go(‘i am looking for this’)’ in console, you would see the progress and hopefully the requested item after x pages.

 

What if my page hasn’t unique identification by class / id?

Then you have to use other methods. You could just search by href in ‘a’ element.

 

Now you have to keep the current page in variable, since you will be searching by the next page url. Everything put in go function looks like this:

 

Happy hunting.

 

(*If you know precisely which pages you can just download(like ‘/search?page=xxx’), you can use the XMLHttpRequest object with promise support(or callback) – more here).

Posted on

Neural net-like canvas animation(2)

How to use html5 canvas to create animation that looks like neural net(part 2)

I took a little harder way to start, since I first introduced a node model, which is more complicated than simple line model. Here it is:

 

Line model is quite easy to grasp. It is a basic line between two nodes. Because all the movements are stored in Node model, it’s not required to do any other actions besides stoic animation, which only follows those two nodes. As a plus point – i don’t need to call init method.

But with the line accessing node’s pos attribute it is obvious we can’t move, because the attribute is updated only after the animation is complete. We have a few options, with 2 listed here:

  • use reference, so the pos attribute would be always the current one(or rather the next one) – preferred
  • update the values every time after iteration

 

Our first app.js script only works with one node. Now it should utilize two nodes and one line:

After you run it, you should see two points moving around while constantly being connected by line. Easy as that.

 

Let’s update our basic recommendations for models and add this:

  • Each animation should be stored as object. This object should have its own context(in terms of attributes).
  • Animation object should therefore consist of blueprint action(basically unique for each model so it could be stored inside of current model instead of another script) and function which prepares this action

 

Render method now utilizes disposableRenderActions. Since there could be time, when we need to queue more animations, but the old splice-to-dispose could broke the for renderActions cycle. With this, all of those finished animations should be removed at the end of render call. For simplicity, it is placed in Node model instead of Base model. If you are wondering about the ‘this.renderActions[i].iteration();’ part, check the code below.

 

Following my notes about models, each animation object should have its render part stored in actions attribute. With this, the model won’t be the one, that repaints the context. In the core, it should be the action of that model. According to new moveTo method, the rendering would be called through bind function. That gives more freedom while rendering, especially with attributes which belongs to animation object.

Animation object has two basic callable methods – iteration and finished. Accordig to previous code, call to finished method would dispose of associated animation right after current render method. As before, this occurs when animation reaches the iteration max counter.

 

With this, the Node model is prepared for another type of method – behavioral.

The point of this method is repeatedly calling some animation. It could be done like this, or it could be made into bigger moveTo method, which resets iteration count and sets a new positions. I wanted it to be separated so i picked the first one. The code above demonstrates how to do it BUT also demonstrates necessity to implement pseudo-asynchronous control flow, which i pointed out in the previous article in Base model(on and event methods).

The thing is, the repeated moveTo call would be disposed even before the increment call. It’s because the callback would be called before the render cycle ends, therefore the disposableRenderAction part would always remove the new animation instead of the old one.

 

On & Event mechanism is example of preventing such thing to happen, but also provides more flexibility in other areas. Please note, that for parallel animations it is still lacking something.

 

And finally app.js

This creates 40 nodes and for each 2 neighboring nodes additional one line. That means even first and last node.
Second example.

Posted on

Neural net-like canvas animation

How to use html5 canvas to create animation that looks like neural net

A few months ago i saw some animation, which you could see even in some movies – looks like neural network with all of these nodes and lines. I thought it would be nice to do it in html canvas. The whole net should be able to generate random movements and as a bonus, it should be able to react to mouse events(demo at the bottom)

For the gist of this idea, you can look at the following image, which serve as the basis http://cdn.phys.org/newman/gfx/news/hires/neuralnetwor.jpg

(*side note – i believe there is some lib on github which is doing the same and even better as a lib)

 

This example doesn’t use any third party lib, just vanilla javascript, so it could be added everywhere without dependencies.

As i planned the structure, i prepared one lib script, and three models for starters:

  1. Base model
  2. Node model
  3. Line model

Base model should contain basic functionality, which other models should inherit.
Node model stands for those dots which are points between lines – like joints.
And finally lines – between each two nodes, there is always line which connects them.

 

Let’s create html template:

There is nothing much to comment – just plain html template with style element and virtual paths to scripts.
I intend to use canvas element by getElementById and pass this element into my small lib. The lib itself should handle all around it. Manipulating and basic interaction with lib while initializing should be placed in app.js, which i cover later.

 

For now let’s create basic class for the lib.

I created basic class with attributes which are essetial(besides animationQueue – let’s ignore it for now).
This lib should accept any given object, which follows some interface(later covered in base model) through addObject method. I don’t want to link everything everywhere, so i am pushing only canvas context and basic color setting into every single new object.

 

Constructor doesn’t have any specialities, just accepts canvas element and initializes some arrays. Color would be later part of more robust default options, which could be altered by some other method.

 

Canvas as html element represents real painting canvas which could be seen everywhere – you pick some brush and paint some lines. When finished or not satisfied, just get the layer and throw it away. So you can’t just pick some object you painted and just alter it or remove it. Simply as that, the lib should provide other basic function renderCycle and clean for painting the whole layer and cleaning respectively. With models which gains reference to context, they could easily manage the rendering on their own. Lib just picks objects in cycle that should be rendered right away.

 

renderCycle should be called and then it should work on its own. That is guaranteed by the requestAnimationFrame part, which basically registers event, which should be fired before the whole browser repaints the scene(on browser level). This guarantees smoother framerate. Also the event is not eternal, that’s why the renderCycle is called again in the callback.

 

For better portability between browsers, you should consider using something like this:

var requestAnimationFrame = window.requestAnimationFrame || window.mozRequestAnimationFrame || window.webkitRequestAnimationFrame || window.msRequestAnimationFrame;

 

Now lets create base model:

Because i want everything to be inherited, i don’t need to initialize anything in constructor. That means for attributes, that should be present in extended object, I had to create special method in prototype which only initializes those attributes. Next there is empty render method. Each and every different model should have it’s own render method, because logic ought to be different. Here I don’t need to specify anything, it works just as an interface in other languages that supports interfaces.

Also I prepared methods on & event which stands for on & emit for example in node.js events module. on registers callback that should be called when specified event is triggered. Additional arguments are just for convenience.

 

So here is the basic plan for models:

  • Each model should follow base model’s render function, which repeatedly, hopefully at 60fps(pc master race!) paints on canvas its part.
  • Model could have animations queues
  • When there is no more animations, the render function is still called. And because the canvas is always cleaned before each render cycle, the object should repaint its current position(let’s call this stoic repaint)
  • Animation should have its starting point and could have ending point. After animation, the current position has to be updated so next animation(even if stoic) uses this new position
  • Each animation could be just called from outside just once. Everything should be covered inside the model

 

Node model:

This is first version of node model. As a single model, it defines own attributes pos and nextPos(since this node should be moving in something like animations). Also it has defaults object which now consists only of value for maximum interations count – more iterations means longer animation. This node model has one animation method – moveTo. It creates anonymous function, which would be called in every iteration. After it ends, it should be disposed of by splicing the animation queue array.
This node model has already its render method in usable form with stoid rendering in case of empty queue.

 

So let’s create the app.js script which finally utilizes the lib:

After running the index.html you should be able to see the first animation which would differ each time you run it. Here is the first example.

I will cover more in the next article.

 

(images: http://rajasegar.deviantart.com/, http://phys.org/ )

Posted on

My first html5 canvas game part2

This is the second part of my tutorial on building basic html5 canvas game (the firts part is here). Last time, we built some easy logic for spawning meteorits and their movement, control of ship and some collision events. Now we would like to add some counters for lives and points and some bonuses to make this game even more entertaining.

Continue reading My first html5 canvas game part2

Posted on

Creating custom control button in leaflet

custom control button

In this post, I would like to shortly describe how to easily create, style and use self-defined control button.

 

L.Control.extend

We should start with extending the original class:

 

onAdd

onAdd method should be now filled with definition of div, its styling and event handlers. For example:

 

map.addControl

Initialization is done by addControl method called on map object:

 

Working example:

See the Pen Leaflet custom control by Eidam (@eidam61) on CodePen.

Also,  I should mention that we can use existing plugins. For example http://cliffcloud.github.io/Leaflet.EasyButton/.

Posted on

My first html5 canvas game part1

In this post, I would like to present you my first attempt to code a javascript game based on canvas. As my experiences with this technology are limited, I had to improvise a bit. My goal was to create a game where a triangle shaped spaceship under the control of player tries to avoid all the asteroids around and pick randomly generated bonuses.

Continue reading My first html5 canvas game part1

Posted on

number to characters function

This is a short snippet for converting number to characters in alphabet. For example, 3 should be D, 7 is H. It should be a problem when trying to convert a number bigger than 25 (length of alphabet is 26, I am counting from 0). It that case, we got more than one character – for example 26 is AA, 150 is FU, 2222 is DHM.

This code should be used for example for converting number of column to its name in Excel.

Code in coffeescript:

 

In javascript:

 

In action:

See the Pen coffeescript function for converting number to string of characters by Eidam (@eidam61) on CodePen.

Posted on

Draggable and Resizable without jqueryUI

I was searching for possibilities to build “windows-like” div’s without the need of importing jquery UI. I found out, there are still not a lot of alternatives when comes to jquery UI and therefore I could find a way to do it by my own.

At the begginig, here is my html code:

 

RESIZABLE

Actually, making div resizable is very easy and you would need only two lines of css:

 

DRAGGABLE

The real problem should be to build your div’s interaction, so when holding .handler element, all .draggable is moving.For this task, I used jquery. To initialize action :

 

 

Inside the mousemove listener, I put code to change css rules for position, values from position of mouse. I also changed the class of dragged divs to be able to manipulate css:

 

window.getSelection().removeAllRanges() unselects all what was selected by moving the div, what should be sometimes a little bit annoying.

I am always moving dragged div to the position, so mouse is in the middle of handler. This is not the ideal case, but it is easy to write. The harder way should be to get relative position of mouse inside the handler and then trying to preserve that values.

 

At this point, I should add a function to stop dragging. Firstly, event listeners for this action:

 

…and now the function itself:

 

 

$(this).off(‘mousemove’) will turn off listener for mousemove. That means, our moving action will stop until one click on the handler again.

 

The another problem should be z-index of our dragged div. As we manipulate with “dragging” class, we should just add some css rules:

 

 

 

Working example of this is here:

 

See the Pen Draggable Resizable without jquery ui by Eidam (@eidam61) on CodePen.