I have found a little more time this evening to fix some bugs and make some improvements to my Chrome extension Recursive. The changes are as follows:
—– v.1.1 ——
- Now only displays recursive icon on tabs with http:// and https://
- Full screen now works (uncommented that line of code, doh!)
- Pause and reseting a recurse now works correctly
- Renamed the title page of the app
- Some Performance improvments
- There is now an option in the settings to define a custom file filter
- There is now a setting to disable removing duplicate files
One of the main new additions is the ability to add a custom filter in the settings which recursive uses when parsing a file:
If any are found they are then displayed in the files dialog:
As I say, just a quick update this evening. I plan on writing some more in-depth blog posts this week explaining some of the nuts and bolts of Recursive.
Wow, well that took longer than expected! 44 days ago I blogged that I had started work on a second version of my Chrome Crawler extension and have only just managed to get it to a state I was happy with enough to release it. To be fair I had been on a trip to New York during that period so perhaps I can be excused. Having said that however I think the time has been well spent and am fairly proud of the result.
TL;DR
Recursive is an experimental tool for visualising the world wide web. Given a URL it downloads the page search for links and then recursively downloads those. The information is then displayed in a node-based graph.
The Name
So what’s this all about? Why is it called ‘Recursive’, why not ‘Chrome Crawler 2′?
Although I would like to have called the spiritual successor to ‘Chrome Crawler’, ‘Chrome Crawler 2′ Chrome’s branding guidelines forbid using the Chrome name or logo (they brought this in since the launch of Chrome Crawler 1).
With that in mind I decided that rather bend Chrome Crawler’s name and logo to fit the guidelines I would create a whole new logo, name and app. The app is a total rewrite from the previous iteration anyway so I thought it justified.
According to dictionary.com there is no definition for “Recursive” or “Recurse” but there is one for “Recursion”:
2. the application of a function to its own values to generate an infinite sequence of values.
So a tool that downloads pages, follows the links on that page to download other pages seemed like a rather apt description of something that is “Recursive”.
Video
Before I go much further, I put together this little video demonstrating some of the extensions core functionality:
Installing
Installing and upgrading is dead simple thanks to how Google Chrome’s extension system works. Just head over to this link and hit install:
Then to launch it visit any website and hit the little icon in the Omnibox:
How it works
Recursive works by taking in a starting URL which it uses to download the page it points to:
Once that page is downloaded Recursive parses it looking for links and files. If it finds things it thinks are files then it records them against that URL. It then proceeds to visit all the links in turn, downloading the page then parsing the for yet more files and links.
This cycle continues until a certain “depth” is reached which is the maximum number of links away from the starting URL. You can set the maximum depth allowed in the settings:
One of the key improvements of Recursive over Chrome Crawler is the way it visualises the data as it is returned:
Every page is grouped by its domain and is represented by a circular “node”.
So for example “http://mikecann.co.uk/personal-project/tinkering-with-typescript/” would be grouped under the “mikecann.co.uk” domain. Any other pages found while running that match this domain are added as little page icons inside the host node.
Any files that are found on a given page are given an appropriate icon and added to that page’s domain node.
As Recursive downloads pages and follows links it records the path it takes. It then draws lines between the nodes that are linked:
Interacting
Using the mouse wheel you can zoom in and out to get a better perspective. Click and drag to move about the recursive space. You can also run the app in fullscreen if you so desire.
If you click on a node it tells Recursive to explore that node for one extra level depth.
Right clicking a node opens a menu that lets you either open all the pages contained in that node or view the files for that node.
Files
By using the context menu for a node you can checkout all the files that Recursive found for that node. The files are separated into various categories which you can toggle on or off:
Then if you wish you can download all the files as a zip.
The tech
I think i’ll leave this section for the next blog post as this one is long enough already.
If you would like specific info on the tech in the meantime however or have some suggestions for features then don’t hesitate to contact me: mike.cann@gmail.com
Well that’s about it hope you like it. I had a blast making it even if it did take alot longer than I was expecting. It doesn’t have much purpose really but there is a lot of cool new tech in there, which is reason enough to make it surely?
TypeScript attracted me for three reasons. Its the new project from Anders Hejlsberg the creator of C#, it brings type safety to JavaScript and im a sucker for new technology.
TypeScript is basically a superset of Javascript much in the same way C++ is to C. So every single JS project is a valid TS project. This is good if you are familiar with JS already, you should be able to pick up TS fast and then get to grips with the other cool bits it bring.
TS compiles down to JS much in the same way that that JS target of Haxe compiles down to JS. Unlike Haxe however the generated code is much more readable and so although there is no integrated debugger (yet) you can just use Chrome’s developer console to debug with without too much pain.
Before I get too much more into the specifics of the language I want to mention the project I am tinkering with TS for. A while back I wrote an extension for Chrome called Chrome Crawler. It is a rather simplistic web crawler written in JS and released as a Chrome extension (because of the cross-domain scripting limitations with normal JS).
Over the intervening couple of years I have returned to the project on occasion with an idea to do a second version, however I never actually completed one. So I thought it may be nice if I gave it a go again but this time using TypeScript and at the same time see how it compares to the Haxe JS target.
Chrome Crawler v2 isnt ready for release but checkout the quick video I put together below to see how its coming along:
The original idea was to lay out the crawling in a left to right fashion. So each crawl depth is put in its own column, but as you can probably tell from the video above, things start getting a little unmanageable when you get to just level 3 of crawler depth. So I think im going to have to rethink how I display the nodes in the crawl graph. I have some ideas, but that’s going to have to wait until I return from my trip to NY next week.
More on that in later posts, lets take a look at a few key TypeScript features:
Typing
One of the key features of TypeScript is that it allows you to structure your code in a type-safe way. You can declare Classes, interfaces (inequivalent to typedef in Haxe) and modules as types then use them later.
TS also has structural typing so you can define a structure as part of a function or variable definition without first declaring a Class or Interface for it. For example:
This is great for many reasons but the big one for me is the way it assists tooling particularly when coupled with the excellent VisualStudio 2012 you get all the things you would expect such as intellisense, go to definition, find references and refactoring. For me it takes much of the pain out of the dynamic nature of JS.
As with AS3 and Haxe typing is optional. You can if you wish declare everything as “any” (dynamic in Haxe or * in AS3). Doing so however would forfeit many of the strengths of TS.
Like Haxe you can ‘define’ things as being available at runtime, this means you can reuse existing JS libraries without having to totally rewrite them in TS. By declaring an “interface” you can just tell the compiler that this type will exist at runtime (we did this with extern’s in Haxe) and thus you can use the existing library in a type-safe way. For example here is a definition for the Facebook SDK:
Alot of the type-safe TS stuff is familiar to me because of Haxe. I think some things in TS are nicer fit because its only designed to compile to JS unlike Haxe which can compile to many different languages. For example TS has function overriding baked into the language also strongly typed functions in my option are a little nicer:
Because TS uses type-inference all the variables in the above are type safe. I really love how terse this syntax is and have had quite a few discussions about introducing it into Haxe.
There is one big difference between function() and lambda definitions however and that is the way they handle scoping of “this”.
Some may consider it rather minor but I find it annoying coming from a Haxe / AS3 / C# perspective where member variable access is implied. I guess the reasoning is because in JS if you dont supply a var before a variable declaration / usage you are referring to global scope, thus in that above example “console.log(url);” would be trying to log global.url which is undefined.
Tooling
The tooling support for TS is pretty good so far. Visual Studio has a plugin for the language and has reasonably good support. There is still quite a bit of room for improvement here however. Things such as integrated debugging, code generation for functions event handlers etc and source formatting would be nice.
Because the TS compiler is written in TS they are able to create really cool things such as the TypeScript Playground:
With it you can mess around with TS and see its generated JS as you type. It even includes a version of intellisese for that type-hinting goodness.
Conclusion
I must admit I really like TypeScript. It provides a light-weight layer over the top of Javascript giving just enough of the type-safe goodness and language enhancements without making it totally alien to a JS coder.
The generated JS looks very similar to the source TS (and will look even more so once ES6 is mainstream) which is important when you are trying to debug a problem. I discovered this to be an important point when developing the Haxe version of my Post To Tumblr chrome extension. Haxe although awesome tends to generate alot of alien looking JS making it tricky to work out where to put a breakpoint.
More important than any particular nuance of the language however is who is backing it. Microsoft is a huge company with a massive following of developers. It pays many people to work on the language and evangelise it. What this results in is a much bigger community. A bigger community means you have more questions asked on Stack Overflow, more API definitions written and more job positions specialising in the language. Also having Anders Hejlsberg, the father of C#, behind it you can be confident that the language will continue to develop in a manner that (if you are a C# fan) makes sense.
I have been having a whole lot of fun in TypeScript, and have high hopes for its future.
Michael is a Professional Flash Developer currently based in Manchester UK. His passion is all things programming, particularly Games and Particle Systems.