Applications for linear algebra: straightening out perspective

Last night I completed a really interesting lab as part of the linear algebra class that I've been studying. The purpose of the lab was to build up – and make use of – some Python code to process an image taken at a certain angle and "straighten out" the perspective of an image. The processing was done using matrix operations to translate between coordinate systems – something that's bread and butter when working with CAD, so I found it particularly interesting.

Here's the image used as an example – a whiteboard with some writing that's difficult to make out when viewed from this angle:

Angled view of a whiteboard

After running the code, here's the resultant image that was generated and displayed (with some visual artifacts that I believe come from the display technology used, which basically creates a humungous HTML page with embedded SVG to generate the various pixels as skewed polygons with an RGB fill).

Straightened view of the whiteboard

The whole process had me thinking that there simply have to be interesting applications of this technique for CAD data, perhaps processing captured image data to de-skew it and map it into raster images in world coordinates. That was about as far as I got, though, even though I've tried to spend time today considering the possibilities.

Hence this blog post, which is really a call for suggestions. Does anyone have some ideas they'd like to share on a possible application for this type of processing? Right now the code is in Python – it would take some work to replicate in a more AutoCAD-friendly language (although that may be a good reason for me to look at IronPython again, even if the code runs fairly slowly, as it currently stands). Please do post a comment to share your thoughts.

19 responses to “Applications for linear algebra: straightening out perspective”

  1. Common use is "flattening out" building facades shot from a single or multiple angles (then glued together). Once facade is de-skewed and propely scaled, it may get digitized using AutoCAD (or simply used for measuring facade elements, like windows or openings).

  2. Something I sorely miss in AutoCad which MicroStation has,
    is a raster image warping tool which maps a series of point pairs ie. image point to monument point, etc...).
    It's most typically useful tweeking skewed googleearth
    images to basemaps.

    I've faked it a few different ways in AutoCad.
    Create a block of the raster, and adjust xy scaling
    and then 3dRotate it if/as required to 'fake' the eye.

    Hard to believe that Autodesk has not implemented this
    in AutoCAD.

  3. Shadow projections onto a planar surface. 3d modeling programs often do projection onto a plane but only along the plane normal. If you want to project solar geometrically, as I did in Heliotrope, you have to project obliquely from the sun. Can still be done with a clever matrix operation.

  4. Hi,

    Very interesting, can you show the code used to de-skew the image?,as for a suggestion, may be an implementation of the Hough transform to detect features like the brackets of the matrix or some edges and then some angles to de-skew or re project the image.

  5. I employ a similar technique with the Projective2d app. Instead of the Matrix manipulation, though, I used the ManagedARX IntersectWith function to contend with all of the Linear Algebra. Part of the reason for that was to facilitatetheo processing necessary for projecting Splines with complete fidelity.

    What may be surprising/interesting – or maybe not – is that processing Rational Splines (Splines that have weighted CPs) is a Perspective (Projective) transform. It’s a bit of a mind frack, as the 4d ControlPoint/Curve element requires projection back to 3D space.

    It is nice to see the Autodesk developers have provided enhanced functionality in this area of AutoCAD. I will point out, however, that the algorithm they used does not translate well to Grip manipulation.

    In the works, a local gallery intends to have numerous graphics applied to ceilings floors and several wall oriented in non-orthographic angles. At just one perspective will all the graphics come together. I had proposed the notion to them shortly after the TTI Street Art blog.

  6. Hi Kean,

    This reminds me the question I've asked about mapping google earth images (keanw.com/2009/04/triangulating-an-autocad-polyface-mesh-from-a-set-of-points-using-net.html).

    I so far could obtain a point cloud (X,Y,Z) with (X,Y) calculated using world file (en.wikipedia.org/wiki/World_file) georeferencing technique and Z calculated from a triangular mesh.

    The problem is that the code is too much slow, so I've only tested it with very small images (16 ko).

    I can send you the code (written in VB.Net) if you think this is interesting.

    Regards,

    Mourad

  7. Will see what I can post... ideally in the context of AutoCAD.

    Kean

  8. Hi Mourad,

    Ah yes, I remember. Colouring a point cloud seems a good way to go. It's a shame it's working so slowly for you.

    Feel free to send it over, but I can't say when I'll find time to take a look.

    Regards,

    Kean

  9. Hi Kean,
    Was reminded of this today when looking into interactive whiteboards for use in my first job as a firefighter.

    This technique could be used with a smartphone camera or similiar to live stream the contents of a normal whiteboard. The camera could be mounted to the top of the whiteboard similar to a short throw projector.

    Something like this would be incredibly useful and cost effective for us, and I think this would be useful to lots of other applications too.

    Regards Rod

  10. Hi Kean,
    It's seems, that application above does linear transforms of the picture.
    Why you choosen python for such purposes?
    I'm working on alignment situation plan stratening tool for Civil3D (aerial photos),and used WPF libraries for image manipulations. Thats was quite enough for my purposes.
    Also, I think, there are many free c# libraries/wrappers for image processing.

    Thank you for your blog!

  11. Hi Ruslan,

    I chose Python because the class I studied taught the technique in Python, and so I had Python code that implemented it. And it seemed an interesting challenge to have a single application with core capabilities in Python, UI in HTML/JavaScript and the glue in C#. It deals with some potentially interesting interop scenarios (getting data into the HTML page, getting data out of the HTML page into Python, getting the results out of Python into AutoCAD...).

    Would we write a core AutoCAD feature that does it this way? Almost certainly not. 🙂

    Cheers,

    Kean

  12. OK. It's really interesting to combine different technologies to get best of each one. Thank you for explanation.

  13. Hi Brian,

    As i gathered, heliotrope projects shadows only on planar surfaces. Is it possible to do so on non planar surfaces like a topography for example?

  14. Hi Kean,

    I am working with toposheets. Selected contours are to be incorporated in dwg file.
    Is it possible to capture RGB information of selected raster pixels using VBA for autocad ?

    Regards

    Bhaskar Rao

  15. Hi Bhaskar,

    Perhaps, although with VBA it'd probably be trickier (it would be for me, anyway).

    I suggest posting your question to the relevant discussion group.

    Regards,

    Kean

  16. Andrea Andreetti Avatar

    some one redirect me to this website,...
    Hi, I'm Andrea Andreetti from DuctiSoft inc, I've made many Lisp tool since 1985, many of them are huge, complete and complex program...
    I've created last week this kind of tool for the exact same reason 🙂
    I use Lisp who create a copy of modified image then,...reloaded each time the image is stretched. I call the tooll "I-Stretch".
    http://www.ductisoft.com/images/is...

  17. here we go....:)

    1. Nice!

      Kean

Leave a Reply to Sean Tessier Cancel reply

Your email address will not be published. Required fields are marked *