Phase 2: Kinect Depth Images to Minecraft Animation

                By: Nathan Viniconis

                1/11/2010-1/20/2010

 

Goal

                Take a sequence of snapshots from a Kinect and transform them into a stop motion animation within Minecraft. This is the second phase of the Minecraft-Kinect integration project.

 

Breakdown

 

Results

Python Scripts

Take quick successive kinect snapshots

Update color algorithm

Ready a world for the action

                How to set the stage

                Lowering the world

                Anchoring the data

                Setting the POV

Automate the world-building process of each frame

Create a stop-motion animation

Add effects

 

 

 

Phase 1: Kinect Depth Image to Minecraft Save

                By: Nathan Viniconis

                12/26/2010-1/9/2010

 

Goal

                Take a snapshot from a Kinect and transform it into a MineCraft save file. The Depth and Color information will be used to approximate the position and types of blocks to best represent the 3D point cloud in MineCraft.

           

Breakdown

           

            Results

Python Scripts

 

            Rough development information (browse at your own risk):

Get color and depth Kinect data saved as files.

                        Open up a minecraft world save file

                        Determine orientation of save so a world render will show the image

                        Save the info to a save file

                        Add color data to each rendered block

                        Modify other data

                        Don’t Profit!

 

 

 

 

 

 

Results

  1. Youtube video:


  2. ->
  3. ->
  4. ->

 

 

Python Scripts

 

Take quick successive Kinect snapshots

 

Update color algorithms

 

Ready a world for the action

 

Automate the World Building Process of Each Frame

 

Create a stop-motion animation

Add Effects

 

 

 

 

 

 

 

 

From Phase 1:

 

Results

 

1.      Me holding a mighty nerf weapon (1cm^3):

a.      ->

2.      High-Five:

a.      ->

3.      Reaching forward:

a.      ->

4.      With the girl:

a.      ->

b.      ->

5.      Playing around with background filters:

a.      ->

 

Python Scripts

 

·         Taking the snapshots:

o   This script takes a X ‘snapshots’ every Y seconds from the kinect and saves each ‘snapshot’ as a series of two images

§  OutputColor.png – RGB color image from the Kinect Camera

§  OutputDepth.tiff – 12 bit disparity information from the Kinect Depth Camera (note: it will look black if opened with a normal viewer)

o   Dependencies:

§  Python 2.5 or higher

§  Python Imaging Library (PIL)

§  The Kinect must be connected to the PC, and the drivers from http://codelaboratories.com/forums/viewthread/442/ must be installed

o   The Script:

§  http://www.orderofevents.com/MineCraft/getSnapshot.py

o   Global Variables:

§  Image names: A five digit index 00000->00001->etc will be prepended to each image

§  numImages: The number of images to be taken, one after another

§  imageWait: The number of seconds between each image taken

 

·         Inserting a snapshot into a minecraft save:

o   This script takes a color and depth image from getSnapshot.py and pushes them into a minecraft save. The statue is anchored to the top of the world by default. The depth and spread of the area is determined by the area of the 3D image capture from the Kinect

§  When testing, set testEdit to True. This will prevent the save from happening, and the mapImage.png file can be checked for results.

o   Dependencies:

§  Images from getSnapshot.py

§  Python 2.5 or higher

§  NBT parser from https://github.com/twoolie/NBT

§  MineCraft

§  PIL and numpy Python libraries

o   The Script:

§  http://www.orderofevents.com/MineCraft/processImages.py

o   Global Variables:

§  Verbose: Set to true to show logging on the console

§  testEdit: When true, nothing is actually saved to the minecraft save. Instead, an image “mapImage.png” is created to show a cutout of what will be saved projected in 2D

§  colorMatch: When true, the various colors of the 3D image are mapped. Currently uses a manhatten distance RGB algorithm.

·         If false, will create statues out of randomized smooth stone, cobblestone, and mossy cobblestone

§  ImageNames: the depth and color image from getSnapshot.py that will be processed

§  pathToSaves: The absolute path to the minecraft save file

§  minmaxDepth: This can be toggled to exclude regions outside these bounds. The background walls, or close static can be removed by toggling these values.

§  blocksPerMeter: When set to 100, each block will represent 1cm^3. Can toggle this to shrink or grow models.

§  startChunksX,Z: where the image should be inserted into the world.

·         Note: All the modified chunks must exist prior to the save

§  Additive: False if blocks should be ‘cleared’ prior to the model insertion.

§  Floorlevel: How deep to clear the chunks too if Additive is set to False

§  FloorType: The type of block to put at the bottom of the cleared chunk

§  PossibleBlocks: A list of all the blocks that can be color matched. The name, ID, and associated RGB values that represent the block. Current values were chosen by sampling the various textures from terrain.png

 

 

 

 

Get a color and depth Kinect data saved as files

 

·         Found a Python and C# project at: http://codelaboratories.com/forums/viewthread/416/

o   The python:

§  Does not appear to work.

§  With .dll in the proper directory, get a Windows Error when reading the memory

§  Run as administrator?

o   The C#: CLNUIDeviceTest.sln

§  Works in Visual Studio 2008

§  If it crashes, the USB controller gets overloaded until a reboot.

§  Takes ~15fps of the color/depth information

·         The real next step is to determine which approach to take. One of the two:

o   Get the python working, keep tweaking the system/dll locations/installs/drivers until it works and you can read out an image..

o   Modify the C# to save out a file using the data that it is correctly getting really quickly. This will require looking into the APIs and figuring out where I can sneak in to take the snapshots.

·         Decided on an approach: Modifty the UI of the MainWindow in C# to include a button that, when clicked, saves the two feeds to files.

o   After trying and failing in many efficient ways to save a BMP, a brute-force and slow method was created.

§  A button is added to the GUI

§  When pressed, ARGB copies of the two camera streams are saved

§  These int[] are converted to byte[]

§  The byte[] is used to set colors that set data in a Bitmap

§  The Bitmap is saved to the output location, overwriting the previous

·         After looking at some images, the RGB data was seen to be slightly green tinted. Reworked the algorithms, made it much smaller, and not it appears to work as intended. Go me!

 

 

           

 

 

A problem of non-conformance is readily apparent. The RGB colormap that is supplied with the drivers must some some algorithm to determine what color to turn the various pieces of the scene into.

 

By asking for the Corrected8 data instead of the raw data, it seems that we now get values we can scale as desired in the minecraft world. Sadly, there is a problem with alignment.

 

Examples of the grayscaled output is:

 

Here, Black signifies nothing, and the darker you get the closer you get to the sensor. Infinite background stuff will be white.

 

It appears a shift to the right and up may be enough to account for the alignment issue, but we can explore then when we try to colorize our blocks (phase 2)

 

Info:

·         Post describing updates in order to get raw depth data: http://codelaboratories.com/forums/viewthread/416/P15/#1448

·         Post asking how to use depth information, hints that it may be non-linear and compressed on-board: http://codelaboratories.com/forums/viewthread/425/

·         Project that maps kinect data to a 3D point cloud (awesome): http://nkinect.codeplex.com/

·         In order to “quad warp / rectify” the images:

http://groups.google.com/group/openni-dev/browse_thread/thread/b73cc93a11b501c4

 

Now, to ‘quad warp’ the images to we get the correct RGB pixel for a depth pixel

 

·         Since the two images are taken from two different cameras, they do not share a viewpoint. In order to be able to know which RGB pixel relates to which Depth pixel, some math will be involved

·         Kinect inner workings and complex equation for rectification explained here

·         Camera 3d reconstruction explained here

 

 

Some Kinect calibration research is done here:

·         To get Depth from Disparity:

·         float raw_depth_to_meters(int raw_depth)

·         {

·           if (raw_depth < 2047)

·           {

·            return 1.0 / (raw_depth * -0.0030711016 + 3.3309495161);

·           }

·           return 0;

·         }

·         Equations to map the depth to a RGB pixel

o   The first step is to undistort rgb and depth images using the estimated distortion coefficients. Then, using the depth camera intrinsics, each pixel (x_d,y_d) of the depth camera can be projected to metric 3D space using the following formula:

o P3D.x = (x_d - cx_d) * depth(x_d,y_d) / fx_d
o P3D.y = (y_d - cy_d) * depth(x_d,y_d) / fy_d
o P3D.z = depth(x,y)

o   Where:

§  cx_d 3.3930780975300314e+02
§  cy_d 2.4273913761751615e+02
§  fx_d 5.9421434211923247e+02
§  fy_d 5.9104053696870778e+02
o We can then reproject each 3D point on the color image and get its color:
§  P3D' = R.P3D + T
§  P2D_rgb.x = (P3D'.x * fx_rgb / P3D'.z) + cx_rgb
§  P2D_rgb.y = (P3D'.y * fy_rgb / P3D'.z) + cy_rgb
o Where:
§  fx_rgb 5.2921508098293293e+02
§  fy_rgb 5.2556393630057437e+02
§  cx_rgb 3.2894272028759258e+02
§  cy_rgb 2.6748068171871557e+02
§  R
§  [ 9.9984628826577793e-01, 1.2635359098409581e-03,
§  -1.7487233004436643e-02, -1.4779096108364480e-03,
§  9.9992385683542895e-01, -1.2251380107679535e-02,
§  1.7470421412464927e-02, 1.2275341476520762e-02,
§  9.9977202419716948e-01 ]
§  T
§  [ 1.9985242312092553e-02, -7.4423738761617583e-04,
§  -1.0916736334336222e-02 ]
§   

o   In order for this to work, valid depth data is needed. Going to need to get the raw disperity information

 

Coding the transforms

o   Now that all the formulas and constants are known, it has to be coded up

§  Done!

o   First test is to take a depth and color image

§   Description: F:\Programming\Talon\Kinect\outimg24.png

§  Depth image is all black due to 0-2048 data per pixel

o   Now play around with the parameters to figure out what X,Y,Z spreads are required

§  Created a new python script to limit various pixels of the depth image to their RGB counterpart, and then trim select regions. After some toying around with the parameters, got the resulting image:

§  Description: F:\Programming\Talon\Kinect\colorizedTry.bmp

 

 

 

Open up a minecraft world save file

 

In order to “open” minecraft save files, an understanding of the save-format is needed. There are a few different formats that minecraft has used throughout it’s life, but the latest is the alpha format described here: http://www.minecraftwiki.net/wiki/Alpha_Level_Format

 

Some tidbits of knowledge are as follows:

·         The terrain is split into 16x16x128 chunks and saved into their own files

·         The world folder can have 64 subdirectories, which can each contain 64 subdirectories

·         Each chunk is identified by it’s xPos and zPos.

o   The chunks name is created by base36ing xPos and zPos.

o   The names of the folders are found by taking xPos and zPos, modulo 64, and converting to base36

o   Ex: to find the cunk at position (-13, 44)

§  The first folder name is base36(-13 % 64). This is base36(51) which is "1f".

§  The second folder name is base36(44 % 64). This is base36(44) which is "18".

§  The chunk file's name is "c." + base36(-13) + "." + base36(44) + ".dat". This evaluates to "c.-d.18.dat"

§  Thus, the chunk at (-13, 44) is stored in "1f/18/c.-d.18.dat"

·         Level.dat stores environmental data like time of day, health, inventory, etc

·         There are 128x16x16 (32768) blocks per chunk

·         There are many bits of light optimization (skylight, blocklight, heightmap)

·         Block format:

o   Blocks are laid out in sets of vertical columns, rows go E->W, columns go N->S

o   unsigned char BlockID = Blocks[ y + ( z * ChunkSizeY(=128) + ( x * ChunkSizeY(=128) * ChunkSizeZ(=16) ) ) ];

§  Simple: char BlockID = Blocks[ y + ( z * 128 + ( x * 128 * 16) ) ]

o   X increases south, decreases north

o   Y increased upwards, decreases downwards

o   Z increases West, decreases East

 

Now that we  have some knowledge of the format, lets fill in some pieces.

·         Calculating base36(int)

o   World.py from overviewer does a good job with having a small simple python script for converting to base36. Use theirs!

·         Scale:

o   Since the minecraft world is only 128 high, a 480 high depth map may not work… So lets scale by 1/8th. This will turn a 640x480 to a 80x60.

§  Sampling for now, but turn to an average if required

o   PIL library data is here:

§  http://www.pythonware.com/library/pil/handbook/introduction.htm

 

 

Determine orientation of save so a world render will show the image

 

Now that we have the data, we have to process is so that the image will properly align in the minecraft universe so that renders of the world, with N to the Bottom Left, will see the image created from the Kinect data.

·         Determine the area of drawing. How deep should these images be?

o   80x60x64 seems decent.

o   0-255 grayscale to 0-64 grayscale via sampling

·         What chunks will be effected?

o   We want it starting at 0,0 for ease of math reasons

o   5 chunks will spread in the E->W direction, so the following chunks will be the ‘z-axis’ of the image

§  0,0   0,1  0,2  0,3  0,4

o   It will go 4 chunks deep in the N->S direction

§  0,0  -1,0  -2,0  -3,0

o   Overall spread is [-3,0]->[0,4]

·         Orientation

o   Turn the normal NSEW 90deg counterclockwise

§  EW is Z axis                        (E)(Z-)

§                                                   |

§  NS is X axis.      (N)(X+) ------------(X-) (S)

§                                                        |######

§                                                        |######

§                                                        |######

§  EW is Z axis                          (W)(Z+)

·         Calculate which chunks actually have to be opened

o   Created a python function that translates the required chunks into the paths it should exist in.

§ 

·         Orientation:

o   X increases South, decreases North

o   Y increases upwards, decreases downwards

o   Z increases West, decreases East

o  

o   This is 0,0:

o  

·         An NBTFile example can be found at:

o   http://pepijndevos.nl/where-to-dig-in-minecraft

o   https://github.com/l0b0/mian/blob/master/mian/mian.py

 

 

 

Save the info to a save file

 

We are now able to dig into an individual chunk save and attempt to modify the blocks within.

 

A summary seems to be at: http://www.minecraft.net/docs/NBT.txt

 

Time to get one chunk, (0,0), updated..

·         Open the .dat file

·         Un-gzip it

o   http://docs.python.org/library/zipfile.html

o   http://code.activestate.com/recipes/148292-controlling-gzipped-io/

o   http://www.minecraftforum.net/viewtopic.php?f=25&t=24585

o   https://github.com/twoolie/NBT

§  Downloaded the source and ran python setup.py install, library should be installed

·         Find the appropriate blocks, update them to obsidian or something obvious

o   The format is specified as:

§  http://www.minecraftwiki.net/wiki/Alpha_Level_Format/Chunk_File_Format

·         After all the data is read in, ungzipped, the TAG_ structures have to be parsed so we can edit the appropriate part of the file

o   Probably not order-dependant.

§  Find a way to find the position, name, and size

o  

o   With the NBT package installed,

·         Update heightmap, skymap

 

The overall flow of the save will be:

·         Open and Scale Kinect images

·         Check Minecraft Save and Open

·         For each X in the width of the image, 0->79

o   Every 16 pixels:

§  Save old strip of chunks if loaded

§  Load the strip of chunks coorelating to next 16 horz pixels

o   For each Y in the height of the image, 0->59

§  Read depth

§  Calculate chunk XZ

§  Calculate Block XYZ

§  Update chunk data

 

 

Attempt a NBTFile creation file from:

 

 

 

 

Preliminary “in the world” save:

            Note: This is saving basic information. Every XY point in the image had it’s disparity found and a Z was calculated. This is just a projection into 3D rather than a true 3D image. It was based off a scaled depth map from the kinect(image 1), and originally turned to a glass sculpture(image2) and then as a stone/wood statue(image3)

             ->

 

           

 

 

Add Color Data to each Block

 

In order to get the color data from the color image to match an existing block we need to get a color approximation for each possible block that can be used.

 

·         The terrain.png as of 12/29/2010 from MineCraft Beta.

o   16x16 Originals                                          2x2 pixel average via MS Paint

o   Description: C:\Users\Revrick\Desktop\MineCraft\terrain.png ->

o   If 1/2ed in MS paint 3 times, we’ll get 2x2 pixel approximations of each texture

§  Can pick and choose which of the approximations match the best

 

FLow:

·         Look through the blocks and pick the ones that can be used, then look through the ‘can be used’ and trim ‘will be used’

o   Be careful of data that has to effect other blocks

§  Solid/not solid

§  Data (for torch rotation, etc)

§  Flows

§  Light (recalcing lightmap will be necessary)

o   Block data can be found at: http://www.minecraftwiki.net/wiki/Alpha_Level_Format/Chunk_File_Format#Entity_Format

§  Known TileEntity ids(skip for now): Furnace, Sign, MobSpawner, Chest

o   ID data can be found here and here:

·         Description: File:ItemslistV110.pngDescription: http://www.minecraftwiki.net/images/1/1a/DataValuesV110.png

·         Data for blocks can be: (4 bits per block – same index as blockID)

o  

§  If these are saved incorrectly, or become incorrect by updating a blockID that had a value that is now invalid, there can be crashes. This data should be set imediately after setting a new block in a pos.

o   Pumpkins: 0123 for ESWN

§  North (3) or East(0) can be used to give an additional block of color

o   Torches: 1234 for SNWE, 5 is standing

o    

·         Lets start with 4 basic colors for the blocks and get the algorithm working

o   Smooth stone

§  BlockID 1, HSL(0,1,115)

o   Wooden Plank

§  BlockID 5, HSL(25,119,103)

o   Diamond

§  BlockID 57, HSL(120, 240, 152)

o   Sand

§  BlockID 12, HSL(33,136,192)

·         Create color different algorithim

o   Initial:  .5dist(H) + .3dist(L) + .2dist(S)

·         Then add access to the Data for each block

o   Then recalc Data while adding blocks. Parallel to “Blocks”

·         The dividing of the depth data by 4 is losing too much resolution. Turning it into a 64 spread

 

First attempt at color matching using straight HLS distance:

   

 

Changed HLS to be 50,30,20 weight:

 

·         Adding more colors:

o   Use multiple HLS’s for each block. Start with 1 each, then add other tints you want that block to represent

o   Print a pallett out so you can see the possible colors being used

o    

Refine the colors!

o   Turn this into a minecraft save:

§  Instead of going through every X and Y pixel of the depth image, and looking for its cooresponding depth, the projection points from the above formula are used.

·         For every X,Y of the image, determine its depth from the disparity

·         Determine it’s 3D X,Y,Z position in space in meters

·         Re-project onto the RGB to get the color of the particular pixel

·         Map to blocks in the minecraft space

·         Find the type for the RGB and change the block

·         Save the file

o   First working attempt, Color mapping data (in real 3D X,Y coords – newer overwrite Z) and MC:

§  Description: F:\Programming\Talon\Kinect\colorizedTry.bmp

§  ->

 

 

 

 

 

 

 

Modify other data:

 

It is annoying to load a world that is night. Modify the world after the save to set it to noon.

Level.dat file format contains the time of day, it’s format can be found here

o   TAG_Long("Time"): Stores the current "time of day" in ticks. There are 20 ticks per real-life second, and 24000 ticks per Minecraft day, making the day length 20 minutes. 0 appears to be sunrise, 12000 sunset and 24000 sunrise again.

o   Set it to 12000 and the sun was going down

o   Set to 18000 and it was night!

·         Creating a heightmap

o   Every chunk that is cleared and saved should have it’s heightmap recalculated. This is required in order for the blocklight and skylight to be calculated correctly

 

TODO: Calculate the skylight and the blocklight

 

 

Don’t Profit!

            -easy enough