It is based on Arducopter, we changed the motor library matrix file to account for six tilted motors, link to the commit on github: 

https://github.com/tatsuy/ardupilot/commit/13dc580757ba802c6a3bd940b0abef425da57152

The rotation of each motors is handled just like any other hexa-copter:

The most important piece of code to understand is located in the motor library, in the AP_MotorMatrix.cpp file: 
case MOTOR_FRAME_OMNI:
            switch (frame_type) {
                case MOTOR_FRAME_TYPE_X:
                    add_motor_raw_6dof(AP_MOTORS_MOT_1, -1.0f,  0.0f, -1.0f,  0.0f, -1.0f, 1);
                    add_motor_raw_6dof(AP_MOTORS_MOT_2, -1.0f,  0.0f,  1.0f,  0.0f,  1.0f, 2);
                    add_motor_raw_6dof(AP_MOTORS_MOT_3,  0.5f, -1.0f,  0.5f,  1.0f, -1.0f, 3);
                    add_motor_raw_6dof(AP_MOTORS_MOT_4,  0.5f, -1.0f, -0.5f, -1.0f,  1.0f, 4);
                    add_motor_raw_6dof(AP_MOTORS_MOT_5,  0.5f,  1.0f, -0.5f,  1.0f,  1.0f, 5);
                    add_motor_raw_6dof(AP_MOTORS_MOT_6,  0.5f,  1.0f,  0.5f, -1.0f, -1.0f, 6);
                    success = true;
                    break;
Here is a video of it flying, pretty cool and trippy: